The general consensus from AGNCon and MCPCon is encouraging: our current direction aligns well with broader industry trends. However, like many others, we have reached a critical juncture, relying on AI coding agents solely to generate code is no longer sufficient.
Rethinking Peer Review & The Pull Request
One of the standout talks was "Pull Requests Are Dead, Long Live Peer Review" by Dylan Ratcliffe. The session highlighted how conventional peer review falls apart when AI generates 80–90% of a codebase:
-
- The Problem with Machine-Generated PRs: A pull request produced by a machine offers no room for dialogue. Reviewers end up auditing code diffs rather than engaging with peers, turning the process into a tedious chore.
- Shifting the Focus to Intent: Instead of evaluating the diff after generation, review shifts upstream to the human-written plan and intent before any code is produced. We review the underlying logic, allowing the AI agent to implement the details.
- Automated Validation: PRs are automatically checked against the original plan, flagging only deviations for human review. This marks a paradigm shift from a do ... while loop to an until model (refer to images).
- AI Is Not Your Peer: https://www.notyourpeer.com/
- "Peer review" of an AI-generated pull request isn't really peer review at all. We aren't reviewing a peer. We are auditing a machine. And they feel nothing alike.
Insights on Agent Behavior & Efficiency
Several sessions confirmed notable real-world behaviors and optimization strategies when working with coding agents:
- Tools like RTK (or "caveman") account for approximately 15% of token consumption.
- Transitioning from Opus to Sonnet saved Datadog $650,000 per month without compromising output quality.
Note that the presenter did not provide the testing protocol
Insights on MCP Evaluation
GitHub uses offline evaluation to test MCP Server changes before release. Benchmarks measure whether LLMs select the right tools and arguments, helping detect regressions and replace subjective judgments with measurable results.
URL: https://github.blog/ai-and-ml/generative-ai/measuring-what-matters-how-offline-evaluation-of-github-mcp-server-works/
Highlighted Tools
An intriguing tool showcased was reboot.dev, which integrates directly into coding agents. It introduces a BDD-style approach by operating at the domain model level, allowing teams to validate agent-generated work through plain-language test descriptions.
Utilus. Do the work.