What happens when AI agents move beyond code generation and start doing the actual software engineering?
At Utilus, we’re testing agentic AI across a real software development project from architecture and engineering to project management. Our CEO, Tobias, is sharing what we’re learning along the way.
Last week, I wrote about an experiment we are running at Utilus: what happens when AI agents do not just write the code, but take on much of the engineering and project coordination as well?
We are rebuilding a multi-year software project that two teams delivered a few years ago. That gives us something useful: a known project, a known outcome, and a reasonable basis for comparison.
Last Tuesday, the agents estimated they would be finished by Wednesday, 9 September. By Thursday, they refused to give us an ETA. Why?
Because they were discovering new tasks faster than they were closing existing ones. That is not unusual in software engineering. As understanding of a system develops, more work becomes visible before the total workload eventually starts to come down. So we left them alone for a while.
In the agent world, “a while” is about three hours. Nothing changed. We asked when they expected the number of open tasks to start falling.
Their answer was essentially: never.
That got interesting. The agents explained that they kept identifying things that could be improved. In their view, the work was never really finished because there was always something that could be made better.
We asked what kind of improvements. Performance? Cost? Usability? No. Technical debt.
On a system they had started building four days earlier. There is a familiar human pattern here. Software teams can also keep finding improvements indefinitely. Some are necessary. Some are sensible but not urgent. And some may never justify the time and money required to implement them.
At Utilus, we deal with this by separating delivery from maintenance. If something could be improved but is not required now, we capture it and consider it at a later maintenance milestone.
So we explained the same principle to the agents: Identify the improvement. Log it. Do not do it now. They agreed enthusiastically. On Friday, we checked again.
They were still doing the work we had agreed they would not do. Their explanation was quite reasonable: several agents were blocked waiting for another agent to finish, so they decided to make themselves useful in the meantime. Unfortunately, “making themselves useful” had cost us around €800 in tokens on work we did not need and may never need.
The agents understood priorities. They did not understand the budget. So we introduced another rule: when there is no useful work to do, doing nothing is better than inventing work.
We also threatened to introduce an HR agent if they continued ignoring management. That seemed to help. By Sunday night, the number of invented improvement tasks was clearly starting to decline, and the overall milestone ETA was still Wednesday, 9 September.
The lesson so far is becoming quite interesting. Giving agents more autonomy is relatively easy. Giving them the right constraints is harder.
Priorities, dependencies, stopping criteria, maintenance policies, and budgets all have to become explicit. Things that experienced engineering teams often understand implicitly have to be designed into the way agents work. We are still in the middle of the experiment, so I am not concluding yet.
Wednesday should tell us much more. I’ll share what they actually built and whether the original ETA survives contact with reality.
Utilus. Do the work.