Over the past year, I’ve been building an agentic software factory inside a mature SaaS company, working with an engineering team that was initially skeptical of AI. We’ve come a long way: from copy-pasting code out of an AI chat to delegating entire features to agents. This isn’t vibe coding; it’s AI engineering: we build a system that consistently delivers results at our quality bar. Today, we are experimenting with a supervised factory.
I’ve mapped this progression into six levels, from manual development to a “dark factory.” This article shows how the changes at each level and helps you assess where your team stands and what it takes to move forward.
Six levels of AI adoption
L0 Manual:
the baseline, with every stage done by hand
At L0 people do every stage of the life cycle by hand, from the requirement to release and watching production.
Two years ago, feature and support requests in our team were triaged by hand, the spec was a page typed in Notion, and the implementation plan lived in the developer's head. Some services followed with no pull requests and no review; where we had review, a fellow developer did it. QA tested by hand on a shared staging environment against the PM's original spec, which might be out of date after implementation. Some projects had decent test coverage and static analysis, which was a good start, because CI is crucial for agentic engineering.
L1 Assisted:
where you chat with a well-read advisor
At L1 you get a ChatGPT subscription and start asking questions of someone who seems to have read every engineering book at once, a wise colleague sitting right next to you. You ask it how to design a feature, draft a spec, explain a code snippet, write an efficient algorithm, or find what is wrong with a piece of code. It can even write a simple script, which you copy back into your IDE.
Then the IDE gets autocomplete or AI Assistant, and you start hitting Tab relentlessly. You're staying tactical, accepting suggested changes snippet by snippet.
People get faster, but the cycle stays the same: the chat never touches the team's artifacts and has no tools.123 Dan Shapiro compares this level to lane-keeping and cruise control: you hand specific, discrete tasks to an "AI intern".4 The organization needs little here: a clear, communicated stance on which AI tools are allowed and how to use them, and access to those tools.5
L2 Augmented:
where you teach your junior agent your house rules
At L2 you get Claude Code or Codex, the agent which can actually do things: it writes prose, specs or code in your repository. But soon you realize that your well-read advisor is a mediocre junior developer who writes specs and code the way the internet does, not the way your team does. The artifacts you get are good enough to impress the business ("look, I built the whole product in one prompt, in one evening"), but it starts to fall apart once real users get it. So you start teaching your junior, and that means building context.
You add a short, hand-written AGENTS.md to each repository with your house rules. The agent loads it into every session, like a long-term memory of the project. AGENTS.md, released by OpenAI in 2025 and now stewarded by the Linux Foundation, is the common format for that file.67 It holds what you would tell a new developer and what they could not easily infer from the code: how the project is organized, architectural boundaries, coding patterns, naming conventions, and domain language and domain-specific knowledge.
The agent still does not always do it your way. Each time it gets something wrong, you add a rule, and your work starts to shift from building a feature to building the system that builds it: run the agent, look at the result, correct the instructions. You also give your junior tools: scripts that run checks, for the company knowledge base, production logs and error tracking, and a browser it can click through the UI with. When AGENTS.md grows too big, you split it into files the agent loads only when needed, and recurring jobs become .
matter as much as context. A gate is an automated check a change must pass before it merges: the build, the tests, static analysis. Without gates your junior is blind: it gets no feedback and does not catch its own regressions. In our team, required CI checks for build, code quality, and tests are what the coding agent verifies its own output against. That is the first loop, the one every junior learns: write the code, run the checks, see the errors, fix them. That loop needs pull requests: trunk-based development without review worked for some of our services while people wrote the code, but it gives an agent's change no place to meet the gates and a reviewer.
QA changes the same way. Once the agent can click through the app in a browser, it can take over the deterministic part of testing, where the scenarios are well defined. The creative part stays with people: questioning the requirements, the design, and the UX, and bringing their own taste and judgment.
The result of L2 is a knowledge base about your project: its architecture, standards, and processes. You build it until the agent produces reasonably consistent results that meet your team’s expectations. You cannot skip this work on the way to L3: even if you start from a template or reuse an existing knowledge base, you still need to adapt the context to the new project.
L3 Delegated:
where personal practices become one process with owners
At L3, your junior grows into a mid-level developer: the team shares a process that lets a developer hand it a whole feature, the way a lead delegates work. At L2, each developer trained their own agent to follow the rules. To make the agent work on its own, you write down procedures and teach it skills.
We started with small skills: open a pull request using our template, plan an implementation, run local checks. Then we combined them into one workflow: the /implement-feature skill. We give it a product-owned spec and an implementation plan with the architectural decisions already made; it returns a reviewed pull request.
The skill runs the L2 CI gates and a review loop: a second agent reviews the PR, and the skill addresses valid comments and requests another review. That second opinion improves the output dramatically. The developer makes the architectural decisions, hands over the task, and reviews the result.
A skill is a written, versioned procedure with instructions, checks, tools, and context. Our feature skill lives in a shared plugin, so updates reach every developer. The repository template places each project’s constraints and quality gates in a standard location, so the skill can find them across repositories. Each part of the process has an owner who keeps it current.
That is the L3 shift:
The team maintains one shared process instead of each developer training their own custom agent.
The product manager’s work changes too. An agent can read the codebase and answer questions about the product in non-technical language. It can interview the PM about initial requirements, using the code to identify details and decision branches for the spec. The PM can also prototype a feature in the real app using its design-system components. Even a vibe-coded prototype that looks right but doesn’t quite work gives the developer a useful starting point. This makes a well-maintained design system in code more valuable. The spec also moves out of Notion into the repository, next to the code, with the plan, the test cases, and the test report, so the PM learns Git basics: branches and versions.
Our QA team built a testing harness. In it, the developer generates test cases from the spec and the implementation, including the differences between them. An agent runs deterministic scenarios through a browser or Playwright, captures screenshots and video, and produces issue reports. QA focuses on creative testing: questioning requirements, exploring the product, and finding awkward experiences and missed cases.
L3 moves the bottleneck to review. In Faros AI’s 2025 telemetry from 10,000 developers, teams with high AI adoption merged 98% more pull requests, while PR review time was 91% higher.8 That is a correlation for teams with high AI adoption, not a measurement of L3, but it matches what happens once agents write whole features. Developer review time now limits how much the team can ship.
At L3, a shared system knows the product, helps elaborate requirements, and can build and test a feature. People still start each workflow themselves, and agents usually run on local machines.
L4 Supervised factory:
where you stop building features and start building the system that builds them
Your mid-level developer produces decent results, but you become the bottleneck: you design the features, start the workflows, and review every change. The , the tooling around the model that gives it context, tools, and permissions and runs its loop, still runs on your laptop.
At L4, the agent moves to the cloud and takes tickets from a shared board. The product manager writes the spec; the agent proposes a plan and raises architectural questions. Once the developer and agent agree on the approach, the agent works independently. It can escalate further decisions or make them under an agreed policy and record them for review. The agent returns a pull request, ready for review and creative QA on staging. Sber similarly rebuilt its autonomous test generator to ask an engineer for confirmation at key points.9
Behind the board, a router decides which agent should handle each incoming signal. An orchestrator manages the queue and available machines, starts a sandboxed coding agent for each feature, and reports when the work is ready.
At L4, the agent doesn't have to wait for a ticket. It watches errors and logs, investigates issues as they appear, and either escalates them or proposes a fix.
Our Sentry skill already demonstrates the investigation part. It examines production errors and logs, reads the code, and runs read-only scripts against production data to test hypotheses. It can reproduce a defect with a failing test and prepare a fix plan; for qualifying issues, an operator can authorize it to proceed to a pull request. Today, a person still starts the workflow. At L4, an incoming error could trigger that investigation automatically.
Through Slack or another messenger, the agent is also available to people outside engineering. Customer support can ask how a feature works; product managers can ask questions without reading code. The agent checks the code and knowledge base, answers from the information available to it, and can turn requests into draft specs for the PM.
Now you ship more, but specs, pull requests, and decisions pile up on your desk. The work becomes exhausting: all design and review, and you’re the bottleneck again. Gradually, you learn to trust the system, loosen your grip on reviews, and move up a level of abstraction.
At L3, the developer reviews the implementation and fixes findings in the pull request. At L4, automated gates check each change. Developers review against agreed architectural principles and use their findings to improve the gates. They do not necessarily review every change, but without architectural review they cannot know whether the gates work or understand the code well enough to maintain the factory.
Not every finding has to wait for a person to spot it. At L4, the factory learns from its own mistakes: it collects every correction, whether a failing test, a review comment, a production error, or a person saying "that's not how we do things", and when a correction repeats, the agent proposes a change to the system rather than fixing one more pull request.10 Your junior stops needing the same reminder twice. The strongest fix makes the mistake impossible in the architecture; next come types, lint rules, and CI checks, then a test, and only last a line in AGENTS.md or a skill, the weakest because the agent can skip it.10 A person approves each proposal before it lands, so the context does not bloat and the loop does not make mistakes unattended.10
As Cole Medin advises, build a workflow you trust before delegating larger pieces of work with less oversight.11 That workflow needs an owner: the AI platform engineer maintains the platform and agent harness.
This is the final shift in mindset: you stop focusing on doing the work and start building and supervising the system that does it.
Stop doing the work. Start building and supervising the system that does it.
Without line-by-line review, security must be part of the gates. Veracode's 2026 report found a 56% security pass rate for AI-generated code on benchmark tasks without security-specific prompts.12 Meta's "" says an agent combining untrusted input, access to private data, and the ability to change state or communicate externally in one session should not run autonomously: it may hold at most two of the three.13 Our Sentry skill keeps to that: its scripts against production data are read-only, and it opens a pull request only when a person authorizes it.
For QA, test cases come from the spec, the suite runs automatically, and QA reads the report. The creative part remains: a person still explores the feature, tries to break it, and spots awkward interactions.
Watch for a false L4: cloud agents alone are not enough. If architectural review keeps finding line-level defects, the gates are not ready and the role is still at L3. Someone must verify that the gates work before people stop reading every line. Metrics such as accepted-change rate and rework help reveal whether the overall workflow is improving.
L5 Dark Factory:
why a supervised factory is where to stop
At L5, no person checks the result against what they wanted, and that is why we stop at L4. Dan Shapiro calls the "a black box that turns specs into software".4 Its advocates do not drop verification: it moves from diffs into precise specs, scenario tests, and quality thresholds that people define in advance.14 Dash0 adds that this needs specs so precise that no one looks at diffs, and that draft-quality specs then ship draft-quality output at full throughput.14
We do not aim for L5, because it removes the second gate, the check that you got what you asked for, which is why you build software in the first place. It is like letting your mid-level developer ship to customers without anyone checking the result against the request. Automated checks can confirm that the result meets criteria written before it existed. They cannot tell you whether those criteria described what you wanted, or that the feature is awkward for the person who will use it. That takes the person who asked, and a QA engineer who uses the feature as a user would. Better models do not change this. In the end it is a question of meaning, and each of us answers it for ourselves. We build software for people. If nobody checks whether we got what we asked for, the work loses its point and turns into AI slop, so we keep both the meaning and its check with people. Where the result is not meant for people, the answer can differ.
Cap each kind of work by the cost of an error
How close each kind of work gets to L4 is each company's choice, made by the cost of an error. Sber's AI-Disrupt PDLC framework keeps a separate, risk-adaptive scale of agent permissions, where the most critical paths stay at proposals that a person accepts.15
Ceiling by cost of error
How far each kind of work goes
| What an error costs | Examples | Highest level | How deep the architectural review goes |
|---|---|---|---|
| Little, and easy to undo | an internal admin page, UI behind a feature flag, a throwaway prototype | L4 | optional on some changes |
| A lot, but reversible | customer-facing product features | L4 | a person stays in the decision loop15 |
| Irreversible or high-risk | authentication, billing, customer data, schema migrations | L3: a person reviews every line | —1415 |
The ceiling is the highest level at which your gates reliably catch the errors you cannot undo. Authentication or billing stays at L3 not because a person reads code better than a gate, but because a missed defect there cannot be rolled back, and the gates do not yet rule that class of error out; when they do, the ceiling can rise. Dash0 puts it the same way: a service earns a higher level on evidence, one service at a time, and high-risk surfaces may never qualify.14
Two human gates: what people still decide and check
Will a human still be needed in this loop? Will you lose your job?
I believe people stay essential, but the bar is rising: your agent takes on more of the developer's work, so you have to be the senior who decides, judges the result, and answers for it. Some jobs will disappear and most will change; that deserves its own article, "The Human in the Loop". Here I stay with what does not change across the levels.
Two gates stay with people at every level below L5: deciding what we want, and checking whether we got it. An agent can find gaps in a request and help turn it into a spec, but people choose the problem, the outcome, and the trade-offs. The person who asked checks the result; at L4 that check moves from every line to the architecture and the product, and QA explores the product as a user would.
Two gates stay with people: deciding what we want, and checking whether we got it.
Between the gates, you delegate to an agent as you would to a developer: with a contract, checkpoints, gates, and a rule for when to ask. The contract grows from a prompt (L1) to AGENTS.md (L2) to a spec and a plan (L3); at L4 the gates carry most of the trust a line-by-line reviewer used to carry. Where the agent must stop and ask is each company's choice, and the platform engineer configures it.
Delegation
What a person checks, and the tool they delegate with
The two human gates, deciding what we want and checking whether we got it, span L0 to L4 and end before L5.
| Level | A person checks | Delegation tool |
|---|---|---|
| L0, Manual | Every stage | None yet |
| L1, Assisted | Every chat answer | A prompt |
| L2, Augmented | Every diff | AGENTS.md and CI gates |
| L3, Delegated | The implementation, line by line | A spec, a plan and shared skills |
| L4, Supervised factory | The architecture and the product | Trusted gates and an action policy |
| L5, Dark Factory | Nothing | A spec only |
Measure AI adoption per role: the self-check
A team is almost never on one level as a whole. When Raiffeisenbank split usage across its 500 engineers by role, developers used agents fairly actively while testers, analysts, and DevOps used them unevenly.16 System analysts were hardest: their first artifact is harder to hand to an agent and check against a template.17 In quality engineering, the 2025 World Quality Report found nearly 90% of organizations pursuing generative AI but only 15% at enterprise scale.18
The slowest role sets the pace. Work queues at each handoff from product to development to QA, so speeding up one stage moves the queue to the next.19 Measure maturity by the lowest dimension, not the highest.20
So the self-check goes role by role. Pick a role in the level matrix below and find the highest level where you can point to evidence for its practices: a file in the repository, a gate in CI, a ticket, a review comment that became a rule. If you cannot point to it, the role is not at that level. A role sits at the level that holds for most of its work, and the cost-of-error ceiling above caps the risky parts.
Three questions place a role faster than any tool list. Where does the agent get our context? What does a person still check, and at what altitude? When a review finds a problem, does it fix this one change or the system that produced it? If the answer to the last is "this change", the role has not reached L4. The levels describe how the process is built, not what it delivers: a level that does not improve lead time, defects, or rework is a false level.
At L4 the roles may not stay three. Someone still owns the requirement, maintains the pipeline, and verifies the product. A company that can afford it keeps three specialists. As I see it in 2026, the trend is to merge them into two roles: an AI Builder, who owns the product and supervises features through the factory, and an AI Platform Engineer, who maintains the platform and the agent harness. A company that cannot afford more may end up with a single AI Builder.
So the decision is per role and per kind of work. Place each role on the level matrix below, take the next transition for the slowest one, and cap each kind of work at the level its cost of error allows. We aim for L4, a factory people supervise, and do not go to L5.
The level matrix: who does each stage, and the practices at every level
The life cycle at six levels

