The engineer's job moved up a level. Most teams measure the old one.

Most engineering teams now use AI to write code. Far fewer have changed what an engineer does all day. The tool moved. The job description did not. That gap is where the productivity goes missing.

This article is a way to find out where your team stands, and what changes at each step. It is built on a framework we did not invent but use every week.

Five levels, one question

Nate B. Jones describes AI coding maturity as a ladder.¹ We use his levels because they make a vague conversation specific.

Level 0 — Autocomplete. The model predicts the next line. The engineer writes everything and accepts suggestions.

Level 1 — Intern. The model produces boilerplate and explains errors. The engineer delegates snippets.

Level 2 — Junior developer. The model writes first drafts across files. The engineer reviews and steers multi-file changes.

Level 3 — Engineer as manager. The model implements a feature on its own. The engineer sets direction and approves the pull request.

Level 4 — Engineer as product manager. The model owns implementation end to end. The engineer writes the specification and checks the outcome.

Level 5 — Dark factory. An autonomous pipeline. The engineer defines the business goal.

One question tells you where a team is: what does the engineer type? Code, and the team is at Level 2 or below. Specifications, and it is at Level 3 or above. Teams tend to place themselves a rung above what their pull requests show. The pull requests are the honest answer.

The dip between the rungs

Moving up the ladder costs something first.

A controlled study of experienced developers on codebases they knew well found them 19% slower with AI tools, while believing they were 20% faster (METR, 2025).² The tool was fine. The workflow around it was the old one: the engineer still wrote the code, now with a second opinion to argue with.

That is the dip. Each rung needs a different workflow, not a better model. A team that buys Level 4 tooling and keeps Level 2 habits pays for both and gets neither.

The gain was never in the typing. It was in the thinking, and the thinking still has to be done by someone.

Three skills the higher rungs need

At Level 3 and above, the engineer's output is a brief, a review, and a decision. Three skills do most of the work.

1. Decomposition

An agent-sized task has a boundary, an interface, and a way to know it is done. "Add export to the reporting service" is not one. "Add a /reports/export endpoint that returns the existing report model as CSV, behind the current auth, with a test for an empty report and a test for a 10,000-row report" is.

The skill is in the cut. Too coarse, and the agent invents the parts you did not specify. Too fine, and you are back to Level 2 with extra steps. The test we use: can one agent finish this without asking a question that only a human can answer? If not, split it or answer it in the brief.

2. Model selection, written down

Not every step deserves the largest model. Planning and cross-service analysis want the strongest reasoning. Scoped implementation against a complete brief wants a fast, cheap model that follows instructions. Exploration — "find every place this type is used" — wants the cheapest one that reads well.

What matters is that the rule is written, not remembered. On our own builds the routing lives in the repository: which tier for which task, and why.³ When an engineer overrides it, the override goes in the brief with the reason. Cost per released feature is then a number, not a surprise.

3. Trust calibration

Delegation is a dial. With the dial turned too far down, the engineer re-reads every line and the speed is gone. With it turned too far up, the agent takes a liberty and nobody notices until production.

Two signals keep it honest. First, the number of steering messages a feature needs. A feature that takes 20 corrections has a bad brief; the fix is upstream, in decomposition. Second, the boundary. An agent may change the module in its brief and nothing outside it. Anything an agent does beyond its task is reverted unless a person adopts it on purpose. "While I was here" is how unwanted code gets in.

What Level 4 looks like on a real repository

This is the setup Olensys runs on its own builds and on client work.⁴ None of it is exotic; all of it is written down.

Roles, not one assistant. An explorer that maps the code and writes nothing. An implementer that works from a brief inside one service. A reviewer that reads the whole branch before any pull request. A chores agent for the mechanical work — worktrees, pull-request prep, cleanup. Each has its own instructions and its own model tier.

Plan before code. Every task starts with a written plan the engineer approves. Agents working across services write their plans first and message each other about dependencies; a lead reconciles them into one. Then they build, in parallel, each in its own directory.

Hard limits, enforced by the tooling. No agent can merge, tag, push to main or force-push. That is a hook, not a policy. Agents open draft pull requests with evidence attached — the command output, the test run, for mobile a screenshot from the device — and stop. A named engineer reads the diff and presses merge.

Sessions that end. One ticket per session. When the context fills, the session writes a handoff file and the next one starts from it. Side findings go to a backlog file, never into the current change. Progress is a line in a ledger, not a memory.

Gates that count. A change is not done when the agent says so. It is done when it has passed the six gates — repo-ready, spec approved, plan approved, verified, production-ready, released and measured — and the last one has a number next to it.

What we do not run

We do not run a dark factory. Level 5 is a real destination and we are not selling it, for three reasons we would rather say now than in week three.

The failure rate is not zero. An agent that is right nine times in ten is wrong on the tenth, and a cross-service mistake has a wide blast radius. The gates exist because the failure rate exists.

Skills atrophy. Engineers who never write code stop being able to judge it. On our pods, engineers still build things by hand. Not for output. To stay able to read a diff and know.

The agent layer can be over-engineered. Some tasks need one prompt and a careful read. Building an orchestration around them is activity, not throughput. The question is always the same: did more valuable software ship?

What to measure instead

If the engineer's job moved, the metrics have to move with it. Lines of code and commit counts measure the typing. These measure the shipping:

• Spec-to-merge time — from a written brief to a merged, reviewed change.
• Changes released without rollback — the cross-service ones especially.
• Rework rate — how often a human corrected or reverted agent output within 30 days.
• Deployment frequency — against the team's own history, not a benchmark.

Each one needs a baseline before anything changes. We take ours from the team's own tracker and git history, and we do not claim a delta against anything else.

Where to start on Monday

1. Place your team on the ladder from the pull requests, not from a survey. What did the engineer type last week?
2. Pick one repeatable workflow — the pre-deploy checklist, the dependency bump, the environment setup — and make it one command or one agent. Measure it before and after.
3. Write the model-routing rule down. Three tiers, three kinds of task, one file in the repository.
4. Count steering messages per feature for two weeks. Where the count is high, fix the brief.
5. Write the next feature as a specification instead of code. Hand it to an agent. Review what comes back the way you would review a colleague's. That review is the new job.

Have something important to release? Talk to an engineer. 45 minutes. Your roadmap. No sales deck.

Sources

1. Nate B. Jones, The 5 Levels of AI Coding (Why Most of You Won't Make It Past Level 2), AI News & Strategy Daily (YouTube, Feb 2026). youtube.com
2. METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Jul 2025): 16 experienced developers, 246 tasks; 19% slower with AI, self-estimated 20% faster, metr.org, 2025; and We are Changing our Developer Productivity Experiment Design (24 Feb 2026), metr.org, 2026.
3. Olensys workspace standards: model-routing rules kept as a file in each workspace (docs/MODEL-ROUTING.md), with the reasoning. Internal; shown to clients on request.
4. Olensys agentic SDLC standards (the eight standards and nine templates behind the Factory Handbook). Internal; the six gates are published on the Velocity Sprint page. olensys.com/velocity-sprint