Software Factory Levels in 2026: Six Axes Instead of One Ladder

TL;DR

  • A software factory is a setup where AI agents take work from a queue, build, test and review it, and a person decides what to build and approves what ships.
  • It has six axes: control, compute, context, trust, learning and runtime. Where the agents run is only one of them.
  • Score each axis from 1 to 4 on its own. A small setup can be high on control and low on compute, and that can be the right choice.
  • Raise the axis that hurts first. The picture in "Where to start" gives the first fix for six common pains.
  • A box per task (one machine per task) is the last step. We have not built one.
  • Give the factory a time budget. About 20% of the week is the limit I would set.

I spent more time on my software factory than on my product. About two months ago I started building it. Every week went more into the factory than into the app, and I am not sure why. I stopped optimizing it, started again from a smaller version, and set myself a limit of about 20% of the week for the factory. One ladder cannot score the setup I ended up with, so I use six axes.

A caution before the details: this is one setup, two months old, and the scores are my own, not measured. I have no cost, return or team-scale numbers, so I give none.

What I mean by a software factory

By software factory I mean a setup where AI coding agents take work from a queue and carry it through build, test and review, while a person decides what is worth building and approves what ships. I do not type every prompt, and I step in when a decision really needs me. I think of it as a production line, because the work moves from station to station without someone carrying it. It is my working definition for this post, not a standard one.

A factory like ours has a few parts, whatever tools sit behind them:

  • a runner that gives each ticket to one agent, in its own workspace;
  • a queue of tickets, and an orchestrator that picks the next one;
  • tests, a separate reviewer and a merge gate in front of main;
  • guardrails that block unsafe commands;
  • a way for the agents to reach a human when a decision is needed;
  • written rules and context that every agent reads.

One coding agent in a terminal that you watch is the start of this, and it is not a factory yet, because you carry the work from step to step. The six axes below describe how far a setup has come from that start. I see a factory as the point where the loop runs without you, and we wrote about the five stages from autocomplete to loop engineering in From Prompt Engineering to Loop Engineering.

What a software factory is made of

One ladder, two questions

I read a ladder with six steps that rates agent setups. It starts with one agent that you watch and ends with an orchestrator agent that splits the work and starts the others, and the steps in between mostly move the agents from a laptop to a server and then to a machine for every task. The ladder cannot place my setup, because it bundles two different questions.

Where the agents run is one question. Who keeps the work moving is another.

In our setup one long-running agent talks with me about the plan, the retro and the task status, and it assigns work to the other agents, which is the top of the ladder on control. The agents run on two machines, which is the middle of the ladder on compute. Scoring each question on its own gives a true picture of that mix, and a single ladder does not.

Where agents run is one of six axes

The six axes

The six axes are our own model, built from running the factory, and each has four levels. The picture below has all four levels for every axis and marks where we are. Here are the two ends of each axis, with our score.

Score your software factory on six axes

  • Control, who keeps the work moving. Level 1: you babysit one agent. Level 4: the lead agent also plans and splits the goal. Ours: 3 to 4.
  • Compute, where the agents run. Level 1: a laptop. Level 4: a box per task, created on demand. Ours: 3, on purpose.
  • Context, what agents know without asking. Level 1: nothing. Level 4: agents keep the context current themselves. Ours: 3 to 4.
  • Trust, what you still check by hand. Level 1: every diff. Level 4: you only see exceptions. Ours: 3 to 4.
  • Learning, does a mistake stay fixed. Level 1: no. Level 4: lessons flow into the context on their own. Ours: 3 to 4.
  • Runtime, can it stay on, safely, without you. Level 1: it runs only while you run it. Level 4: durable across restarts, with a credential proxy that keeps real keys away from agents. Ours: 3.

At level 3, runtime means an inbox (a queue of events from outside, such as webhooks, alerts and schedules, that loses nothing), timers, memory, permission guards and an outbox (a controlled way to reach you). These are the same five parts Anthropic names in its article Building always-on agents (William Hu, 13 November 2026), which also says that permissions are where always-on agents carry the most risk. Its description of the top end is the one I used for level 4: the agent sends requests without credentials, and a proxy injects the real credentials before forwarding them.

Signals from outside come in through that inbox too. In our first version a triage agent read error and latency signals and filed tickets for them, and what to do with each signal was the lead agent's call. Two things keep our runtime at 3. We have no credential proxy yet, and our inbox may lose a message when it restarts, which we have not tested.

Compute sits at 3 on purpose. We run two machines and self-hosted runners, and nothing in our work needs a machine for every task yet. Control is higher because one agent plans and assigns while I approve. Context, trust and learning sit between 3 and 4 because of the written wiki, the merge gate and the lessons log described below, and runtime sits at 3 because we use timers, a command guard and decisions batched into one post a day. In my experience compute and runtime are the easier axes to raise, because you can install them, while the other four take thinking and writing. That is what I have seen in our own factory, and I have no data for it.

Where to start: raise the axis that hurts

Look at what hurts today and raise that axis first. The picture gives the established step first for each pain.

Where does it hurt? Raise that axis first

The first four fixes come from the ten rules we run. The last two come from the level descriptions in the picture above, so treat them as a reasonable first step and not as a result we proved.

Our setup

I use Orca, an open source app (MIT license) that runs coding agents in parallel, each in its own worktree, with a phone app that pairs with the desktop. The agents run on two machines, and the self-hosted runners use K3s, a Kubernetes distribution shipped as a single binary. Caching and a package registry came later, as fixes for problems we met while the factory was running, and we tuned them more after that. We still use Slack, in different ways at different stages and in different projects.

The first version was bigger. It ran Claude Code with one agent per ticket in its own git worktree, an orchestrator on a systemd timer, a merge gate, Playwright tests, a reviewer agent, a Slack bot and OpenObserve for cost and health. An agent can write each of those scripts in an afternoon, and that version is the one that ate my time.

What took the weeks was context

Tools took us minutes. The written context took weeks.

That context is the wiki structure, the architecture, the standards, the rules and the design documents that every agent reads before it starts. I think this is the axis that decides how much the other five give you. It is a guess, with only our two months behind it. Anthropic names the same problem for long-running agents: each new session begins with no memory of what came before, and its fix is a progress file and a git history that the next session reads first (Effective harnesses for long-running agents, Justin Young, 26 November 2025). Our first version did it with files too: each ticket had a status file and a log, so agents and people read the same record. We have argued before that AI adoption is a workflow decision, not a tool decision, and the weeks we spent on context are the same argument at the scale of one team.

The rules we run are the part of that context I would copy first. We keep ten:

  1. Building is cheap and cleanup is not, so optimize for your review time, and not for the agent's speed.
  2. Two gates for every task: is this the right thing, before coding, and is it done right, before merging.
  3. No fix without a reproduced root cause, and the agent points to the exact file and line.
  4. One agent, one ticket, one isolated workspace, and no two agents share a branch.
  5. Agents never merge. A single merge gate does, after the tests pass.
  6. Tier your tests: only the affected tests per task, and the full suite once per batch.
  7. A separate reviewer agent checks tooling changes, because the builder should not grade its own work.
  8. Hard guardrails and no polite prompts: block force-push, branch deletion and destructive commands at the tool level.
  9. Batch the questions agents ask humans. One decision post a day beats 30 pings.
  10. Keep a lessons log. Every mistake becomes a rule, so it does not repeat.

Together these show what the trust and context axes hold in practice. The merge gate is one script on the way to main, and it runs after the tests pass. The wiki holds architecture, standards, rules and design documents, and the lessons log feeds new rules back into it. How we split review between agents and people is in our post on the 4-layer review model. None of this needs a bigger machine, and all of it needs someone to write it down and keep it current.

What a box per task would mean

Level 4 on compute is a box per task: one machine for each task, created on demand and removed after the merge. We have not built one, so this is information only. The closest option to our setup is a pod per task on Kubernetes, since we already run K3s, a CNCF sandbox project, and for code you do not trust you could add gVisor, an application kernel between the application and the host operating system, or Kata Containers, which run lightweight virtual machines as a container runtime. Another route is Firecracker, open source virtualization built at AWS that boots in under 125 ms, or Lambda MicroVMs, which we covered in our Lambda guide. A standard Lambda function runs up to 15 minutes and a Managed Instances function up to 90 minutes when invoked asynchronously, while the AWS quota page lists up to 8 hours per MicroVM, which removes the 15 minute barrier. We have not checked whether a MicroVM gives a shell and reachable ports, which a coding session needs, so check that before you plan around it.

A fourth route is to let a vendor run it. Claude Managed Agents is in beta, and its docs say behavior can change between releases. It runs the agent loop in an environment that is either an Anthropic-managed cloud sandbox or a sandbox on your own infrastructure. Its secure deployment guide tells you to keep every secret in a vault, so the sandbox sees a placeholder and the real secret is added only on outgoing requests to hosts you allow, and scheduled deployments start sessions from a cron schedule. It is not eligible for zero data retention, except when it is self-hosted. We compared it with our setup on 17 November 2026 and decided not to move now: our gaps are small, and we would have to rebuild our harness, reach our local test resources some other way, and send code off our machines. We would revisit it if we needed a box per task, or if a teammate's run needed credentials we do not want on our machines. That comparison was not checked against live prices.

Six is a choice, and you can change it

There is more than one right way to cut this up.

A model is a tool. Judge it by the job it does.

The ladder I read does one job: it tells you where an agent setup runs. Another way to cut it rates how much the system does without you. Our six axes do a third job, which is to tell you what to raise next.

For that job I use three checks before I split or merge an axis. First, does it have its own first fix? If two pains need different fixes, they probably belong to different axes. Second, would two people give it the same score? If not, the levels are too vague and need sharper words. Third, can you remember the list? I find six easy to hold in my head, and I would expect a longer list to get skipped.

That is how security and visibility ended up without an axis of their own. Security sits inside runtime today, as permissions, a command guard and, at level 4, a credential proxy. If safety matters more than uptime for you, score it as its own axis, because the two can move apart: a factory that runs all night with broad access scores high on runtime and carries more risk. Anthropic's article on always-on agents makes the same point from the other side: it lists permissions as one of five parts and says that is where those agents carry the most risk. Visibility, meaning whether you can see what the factory costs and produces, is something we do not score yet, because the measure I want is still early. Signals, meaning what the factory picks up from outside and turns into work, are part of runtime as well: they arrive through the inbox, and the lead agent decides what to do with them. Intent, how clearly you state what to build, matters at every level, and I could not place it in one axis without guessing, so I leave it unscored.

Feedback is not an axis either. It runs through three of them: tests and reviews in trust, retros and lessons in learning, and timers and guards in runtime. So treat the six as our cut. Split runtime if safety is your worry, add intent if your specs are the bottleneck, and drop an axis that never changes your next step. Expect the levels to age too. Anthropic's engineering team notes that a harness encodes assumptions about what the model cannot do, and that those assumptions go stale as models improve (Scaling Managed Agents, 8 April 2026), so I would reread the level descriptions whenever your tools change.

What we still measure badly

The number I want is hours of useful agent work per hour of my attention. It is early, and I am still working out how to capture it. My guess is that more machines move that number a little, and better written context and a gate I trust move it a lot, but that is a guess.

I also use two tests for a real level 4. Can I merge a ticket without opening the diff, and a week later none of those merges needed a rollback? And does a mistake made once stay fixed, because a lesson stopped it that I did not write myself? My scores sit between 3 and 4, so I count both tests as not passed yet.

Close

Which costs you more attention today: explaining the same thing again to every agent (context), or checking every change by hand (trust)? I would like to know which one you picked, and what you added first.

Further reading

Similar articles

A modern development stack in 2026

What a Modern Development Stack Looks Like in 2026

Many posts explain how to choose a tech stack. Here is what we use in real projects in 2026, our preferences, where each default bends, and the five questions that decide when it should.

Scaling and Integrating Health Management Solutions: Overcoming Tech Challenges in Healthcare

Scaling and Integrating Health Management Solutions: Overcoming Tech Challenges in Healthcare

See how INNOMIZE helped a health platform overcome technical challenges to improve scalability, integration, and user experience in managing chronic conditions.

How To Set Up A Multi-version Odoo Development Environment Locally

How To Set Up A Multi-version Odoo Development Environment Locally

Configuring the local development environment to run multiple versions of Odoo that need different versions of Python dependencies