Are we meat proxies?

Recently I came across this post. It bothered me a little how cynical it was about the human role in building the future. It struck a nerve for sure: building software and grappling with engineering problems is not only what I enjoy doing on my day to day, it’s also what has provided a livelihood for my family.

So, just to be on the safe side, I entertained the idea the post proposes. Are we really just meat proxies? Maybe there was an ironic tone I missed, and that’s fine if that’s the case, but let’s assume there isn’t one for the sake of this piece.

Let’s walk over and extreme example of a software company, think: Amazon. Let’s forget about the pesky physical world (fulfillment centers, the real computers behind the AWS cloud) and focus on the software development side. Fable from Anthropic is really great at software development, no secret there. Just lately I’ve been able to ship a lot of bug fixes and even a whole greenfield project (a mobile app backend for field workers, a few Go services on AWS with the usual Bedrock and Step Functions plumbing), ticking all the bureaucracy checkboxes along the way (Jira tickets, documentation, testing), much faster thanks to integrating AI into my workflow. The speed is amazing, and I don’t think I’m losing quality. I just need to focus on the critical parts of the code and the process. So my natural question is: how far could a single person scale this? Could the whole software branch of Amazon, for example, be run by AIs only? Let’s put a single (very well paid) human at the top for good measure. After all, we don’t want to hand those suspiciously friendly machines the keys to the kingdom just like that! What would this human’s day to day look like?

This human would need to context switch between so many tasks and projects. Think Amazon.com, Whole Foods, Amazon Pharmacy, One Medical, Amazon Logistics, Amazon Robotics, AWS (EC2, S3, Lambda, DynamoDB, Bedrock, and hundreds of other services), Prime Video, Twitch, Audible, Amazon Games, Kindle, Goodreads, IMDb, Ring.

I tried to estimate the number of decisions this human would have to make in a single day, and actually asked my AI companion to do it for me. We already agreed they are really good at some tasks, and this can be one of them for sure. He said (trimmed for length):

Amazon has somewhere around 50,000–70,000 software engineers. Call it 30 real decisions per engineer per day, and you’re at roughly 1.5–2 million engineering decisions a day.

Now put AI agents on all of it and one human on top. The human doesn’t need to make the two million decisions, only the ones agents can’t or shouldn’t make on their own. What lands on their desk?

• Decisions that need accountability, not intelligence. Rotate keys across tens of millions of devices. Push firmware to cameras in people’s homes. Regulators, courts, and customers need a name attached, and “the agent decided” isn’t a name.
• Decisions where the objective itself is unclear. Should Ring share footage with police? Those are value judgments and someone has to own them.
• Conflicts between agents. The AWS agents want to deprecate an API; the Stores agents depend on it. Somebody adjudicates.
• Novel situations. A new regulation, an outage no runbook covers. The pattern-matcher has nothing to match.
• Catching the agents being confidently wrong. Even at a 0.1% error rate, that’s 2,000 wrong decisions a day, and the ones that matter compound silently until someone notices.

Even if you filter aggressively, that’s thousands of decisions a day. One human physically can’t. You’d end up with a few thousand people, and now you’ve reinvented an engineering org, just a smaller one with a different job description.

End quote. Human back here 👋

That bullet point about key rotation was part of my job at Ring for six years, and back then there were no AI agents to be accountable for a decision like that. Even if there had been, I don’t think we would ever have left a decision and implementation like that to an agent alone. We could have implemented it much faster, for sure, and even used it to review our code. But I don’t think we are anywhere close to fully automating a secret key rotation that could brick millions of devices that families rely on for their home security. Maybe I’m being a bit old fashioned here, but I don’t think we should ever do it. Just read Dune if you lack the imagination for why that’s a bad idea.

OK, back to the “single human running Amazon” experiment. As you can see, that’s just too many decisions for a single human to handle, let alone handle well, since each one needs real context to get right. I don’t know what the ceiling is for good decisions per day, and I’m not sure anyone does, but I’m confident it’s a lot closer to a few dozen than to a few thousand. So our human would have to pick which decisions to actually look at (which is a decision in itself, by the way) and push “Accept all changes” on everything else. Not good.

So I think it’s fair to say no single human could handle this, but a couple of thousand probably could. Is the conclusion, then, that we don’t need as many humans working in software? For a particular company, probably not. But for software development as a sector, maybe there is still room for everyone, and then some.

Enter: Jevons paradox.

Long story short, this Jevons fellow was the first to notice that as steam engines got more efficient, coal consumption went up instead of down, which seems contradictory. The thing is, as efficient engines got cheaper, suddenly it made sense to put an engine where previously it didn’t make economic sense. Software could be following the same path: an expensive thing getting cheaper to develop, so instead of needing less software, we start putting it in places that never made economic sense before. Let me share an anecdote to illustrate this. I distinctly recall one of my first jobs, at a major software company that dealt mainly with big international clients, think Disney, Electronic Arts, etc.

One day a small business owner from the town where this particular office was located came knocking on the front door. I happened to be sitting very close to the entrance (as juniors did) and heard the whole exchange. The business owner was politely escorted out of the building after being told that this software company didn’t deal with small businesses, and that he should hire some local developers instead.

Many years later I still think about that exchange. Of course it didn’t make sense for this big company to take on such a small client, but it still felt wrong to see someone asking for help and know we couldn’t help him in any way that made economic sense. I was lucky enough to witness it first hand, and I know it probably happened many, many times at many other large software companies. The irony is that nowadays that software company has hit a bit of a rough patch in the stock market, while that small business has grown a lot and is thriving. To me that’s a lost opportunity for the big company and the small business alike, an efficiency mismatch that in turn means lost economic growth.

The point I want to make after all this yapping is that maybe now all that demand for software services can finally be served by willing engineers who can tackle custom, tailor-made solutions at a much faster pace, with less effort and time, so it finally makes economic sense for both parties involved.

Sure, business owners could try to do it themselves with AI, but they still have to run their actual business, and that alone takes a lot of context. They may already be maxing out their decision-making bandwidth for the day, so having a fellow human with some experience in the field could help, especially if he’s also nice to have around. So maybe now’s the time for us software engineers to get better at the human part of the job: listening, explaining, not making people feel stupid for not knowing what an API is. Crazy times, I know, but a little empathy goes a long way.

Maybe I’m biased here because I’m the deer about to get hit by some blinding lights. I’d like to keep working on interesting problems and providing for the family I love. But maybe there’s nothing to worry about. Maybe those bright lights are just the bright future ahead. Either way, the lights are fast approaching. Good luck to everyone!