At Square, I built fraud detection models. I also reviewed fraud cases.

We called them suspicions. I had a small case load, but it was enough that I occasionally called up merchants and asked them to explain their business.

Fraudsters could be remarkably creative. Travel agency was a recurring genre. There was always some elaborate explanation for why a merchant had just run a pile of unrelated credit cards in rapid succession.

They were also lazy at the weirdest points. Supposedly unrelated accounts would share an IP or device. Email addresses would technically be different but follow the exact same pattern.

The point of my job was to teach the computer to notice those things too.

At one point we got hit pretty hard by gift-card fraud. Someone would steal a stack of gift cards, run them through Square, and effectively pay themselves. One useful feature turned out to be almost trivially simple: steal a consecutive stack of gift cards and the card numbers themselves were often consecutive too.

That was how a lot of fraud detection got built. A human noticed something weird. That became a hypothesis. We turned the hypothesis into a feature and tested it across a much larger set of transactions. If it held up, the model learned to notice it. Then the human didn’t need to anymore.

Our best risk analysts weren’t just good at working through their queues. They surfaced patterns, corrected mistakes, added labels, and showed us where the models were weak. We trained new models constantly and ran them in shadow mode before putting them into production. Slowly, the models handled more of the ordinary cases. The humans handled the exceptional ones.

There was a paradox to the job. Part of being great at it meant teaching the model how not to need you.

By 2017, Jack Dorsey could say on an earnings call that Square automated risk assessment for more than 99.95% of transactions. That was a long way from the manual operation I joined in 2011.

We carried a similar playbook into home pricing at Opendoor.

Our internal valuation tool was where pricing analysts and pricing models met. Humans thought in comps, and for a long time our models did too (until we relearned the bitter lesson). An operator could pull up a property, inspect comparable sales and the underlying data, annotate something the model had missed, and disagree with its answer.

Between Square and Opendoor, the loop was straightforward: do the work → turn what we learned into features and labels → train the model → review the errors and exceptions, both in backtests and in production → repeat.

I used to call this Error-Driven Development.

A lot of AI diffusion today feels like the same progression – except the distance between doing the work and teaching the machine has drastically compressed.

You can already see the next phase in customer support. AI handles more of the routine questions. Humans take the escalations – and increasingly spend their time correcting bad answers, annotating edge cases, and teaching the system what good looks like.

The doer becomes the reviewer. The reviewer becomes, in part, the trainer.

Looking back, the people who navigated these transitions best weren’t protective of the work they happened to be doing at the time. They were protective of the outcome. They were happy to teach the model something they knew, because then they could move on to the next thing it didn’t.

The best operators didn’t make themselves indispensable to the queue. They helped make the queue disappear.