Managing AI Contributions: One Ticket Traced From Filing to Merge
Futurism reported this month on what happened at Shopify after CEO Tobias Lütke made AI use a baseline expectation of the job. Staff started shipping AI output nobody had checked, and colleagues downstream had to clean it up. Lütke's own term for it, in his words: "We call those 'slop grenades' that people toss at each other."
The piece cites an April survey of 1,150 office workers in which 40 percent had encountered what the researchers call workslop — AI-generated work that looks finished and is not. The same survey put the cleanup cost at 3.4 hours a month per worker, which the article extrapolates to a monthly loss of $8.1 million for an organization of 10,000 people.
Those numbers are the visible part. The interesting part is the mechanism underneath them, because it is not really about AI quality.
The actual failure
A slop grenade is not a bad output. It is an unattributable output. Somebody produced something, the provenance of it is gone by the time it reaches you, and so the only way to find out whether it is right is to redo the work. That cost lands on the reviewer, not the author, which is why it compounds — the person who saved twenty minutes is never the person who spends three hours.
Mandating AI use without a system that governs its output does not create the problem. It industrializes it. You have taken the one workflow property that made review affordable — knowing who did what and why — and removed it at scale.
Our position is the narrow one: manage AI contributions, don't blindly enable them. Every AI change here is dispatched against a tracked issue, runs as an identified role, produces a reviewable artifact, and is recorded whether it succeeded or failed. None of that makes the model smarter. All of it makes the output attributable, which is the property that was actually missing.
The rest of this post is one ticket, traced from filing to merge, naming the artifact at each step. It is a boring ticket on purpose — a dependency upgrade, the kind of work nobody writes a case study about.
Specifics in this walkthrough — issue keys, dispatch numbers, commit hashes, pull request numbers and package names — have been changed to protect the enterprise user. The mechanism, the sequence and the artifacts are unchanged.
One change, end to end
1. Issue first
ATLAS-312: "Dependency currency: lockfile five months stale, 43 high and 4 critical advisories, an unused framework tree pulled in as a transitive peer, three majors behind on the build toolchain; bring everything to latest and pin the runtime version."
No dispatch happens without a tracked issue carrying scope and acceptance criteria. There is no path where an agent picks up ambient work. This is a constraint on the humans as much as on the agents: if the thing is worth an agent's time, it is worth a ticket.
2. Approval gate
A human moves the issue into the dispatchable state. Nothing self-starts, and an issue sitting in the backlog with a perfectly good spec stays there until somebody decides otherwise. The gate is intentionally dumb — it is a status transition, not a heuristic — because a gate that reasons about whether to open itself is not a gate.
3. PM review
Dispatch #1184, role pm, ran against ATLAS-312 and posted a machine-readable verdict. The interesting part is that it did not take the ticket's word for anything. It checked the claims against the repository at that moment:
Verified against current repo state (HEAD
a3f0c21): the manifest confirms the versions named in the description … the lockfile confirms the unused framework resolved as a transitive peer under the analytics package, exactly as reported … no runtime pin anywhere in the repo — no version file, no engines field, no platform config — so that part of the description holds too.
Then a recommendation — Coding — and its own verdict line:
Dispatch #1184 complete — role: pm, attempt: 1 Verdict: ✓ PASS Reason: Verified ATLAS-312's claims against live repo state, posted full PM spec comment with implementation steps and definition of done
A PM run that finds the scope unclear holds the ticket for its author instead, and says why. The verdict decides; it does not advise.
4. Coding dispatch
Dispatch #1191, role code, attempt 1. It runs in an isolated checkout, against ATLAS-312 and nothing else, on a branch whose name is assigned rather than chosen: code/ATLAS-312. The branch name is the join key. An operator that pushes to a branch of its own invention is reported as an error naming both names, because a change nobody can trace back to an issue is exactly the artifact this whole apparatus exists to prevent.
5. Evidence, not claims
This is the step that does the most work, and it is the one most easily skipped.
The dispatcher records the working branch's head SHA before the operator is spawned and reads it again after the operator exits. If it did not move, the run produced nothing, and it is recorded as a no-op regardless of what the operator said about itself. An agent's self-report is an input to the record, never the record.
For ATLAS-312 the SHA moved, to 7b42d0e, and the close-out comment is specific about what was and was not verified:
| AC | Result |
|---|---|
| Everything to latest | One package held back, and the reason is named: the type checker's companion plugin throws on the new major. |
| Unused framework tree gone | Dependency resolver reports nothing; zero matching entries left in the lockfile. |
| Audit | 43 high / 4 critical → 1 low, with the upstream tracking issues named. |
| check / build | Type check 0 errors, 0 warnings; build exit 0 on the pinned runtime. |
And, in the same table, the thing that did not happen: "Preview deploy not done — operators don't run deploys." A close-out that only lists successes is a close-out you have to re-derive later.
6. PR and review
Pull request #57, from code/ATLAS-312. Merging is a human decision and is not delegated anywhere, which means a correct, passing, fully-evidenced change can still sit unmerged — and here it did. Thirty-two minutes later the stall monitor said so on the ticket, unprompted:
⏳ NAICOM:
codepassed and opened a PR for ATLAS-312, but the PR has sat unmerged for 32 minutes … nothing is actively watching it, so the ticket reads as idle with zero active dispatches.
A system that only notices failure will quietly accumulate finished work nobody merged.
7. Close-out
The operator's verdict is written to the issue in a form that refuses to overstate itself:
Dispatch #1191 complete — role: code, attempt: 1 Verdict: ✓ PASS — PR SUBMITTED Reason: dependency upgrade, unused framework removal and runtime pin committed (
7b42d0e); PR #57 submitted, preview deploy still needed before merge … Submitted is not merged: the issue is held at the awaiting-merge status and advances to Done once that PR closes.
"Submitted is not merged" is there because the distinction erodes on its own. The verdict, the transcript and the dispatch record persist against the issue key, alongside the session log — which we treat as an SDLC artifact in its own right, not a chat scroll.
ATLAS-312 reached Done when a human approved and merged #57 the following morning.
8. The corrections record
Every step above is a mechanism for catching output. None of them records that the reasoning was wrong. So there is a separate, first-class record for that, and it is the piece nobody else seems to show.
When an operator or the assistant gets something wrong — anything that had to be said twice or reversed — it is written to a counted record, classified, and attributed to the dispatch and issue it happened on. An entry looks like this:
#128 · class
wrong-mechanism· HELIX-204 · rolecodeObserved: Passed the database connection string, password included, as a command-line argument — where any process listing on the host can read it Correction: The adjacent branch of the same function already routes its password through an environment variable for exactly that reason. The fix follows it, and passes the rest of the connection string through unchanged.
Because entries are classified, they count. As of this writing the record holds 402 entries. The largest class is unverified-claim at 118 — a claim stated without running the check that would have falsified it. stale-state follows at 58, wrong-mechanism at 49. By source it splits assistant 277, operator 125. Four entries repeat an earlier one.
Those counts are the useful output. A single correction is an anecdote; a class with 118 entries and a rising weekly trend is a design problem, and it tells you which guard to build next. The classification even feeds back into the dispatch prompt: an operator in a role with a heavy unverified-claim history is handed that history, with the check each class implies, before it starts.
What this does not do
It would be easy to over-claim here, so:
This catches unreviewed output and it preserves provenance. It does not make the model correct. Dispatch #1191 could have shipped a confidently wrong dependency pin, passed every gate, and been merged — the apparatus would have recorded precisely who did it, when, against what issue, with what evidence, and it still would have been wrong. What the record buys is that the error is findable in minutes rather than archaeology, and that the class of error becomes visible across hundreds of runs instead of dying in one person's memory.
It also costs something real. Every one of those steps is latency, and a fair amount of the machinery exists to protect against failure modes that a careful human would not hit. That trade is worth making at agent volume and is probably not worth making for one person with one assistant.
The question worth asking
The Shopify story is usually read as a story about AI quality. It reads better as a story about attribution. Nobody in it could answer what AI did, on whose instruction, reviewed by whom, and with what result — and once you cannot answer that, every downstream reader pays the review cost in full, every time.
So the question for anyone rolling out an AI mandate is not whether the output is good. It is: pick any change that shipped last week, and see how far back you can trace it.