Shopify CEO Tobi Lütke said on the Sources podcast that it "didn't matter" how employees answered the AI question in the company's reviews. He described it as a peer-rating item about how reflexively a colleague shows up with an AI prototype, and said "sometimes you play a little bit of tricks there." (Search Engine Journal)
The lesson for your team is narrow. Search Engine Journal points out that an adoption question can't tell a lead whether anyone checks AI drafts before they reach an editor or client. If you manage writers or analysts, you need a review that measures that gap directly. Here is a repeatable way to build one.
Step 1: Define what good AI use means for your team
Start with one sentence you can defend: AI-assisted work is good when it reaches the next person already checked, sourced, edited and fitted to the brief. Usage volume is not part of that sentence.
Search Engine Journal also reports that Shopify staff have a name for AI output handed to a colleague without being read: a "slop grenade." That is the failure you want to detect, and heavy and light users can both produce it.
Tell your team up front that disclosing AI use carries no penalty. If disclosure costs people something, they will stop disclosing, and you lose the evidence you need.
Step 2: Replace the adoption question with verification questions
The questions below cover self, peer and manager reviews. Each asks for something you can inspect, not a self-rating of enthusiasm.
- Self: Pick one recent AI-assisted deliverable. Which claims did you verify, and where did you verify them?
- Self: Describe one time you rejected or heavily rewrote AI output. What made it unusable?
- Self: How do you record AI use on a draft so an editor knows what to check first?
- Peer: When you received this person's AI-assisted work, how often did you find unchecked claims, dead links or off-brief sections?
- Peer: Does this person flag what they are unsure about, or does uncertainty surface only after you catch it?
- Manager: In the sampled drafts, what share of checkable claims had a working source that supports them? (Use the scorecard in Step 3.)
- Manager: How much editor rework did this person's AI-assisted drafts need compared with their non-AI drafts?
- Manager: Did this person improve a prompt, template or checklist that made the team's output more reliable?
Questions 4 and 5 rely on peers, so treat them as conversation prompts, not scores. Lütke's remark about playing "tricks" with a peer-rating question is a reminder to handle peer ratings with caution.
Step 3: Build a draft QA scorecard
Score each draft on five criteria, 0 to 2 each, for a maximum of 10. Anchor every score to something visible in the draft or its working files.
| Criterion | 0 | 1 | 2 | |---|---|---|---| | Facts | Claims unchecked or wrong | Most checked, minor gaps | Every checkable claim verified | | Sources | None, broken, or not supporting the claim | Present but weak or secondary | Each key claim backed by a source that says what the draft says | | Edits made | AI text largely unchanged | Surface wording changes | Substantive changes: restructured, cut, original input added | | Intent fit | Misses what the searcher or brief needs | Partly fits | Matches the intended search task and brief | | Brand voice | Generic or against the style guide | Mostly on voice | Clearly on voice |
Add one hold rule: a 0 on facts or sources sends the draft back regardless of total. A polished draft with an invented statistic should not pass because its voice scores well. This pairs well with see also: how to audit content for chatgpt information seeking.
Use three bands: 8 to 10 is ready for the editor, 5 to 7 needs targeted fixes, and 0 to 4 goes back to the writer.
Step 4: Sample drafts and gather evidence
Pull 10 AI-assisted drafts from the review period. Choose them by a repeatable method, such as every third item in the content tracker, so you are not cherry-picking. Spread them across writers when you can.
For each draft, ask for three things: the source list, the tracked changes or version history, and a short note on which parts AI produced. Then check each claim yourself. Open the links, and confirm each page says what the draft says it does.
Worked audit (hypothetical example)
This example is illustrative. The drafts, scores and outcomes are invented to show the method and are not real data.
A content lead samples 10 AI-assisted drafts and scores them with the scorecard.
| Draft | Facts | Sources | Edits | Intent | Brand | Total | Band | |---|---|---|---|---|---|---|---| | D1 | 2 | 2 | 1 | 2 | 2 | 9 | Ready | | D2 | 1 | 1 | 1 | 2 | 1 | 6 | Fixes | | D3 | 2 | 2 | 2 | 2 | 2 | 10 | Ready | | D4 | 0 | 0 | 0 | 1 | 1 | 2 | Return | | D5 | 1 | 2 | 1 | 2 | 2 | 8 | Ready | | D6 | 2 | 1 | 1 | 1 | 1 | 6 | Fixes | | D7 | 1 | 0 | 0 | 2 | 1 | 4 | Return | | D8 | 2 | 2 | 1 | 2 | 1 | 8 | Ready | | D9 | 1 | 1 | 2 | 1 | 2 | 7 | Fixes | | D10 | 2 | 2 | 2 | 1 | 2 | 9 | Ready |
The totals add to 69, an average of 6.9. Five drafts land in the ready band, three need fixes and two go back. D4 and D7 trip the hold rule.
The average matters less than the pattern. Intent fit is the strongest criterion (16 of a possible 20 points), while edits (11) and sources (13) score lowest. In this made-up case, the lead would conclude the team writes on-brief but verifies unevenly. Coaching would target source checking and substantive editing, not prompting skill. None of this depends on how often anyone opened an AI tool.
Step 5: Verify your scoring is consistent
A scorecard is only useful if two careful people would score a draft similarly. Test that in two ways.
Second-reviewer re-score
Have another editor or lead re-score a sample of the drafts without seeing your scores. Aim for at least four of the ten. Compare totals and bands, then discuss any draft where totals differ by 2 or more points or the band changes.
Suppose the second reviewer re-scores D1, D4, D6 and D9 and returns 9, 3, 4 and 7. D1 and D9 match. D4 differs by one point and stays in the same band.
D6 drops from 6 to 4 and moves from Fixes to Return. The gap traces to facts (2 versus 1) and sources (1 versus 0). The reviewers reopen the draft and find one cited page that does not support the claim attached to it. They agree the lower score is right and tighten the scorecard anchor: a link that does not support its claim scores 0.
Compare scores with later editor rework
Log what editors fix after drafts pass through. Count corrections or note rework time on the same drafts, using whatever your team already records. Then check whether low-scoring drafts drew more rework than high-scoring ones.
In the hypothetical, editors logged heavy rework on D4, D6 and D7 and almost none on D1 and D3. That pattern supports the scorecard. If it had run the other way, you would revisit the criteria or the scoring.
Also read: how to audit google ads for wasted spend in the ai era in depth
Limitations and how people will game this
Small samples mislead. Ten drafts from a team of five can swing on one unusual assignment. Do not treat a single audit as a verdict on a person. Repeat it each cycle and look at trends.
Scoring is subjective. Anchors reduce disagreement but do not remove it, which is why the second-reviewer step exists. Judging whether a source truly supports a claim still takes reading and judgment.
Any metric invites gaming. Writers can pad drafts with citations they never read, make cosmetic edits to lift the edits score, or save their best work for sampled items. Counter this by verifying sources yourself, varying the sample, and weighing the editor-rework comparison. Do not tie pay or promotion to a single audit score.
The audit says nothing about search performance. A well-checked draft is not a ranking or traffic promise, and this method measures neither. It measures whether work arrives verified, which is the part you can control.
What to expect next
Your first audit will mostly expose inconsistency in the scorecard, not in your people. Expect to rewrite some anchors after the second-reviewer comparison.
Over several cycles, aim to move the review conversation from "do you use AI?" to "show me what you checked." The rework log will tell you whether that shift is real.
