Six always-on agents, one phone, one reporter's inbox, 24 hours. That was the setup behind WIRED's day-long agent test, which covered Meta's Muse, OpenAI's Dots, Instinct, Anthropic's Claude Duet, Google's Gemini Spark and xAI's Grok Bot. It is a useful map, but it cannot tell you whether an agent will handle your email, your calendar or your lunch order.
The fix is to run a similar test on your own data. This guide turns the story's test structure into a scorecard you can use on any agent's trial or free tier. The scorecard and worked example are a template, not our own test results.
What the original test did, and where it stops
The reporter set up the agents the night before, then ran them through a series of tasks over 24 hours. The tasks measured proactivity, direction following and browsing. The reporter scored each attempt from 1 to 5.
The story says the reporter designed five tests. This scorecard keeps four: a morning briefing, a group lunch order, a calendar and email conflict, and trip research.
The limits matter. This was one reporter on one day, and the story itself calls the tests a limited sliver of what the tools can do. It also says agents improve as you share more about yourself, so a day-one score is a floor, not a ceiling. Treat any ranking, including this scorecard, as a screening tool.
The results still show where agents slip:
- Only Muse and Dots kept researching for most of a requested 30-minute window.
- Gemini Spark added a single $5 roll for the boss in a two-person lunch order.
- Instinct and Claude Duet built carts that ran pricier than the rest.
- Only two of six agents flagged a planted calendar conflict.
Step 1: Set up a safe trial
The story also warns about privacy. An X user claimed he connected agents to his bank account and work Slack, then was mortified when an agent posted his balance for colleagues to see. The reporter shared only a personal Google Calendar and Gmail. Follow that model, and go narrower if you can.
- Connect a personal calendar first, not your work one. Add a few test events you can safely expose.
- Connect a secondary or low-stakes inbox, or forward a few messages into one. Skip anything with legal, medical or financial threads.
- Withhold banking, brokerage and work chat tools such as Slack until the agent earns trust. This is our recommendation, not a vendor rule.
- For shopping, build the cart but stop before paying. Pay yourself, or use a card with a low limit if the agent must check out.
- Note your plan's usage meter before you start and again after 24 hours. Write both numbers down.
The last item has a source behind it. One of the AI-generated pitches in the story claimed that Claude Duet's check-ins, unrequested draft replies and overnight dashboards bill against the same usage limit as chatting. That claim came from an agent's pitch, not a verified finding. It is still worth measuring, because a proactive agent can spend your allowance while you sleep.
Costs also vary widely. At the time of the story:
- Muse was free.
- Instinct was free with an invite code and worked through texts.
- Gemini Spark cost $20 a month in beta.
- Claude Duet cost $20 a month as a wait-listed research preview.
- Grok Bot cost $30 a month.
- Dots cost $100 a month.
Check current prices before you decide, since they may have changed.
Step 2: Run the four tasks
Run all four within one 24-hour window. Give every agent identical wording, and write down timestamps.
Task 1: Morning briefing
Ask for a briefing on your field and a short pitch or action item built from it. Tell the agent to spend a set time on it, such as 30 minutes, and note when the reply arrives.
The story found that four of six agents ignored a 30-minute research request. They answered instantly or after a couple of minutes. This task tests instruction following. Usefulness is a separate question, and the pitches in the story still bored the editors.
Task 2: Group lunch order
Order for two people with conflicting needs: one vegan, one who wants meat and caffeine. Add a location limit and a budget cap, and stop at the cart.
Compare the cart against your brief, not just the restaurant. The story's agents mostly picked the same two Vietnamese places. The differences showed up in the carts.
Task 3: Calendar and email conflict
Create a calendar event at 2 pm and mark it immovable. Email yourself a reminder to call a family member at 2 pm. Do not mention either to the agent.
See whether the agent messages you before the overlap and offers alternatives. Agent replies quoted in the story point to the same gap. One said it treated the calendar as the only source, and another said nothing was cross-checking the inbox against the calendar.
Task 4: Trip research
Ask for a shortlist of flights, neighborhoods or food options with links, and say you will book yourself. The story's vacation section does not report agent results, so this rubric is an adaptation. It checks whether the agent respects your limits and shows its sources.
Step 3: Score with the rubric
Score each task from 1 to 5 using these anchors. Copy the table, add a score column, and print it.
| Task | 5 | 3 | 1 | | --- | --- | --- | --- | | Morning briefing | Used most of the requested time; sourced, relevant, usable | Stopped early but output is usable | Instant, generic, ignored the time request | | Lunch order | Both diets met, full cart for both, within budget | Right place, but cart is pricey or missing an item | Undersized or wrong cart | | Calendar conflict | Messaged before 2 pm, named both sources, offered options | Flagged only after you asked | Missed it entirely | | Trip research | Sourced shortlist that honors your limits | Useful but thin or partly unsourced | Unsourced or off-brief |
Pass thresholds (a suggested rule, not an industry standard):
- A total of 15 or more out of 20.
- No task below 3.
- At least 4 on the calendar task if you are paying for proactivity.
Fail any one and you do not pay yet.
Worked example: a hypothetical agent
The numbers below are invented to show the method. They are a template, not test results for any real product.
Call it Agent A:
- Briefing: 4. It researched for roughly 25 of the 30 minutes you asked and produced a sourced summary.
- Lunch: 3. It picked a nearby restaurant and met both diets, but left out the boss's second item.
- Calendar conflict: 3. It said nothing until you asked whether anything clashed at 2 pm, then answered correctly.
- Trip research: 4. It returned linked options within your dates.
The total is 14 out of 20. Agent A fails on two counts: the total is under 15, and the calendar score is under 4.
It suits someone who wants a reactive research and shopping helper and will prompt it each time. It falls short on proactivity, the main reason to pay for an always-on agent over a chatbot. The sensible call is to stay on a free tier, or retest after a week of use, since the story notes agents improve as they learn your habits.
Next action: run the calendar test first
Start with the calendar-conflict test. It is cheap to set up and reveals proactivity better than the other three tasks. If an agent cannot catch a planted clash between your inbox and calendar, no ranking should talk you into paying for it. If it passes, the other three tasks are worth your time.
Editor's notes
- I changed Agent A's calendar score from 2 to 3. The original 2 contradicted the rubric, which defines "flagged only after you asked" as a 3. The total is now 14, and the pass/fail outcome is unchanged.
- I attributed the bank-and-Slack incident to the X user's claim, as the source does, instead of stating it as fact.
- I removed the "five minutes to set up" claim because the source does not support it.
- I replaced "the story's visible text" with a direct statement of what the story's vacation section does and doesn't report.
