The AI Automation Agency Trust Test: 12 Questions Operators Should Ask Before Signing

The "AI automation agency" category is, candidly, a mess. The dominant content on the SERP is for aspiring agency founders — courses on how to start one, listicles of which one to join, Skool communities of would-be operators selling each other Zapier templates. Almost none of it is for the buyer.

This is for the buyer. Specifically, for an operating partner or CEO at a $10M–$200M company who has been pitched by three "AI automation agencies" in the last quarter and cannot tell which one is real.

Here are twelve questions. The right answers are specific. Vague answers are red flags.

1. "Show me a system you built that has been running, untouched, for at least six months."

The right answer is a screen-share of a production system, with metrics, that the agency built and the client still uses. The wrong answer is a portfolio page, a case study PDF, or a screenshot of a Zap. If they cannot show you something running today, they cannot build something that will be running in a year.

2. "Who owns the system after you hand it off?"

The right answer is a named human at the client, with a documented runbook, with an escalation path if something breaks. The wrong answer is "you do" with a shrug, or "we'll be there" without a contract that defines what "be there" means.

3. "What happens when the underlying vendor changes their API?"

This happens constantly. OpenAI deprecates models, HubSpot changes a webhook signature, Twilio updates an SDK. The right answer describes a monitoring layer that catches the break, an alerting flow that notifies them, and an SLA on how fast they fix it. The wrong answer is "we'll cross that bridge when we get there."

4. "How do you handle errors in production?"

The right answer is structured: every workflow logs to a queryable store, errors trigger alerts, retries are bounded, and humans get involved before the customer notices. The wrong answer is "we test thoroughly" — which means they don't have a plan for when production reality diverges from their tests.

5. "What does 'managed' mean in your contract?"

The right answer specifies uptime, response time, monthly hours of iteration, and what happens when scope changes. The wrong answer is a flat retainer with no defined deliverables — which is how managed services quietly become "we are paying them and nothing is happening."

6. "What is the ROI calculation we will use to evaluate this in 90 days?"

The right answer is a single, agreed-upon number — dollars saved, hours returned, leads recovered, deals closed — with a baseline measured before the build. The wrong answer is "you'll see the impact" or "ROI is hard to measure."

7. "Show me the system architecture diagram you'll build."

The right answer is a diagram with named systems, named integrations, named data flows. It does not have to be pretty. It does have to be specific. The wrong answer is a slide that says "AI" with arrows.

8. "Who are the engineers actually doing the build?"

The right answer is named humans, ideally on the call, with backgrounds you can verify. The wrong answer is "our team" without specifics — often a polite way of saying "we'll subcontract this to a developer in a Telegram group."

9. "What happens if we fire you?"

The right answer is: you keep the systems, the credentials, the documentation, the runbooks, and any custom code, with a clean handoff process defined in the contract. The wrong answer is anything that involves vendor lock-in to their proprietary platform.

10. "What's your stack and why?"

The right answer is opinionated and specific — for example: "HubSpot or Salesforce as system of record, Make and n8n for orchestration, OpenAI and Anthropic for reasoning, Twilio and ElevenLabs for voice, Postgres and Supabase for data." With reasons. The wrong answer is "whatever the client wants," which usually means they don't have enough reps to have an opinion.

11. "What's a project you walked away from, and why?"

The right answer is real — a specific situation where the agency declined work because the underlying process was broken, the data was unfit, the timeline was unrealistic, or the engagement model was wrong. The wrong answer is "we've never had to," which means they take everything and ship inconsistently.

12. "What does month 13 look like?"

The right answer is a real, evolving operating relationship — quarterly reviews, ongoing iteration, new workflows added as the business changes, with a contract structure that supports it. The wrong answer is "we'll cross that bridge when we get there," which is how a $40K build turns into a $0 second year.

What this list isn't

This isn't a list of gotchas designed to make every agency look bad. Plenty of agencies will answer most of these well. The point is to make the conversation specific. If you ask these questions and the agency welcomes them, you are probably in the right room. If the answers get vague or the rep gets defensive, you have your answer.

For more on the operator side of this, see our buyer's framework for choosing an AI voice agent and the speed-to-lead playbook for mid-market sales teams.

If you'd like to put us through this list, schedule a Discovery Call and we will answer all twelve in writing.