12 Questions to Ask an AI Automation Agency Before Hiring (and Red-Flag Answers)

The short answer

Before you hire an AI automation agency, ask twelve questions in four groups: where your data goes, who owns what gets built, where humans stay in the loop, and how success and exit are measured. MIT's 2025 research found that 95 percent of organisations saw no measurable return from generative AI pilots, and the difference between the 5 percent and the rest was approach, not technology. A good agency will answer every question below in plain language. A bad one will answer with a demo.

Why an AI automation agency demo proves very little

The MIT NANDA report The GenAI Divide: State of AI in Business 2025 (July 2025), based on a review of more than 300 public AI initiatives, interviews with 52 organisations and a survey of 153 senior leaders, found that despite US$30 to 40 billion in enterprise spending, "95% of organizations are getting zero return" and "just 5% of integrated AI pilots are extracting millions in value." The authors attribute the gap to approach rather than model quality: tools that do not learn from feedback, do not fit the workflow and have no owner.

Two of the report's findings are directly useful when hiring. External partnerships with customised, learning-capable tools reached deployment about 67 percent of the time in its sample, against about 33 percent for internal builds, so hiring help is reasonable. And while roughly half of AI budgets went to sales and marketing, "some of the most dramatic cost savings we documented came from back-office automation", with faster payback. If an agency only wants to talk about your front end, ask about the back office. Our articles on why AI pilots fail and what AI automation really costs a small business set the context; the questions below are the hiring checklist.

Questions 1 to 3: where does my data go?

1. Which systems and records will the automation read, and which can it write to? You want a list: CRM contacts, inbox, calendar, invoices, customer files. Red flag: "whatever it needs." Scoped access is the first control, and our article on AI agent permissions explains why an agent should have its own login with the minimum rights.

2. Where is the data processed and stored, by which AI provider, and is any of it used to train models? The answer should name the model vendor, the region, the retention period and the training opt-out. Canada's privacy regulators' principles for generative AI include limiting collection, use and disclosure to what is necessary, and keeping the organisation accountable regardless of which tool it uses. If data leaves Canada, the Privacy Commissioner's cross-border guidelines expect comparable protection by contract and disclosure to customers. Red flag: "it's all encrypted" as a complete answer. Encryption says nothing about who can read the data or whether it trains a model. This is general information, not legal advice.

3. What happens when the AI is wrong about a customer? You are asking about accuracy, logging and correction. The regulators' principles call for accuracy appropriate to the purpose and for individuals to be able to access and correct information, including in outputs. Red flag: "it's very accurate." Ask for the error rate they measured on a comparable deployment, and how a customer gets a mistake fixed.

Questions 4 to 6: who owns what gets built?

4. Do I own the workflows, prompts, code and configuration when I pay? Under Canada's Copyright Act, the author is the first owner of a work and an assignment is valid only "in writing signed by the owner." If the contract does not assign the custom work to you, the agency keeps it. Red flag: "you have a licence to use it while you're a client."

5. Which parts run on platforms I would have to keep paying for, and what do they cost at twice my volume? Most automations sit on third-party tools and model APIs with per-task or per-token pricing. You want the stack listed with its price per unit. Red flag: a single bundled monthly fee with no breakdown.

6. If we part ways, what do I walk away with, and how long does the handover take? The answer should include exported workflows, documentation, credentials for every account in your name, and a defined transition period. Red flag: the accounts for the AI provider, the automation platform and the integrations are all in the agency's name.

Questions 7 to 9: where do humans stay in the loop?

7. Which actions will the system take on its own, and which will it queue for a person to approve? There is a right answer for each action, and it is not "automate everything." Sending a quote, issuing a refund, changing a booking and replying to a complaint have different consequences. Our guide to human-in-the-loop AI workflows maps where approvals belong. Red flag: an agency that cannot say which actions need approval has not thought about your downside.

8. Who is accountable when it goes wrong, and how will customers know they are dealing with AI? The British Columbia Civil Resolution Tribunal's decision in Moffatt v. Air Canada, summarised by law firm McCarthy Tétrault, held the airline liable for its chatbot's wrong answer about bereavement fares. The tribunal called Air Canada's argument that the chatbot was a separate legal entity "remarkable" and stated that "it should be obvious to Air Canada that it is responsible for all the information on its website." The regulators' generative AI principles also call for openness about when content or decisions come from AI. Red flag: a vendor who says disclosure hurts conversion and should be skipped.

9. How does the system learn from corrections, and who reviews what it learned? The MIT report's central finding is that tools that do not learn from feedback stall. But learning without review is how a system drifts. You want a described loop: corrections captured, reviewed by a named person, applied deliberately. Red flag: "it improves automatically."

Questions 10 to 12: how will we measure it, and how do I leave?

10. What is the baseline today, what is the target, and when will we measure it? A real answer names a number you can already see: hours spent on the task, time to first reply, invoices chased, no-shows. Red flag: targets expressed only in percentages, with no baseline and no date.

11. Under what conditions would you tell me to stop or scale back? This is the most revealing question on the list. An agency that has an answer (volume too low to justify the platform fee, error rate above a threshold, a process that should be fixed rather than automated) is selling outcomes. One that does not is selling hours.

12. What breaks if your company disappears tomorrow? Which accounts are in whose name, where the documentation is, which parts another developer could pick up, and what runs on the agency's own servers. Red flag: anything essential that only exists inside the agency.

How to read the answers

Good agencies are not threatened by this list. Listen for specifics: named systems, named model vendors, named people, numbers with dates. Listen also for honest limits. An AI agent that handles 80 percent of routine enquiries and routes the rest to a person is a credible claim; one that handles everything is not.

For what it is worth, this is how our own business automation and AI agent engagements are set up: scoped access per system, a fixed price in writing, the workflows and code assigned to the client, accounts in the client's name, approval steps agreed before anything sends, and a baseline measured before the build. None of that is proprietary, and any agency you consider should be willing to put the same things in a contract.

Key takeaways

• MIT's 2025 research found 95 percent of organisations saw no measurable return from generative AI pilots; approach, not technology, separated the rest.
• Ask where data goes, who the model vendor is, what is retained and whether anything trains models; "it's encrypted" is not an answer.
• Insist on written assignment of workflows, prompts and code, a priced list of the platforms underneath, and accounts in your name.
• Agree which actions need human approval before launch; the Air Canada chatbot ruling shows the business, not the bot, carries the consequences.
• Measure a baseline before the build, and prefer the agency that can tell you when to stop.

Frequently asked questions

What should I ask an AI automation agency before hiring?

Cover four areas: data (what the system can access, where it is processed and whether it trains models), ownership (who owns the workflows and code, what the underlying platforms cost, what you keep if you leave), human oversight (which actions need approval, who is accountable, whether AI is disclosed) and measurement (the baseline, the target, and the conditions under which the agency would tell you to stop).

Who is liable if an AI agent gives my customer wrong information?

Your business. In Moffatt v. Air Canada, a British Columbia tribunal rejected the airline's argument that its chatbot was a separate legal entity and held Air Canada responsible for all information on its website. Design approval steps for consequential actions and disclose that customers are dealing with AI. This is general information, not legal advice.

Should I hire an agency or build AI automation in-house?

MIT's 2025 GenAI Divide report found external partnerships with customised tools reached deployment about 67 percent of the time in its sample, against about 33 percent for internal builds, based on self-reported outcomes. An agency is a reasonable choice if you keep ownership of what is built, keep the accounts in your name and measure results against a baseline.

Sources

1. The GenAI Divide: State of AI in Business 2025 — MIT NANDA (Challapally, Pease, Raskar, Chari), July 2025
2. Principles for responsible, trustworthy and privacy-protective generative AI technologies — Office of the Privacy Commissioner of Canada and provincial and territorial privacy regulators, December 2023
3. Guidelines for processing personal data across borders — Office of the Privacy Commissioner of Canada, January 2009
4. Copyright Act (R.S.C., 1985, c. C-42), section 13 — Government of Canada, Justice Laws Website
5. Moffatt v. Air Canada: A Misrepresentation by an AI Chatbot — McCarthy Tétrault, February 2024

How Flowmatix can help

We put this into practice for businesses in Toronto and around the world through our AI Agents & AI Solutions, business process automation services. Want to know what it would take for your business? Get a personalised quote in about 3 minutes.