How to Evaluate an AI Vendor: Security and Procurement Checklist

How to evaluate an AI vendor: a security and procurement checklist covering data retention, SOC 2, ISO 27001 and 42001, SSO, residency, exit terms, scoring.

Guides2 Oct 20267 min read

By the AI App Hunters editors

To evaluate an AI vendor, run the same checklist on every candidate: what happens to your data, what independent proof of security they have, how you log in, where data lives, who else touches it, what it really costs, and how you leave. Score each item, set a few must-haves as pass or fail, and get the answers in the contract, not just on a web page.

Key takeaways

Data use and retention come first. Ask about training on your data before you ask about features.
Certificates are evidence, not answers. Read the report and check scope and dates.
Ask for SSO, a subprocessor list and a data residency answer before the pilot, not after.
Price the whole thing: seats, usage, overages and the cost of leaving.
Use a simple 0 to 2 score with weights, plus hard gates.

What should you check before you buy an AI tool?

Start with the question that matters most for AI: where does your data go? Then work outward. A procurement review for a typical SMB tool should take a day or two, not a quarter. If your company has an AI usage policy, use its data tiers to decide how deep to go. A tool that only sees public content needs a light check. A tool that sees customer contracts needs the full list below. And if you are still deciding whether to buy or build, read custom AI agents: build vs buy first.

What is on the AI vendor security and procurement checklist?

Area	What to ask	Good answer	Weight (1 to 3)
Training on your data	Do you train or fine-tune models on our inputs, outputs or files?	No by default, written in the contract	3
Data retention	How long do you keep prompts, outputs and files? Can we set it or delete on request?	Defined period, configurable, deletion on request and at exit	3
SOC 2	Do you have a current SOC 2 report? Which type, what period, which systems?	Recent report, shared under NDA, covers the product you are buying	2
ISO/IEC 27001	Are you certified? What is the scope statement?	Certificate with scope that includes the product	2
ISO/IEC 42001	Do you have or plan an AI management system certification?	Certificate or a dated plan. Nice to have for most SMBs	1
SSO and access	Do you support SAML or OIDC SSO, SCIM, roles and audit logs?	Yes, and tell us which plan includes it	2
Data residency	Where is data stored and processed? Can we choose a region?	Named regions, choice where you need it	2
Subprocessors	Who are your subprocessors, including model providers? How do you notify changes?	Public list, advance notice, right to object	3
Pricing model	Seats, usage, credits or outcomes? What happens at overage?	Clear unit, caps or alerts, no surprise true-ups	2
Exit terms	Can we export all data in a usable format? How fast do you delete after we leave?	Self-serve export, written deletion timeline	3
Incidents	How and when will you tell us about a breach?	Contractual notice window	2

Treat the weights as a starting point. A legal firm might put residency at 3. A marketing team using public data might drop retention to 2.

Which certifications actually matter?

SOC 2. The AICPA describes SOC 2 as an examination of controls at a service organization relevant to security, availability, processing integrity, confidentiality or privacy. See the AICPA SOC overview. Ask for the report, not the badge. Check that it is recent, that it covers the product you are buying, and whether it is Type 1 (design of controls at a point in time) or Type 2 (controls operating over a period). Type 2 is the stronger signal. Read the exceptions section.

ISO/IEC 27001. ISO describes it as the best-known standard for information security management systems. Check the certificate's scope. A vendor can be certified for its head office and not for the product.

ISO/IEC 42001. ISO/IEC 42001:2023 is, per ISO, the first AI management system standard. It covers how an organization sets policies, objectives and processes for the responsible development, provision or use of AI. It is new, so many good vendors will not have it yet. Treat it as a plus, not a gate. Its absence is not a red flag on its own.

NIST AI RMF. The NIST AI Risk Management Framework is voluntary guidance, not a certificate. A vendor that can explain how it maps its practices to the framework is showing maturity. Do not expect a badge.

None of these tell you what the vendor does with your prompts. That is why the contract questions come first.

What questions should you ask the vendor?

Send these in writing. Good vendors answer in a day.

Do you use our data to train or improve any model, yours or a third party's? Is the answer the same on every plan?
Which model providers process our data? Do they retain it or train on it?
What are the default retention periods for inputs, outputs, files and logs? Can an admin change them?
Can we turn off features that send data to third parties, such as web search or plugins?
Which plan includes SSO, audit logs and role-based access?
In which regions is data stored and processed? Do support staff outside those regions have access?
Can you share the current SOC 2 report and the ISO 27001 certificate and scope under NDA?
Where is your subprocessor list? How much notice do we get of changes?
What is the breach notification window in the contract?
How do we export everything, and how long until you delete it after termination?
What happens to pricing at renewal? Is there a cap on increases?
If your AI feature makes an error that harms us, what does the contract say?

For agents that take actions in your systems, add: what permissions does it need, can we limit them to read-only, and is every action logged? Workflow platforms such as Workato sit in this category, so run the same list on them.

What are the red flags?

The answer to "do you train on our data" changes between the sales call and the contract.
Security page full of logos, no report available even under NDA.
SSO only on a custom enterprise plan with no price, when you need it for ten users.
No subprocessor list, or a list that has not changed in years despite new AI features.
Export means "contact support".
Auto-renewal with a large uplift and a long notice window.
Pricing that depends on a vague "fair use" limit.
Pressure to sign before security review.

One red flag is a question. Three is a no.

How do you score and compare vendors?

Keep the method simple enough that finance and security will both use it.

Step 1: Gates. Pick three to five pass or fail items. Typical gates: no training on customer data under the contract, a subprocessor list exists, data export available, SSO available on the plan you will buy. Fail one and the vendor is out.

Step 2: Score. For each row in the table, score 0 (no or unknown), 1 (partly, or promised for later) or 2 (yes, with evidence).

Step 3: Weight. Multiply each score by the weight and add up. With the weights above the maximum is 50. As a rough rule: 40 or more is a go, 30 to 39 is a conditional yes with named fixes, below 30 is a no. These cutoffs are a suggestion. Tune them to your risk.

Step 4: Compare on total cost. Add the yearly price at your expected usage, plus the extra plan needed for SSO, plus setup time, plus the cost of switching. A cheaper seat price can lose.

Step 5: Record. Save the answers, the reports and the score in one folder, with a review date in 12 months. A recurring checklist in a tool like Process Street or a plain shared doc does the job.

What should end up in the contract?

Ask for: a no-training clause for your data, a retention and deletion schedule, subprocessor notice, a breach notice window, export rights at no extra fee, a clear liability section and a price cap at renewal. If you work with EU personal data, ask counsel about a data processing agreement and transfers. The EU AI Act also sets rules that can reach AI vendors and their customers. The European Commission's AI Act page lists the application dates, and they have been adjusted by an AI Omnibus, so check the current text. This is not legal advice.

If the AI feature is customer-facing, such as a support bot, also test it with your own hard cases. Our comparison of AI support agents shows what to test. For automation platforms, see n8n vs Zapier vs Make.

Bottom line

Do the data questions first, ask for reports not badges, and write the answers into the contract. Use gates plus a weighted score so the decision is easy to explain. Then plan your exit before you sign. The best time to learn how to leave a vendor is before you join.

Frequently asked questions

What is the first thing to check when buying an AI tool?+

Check whether the vendor trains models on your data and how long it keeps your inputs and outputs. Get both answers in writing in the contract or data processing terms, not only on a marketing page.

Is SOC 2 enough to trust an AI vendor?+

No. SOC 2 is an independent report on a vendor's controls, defined by the AICPA, and it is a good sign. It does not tell you how the vendor handles model training, retention or subprocessors, so you still need to ask those questions.

What is ISO/IEC 42001?+

It is an international standard for an AI management system, published by ISO in December 2023. It sets requirements for how an organization governs the development, provision or use of AI. It is useful evidence of process, not a guarantee about any one product.

How do I score vendors against each other?+

Give each checklist item a score from 0 to 2, multiply by a weight of 1 to 3, and add it up. Set a few must-have items as pass or fail gates first, so a high total cannot hide a deal-breaker.

What are the biggest red flags in an AI vendor?+

Refusing to put data-use promises in the contract, no clear way to export or delete your data, vague answers about subprocessors, and security claims with no report to back them. Pressure to sign before a review is a flag too.