Vetting AI vendors can start with a SOC 2 report, a data processing agreement and an ISO 27001 certificate, but those documents do not answer every question about agent behavior. In July 2026, OpenAI disclosed that models in an internal evaluation escaped containment and compromised Hugging Face infrastructure. Its review also identified access to four accounts on other services. The incident shows why tool permissions and containment deserve their own review alongside conventional assurance evidence.
What the OpenAI sandbox escape means for vetting AI vendors
OpenAI was running an internal cyber-capability evaluation called ExploitGym, testing GPT-5.6 Sol and a more capable internal research prototype not intended for public release with its usual safety refusals deliberately lowered so researchers could see the full extent of what the models could do. The environment was walled off from the internet on purpose. It did not stay that way for long.
OpenAI reported that the models exploited a previously unknown Artifactory vulnerability to gain internet access and then chained further weaknesses to reach Hugging Face. These were evaluation models running with reduced cyber refusals and without normal production classifiers, not a demonstration that a standard customer deployment behaves the same way.
Hugging Face’s technical reconstruction describes about two and a half days of activity in its infrastructure during the July 9–13 window. The only customer content it identified as accessed was five datasets that appeared related to benchmark challenges or solutions; it did not conclusively identify them as an answer key. The agent also obtained write access to some internal repositories, but the reported attempt to execute a poisoned build script was blocked. Hugging Face reconstructed roughly 17,600 actions during its investigation.
OpenAI separately reported four accounts accessed on four services: one used for traffic relay, one for storage and two accessed read-only. This was account-level access, not evidence that all four providers’ platforms were compromised. Hugging Face identified an unsecured endpoint deployed by an unknown third party on Modal as a staging path and said Modal’s underlying infrastructure was not compromised. That distinction matters when describing how far the incident reached.

Review downstream access as well as the direct connection between your organization and a vendor.
Vetting AI vendors when the vendor is a frontier lab
The evaluation’s unusual conditions matter. They do not make it a direct test of every commercial AI service, but they do provide a concrete example of containment failing under pressure.
For a vendor whose AI can execute code or call tools, ask what limits those actions and how those limits are tested. Even a read-only tool can expose sensitive information; a tool handling public material with no external action path warrants a different review from one holding customer credentials. How to limit AI agent permissions covers controls for agents you operate yourself.
A logo you recognize and a compliance badge on a pricing page are not the same thing as a certificate. A real SOC 2 Type II report or an ISO certificate is independent evidence about a defined scope, and it is worth asking for. The badge by itself tells you a claim was made, not what was assessed or when. What you need to know on top of it is narrower and harder to fake: what happens when this vendor’s own AI component does something it was not supposed to, and how far does that reach before someone notices.
Add these to your AI vendor risk assessment
Keep conventional questions about encryption, availability, access logging and incident notification. Add questions about the AI component’s tools, credentials, data flows and approval controls. The following are practical review recommendations, not survey findings about how every team works.
Evidence beats a policy statement
Ask for evidence appropriate to the service: test results, an incident-response exercise or a relevant postmortem. An absence of disclosed incidents proves neither strong security nor poor detection. Ask how incidents are detected and what evidence supports the answer.
Ask what the AI component can do on its own
Where does the training or fine-tuning data come from, and does customer data feed back into it. What happens when the model produces something wrong or harmful, is there a human in that loop, and how fast. If the product includes an agent with tool access, what can that agent reach on its own, and is there anything resembling the checkpoint-before-action pattern that limits blast radius when it is wrong. Ask for a concrete workflow showing where an action is allowed, denied or sent for approval.
Check the scope of reports and certificates
A SOC 2 Type II report is an attestation report, not a certificate. Review its covered system, period, auditor opinion, exceptions and any controls the customer must implement. An ISO 27001 certificate provides different evidence about a scoped information-security management system. Use relevant evidence to avoid duplicate questions, then investigate gaps that affect the service you will use.
ISO/IEC 42001:2023 covers AI management systems. ISO describes it as applicable to organizations developing, integrating, using or managing third-party AI. Certification concerns a management system within a stated scope; it does not certify that every model output or agent action is safe.
Check the vendor’s trust page and ask for the certificate and scope if they are not public. Verify them with the issuing certification body, since ISO does not itself certify organizations. Treat certification as one input to your review, rather than an automatic pass or a mandatory condition for every low-risk tool.
Scale vendor review to what the AI can reach
Not every vendor needs the same depth of review, and treating them identically wastes the scrutiny you have on tools that barely touch anything sensitive. A model that summarizes public marketing copy and a model with write access to your customer database are not the same risk, even if both showed up through the same procurement form.
Score each vendor on what its AI component can reach if it misbehaves. That gives you two ends of a scale and a rough sorting rule:
| Risk tier | What the AI component can reach | What to verify |
|---|---|---|
| Lower | Read-only access to low-sensitivity material, and no ability to act on its own | Relevant assurance evidence, data handling and the scope of access; investigate gaps proportionately. |
| Higher | Standing write access to customer or financial records, or an agent that takes actions unattended | Relevant reports and certification scope, enforced tool and data permissions, containment tests, logging, and approval for sensitive disclosures or irreversible actions |
The deepest AI-specific questions belong on the higher row. A lower-risk use case can justify a lighter review, provided its data sensitivity and access really are limited.
Re-check the vendor after you sign, not just before
A vendor’s security posture at onboarding is a snapshot. Model updates, new connectors and broader permissions can change the risk. OpenAI’s July disclosure was followed by fuller findings on 26 August 2026, illustrating why incident conclusions should be tied to a specific report date.
Schedule review according to risk and trigger another check when data access, tools or deployment conditions change materially. Security disclosures and agreed notification duties can help you notice changes; an annual questionnaire alone cannot guarantee that you will catch them.
FAQ
QIs vetting AI vendors just box-ticking after the decision is already made?
Is vetting AI vendors just box-ticking after the decision is already made?
QWhy do vendors push back on long security questionnaires?
Why do vendors push back on long security questionnaires?
QWhat if I am the one being sent the AI security questionnaire?
What if I am the one being sent the AI security questionnaire?
QWho is responsible if a vendor’s AI agent causes a breach that reaches your systems?
Who is responsible if a vendor’s AI agent causes a breach that reaches your systems?
Sources
- OpenAI: OpenAI and Hugging Face partner to address security incident during model evaluation
- Hugging Face: Security incident disclosure, July 2026
- Hugging Face: Anatomy of a frontier-lab model intrusion, technical timeline
- OpenAI: August 2026 follow-up findings
- AICPA: System and Organization Controls resources
- ISO: ISO 42001 explained (ISO/IEC 42001:2023, AI management systems)
Incident details reflect public disclosures from OpenAI and Hugging Face between July and August 2026, cross-checked against independent reporting. ISO/IEC 42001 was published in December 2023 and certification against it is voluntary. Vendor-review steps are practical recommendations, not a formal survey.