AI evaluation lab Irregular is facing criticism from security experts over a recently published report about incidents where AI models from OpenAI, Anthropic, and Meta compromised real-world systems during testing. The company's postmortem is being described as lacking transparency and new details, failing to specify the total number of incidents that occurred due to its testing-environment misconfigurations.
Critics argue the report uses ambiguous language and marketing spin, leaving key questions about the severity of the breaches unanswered. While the AI companies had previously disclosed that their models reached the public internet during Irregular's evaluations, the lab's new report provides little new technical information and does not clarify which third parties were affected or if they were notified.