White House AI commitments connect model testing to release evidence

Seven AI companies have agreed to a shared set of testing, security and transparency commitments at the White House. The July 21 package brings together Amazon, Anthropic, Google, Inflection, Meta, Microsoft and OpenAI, with provisions that connect model development to release documentation and later vulnerability reporting.

File photo: US Flag Backlit, ©2009 Joshua Nathanson (Jnn13), via Wikimedia Commons. CC BY-SA 3.0. Cropped/resized; responsive adaptations carry the same licence. Photographed March 12, 2009, not at the AI announcement.

The official announcement says the companies are undertaking the commitments immediately. The accompanying original text explains which systems they cover and where implementation depends on further technical work.

The model boundary matters

Where the commitments refer to particular models, their scope is generative models that are more capable overall than the industry’s current frontier. Its July 2023 reference points are GPT-4, Claude 2, PaLM 2 and Titan, plus DALL-E 2 for image generation. It gives a comparative capability boundary rather than a numerical training-compute threshold.

For major public releases within that scope, companies will develop internal and external adversarial testing, drawing on independent specialists. The work covers risks from misuse, cybersecurity, tool interactions and societal effects. Evaluation methods themselves remain an area of research, so the agreement also supports better testing and interpretability work.

Companies commit to publishing reports for all significant new public model releases within scope. Those reports should describe capabilities, appropriate and inappropriate uses, important performance limitations, societal risks such as fairness and bias, and testing results, with dangerous-capability information disclosed where it’s responsible to do so. Separate vulnerability-reporting provisions allow bounties, contests, prizes or extensions of existing bug-bounty programmes.

Provenance has an implementation sequence

The audiovisual provision allows provenance information, watermarking or a combination. It covers content from publicly available systems within scope introduced after the watermarking system is developed, and calls for tools or APIs that can identify content produced by the company’s system.

The text has explicit exceptions for material that’s readily distinguishable from reality or designed to be recognisable as a company’s AI output, such as an assistant’s default voice. Identification data should name the originating service or model; personal information identifying the user isn’t required. These details give developers a more specific engineering brief than a general instruction to label AI output.

Microsoft’s separate July 21 commitments illustrate one implementation path. The company plans to use the C2PA specification to automatically sign images from Microsoft Designer and Bing Image Creator with origin metadata. It expects engineering to finish by the end of 2023. That’s a company-specific target, rather than a completion date agreed by all seven participants.

Shared practices and company-specific promises

The common package also calls for a forum or mechanism to develop shared safety practices, protection for unreleased model weights, research on societal risks and work on beneficial applications and education. Weight protection includes limiting access by job function and safeguards against unauthorised disclosure.

Microsoft adds its own plans for annual responsible-AI reporting and implementation of the NIST AI Risk Management Framework. Its support for a future licensing regime and a national high-risk-system registry is separately stated policy advocacy. Those proposals don’t become requirements through the seven-company agreement.

The companies intend the voluntary commitments to continue until regulations addressing substantially the same issues take effect. For now, the practical connection is between evidence gathered before release, documentation supplied with significant releases and channels for discovering problems afterwards, alongside technical work to identify generated media.