The Defense Advanced Research Projects Agency (DARPA) has released Amendment 04 for the "Artificial Intelligence Quantified" (AIQ) program under Broad Agency Announcement HR001124S0029. This program aims to develop innovative approaches to assess and understand the capabilities of AI systems, particularly focusing on enabling mathematical guarantees for AI performance. The program addresses three capability levels—specific problems, classes of problems, and natural classes of problems—through two Technical Areas (TAs). TA1 will focus on developing rigorous mathematical foundations, while TA2 will empirically verify and scale these results using open-source AI models.

Proposals must align with one of these TAs, and proposers are encouraged to demonstrate expertise in mathematics, AI evaluation, and empirical testing. Proposers may not combine TAs within one submission. Multiple awards are anticipated, and various types of contracts and agreements may be offered, including procurement contracts and cooperative agreements. The program is structured into two phases of 18 months each, with key milestones including PI meetings and evaluations.

The proposal process includes a required abstract phase, with a deadline for abstracts on June 28, 2024, and full proposals due on August 20, 2024, for those invited. Evaluation criteria prioritize scientific and technical merit, relevance to DARPA's mission, and cost realism. Open-source methodologies and collaboration among teams are emphasized, with performers expected to contribute to open-source repositories.

For questions or assistance, DARPA encourages potential proposers to engage with the provided resources, including DARPAConnect and Project Spectrum.