←
Artificial Analysis4,233 posts
Artificial Analysis avatar✓

Artificial Analysis ✓

@ArtificialAnlys

Independent analysis of AI

◷ Joined January 2024▧ 3,025 Photos and videos
Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·6 Jul 2024

Thanks for the support @AndrewYNg! Completely agree, faster token generation will become increasingly important as a greater proportion of output tokens are consumed by models, such as in multi-step agentic workflows, rather than being read by people.

Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·8h

Can’t get to Miami for the GTA VI launch? Two AI models built the city for you We asked Claude Fable 5.2 from @AnthropicAI and GPT-6.2 Astra from @OpenAI to build Miami as it looks this week, dressed as Vice City for the GTA VI launch ➤ One prompt each, in the AA-3D sandbox with no internet access ➤ Both lined Ocean Drive with real hotels the prompt never named, including the Colony and the Avalon ➤ Each scene can be explored by car, speedboat or blimp, at sunset or at night

0:19
Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·16h

GPT-6.2 Astra ranks #1 on AA-3D, our benchmark measuring how well models can build interactive 3D experiences. Humans preferred these scenes in 139 of 152 head-to-head votes against other models

0:19
Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·18h

Microsoft-Decision-1.1 improves by 5 points on AA-Omniscience-Decision v1.1 and 7 points on HLE-MultiChoice v1.1 compared to its predecessor.

Post attachment
Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·18h

Microsoft-Decision-1.1 scores 68 on the Artificial Analysis Decision Index v2.1 @Microsoft has released Microsoft-Decision-1.1, an update to their existing decision model, Microsoft-Decision-1. It remains priced at $0.042 per one million input tokens. Key results: ➤ Microsoft-Decision-1.1 is the #8 decision model. On the Artificial Analysis Decision Index, Microsoft-Decision-1.1 scores 2 points ahead of Elephant 9B and Jev 1.14 but 1 point behind d1.2. ➤ Microsoft-Decision-1.1 improves by 6 points over its predecessor. Microsoft-Decision-1.1’s score increases by 6 points to 68 from Microsoft-Decision-1’s 62, improving across all 6 individual evaluations that make up the Artificial Analysis Decision Index. ➤ Microsoft-Decision-1.1 is the #5 fastest model on a per task basis. Microsoft-Decision-1.1 has a Time per Task on the Artificial Analysis Decision Index of 123 ms, faster than d1.2 (150 ms) but not as a fast as Mammoth 27B (106 ms). It is more than 2x as fast as Microsoft-Decision-1 (274 ms).

Post attachment
Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·18h

GPT-6.2 Astra is the new leader in AA-Robotics v0.1, scoring 79, up 10 points on GPT-6 Astra and ahead of GPT-6.2 Sol (78), Claude Fable 5.2 (75) and Gemini 4 Hydrogen (58) Controlling a real robot arm through Codex, it passed six of our eight physical tasks, partly completed the pouring liquid, and failed only to measure the box.

0:30
Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·18h

MAI-Image-2.7 ranks #1 for 4 of 10 Image Editing use cases: Architecture & Real Estate, Consumer, Frontier, and Marketing & Advertising Use cases measure the workloads end users run. We measure the same 10 use cases for Image Editing and Text to Image. In Text to Image, MAI-Image-2.7 ranks #2 for Marketing & Advertising and for Productivity & Knowledge Work. MAI-Image-2.7-Flash's highest use-case ranks are #3 for Frontier and Animation & Gaming edits, and #5 for Productivity & Knowledge Work in Text to Image. ➤ Marketing & Advertising covers ad creatives, campaign visuals, posters, and brand imagery. ➤ Architecture & Real Estate covers interior and exterior visualization, floor plans, and staging. ➤ Frontier covers capabilities at and beyond the current frontier.

Post attachmentPost attachmentPost attachmentPost attachment
Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·18h

MAI-Image-2.7 ranks #1 on 4 of 7 editing actions and on Knowledge in Text to Image. MAI-Image-2.7-Flash's strongest editing actions are Scene & Style Edit (#3) and Enhancement & Restoration (#4) Our Image Arena taxonomy measures 7 editing actions for Image Editing and 9 capabilities for Text to Image. Editing actions measure what an edit asks the model to do. Capabilities measure what a prompt asks the model to get right. MAI-Image-2.7's largest gaps to the leader are Object-Level Edit in Image Editing and Lighting in Text to Image. ➤ Text or Symbol Edits covers editing text, typography, logos, and symbols within the image. ➤ Enhancement & Restoration covers denoising, retouching, artifact removal, and repair of damaged or degraded images. ➤ Knowledge covers real landmarks, species, and domain facts across science and common sense.

Post attachmentPost attachmentPost attachmentPost attachment
Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·18h

MAI-Image-2.7 and MAI-Image-2.7-Flash both set new points on the Quality vs Price frontier, in Image Editing and in Text to Image Released November 17, MAI-Image-2.7 and MAI-Image-2.7-Flash are Microsoft AI's new models for text to image generation and image editing, now in public preview. MAI-Image-2.7 follows MAI-Image-2.6 and the editing-only MAI-Image-2.7-Preview from October 27. We benchmarked both models through Microsoft's API with square outputs. ➤ Ranks: MAI-Image-2.7 is #1 on AA-Image-Editing v2.1 and #4 on AA-Image-T2I v2.1. MAI-Image-2.7-Flash is #7 and #11 ➤ Frontier: MAI-Image-2.7 is the new top of the Image Editing frontier, replacing GPT Image 2.6 Sunburst (max). MAI-Image-2.7-Flash sets a new lower-price point on both boards, pushing Seedream 5.5, MAI-Image-2.6-Flash and Canva Craft off the frontier ➤ Price: $34.82 per 1k images for MAI-Image-2.7, down from $38.91 for MAI-Image-2.6. $10.24 per 1k images for MAI-Image-2.7-Flash, down from $19.46 for MAI-Image-2.6-Flash. Both at Microsoft Foundry public preview rates ➤ Availability: Microsoft Foundry, MAI Labs (including the new Canvas experience, powered by MAI-Image-2.7-Flash), and providers fal, Genspark, OpenRouter and Vercel

Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·18h

Microsoft AI has launched MAI-Image-2.7, the new #1 in AA-Image-Editing v2.1 at one sixth the price of the four models behind it MAI-Image-2.7 and MAI-Image-2.7-Flash both take new places on the Quality vs Price frontier in Image Editing and Text to Image, at $34.82 and $10.24 per 1k images. Both perform best in Image Editing: MAI-Image-2.7 ranks #1 for Text or Symbol, Scene & Style and Enhancement & Restoration edits, and MAI-Image-2.7-Flash ranks #3 for Scene & Style edits. Congratulations to @MicrosoftAI on the launch of MAI-Image-2.7 and MAI-Image-2.7-Flash! See below for our detailed analysis of both models in Image Editing and Text to Image 🧵

Artificial Analysis avatar
Artificial Analysis@ArtificialAnlys·19h

OpenAI leads Harvey LAB-AA v1.1 with Astra for Law 6.2, a specialized GPT-6.2 Astra configuration with an integrated legal search index OpenAI describes Astra for Law 6.2 as a model configuration of GPT-6.2 Astra that pairs the model with an expanded legal search index and instructions for researching a question, weighing the authorities, and writing the answer. Our implementation of Harvey’s LAB benchmark tests real-world legal work across 120 private tasks spanning 24 practice areas, penalizing hallucination in its headline metric, Hallucination-Gated All-Pass Rate. Astra for Law leads Harvey LAB-AA v1.1 with a score of 20.8%. GPT-6.2 Astra (max) comes in second with 15.0%.

Post attachment