Artificial Analysis ranks Grok 4.7 first on its hallucination checked Harvey legal agent benchmark
Today we are announcing Harvey LAB-AA v1.1 in collaboration with Harvey. This updates our scoring methodology for the Legal Agent Benchmark (LAB) to add a hallucination check and require correct responses to not include material misstatements. LAB-AA v1.1's new headline metric, Hallucination-Gated All-Pass Rate, only credits a task when the deliverables satisfy every rubric criterion and contain no material hallucinations. We define a material hallucination as one that would mislead a reader on a substantive point, such as a wrong contractually required date, while a minor hallucination is a real error that is unlikely to meaningfully affect the legal interpretation of a deliverable. Minor hallucinations are reported separately and do not affect the headline score. Grok 4.7 (xhigh) leads at 9.4% Hallucination-Gated All-Pass Rate, and >60% of otherwise passing results across the models tested at launch contain a material hallucination. This is the first step in enhancing the methodology for Harvey LAB-AA. In future updates, we’re working with Harvey to better account for the full set of factors lawyers value, including usability features like style and tone. Key takeaways: ➤ Top of the leaderboard: Grok 4.7 (xhigh) from @SpaceXAI leads at 9.4% Hallucination-Gated All-Pass Rate, narrowly ahead of Muse Spark 1.3 (max) from @AIatMeta at 8.9% and GPT-6 Astra (max) from @OpenAI at 8.6% ➤ Hallucinations reshape the leaderboard: without the hallucination gate, Muse Spark 1.3 (max) would lead clearly with a 26.7% all-pass rate, but two thirds of those passes contain one or more material hallucinations, resulting in a Hallucination-Gated All-Pass Rate of 8.9%. GPT-6 Astra (max) retains almost all of its passes after the hallucination gate (8.9% to 8.6%) and moves from joint 10th to 3rd ➤ The GPT-6 model family is the most grounded: GPT-6 Astra (max) averages 0.03 material hallucinations per task (4 across all 120 tasks) and GPT-6 Sol (max) averages 0.07. When comparing six checker models on a 20-task subset, GPT-6 Astra had zero material hallucinations under every checker, including Claude Opus 5.5 (high). By comparison, Gemini 3.8 Flash (high) averages 13.96 material hallucinations per task. ➤ Top-scoring models aren’t the most expensive: Grok 4.7 (xhigh) takes the top spot on the leaderboard at a cost of ~$9.50 per task, under half the cost of Claude Fable 5.1 (max with fallback) at ~$21.70, the most expensive model. Muse Spark 1.3 (max) represents strong performance for its cost, landing in second for ~$4.20 per task Thank you to @nikogrupen and @ItsJulioPereyra from @harvey for their work on Harvey LAB and collaboration.
Context
Once answers with material hallucinations are failed, the best models complete fewer than 10% of legal tasks, and the ranking reshuffles.
Harvey's Legal Agent Benchmark tests AI agents on legal work, and Artificial Analysis runs its own version, Harvey LAB AA, on 120 tasks. Version 1.1, announced with Harvey on October 8, adds a hallucination check to the scoring. Elon Musk quoted the announcement, writing that Grok 4.7 ranks first in legal matters.
Deep dive
Artificial Analysis says its new headline metric, Hallucination Gated All Pass Rate, credits a task only when the deliverables meet every rubric criterion and contain no material hallucination, meaning an error that would mislead a reader on a substantive point, such as a wrong contractually required date. Minor hallucinations are reported separately and do not affect the score.
On that metric, Artificial Analysis says Grok 4.7 (xhigh) from SpaceXAI leads at 9.4%, narrowly ahead of Meta's Muse Spark 1.3 (max) at 8.9% and OpenAI's GPT 6 Astra (max) at 8.6%. Without the gate, Muse Spark 1.3 would lead clearly at 26.7%, but Artificial Analysis says two thirds of those passes contained at least one material hallucination. Across the models tested, it says more than 60% of otherwise passing results contained a material hallucination.
Artificial Analysis calls the GPT 6 family the most grounded. GPT 6 Astra (max) averaged 0.03 material hallucinations per task, 4 across all 120 tasks, while Gemini 3.8 Flash (high) averaged 13.96. On cost, it puts Grok 4.7 at about $9.50 per task, less than half the roughly $21.70 of Claude Fable 5.1, the most expensive model, while Muse Spark 1.3 took second place at about $4.20 per task.
What to expect next
Artificial Analysis says this is the first step in enhancing the Harvey LAB AA methodology, and that future updates with Harvey will try to account for more of what lawyers value, including style and tone.
The margins at the top are small, with less than one percentage point separating the first three models, so the order could change as labs release new versions or Artificial Analysis retests existing ones.