Coinbase Finds Newer AI Models Fall Behind Predecessors In Fraud Detection Benchmark
In Brief
Coinbase tested six AI models on 16,140 transactions and found newer versions underperform older ones at detecting fraud, prompting a shift to open-weight models.
Coinbase has published findings showing that the latest generation of frontier AI models performs worse than their predecessors at detecting fraudulent transactions, prompting the exchange to move its risk-control operations toward open-weight models with domain-specific post-training.
In the second installment of its “Owning Intelligence” series, released on October 7, Coinbase described evaluating six large language models on a fixed benchmark of 16,140 transactions spanning 7,293 users. The dataset was drawn from nine weeks of production data prior to the deployment of the company’s LLM-powered risk agent, ensuring that the agent’s own interventions would not bias observable fraud outcomes. The cohort contained 813 confirmed fraudulent transactions.
The evaluation compared Opus 4.5 against Opus 5, Sonnet 4.6 against Sonnet 5, and GPT-5.4 against GPT-5.6. Across all three pairs, the newer versions recorded lower recall, F1 scores, and dollar-weighted recall — the share of total fraudulent transaction value successfully identified, a metric Coinbase considers particularly important for payment risk.
The magnitude of regression varied considerably. Sonnet 5’s recall fell by 22.2 percentage points relative to Sonnet 4.6, with dollar-weighted recall dropping 22.9 points; both precision and recall declined, indicating a weaker overall ability to distinguish fraud from legitimate activity rather than a simple trade-off between the two. Opus 5 showed a more modest recall decline of 0.8 points. GPT-5.6 followed a different pattern: precision improved by 11.5 points, but recall fell by 20.7 points and dollar-weighted recall by 21.8, consistent with more conservative classification behavior.
Implications for Model Selection and Coinbase’s Response
Coinbase cautioned that its benchmark demonstrates a regression without establishing its cause, noting that without access to providers’ training data or objectives, explanations remain speculative. Plausible factors include shifts in decision boundaries and prompt behavior, trade-offs introduced by post-training processes, and changes in training-data composition and task priorities.
The findings reshape how the company approaches model upgrades. Rather than asking which release is newest, Coinbase argues that teams should evaluate whether a candidate improves the overall trade-off between precision and recall, whether observed differences stem from the model itself or from changes to surrounding prompts and decision policies, and whether the model meets latency, reliability, and cost requirements for live payment flows.
The results motivated a strategic shift toward model ownership. Coinbase has moved the risk-control agent for its fiat deposit service, Onramp, toward open-weight models and domain-specific post-training. According to the series’ third installment, the company post-trained Qwen3.5-9B using SkyRL on Anyscale, producing a model that outperformed Opus 4.5 on all four fraud-detection metrics while reducing end-to-end serving latency by 55%.
The move builds on earlier results: in Part 1 of the series, Coinbase reported that layering an LLM review agent onto its existing machine-learning model and rules engine — deployed without replacing the existing risk stack — reduced fraudulent transactions by 30% and fraud value by 22% in an online A/B experiment.
Disclaimer
In line with the Trust Project guidelines, please note that the information provided on this page is not intended to be and should not be interpreted as legal, tax, investment, financial, or any other form of advice. It is important to only invest what you can afford to lose and to seek independent financial advice if you have any doubts. For further information, we suggest referring to the terms and conditions as well as the help and support pages provided by the issuer or advertiser. MetaversePost is committed to accurate, unbiased reporting, but market conditions are subject to change without notice.
About The Author
Alisa, a dedicated journalist at the MPost, specializes in crypto, AI, investments, and the expansive realm of Web3. With a keen eye for emerging trends and technologies, she delivers comprehensive coverage to inform and engage readers in the ever-evolving landscape of digital finance.
More articles
Alisa, a dedicated journalist at the MPost, specializes in crypto, AI, investments, and the expansive realm of Web3. With a keen eye for emerging trends and technologies, she delivers comprehensive coverage to inform and engage readers in the ever-evolving landscape of digital finance.
