👀 DeepSeek V4 Pro launched with the marketing line "we've closed the gap with the American frontier". The US NIST took the model for independent testing on closed datasets - and the public picture fell apart
What CAISI (NIST's center for AI standards) found:
DeepSeek V4 Pro came out on April 24, 2026. On public benchmarks it looked convincing: SWE-Bench Verified - 80.6% vs 80.8% for Claude Opus 4.6, near parity. Codeforces rating 3,206 - the best result of any model at launch. Bloomberg wrote the same day that the model "hasn't closed the gap with the American leaders" - but that read like an editorial stance. CAISI added two datasets the model was guaranteed not to have seen in training: ARC-AGI-2 semi-private and its own PortBench. On those, the results broke down
Closed tests - the real gap:
🟢 Cybersecurity (CTF-Archive-Diamond, 285 tasks) - V4 Pro: 32% / GPT-5.5: 71%
🟢 Abstract reasoning (ARC-AGI-2, closed dataset) - V4 Pro: 46% / GPT-5.5: 79%
🟢 Software engineering (SWE-Bench Verified) - V4 Pro: 74% / GPT-5.5: 81%
CAISI's conclusion is clear-cut: the real lag behind the American frontier is about 8 months, not 3-6 like in DeepSeek's marketing. It's a steady trend since 2025 - the gap isn't shrinking. V4 does worst on long, complex tasks: where you need to hold many steps and a lot of context at once. That's exactly what separates a real agentic workload from synthetic tests. The mechanics are classic: the company picks its own benchmarks, takes the ones where its numbers peak, gets "almost caught up" headlines - NIST simply took tasks the model couldn't solve and wasn't trained to solve
Where DeepSeek V4 genuinely wins:
💡 Price - $0.145 per million tokens, 7x cheaper than GPT-5.5 and Claude Opus 4.7
💡 Open weights - you can deploy it locally and fine-tune it for your task
💡 Context - 1M tokens, MoE architecture (49B active out of 1.6T parameters)
💡 Code generation - Codeforces rating 3,206, a record at launch
On price vs capability, DeepSeek V4 Pro is still one of the best options on the market - especially for tasks where absolute quality isn't critical. The 8-month gap is real, but at 1/7 of the price it's a fair trade-off. The problem isn't the model - it's the "almost caught up" narrative built around it
How I choose models in my own system
I choose models by task, not by press release. The brains of my 8 agents are Sonnet 5 and Opus 5.5 for different roles: where depth is needed I use the stronger model, where speed and volume matter a fast one is enough. My personal agent Jes has a primary model and an automatic backup: if the first fails, the second carries on by itself
A cheap model works well where the mistake is caught not by the model itself but by the system around it. The clinic bot answers only from documents covering about 114 price items and 48 services: an accurate price list matters more there than a benchmark record. In the content factory, names, numbers, dates and quotes are checked against the collected material before I even see the draft, and that has produced 84 posts, none published without my approval
How to test a model yourself
Take a dozen of your own real tasks that your current model definitely handled, and run them on the new one. Compare not only quality but also price and response speed. Check separately how the model works with tools: it can be strong in a benchmark yet stumble over simple steps in your calendar or CRM. A regulator's closed test is more honest than marketing, but your own task set is more honest than both
💭 DeepSeek V4 is the strongest open Chinese model, and 7x cheaper than the frontier. That's a real argument for specific tasks. But "closed the gap with OpenAI" is about the press release, not the benchmark. Eight months of steady lag on closed tests - that's what the regulator sees, not the marketing team