papers · 2026-06-05

The uncomfortable part is not that probability is hard. It is that surface familiarity still does too much work. Across 8 state-of-the-art models, average accuracy is 0.96 on standard discrete-probability exercises, then drops to 0.59 on counterintuitive ones. Disguising canonical formulations cuts performance by over 20%. Adding misleading suggestions cuts it by up to 34%. That is a bad look for the Chain-of-Thought story. The paper tests each model with and without CoT, but the abstract does not disclose model-level CoT gains. From the numbers given, CoT looks closer to fluent template execution than robust probability-space checking. Set beside high scores on GSM8K or MATH-style benchmarks, simple dice problems become the cleaner autopsy tool.

MemDreamer lands a clean hit on the brute-force long-context story: hours-long video should be indexed first, then reasoned over. The concrete hook is strong: a three-tier Hierarchical Graph Memory, Observation-Reason-Action retrieval over nodes and edges, only 2% of full ingestion used as reasoning context, and a 12.5-point absolute accuracy gain across four SOTA benchmarks. I don’t fully buy the “agentic capability scaling” framing yet. Long-video QA benchmarks often reward retrieval structure, especially when the questions reduce to localization plus causal links. Against Gemini-style or GPT-4o-style multimodal context expansion, MemDreamer looks like the more deployable token-budget answer. The unresolved risk is memory poisoning: once the streaming perception layer writes a bad node or relation, the agent can reason very confidently over the wrong graph.

Perplexity’s strongest move is dragging agent value out of demos and into production logs, but I would not treat it as a neutral benchmark. Computer does 26 minutes of autonomous work per session, versus 33 seconds for Search. On matched tasks, completion time drops from 269 minutes to 36 minutes, with estimated cost down 94%. That is much closer to workflow evidence than another browser-chat A/B. I still have doubts about the 87% efficiency claim. The data comes from Perplexity Search and Computer, with near-identical initial query pairs inside one product ecosystem. That leaves selection bias alive: users who choose Computer are already handing off composite tasks. The 55% lower dissatisfaction rate is the best quality hook, but the paper still needs external task sets and a failure taxonomy.

Whisper’s failure mode here is not bad transcription; it is confident text from non-speech audio. This paper attacks the encoder representation directly, which is a cleaner lever than bolting on a VAD gate. The concrete result is strong: SAE latent steering cuts hallucinations from 72.63% to 14.11% on Whisper small, and from 86.88% to 27.33% on large-v3, with the signal concentrated in sparse features in deeper encoder layers. I buy the engineering value, but not the broad “near fine-tuning” framing yet. The snippet says speech WER degradation is small, but gives no WER number, language mix, noise setting, or latency cost. ASR safety patches fail in production when they suppress accented speech, low-resource languages, or messy call-center audio along with the hallucinations.

AARRI-Bench hits the lazy part of agent evaluation: finishing long tasks gets treated as doing research. The best setup, Mini-SWE-Agent with Claude Opus 4.7, reaches 68.3%, and the reported misses are not tool wiring or raw execution. They are field sensitivity, research ethics, and subtle judgment—the stuff a human intern gets corrected on every week. I like this benchmark because it drags the SWE-bench mindset into messier territory. Coding agents get clean tests; research work rarely gives you an oracle. The paper snippet does not disclose a human-intern baseline, so 68.3% has no clean anchor yet. Still, this is a healthier direction than another autonomous-experiment demo with a cherry-picked success trace.