AI High Signal

Making novel variations of evals is very useful, public evals tend to get memorized!

I see some are drawing the wrong inference from this study: they are focusing on this meta benchmark as yet another benchmark for model picking, i.e. open vs closed models, model families etc. That’s too narrow of an interpretation! You need to do your own private evals on

holy shit, an art form of scientific grift next METR/Epoch/AA employee right here you all need to work on your epistemic hygiene

GPT-5.6 Sol and Luna were built by roughly the same people using roughly the same infra, I don't think we can blame the 20 point difference in Luna just on memorization. There's something else being tracked here in addition to memorization. Maybe MaxxBench questions are slightly

I'm both surprised and also not at all

i disagree with the first point, there is nothing that says that scaling up current transformer recipe is not enough for "intelligence". you take kimi K3 or your favorite open arch + some stability fixes, you scale it to 50T parameters, you do classical training end to end, you

We need a model that learns *in spite* of its inductive bias, not thanks to it. We have to stop being cross-entropy fetishists.

True! This is the era of the RL objective. The biggest economy on this earth (US stock market) runs on subscriptions. Subscriptions require vendor lock-in. Currently, the frontier is pushing “Variance” learning from customer data, like how Google won the search market. But I

The Bitter Lesson 2.0

I really love the full bandwidth transformer research - One of the beautiful works of art of this year! “For example, the Bytedance experiments suggest a variant of the Full Bandwidth Transformer might be comparable to a model of twice the size, implying significant savings in

nous shoulda just called it a wasian finetune and sidestepped all the criticism

Is this the largest ablation of harnesses and models ever performed?

neat

Hermes supremacy tweet #9138

mini-swe-agent is 130 lines of code and it keeps on winning in harness comparisons.

thom just took away karpathy’s mote. 🤍

wtf, this is excellent and actually somewhat scary. Thinky did this on a tiny compute and token budget, without any fancy architecture or RL tricks. This is the same level of achievement as 1017. Maybe better. This is FINALLY what can be expected of top-tier ex-Frontier staff.

Thanks @arcprize! One of my favorite eval for monitoring RL health🫡

What an American open-weight frontier lab looks like. Inkling v1 -> v2 was a feat in itself. I can’t wait to see where v3 lands, and will be shoving 2 Small on the 6000’s once the weights drop

Inkling 2 small is an extremely impressive release, especially for its size. It is a Mixture-of-Experts model with 594B total parameters, 17B active, and supports image inputs and variable effort. In coding benchmarks, it easily keeps pace with significantly larger models. What

ok this is just ridiculous

Proud to release our new model, Inkling 2 Small. Check out our blog post -- it includes a lot of detail about our training process. To me, the most interesting part of this release was what we learned about alignment, which you can read about in the sections "Safety and

Inkling 2 small is very strong on vision perception, and also good at planing/coding/acting.

I haven’t seen Nouamane in person. for more than a year. Now I know why

Nouamane cooked again 🔥

After the Ultra Scale Playbook, Nouamane is back with this crazy guide to train MoEs! It's probably the most comprehensive resource if you want to get into large scale MoE training and an absolute gem packed with beautiful visualization!

benchmark 14 and 15 are my favorite

HOLY FUCK FINALLY some information on Anthropic internal models: Model 4 finished training in October 2026 and scores 178.6 on the Anthropic ECI this is 4 AECI higher than Opus 5.5 or about 3 months of frontier progress

METR estimates that Anthropic is currently making progress in the capabilities of its models about twice as fast as it used to, and that the use of AI itself is driving a large part of this acceleration. Independent evaluator METR found that Anthropic’s model benchmark scores

Distilling most of our expertise in kernel and inference engine optimization into an agent loop that powers our production. A 2-year project that we're finally able to share

Together Compiler discovers the most efficient inference runtimes with continuous compilation for any combination of workload, model and silicon. Excited to share the details about the research and engineering that went into it. Together Compiler (or TKCC) is a production

Great work that pushes our understanding of how agents and experts can collaborate on optimizing system performance Beyond the strong results, the team also found some surprising cases where our human inductive biases end up limiting what agents can achieve

The Together Compiler (TKCC) from @simran_s_arora and her team is honestly one of the coolest things ever, and we have validated one of the InferenceX/AgentX results! The ability to continuously improve performance automatically while ensuring accuracy is great enough to quickly

Introducing Joules-to-GDP: a new metric and framework for measuring how much economic value AI generates per unit of energy. Key insight: selecting the right model on the right accelerators can deliver several times more value per joule for the same task. We believe as AI

Fascinating research that finds engineering, CS & Math and Management are the highest value per joules application of AI, while office admin (personal assistants?) being the lowest Cool work by @Avanika15 @JonSaadFalcon