AI High Signal

Ce qui vient de franchir un palier de reprise, le plus récent en tête.

Depuis 2 h 6 h 24 h 3 j
À partir de Tous 2 relayeurs 3 relayeurs 4 relayeurs 5 relayeurs Tout remettre
  • 11
    de 10 à 11 il y a 2 h
    J
    Justin Lo @loresjpg
    Nice DeepSWE score! Does it survive a lowercase filename? Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement. Models clustered around 74% on DeepSWE end up 23 pp apart. Good
    Qui l'a repris
    J Julian Schrittwieser” Hamel Husain” Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)” Ofir Press” Lisan al Gaib Mat Velloso”
    Ce qu'ils en ont dit
    J
    Julian Schrittwieser @Mononofu a cité · 01:37
    Making novel variations of evals is very useful, public evals tend to get memorized!
    Hamel Husain @HamelHusain a cité · 01:16
    I see some are drawing the wrong inference from this study: they are focusing on this meta benchmark as yet another benchmark for model picking, i.e. open vs closed models, model families etc. That’s too narrow of an interpretation! You need to do your own private evals on
    Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) @teortaxesTex a cité · 08/11 22:00
    holy shit, an art form of scientific grift next METR/Epoch/AA employee right here you all need to work on your epistemic hygiene
    Ofir Press @OfirPress a cité · 08/11 21:26
    GPT-5.6 Sol and Luna were built by roughly the same people using roughly the same infra, I don't think we can blame the 20 point difference in Luna just on memorization. There's something else being tracked here in addition to memorization. Maybe MaxxBench questions are slightly
    Mat Velloso @matvelloso a cité · 07/11 21:20
    I'm both surprised and also not at all