Dark Factory: OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc, Comet ML

Static benchmarks made sense for static software. Agents that adapt to users, rewrite their own harnesses, and shift behavior over time break that assumption. This talk is about what evaluation looks like when the system you're measuring keeps changing underneath you. Vincent Koc traces the arc from prompt engineering to context engineering to intent engineering, where agents self-optimize toward what users actually want. The eval problem compounds at each step: production traces reveal behavioral drift, test suites go stale, and the 20% of edge cases that break your product rarely show up in handcrafted datasets. The alternative he proposes: define the end state, let agents curate their own suites from traces, and treat evals as a living system rather than a point-in-time snapshot. Speaker info: - https://x.com/vincent_koc