How to do Distributed RL Training for LLM? feat. Eric Yang from Gradient

Currently most of the post-training of large language models are done via reinforcement learning in a centralized cluster of GPUs. Interestingly about 80% of the wall-clock time is spent on rollout generation which can are kind of simple in the sense that they are inference-only forward passes. This means that quite a lot of compute time is wasted. In this video, I’m going to present you a method to better use centralized data center GPUs for post-training by offloading these roll-out generations to a distributed fleet of workers all across the world. Interesting Articles and Links: 📌 Echo 1: https://arxiv.org/abs/2508.05387v1 📌 Echo 2: https://arxiv.org/abs/2602.02192 📌 blog about the framework: https://gradient.network/blog/echo-2-unlocking-the-second-scaling-law Also for beginners: 📌 learn to code from full-stack to AI with Scrimba https://scrimba.com/?via=yacineMahdid (extra 20% off pro with my link, great resource, I love the team) # Table of Content - current state of RL training 0:00 - overview of echo methodology: 1:46 - Echo 1 breakdown: 4:45 - Echo 2 breakdown: 7:44 - Eric Yang Interview: 19:17 - what does General Collective Intelligence means: 26:26 - how does the gradient stack fit into that vision: 33:10 - intuition behind the p2p broadcast system: 38:14 - why Echo 2 focus is primarily on distributed roll-out: 41:47 - was there any specific research direction that motivated echo2: 45:15 - extending distributed training beyond LLM finetuning?: 47:20 - how tightly coupled with parallax the system is? 50:09 - stance on off policy RL finetuning?: 55:37 - how does long roll-out fit in this paradigm?: 59:55 - how would these result hold up with more parameters?: 1:02:41 - how to manage the staleness knob for user? 1:04:50 - what does day 1 for a startup training with echo2 a 7b looking like: 1:06:53 - what are the next step in this research direction?: 1:08:20 - conclusion: 1:09:54 About Echo: ECHO-2 is a distributed reinforcement learning framew