How We Cut LLM Batch Inference Time in Half with Dynamic Prefix Bucketing

TL;DR LLM batch inference is often difficult, costly, and slow - but it doesn't have to be...