FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline

1. 1Identify the limitations of existing role-playing benchmarks
2. 2Design a scalable multi-agent collaboration pipeline to construct customizable benchmarks
3. 3Implement FURINA-Builder to automatically generate benchmarks at any scale
4. 4Evaluate large language models using the generated benchmarks

AI researchers and engineers on a team benefit from FURINA as it provides a flexible benchmark for evaluating role-playing tasks, while product managers can utilize it to assess language models for various applications

💡 FURINA enables the creation of fully customizable role-playing benchmarks, addressing the limitations of existing benchmarks

🤖 Introducing FURINA: a customizable role-playing benchmark for large language models via multi-agent collaboration pipeline