Samsung Electronics on September 25 introduced TRUEBench, a benchmark created by Samsung Research to measure how large language models (LLMs) perform in workplace productivity applications. The tool aims to address limitations of existing evaluations by incorporating multilingual, multi-turn dialogue scenarios and a broader range of enterprise-related tasks.
TRUEBench assesses models across 10 categories and 46 sub-categories, covering areas such as content generation, summarization, translation and data analysis. To better reflect workplace conditions, the benchmark includes 2,485 test sets in 12 languages, ranging from short prompts of a few characters to lengthy documents of more than 20,000 characters.
The evaluation framework uses criteria developed through a combined process involving human annotators and AI systems. This iterative approach is designed to minimize subjective bias and ensure precise scoring, with all conditions required to be met for a model to pass.
Samsung positioned TRUEBench as a way to provide more comprehensive insights than existing benchmarks, which it said often focus on English-only datasets and simplified single-turn interactions. By incorporating cross-linguistic and multi-step scenarios, the benchmark is intended to capture how AI performs in complex, real-world environments.
Datasets and leaderboards for TRUEBench are available on the open-source platform Hugging Face. The system allows comparisons across up to five models at a time and publishes data on both accuracy and efficiency, enabling a clearer view of overall performance.
Source: Samsung