Samsung isn’t a company entirely focused on AI like Perplexity or OpenAI. However, its current and long-term strategy involves integrating artificial intelligence into the vast majority of its products across all possible segments. The features of the Galaxy AI suite in the company’s smartphones and tablets are a clear example. Given this scenario, Samsung Research has developed a new proprietary benchmark for evaluating AI productivity called TRUEBench.
TRUEBench: Samsung’s AI Benchmark for real-world AI performance
Meet Samsung’s new TRUEBench (Trustworthy Real-world Usage Evaluation Benchmark). The company stated that it created the tool to address limitations found in existing AI benchmarks. The company claims that current tools focus too much on English. Also that they often rely on simple, single-turn question-and-answer structures. This restricted their ability to accurately reflect how AI is used in a real-world work environment.
To overcome these limitations, TRUEBench aims to provide a more comprehensive and realistic evaluation of how large language models perform in professional settings. The benchmark assesses AI performance on ten commonly used enterprise tasks. The list includes content generation, data analysis, summarization, and translation. These tasks are part of a massive collection of 2,485 test sets that span ten categories and 46 subcategories and incorporate diverse dialogue scenarios across twelve languages.
The test sets themselves range from as short as eight characters to over 20,000 characters to reflect a variety of tasks. They range from simple requests to summarizing lengthy documents.
Evaluation system also involves humans
To ensure reliable and precise scoring, TRUEBench utilizes a unique collaborative evaluation system that involves both human and AI review. First, human annotators establish the initial evaluation criteria. Then, an AI reviews the criteria to check for errors, contradictions, or unnecessary constraints. Human annotators then refine the criteria based on the AI’s feedback, repeating this process to create an increasingly precise evaluation standard. This cross-verified system aims to minimize subjective bias. To past this test, an AI model must satisfy all of the required conditions, which enables more detailed and precise scoring across tasks.
Samsung has made TRUEBench’s data samples and leaderboards available on the open-source platform Hugging Face. The platform allows developers and researchers to compare the performance and efficiency of up to five different AI models simultaneously. The company will also publish the data on the average length of a model’s responses. This will provide an at-a-glance comparison of both performance and efficiency.
According to Samsung, the goal of TRUEBench is to establish new evaluation standards for productivity and reinforce the company’s technological position. Basically, it promises more realistic and reliable AI metrics.