SecurityScorecard Unveils AI TPRM Benchmark to Assess AI Supply Chain Security
TECHWORLD ·
✦ AI Summary
SecurityScorecard announced on the 6th the launch of the AI TPRM Benchmark.
The benchmark is designed to evaluate AI models’ third-party risk management performance and cyberattack response capabilities.
Its evaluation items include Security Telemetry interpretation, security exposure prioritization, and responses to Prompt Injection and Jailbreak.
BizCrushWant to follow Korean talks and live streams in real time?Try BizCrush Live
As AI adoption expands in corporate security operations, the importance of verifying an AI model’s ability to handle real security work before deployment is growing.
In response, SecurityScorecard announced on the 6th the launch of its AI TPRM Benchmark. The AI TPRM Benchmark is designed to evaluate an AI model’s ability to perform third-party risk management and respond to cyberattacks. It also provides a standardized framework that helps companies objectively assess a model’s capabilities and resilience before deploying it across core third-party risk management tasks.
The benchmark was designed to address the limitations of existing AI benchmarks, which do not reflect real TPRM operating environments. The tasks it covers include interpreting Security Telemetry, prioritizing security exposure risks, and evaluating incomplete, misleading, or manipulable vendor-provided information.
SecurityScorecard said it created the TPRM benchmark using its proprietary supply chain risk analysis data and expertise, building it around real third-party risk scenarios. Through this, it set out to evaluate AI’s decision-making response, handling of conflicting information, and resistance to adversarial inputs in enterprise TPRM environments.
The TPRM benchmark integrates two key areas of AI evaluation. Core area 1 is AI Supply Chain Security Capability, and core area 2 is AI Model Resilience.
AI Supply Chain Security Capability uses actual TPRM and supply chain security work performance as the evaluation criterion. The assessment items consist of Exposure Triage, Breach Correlation, Vendor Risk Concentration Analysis, and Vulnerability Prioritization.
AI Model Resilience assesses a model’s ability to defend itself against Prompt Injection, Jailbreak, and deceptive vendor inputs intended to manipulate the model’s conclusions. These defensive targets fall within the categories of common supply chain attacks and adversarial attacks.
Aleksandr Yampolskiy, cofounder and CEO of SecurityScorecard, said putting an AI agent into a vendor risk management program without testing is similar to giving privileged access to a new hire on the first day. He framed two key questions for security leaders: first, whether the model can actually do the job, and second, whether it can be weaponized by attackers. He added that if those two questions cannot be answered, it will be impossible to gain a competitive edge through AI, and that deploying AI in such a state creates a new attack surface, underscoring the importance of validating AI capabilities.
Most companies adopting AI in supply chain operations have no way to verify the adversarial attack resistance of AI models used for supply chain work. Their main optimization targets are accuracy, cost, and speed, while cases of resilience evaluation remain rare.
Against this backdrop, academic experts highlighted the benchmark’s rigorous evaluation framework as a differentiator. Yevgeniy Dodis, a computer science professor at New York University and an IACR Fellow at the International Association for Cryptologic Research, said the benchmark stands out because it evaluates AI through supply chain scenarios based on real security data. He also said a rigorous and reproducible evaluation framework could help close the gap between benchmark performance and real-world security as AI adoption accelerates in critical infrastructure sectors.
SecurityScorecard said it requested the participation of selected AI labs, developers, security companies, and researchers to co-develop the testing and evaluation methods for the benchmark. To encourage participation, it open-sourced the benchmark framework and said participants can use it to test AI models and agents.
It also said participants can share a Public Leaderboard after submitting test results. Mike Mylrea, SecurityScorecard’s CISO and Chief AI Officer, said early benchmark test results showed that open-weight models have narrowed the gap with major state-of-the-art models in supply chain security tasks. However, he said many of those models remain highly vulnerable to adversarial attacks.
He said that as AI increasingly takes on more critical security tasks, enterprise model evaluation is not limited to simple performance comparisons. Companies need to identify the top-performing model for each task and verify whether the model continues to function properly even when someone intentionally tries to manipulate it.
Source: TECHWORLD · Kim Hye-jin
Original: https://www.epnc.co.kr/news/articleView.html?idxno=409372
References
This article was produced with the help of an automated content generation algorithm.
Source: TECHWORLD
View originalThis article was summarized and organized by BizCrush based on the original article from TECHWORLD. For exact quotations and full details, please refer to the original article.
Keep reading
BizCrush