[Audit] Doubtfamo math data ‘built in 4 months, reviewed by 2 people’ criticism draws scrutiny… NIA says it conducted a full review
IT DAILY ·
✦ AI Summary
Questions have been raised over whether the verification of the math dataset used to evaluate Doubtfamo was sufficient.
Lawmaker Kim Jang-gyeom pointed to the limits of a 4-month build period, review by 2 people, and the verification of more than 1,000 items.
NIA said it completed a full review through outside experts and quality verification institutions.
Questions have emerged over whether the math items used to evaluate Doubtfamo were sufficiently verified, prompting calls to reexamine the review process. At the National Assembly audit held on the 15th at the National Assembly in Yeouido, Seoul, People Power Party lawmaker Kim Jang-gyeom questioned NIA Chief Kim Hyeong-cheol over the build and management of the Doubtfamo evaluation math dataset, “Math (SOOHAK).”
The criticism was based on a tight production schedule and limited review manpower. In particular, it was noted that the dataset was built over 4 months and reviewed by just 2 people, and concerns were raised that such constraints on time and staffing could affect the reliability of the evaluation.
In response, NIA said it had completed a full review involving outside experts. At the audit, claims about the reliability of the math dataset used in the Doubtfamo evaluation were met by NIA’s explanation that the build schedule and limits on review manpower had already been addressed through a full verification process.
Lawmaker Kim cited the researchers’ paper as evidence, linking the build and review process for the math dataset to manpower and time constraints. Kim Jang-gyeom said the paper clearly stated these limitations.
The math dataset was overseen by Company A. Company A was responsible for the entire process, including proposal preparation, administration, contract procedures, recruiting field experts, collecting items, and pre-reviewing questions, and the work period was within 4 months.
People Power Party lawmaker Kim Jang-gyeom pointed out that the math items reviewed for Doubtfamo’s evaluation numbered around 1,000, while the number of in-house researchers was only 2. He also said the in-house review method consisted not of solving the problems directly, but of comparing the creators’ answers with AI answers, and that this was stated in the paper. He criticized it as a de facto confession rather than a paper.
Kim Jang-gyeom added that the researchers had asked an outside institution to conduct a sample review of some of the items. He said that as a result, issues were found in about 5% of them.
Kim said the math questions that determined whether companies would make it into the national AI team were finalized without sufficient verification because of time constraints. He said this made questions about the fairness of the evaluation unavoidable.
NIA explained that, in addition to the contractor’s internal review, it carried out an additional verification process through outside experts and quality verification institutions. Kim Hyeong-cheol, the head of NIA, also said that after the questions were produced, NIA conducted a separate full review through experts including KAIST professors and other quality verification institutions.
Source: IT DAILY · Yang Seung-gap
Original: https://www.itdaily.kr/news/articleView.html?idxno=242234
References
This article was produced with the help of an automated content generation algorithm.
Source: IT DAILY
View originalThis article was summarized and organized by BizCrush based on the original article from IT DAILY. For exact quotations and full details, please refer to the original article.