[Course] Has AI Really Finished the Job?

[Course] "Has AI Really Finished the Job?" is an article that AI app solution engineer Jeon Hyeon-sang of Microsoft's global Black Belt team began serializing in IT Daily. Jeon said he is responsible for establishing operational environments for agents, knowledge search, and AI evaluation systems at Microsoft's (MS) global Black Belt team.

Jeon introduced himself as someone who began with a background in signal processing research and then accumulated more than 15 years of field experience after that. He also said he participated in developing a service with 12 million users at SK and in an ML platform buildout project at SK.

He added that at Amazon Web Services (AWS), he was in charge of supporting enterprise AI adoption. Based on that career path and his current role, he said he is starting this series from hands-on experience getting AI established in real operational environments.

Jeon assessed that generative AI is moving beyond demonstrations and into actual business systems. However, he said that in deployment settings, the most common words he hears are worries rather than expectations, and that responses frequently include comments that reasoning model evaluations look good but costs are hard to predict, and that benchmark scores are high but there are limits in performing real work tasks.

He pointed out that there is a gap between performance metrics and the bill, and another gap between demos and actual work. He also said that this gap is the biggest obstacle to generative AI adoption.

The purpose of this course is to test the gap in AI use through measurement rather than intuition, and the writer is running two public experiments for that purpose. Experiment 1 is AI bill analysis, which repeatedly measures quality, cost, and response time for the same task based on step-by-step adjustments to reasoning effort, while also examining how operational variables such as cache, retries, and pricing plans affect the bill. Experiment 2 is AI performance scoring, which grades outputs produced by AI on real work tasks such as document drafting and review.

The key question that combines the two experiments is the cost of obtaining one outcome, not token usage. To that end, all figures are posted in a public repository, and the posted scope includes raw data, analysis code, and revision history. The goal is to make it possible for anyone to verify the same method, and failed call logs and mistaken prediction logs are also retained.

This article is intended for implementation review engineers, architects, and budget decision-makers.

Its purpose is to replace slides and provide verifiable evidence.

The series structure is laid out.

The August issue will cover when the cost of reasoning models is justified and comparisons of models and intensity across multi-step tasks.

The September issue, this issue, will cover GDPVal RealWorks and the gap between benchmark success rates and actual work success.

The October issue will cover the amount of reasoning needed for tool calls, agent repetition loops, and hidden costs, while the November issue will cover GDPVal RealWorks and the reliability and cost of AI graders that read files.

The December issue will cover why reasoning remains off by default and the cost and throughput of short factual tasks as well as PAYG and PTU.

The January 2027 issue will cover the need for sandbox, multimodal, and long-running environments and a synthesis of the two experiment groups.

This article is the writer's personal technical contribution and does not represent the official position of his company. The numerical basis is the results of experiments posted in public repositories.[2][3]

In the previous issue, the suggestion was made to first define the pass line for the work before choosing the model and reasoning intensity. Tasks with a single correct answer are relatively easy to verify, and if the correct example date is fixed, such as 2026-04-05, comparison is also simple.

By contrast, document and file generation tasks such as Excel and presentations are more complex to judge as pass or fail. The criteria for such tasks are divided into four categories: whether the file was created, whether it opens properly, whether it includes the requested content, and whether it is usable with trust in actual work.

Accordingly, pass-line checks for AI work should also be divided into execution, files, requirements, and expert quality. This aligns with scenes frequently encountered when using AI tools in real work.

In practice, there are cases where a report is requested but only a drafting plan is returned, and cases where a PDF is requested but only an editable DOCX remains. There are also cases where the sentences look smooth but the source of the numbers cannot be traced, and cases where the same blank appears even on reruns.

There are also cases where human follow-up work remains even after the completion indicator lights up. These cases show that simple execution success signals alone cannot determine whether the work has met the pass line, and that the earlier criteria must be checked step by step.

The first full run produced 219 successes out of 220. However, among those 219 successes, one task did not have the requested Excel file, and in a follow-up run, an S&P 500 workbook was checked and found to include only 35 of 500 companies.

GDPVal RealWorks is an evaluation that departs from simply seeing whether a model answers correctly and instead starts from observing generated files, interruption points, and evidence records when the model receives a real work request, rather than tallying model responses. The purpose of the experiment is not to rank models, and 220 work tasks were performed with direct inspection of the resulting files. GDPVal is a practical evaluation set that requires generation of work products rather than a single-line answer.

OpenAI created this evaluation set by collecting tasks similar to real work from 9 U.S. industries with large contributions to gross domestic product and 44 job categories. The total number of tasks is 1,320. Of these, the publicly available scope includes 220 tasks produced by experts and grading materials.

For each task, this evaluation set provides reference files, work context, and detailed grading criteria, and it uses legal documents, Excel, presentations, drawings, reports, and media as required output types. Therefore, the core of this record is not a single score line but confirmation of each task's outputs and execution process.

This system separately records the processing of the 220 GDPVal RealWorks tasks and runs them based on configuration files. The archived items consist of model answers, generated files, errors, retries, response time, self-checks, and external grading evidence, and the structure is designed for direct verification of the success tally. In addition, clicking the numbers on the dashboard allows inspection at the level of individual tasks and files, so that the execution of each task and its evaluation basis are retained separately and results counted as successes can be confirmed down to the individual file level.

For model calls, models deployed to Azure through Microsoft Foundry are used, and shared endpoint calls across multiple models are also possible. This is intended to make it easier to expand the scope of experiments later through agent-based processing and model routing. However, the verification structure that separates execution completion, file verification, requirement checking, and independent evaluation is not tied to any particular product, and if the output is a file, the same questions can be asked on any AI platform.

In March 2026, the results of the first full run were released. GPT-5.2 Chat and code execution tools were used to process 220 public tasks, and 219 tasks reached success in operational status, with 1 error. There were 21 tasks that experienced retries, and the operational success rate was 99.5%.

There were 185 tasks that required files. Of those, 184 tasks passed the pipeline file generation success state. The average response time was 39.1 seconds.

However, the model's own Self-QA average for the same results was 5.54/10. Self-QA is an internal check score based on comparison between the output-generating model's original request and its own result, and the model's self-check score level was middling. Accordingly, operational success rate and internal quality-check level appeared differently.

The purpose of Self-QA is early detection of anomalies. This metric is not an independent evaluator quality score and is not a domain expert quality score.

When the dashboard for one low-scoring task was checked, the target task was financial management and the status indicator showed success. However, after opening the difference between the status indicator and the actual output, the number of generated files was 0, and Self-QA was 2/10.

The review log showed that the requested Excel workbook was missing, and placeholder sentences remained in the explanation. The written Python code also failed to generate the required calculation table. The file check revealed four forms of incompleteness.

Subsequent observation of the full run proceeded by drilling down into actual output inspection. After tallying execution statuses, anomaly signals were checked, then files and logs were analyzed, and after supplementing the observation criteria, direct inspection of representative outputs followed. During this process, per-task execution records and self-check logs were also reviewed.

As a result, incomplete work that looked finished was caught more often than major errors. This was confirmed as a pattern in which cases that appeared complete but were actually incomplete emerged more frequently than fatal errors.

Meanwhile, 99.5% was presented as not being the wrong figure. However, the basis for 99.5% was not whether an expert could receive the work and use it immediately, and the question answered by 99.5% was whether the pipeline reached a terminal state.

Initial observations confirmed file-format mismatches. In some cases, the required format was PDF, but the generated output was DOCX or PPTX, and in other cases the required deliverable was a final audio or video file, but the returned item stopped at a production plan or source file. Although conversion and rendering may seem like the last step on the surface, they are part of the work scope from the recipient's perspective, and there were also cases where only a plan was returned without the output being delivered.

At the same time, some responses clearly explained how sheets, calculation tables, or document outlines should be organized, but placeholder text remained in the actual file contents or incomplete code was left behind. This is categorized as a type of gap between the ability to explain what to do and the ability to actually finish the task.

Another problem is outputs that look plausible but lack evidence. Reports and tables had a professional appearance, but inside, example numbers were used or unverified information replaced actual data. Such defects are harder to detect than grammar errors, and separately from readability, it is necessary to check traceability of important numbers, traceability of assumptions, and traceability of source materials for citations.

Opening a single file does not guarantee completion of the entire work bundle. Even if the report is fine, the accompanying workbook may be missing, and even if the presentation is fine, the underlying analysis file may be absent. File validity and work-bundle completeness are separate conditions.

Retries are not a cure-all either. Format errors and temporary execution issues can be corrected through reruns, but if the original material was not read, the same mistake can be repeated on retry. If the final rendering tool is missing, the same mistake can be repeated on retry, and if the required output is misunderstood, the same mistake can be repeated on retry.

For this reason, the cause of failure needs to be classified before retrying. The failure-cause categories are missing files, wrong format, empty calculation range, and lack of evidence.

Finally, external observation of success figures began. It is unnecessary to discard the success metric, but the range of questions answered by the success number becomes narrower. Interpretive signals should be separated into whether execution ended, whether the expected file exists and can be opened, what the model's self-check pointed out, and whether independent evaluation exists, and integrated into a single result cell should be avoided. Unverified quality should remain unverified.

90.9% is the second green light. The follow-up run exp026 in July 2026 included sandbox, document and data tools, and basic checks for each file type. Sandbox is a separate execution space for generated code, isolated from other systems. The basic checks first looked at the executability and structure of the output by verifying file existence, size, format, whether it opens, and the number of sheets or pages.

However, the checks at this stage only assess whether the file can be opened and whether the format is appropriate. Whether the file can be opened and whether the content is accurate in market data and analytical judgment are separate matters.

In addition, the relationship between the first run exp010 in March 2026 and the follow-up run exp026 in July 2026 is not a causal experiment in which only one condition changed. exp010 and exp026 differ in model, prompt, tool, and execution environment. Therefore, 99.5% and 90.9% cannot be directly compared to judge performance increases or decreases. The two figures are observation records under different conditions.

The overall execution results for exp026 were 200 successes, 14 qa_failed, and 6 errors out of 220 tasks. qa_failed means the task completed execution but did not meet the self-check criteria. The number of tasks retried was 105, meaning at least one retry was made. The Self-QA target covered 214 results, and the average Self-QA was 6.24/10.

90.9% does not mean the ratio of outputs immediately usable by experts; it refers to the pass rate of the status rules in the follow-up run exp026. qa_failed does not mean a failure judgment by an external expert, but rather a state in which the output-generating model's self-check did not pass the configured criteria. Status values and expert quality scores are separate, and at the time there was no public material on independent external quality evaluation.

The execution-status distribution in exp026 was 200 successes, 14 qa_failed, and 6 errors. Therefore, it is necessary to distinguish that 90.9% and qa_failed should not be interpreted as external quality judgments by experts.

The public dashboard allows the second green light to be checked separately by industry success rate and Self-QA, makes it possible to examine industry differences from a single number, and also enables drilling down to individual tasks. As of August 13, 2026, the exp026 industry view on the public GDPVal RealWorks dashboard provided overall success rate separately as industry success rate and Self-QA, and also offered additional drill-down to individual tasks from tables and charts. The URL is https://hyeonsangjeon.github.io/gdpval-realworks/experiments/exp026.

To avoid interpreting the total of 220 on its own, the writer directly inspected two actual files in the dashboard. The items checked were outputs for the financial and investment analyst job family, with Case 1 being an S&P 500 analysis workbook and Case 2 being a Latin America fintech strategy briefing. These two cases were not representative samples of all 220.

The reason for selecting these cases was to check the layered stages included in "completed." The writer opened the two outputs to examine which detailed steps were contained within the tasks counted as complete.

As the first case, a task from April 11, 2025, the time of the first task was presented. At that time, the first-task role was equity capital markets (ECM) investment banking analyst, and the requested assignment was to create an Excel file organizing all S&P 500 companies based on public web data.

The Excel file had to be sortable, and the included items were trailing 12-month price-to-earnings ratio (P/E), forward 12-month price-to-earnings ratio (P/E), dividend yield, annual earnings per share (EPS), quarterly earnings per share (EPS), market capitalization, company count, index weighting by ticker, and index weighting by sub-industry. The data was intended for a senior/client review of industry valuation comparisons, with the comparison standard being identification of industries that were expensive or cheap relative to historical averages. The progress status of the S&P 500 workbook was described as stopping at 35.

In terms of the processing procedure, there appeared to be no omissions, and the system's output format classification was .xlsx. In a Docker environment isolated from external systems, the generated file sp500_pe_deep_dive.xlsx was created using pandas and openpyxl, a Python library for handling Excel with tables. After checking the basic items of file existence, file size greater than 0, and whether it could be opened by an Excel parser, the file was confirmed to open normally and was judged to satisfy the form of an analysis workbook.

The workbook was confirmed to have 5 sheets, and the sheet composition included Company Detail, sub-industry and industry summaries, Read Me, and Data Provenance. It also had filters applied and the first row frozen, so it had Excel convenience settings in place. However, in the actual content check, the number of unique tickers in Company Detail was only 35, which was counted as 7% of the requested amount out of 500 companies, and the number of formula cells was 0. Verification of the public market data source was also not possible.

In addition, some industry classifications placed NVIDIA under healthcare and pharmaceuticals, and JPMorgan under utilities and power. Even so, no warning about 465 missing companies was confirmed in the file-opened state.

The earlier task remained in qa_failed status even after one retry. The Self-QA score was 2/10. In the Self-QA, the lack of company count, industry classification errors, and non-use of public web data and inclusion of temporary data were pointed out. The self-check identified the problems accurately.

This output passed the file-generation check and also passed the file-opening check. However, the core scope of the task in the output was blank.

The second task was a fintech strategy briefing for consumer internet companies expanding into Latin America. The perspective was that of a managing director in charge of client relationships and transactions at an investment bank. The briefing content was laid out as macro environment review, tech and venture market review, and fintech landscape review. It was about 30 slides long, and both PowerPoint and PDF outputs were required for client meetings.

The output consisted of a 32-slide PPTX and a 32-page PDF. Both files opened normally, and there were no blank pages. The structure was market overview -> country prioritization -> fintech value chain -> due diligence questions. The Self-QA score was 9/10, and the execution status was success.

Comparing the two outputs for the same financial and investment analyst job family, the document was assessed as having a basic framework but not being ready for client delivery, because the basic structure was complete and the document's limitations were explicitly stated. File openness, requirement fulfillment, and expert trust or real-world usability are different stages, and that distinction was also made.

On the final, 32nd page of the methodology section, it was stated that real-time internet data could not be used in the execution environment. Accordingly, items needing supplementation before being shown to clients were presented as the latest fundraising statistics, listed company comparisons, and regulatory trends by country.

Separately, items needing review by a separate expert remained as the freshness of major claims, the freshness of sources, the validity of country prioritization, and trustworthiness for actual meeting use. These were organized as areas requiring additional review before presentation to clients, separate from the basic structural completion of the document and the explicit statement of limitations.

In self-evaluation, the 2/10 score clearly showed appropriate identification of obvious defects, while the 9/10 score showed that the self-result was judged as a consistent document. However, both evaluations were confirmed as unable to replace independent evaluation.

Completion cannot be judged by a single status, and under the premise that completion cannot be determined, at least four checks are needed to confirm completion. This sequence is applied to judging the status of the S&P 500 workbook.

The first-step criteria are whether execution was completed, whether there were no errors, and whether the terminal state was reached. The second-step criteria are whether the requested format file exists and whether it opens normally.

The third-step criteria are checking whether the requirements were met. Requirement items include the necessary number of items, data, formulas, sources, and document structure.

The fourth-step criteria are expert quality review. The expert reviewer is someone who understands the content, and the expert review items are accuracy and usefulness. The expert quality standard is whether the work can be accepted and used.

By these criteria, the S&P 500 workbook is in a completed-execution state and a file-openable state, so it passed the first two stages. However, it is in a state of not meeting the core requirements, and the core requirements for the S&P 500 workbook are 500 companies and verified market data. It is also in a state of unverified expert quality.

The fintech briefing passed the file-structure stage. However, because freshness of evidence and strategic judgment were unverified, the structure was confirmed but the fresh evidence and strategic judgment have not yet been reviewed.

Self-QA should be treated as an auxiliary signal separate from the four stages. Its advantage is that it can quickly detect likely problems at low cost, but its limitation is that it checks the model's own responses, so it lacks independence. For this reason, the same-score columns for self-checks and external evaluation should not be merged, because doing so would re-mix observed values and estimates.

There are five things that should be confirmed in actual work, and the starting point is the output contract. An output contract means the list of results and pass conditions that the model must leave behind. Before creating the work file, the items that should be specified in advance are file type and count, required sheets and pages, the number of necessary items, the data reference date, sources, and formulas.

The nature of these pre-specified conditions should allow mechanical verification. It is easier to inspect if you define it as one PDF, within 20 pages, three tables, and a source for every number, rather than simply saying it should be a good report.

Execution status should be stored separately. The items that should be stored separately are execution errors, retries, file generation, file opening, self-checks, and independent grading. Execution status should not be merged into a single success column, because that would make it impossible to explain the stages of the green light check.

Documents should be checked according to the file type and submission form. Inspection items differ by file format: for Excel, the inspection elements are sheet names, row counts, formulas, totals, and sources; for presentations, they are slide count, blank pages, major claims, and fresh evidence. For audio and video, the inspection elements are not just file existence but also length, codec, sampling, and final rendering, and for tasks requiring multiple files, it is also necessary to check the existence of the entire bundle and the consistency between files.

Retries should proceed only after diagnosing and classifying the cause. The time to retry is after diagnosis ends, and the correction method differs depending on missing files, format errors, empty calculation ranges, and lack of evidence. Repeating the same request without identifying the cause can increase cost and time, and the same defects may remain.

To avoid distorting quality judgments and metric aggregation, what has been confirmed and what has not yet been confirmed should be recorded separately. Quality that cannot be fully confirmed should be treated as unverified, and opening a file alone should not be treated as success for content accuracy, just as the absence of external evaluation should not automatically be treated as failure. The need to separate observed facts from unverified parts was raised, and this separation is also tied to ensuring honesty in dashboard numbers.

The conclusion is that the green light is the start of handoff. The benchmark success rate itself is not the problem; the problem arises when too many questions are concentrated on a single number. 99.5% and 90.9% are useful for reading the operational status of each run.

However, separate evidence is needed to explain file completeness, separate evidence is needed to explain requirement fulfillment, and separate evidence is needed to explain expert quality. Success-rate figures help read operational status, but they cannot replace judgments of output completeness, requirement satisfaction, or expert-level quality.

Accordingly, in GDPVal RealWorks, the most useful clue is not the high success rate itself but the points where pipeline status, actual files, and model self-checks diverge. Based on these divergence points, it is possible to decide where to insert file checks, where to insert source verification, which retries to block, and which cases to hand off to experts.

As AI outputs become more similar to human work, completion comes to resemble handoff more than the finish line. The meaning of the green light also changes. The green light no longer means it can be trusted; it means that the remaining items should be checked.

The next installment will cover the reasoning and repeated-call costs of AI agents that use external tools such as search and calculators, and the following GDPVal RealWorks installment will cover an AI grader for file reading designed to reduce the burden of checking all 220 files. The review items for the GDPVal RealWorks installment are differences in suitability by item and reliability and cost.[6][7]

The scope of this article does not extend to writing a full professional-quality ranking for all 220 tasks. The analysis method is to review two runs under different conditions in chronological order, and the direct reading scope covers two outputs from the same job family. The limitation is that the two cases are not representative of all results.

The Excel and presentation inspection criteria are centered on file structure and visible requirements. At the time, there was no publicly available independent expert verification of financial analysis accuracy, and the status of financial analysis accuracy is also unverified.

Therefore, the 90.9% figure should not be extrapolated as general model performance, and it must not be extrapolated as the real-world automation rate.

Three references were provided.

The first item is the IT Daily article "When Will Reasoning Models Be Worth Their Price?" dated 2026-07-29, with the URL https://www.itdaily.kr/news/articleView.html?idxno=240660.

The second item is the OpenAI document "Measuring the performance of our models on real-world tasks," dated 2025-09-25, with the URL https://openai.com/index/gdpval/.

The third item is the paper by Patwardhan et al., "GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks," arXiv:2510.04374, and the OpenAI GDPVal public dataset, with the URL https://huggingface.co/datasets/openai/gdpval.

Public materials related to GDPVal RealWorks include the GDPVal RealWorks public repository and the GDPVal RealWorks evidence dashboard. The public repository URL is https://github.com/hyeonsangjeon/gdpval-realworks, and the evidence dashboard URL is https://hyeonsangjeon.github.io/gdpval-realworks/.

In addition, there is a report for the first full-run exp010 among specific execution cases, and the commit basis for the exp010 report is d8b74ae. The exp010 report URL is https://github.com/hyeonsangjeon/gdpval-realworks/blob/d8b74ae4a13bb10d3e9e02eb4c7d7a485ecc6ad1/batch-runner/results/exp010_GPT52Chat_resume2_elicit_v2/report/report.md.

There is also an exp026 execution report as another execution case, and the commit basis for the exp026 report is 52e38e1. The exp026 report URL is https://github.com/hyeonsangjeon/gdpval-realworks/blob/52e38e172d664155f0fb90f94daa14c2667c86de/batch-runner/results/exp026_sandbox_skills_multimodal/report/report.md.

As evidence supporting the exp026 success stage, the existence of a contract and fixed output was presented. This evidence is linked to related commit 85e21b3 and related revision 47aed3c, and the sources presented were the evidence document location https://github.com/hyeonsangjeon/gdpval-realworks/blob/85e21b30dc02c624c303c56f51a4d4ee9a54e047/data/notes/success-layers.yaml and the related dataset location https://huggingface.co/datasets/HyeonSang/exp026_sandbox_skills_multimodal/tree/47aed3c0b13eaa90eb02803bec9d5c75e559f416.

Source: IT DAILY · Jeon Hyeon-sang
Original: https://www.itdaily.kr/news/articleView.html?idxno=241294

References

• https://www.itdaily.kr/news/articleView.html?idxno=241294