AI Moves Beyond a “Tool” to an “Emotional Companion” ... Need for Safeguards Against Overreliance
IT DAILY ·
✦ AI Summary
Park Hyung-bin, a professor at Seoul National University of Education, said the safety issues surrounding generative AI should not be limited to physical risks.
He suggested that sustained conversations can create emotional relationships, cognitive dependence, and changes in sociability, making safeguards necessary for vulnerable groups.
Domestic tasks were presented as legislating safety standards for emotionally reciprocal AI, age verification and age-appropriate design, a national AI literacy standard, and regulations to protect neural data and neural rights.
As generative AI spreads, emotional dependence and other new risks beyond physical safety are emerging.
Park Hyung-bin, a professor at Seoul National University of Education, emphasized the issue at a forum titled "Policy Tasks for Citizens’ Mental Health and AI Literacy in the Generative AI Era," held on the 10th at the National Assembly Members’ Office Building in Yeouido, Seoul.
Park said the safety issues surrounding AI technology development should not be limited to physical risks. He added that as users engage in sustained conversations with generative AI, emotional relationships between users and generative AI are beginning to form.
Park said emotional effects, cognitive dependence, and changes in sociability should be examined together, and suggested safeguards for vulnerable groups. The photo shows Park Hyung-bin, professor at Seoul National University of Education, speaking. The photo was taken by Yang Seung-gap, a journalist.
It has been pointed out that overdependence on generative AI may differ from the excessive immersion seen in existing game and video content. That is because interactions continue in which the AI responds to users’ comments and appears to empathize, raising the possibility that perceptions of generative AI could shift from a mere tool to a kind of "relationship." Park said that at 3 a.m., it is hard to call a close friend, but generative AI is always available and never gets irritated.
Park cited as a problem of this characteristic the fact that AI often replaces interaction with other people. He then argued that safeguards are needed to reduce risks in advance, rather than responding afterward.
According to Park, discussions related to AI and online platforms in major countries overseas are becoming more granular, moving beyond the simple question of whether to regulate. Specifically, topics under discussion include measures to make age-verification standards more concrete, age-appropriate service design, and obligations to respond to crisis situations such as suicide and self-harm.
These proposals summarize tasks for improving domestic institutions, and the four tasks presented were: legislating safety standards for emotionally reciprocal AI, establishing age verification and age-appropriate design systems, setting a national AI literacy standard, and protecting neural data and regulating neural rights.
In particular, it was suggested that emotionally reciprocal AI needs a separate definition and safety standards tailored to its characteristics. It was also proposed that response protocols be required when suicide or self-harm risks are detected, and that users be notified that AI is a non-personal entity. In addition, restrictions on dark patterns that induce service use by stimulating users’ emotions and stronger safeguards for minors were proposed.
Park emphasized that children are a vulnerable group because their brains are still developing, and that safety standards for emotionally reciprocal AI need to be more detailed and systematic than those currently in place.
The broad thrust of the proposal focused on protecting minors and strengthening AI literacy. To that end, it is necessary to refine the age-verification system, and it was pointed out that a method based on direct age entry has limitations. There was also a proposal to establish an age-verification system premised on minimizing privacy infringement.
There was also an opinion that a high level of protection settings should be applied by default to accounts used by minors. For services that have a large impact on teenagers, separate age-based safety design is also needed, it was suggested. The idea is that age verification should lead to protection settings and safety design while reducing privacy infringement.
On the education side, the proposal called for establishing a national AI literacy standard. School education should include deepfake detection, labeling of AI-generated content, information verification, emotional relationships with AI, and the risks of overdependence. Park said AI literacy education is needed for all age groups, including parents, and that institutionalization is necessary to make education and self-protection possible for everyone.
It was also pointed out that preparation is needed for the combination of brain-computer interfaces (BCI) and AI. As advances in neurotechnology are expanding the possibility of inferring thoughts and mental states based on individuals’ brain signals, institutional measures related to neural data protection systems and discussions on neural rights are needed, it was suggested. There were also opinions that institutional review of the scope of information collection, use, and access is necessary.
Park Hyung-bin, a professor at Seoul National University of Education, emphasized that rather than excluding new technologies, it is important to create conditions under which they can be used safely. He also stressed the need to review usage conditions and design for generative AI and brain-computer interfaces.
Government officials who took part in the discussion that day also agreed on the need to supplement institutions for AI safety and user protection. At the site, related discussions led toward institutional responses.
Jung Joon-wook, team leader for digital inclusion policy at the Ministry of Science and ICT, said age-specific guidelines based on AI ethics rules are needed. He said the age-specific guidelines will be used for preventive education and professional counseling.
Jung Joon-wook, team leader for digital inclusion policy at the Ministry of Science and ICT, said he plans to develop age-specific guidelines together with experts. The people in the photo are Kim Joo-young, head of the AI Education Promotion Division at the Ministry of Education; Park Hyung-bin, professor at Seoul National University of Education; Jung Joon-wook, team leader for digital inclusion policy at the Ministry of Science and ICT; and Kim Hye-sook, head of the AI User Protection Division at the Broadcasting and Media Communications Commission. The photo credit is Yang Seung-gap.
Kim Hye-sook, head of the AI User Protection Division at the Broadcasting and Media Communications Commission, pointed out that the positive features of AI services can turn into negative effects depending on the situation, and said that even elements once seen as enabling smooth, frictionless use also carry the potential for adverse effects. She added that institutional measures are needed to preserve the benefits while addressing the downsides.
Source: IT DAILY · Yang Seung-gap
Original: https://www.itdaily.kr/news/articleView.html?idxno=241528
References
This article was produced with the help of an automated content generation algorithm.
Source: IT DAILY
View originalThis article was summarized and organized by BizCrush based on the original article from IT DAILY. For exact quotations and full details, please refer to the original article.