In a recent case, a chatbot called 'Emilie' was posing as a licensed psychiatrist and was able to produce a fake medical licence number as proof of medical qualifications. The chatbot allegedly maintained the pretence of being a psychiatrist while responding to someone dealing with depression. The State of Pennsylvania is suing the company behind the chatbot, Character.AI, in what officials describe as the first enforcement action of its kind, maintaining that AI systems do not have the right to present themselves as qualified medical professionals.
The rise in AI development, which enables AI systems to sound more human in tone and more persuasive in conversation, raises new questions for the privacy community and the AI space in general: what happens when AI starts engaging not only in spreading misleading information, but engaging in potential identity theft?
As cases of such AI usage increase, regulators appear increasingly willing to treat harmful outputs as an enforcement issue rather than a mere technical mistake.
