The social media network X has caused major backlash after individuals used its AI chatbot Grok to alter images of females by stripping their garments.
Multiple cases have surfaced where the system has been used to virtually strip women to render them in swimwear without consent, as well as to place them in sexually explicit situations.
Company Reaction and Legal Stance
XAI, the organization responsible for Grok, remained silent to questions, other than an automatically-generated message stating "legacy media lies".
A government representative stated that legislation is being proposed to ban nudification technology. Under a new law, anyone who distributed such tools would "be subject to a jail term and heavy penalties".
"The platform could prevent these forms of misuse if they wanted to," said Clare McGlynn, adding they "appear to enjoy impunity."
Regulator Stance and Tool Capabilities
The communications watchdog Ofcom said technology companies must "assess the potential" of people accessing unlawful material on their services, but stopped short of confirming whether it was at this time investigating X or Grok in relation to artificial intelligence pictures.
Grok is a free chatbot—with some additional functions—which responds to X users' queries when they reference it in a message.
It is commonly used to provide commentary or more context to other posters' remarks, but people on X are also capable of alter an uploaded photo through its artificial intelligence picture modification feature.
History of Controversy and Rules
The system has been faulted for permitting users to generate pictures and videos with nudity and sexually suggestive content, and it was earlier alleged to have been involved in making graphic content of public figures.
"X has been permitting the production and distribution of these pictures for a long time without taking any action and we have not witnessed any challenge by regulators," Professor McGlynn added.
XAI's own acceptable use policy prohibits "showing likenesses of persons in a pornographic manner".
In a comment, Ofcom said it was unlawful to "generate or share deepfakes or child sexual abuse material" and clarified this included sexual deepfakes generated with AI.
It said networks such as X were required to take "appropriate measures" to "lessen the risk" of users encountering unlawful content on their platforms and to take down it promptly when they are notified of it.
