On Jan. 10, Indonesia became the first country to announce it was temporarily blocking access to Grok, followed soon after by Malaysia. Other governments, including the United Kingdom’s, have promised to take action against the chatbot and its related social media site X, on which the sexualized images have been shared.

But while outright national bans can limit casual use of the chatbot, such bans are easily bypassed using virtual private networks (VPNs) or alternative routing services. These mask the user’s real location and make it appear they originate from a location that allows access to the service.

As a result, country-level bans tend to reduce visibility rather than eliminate access. Their primary impact is symbolic and regulatory, placing pressure on companies such as xAI rather than preventing determined misuse. And content generated elsewhere can still circulate freely across borders via encrypted social media platforms and on the dark web.

In response to the controversy, X immediately moved Grok’s image generation features behind a paywall, making them only available to subscribers and promised to take action against “illegal content on X, including child sexual abuse material”. Then on Jan. 15, it announced it would geo-block the ability of all users, including paid subscribers, to generate such images of real people in jurisdictions where this was illegal.

While not all chatbots have image generation capabilities, most mainstream providers, including OpenAI, xAI, Meta and Google, provide this service.

Modern AI image generators are typically built using diffusion models, which are trained by taking real images and gradually adding random visual distortion, known as “noise”, until the original image is no longer recognizable. The model then learns how to reverse this process, step by step, reconstructing an image by removing noise.

Over time, it will learn statistical patterns representing faces, bodies, clothing, lighting and other visual features. These patterns are organized within the model so that visually similar concepts sit close together. Because clothed and unclothed human bodies share very similar shapes and structures, the changes required to move between them can be relatively small.

So when an existing image is used as the starting point and identity-preserving features are retained, transforming a clothed photograph into an unclothed one becomes technically straightforward. Of course, the AI model itself has no understanding of identity, consent or harm. It simply produces images that resemble what it has learned, in response to user requests.

However, after the core model has been trained, companies can apply “retrospective alignment”: rules, filters and policies that are layered on top of the trained system to block certain outputs and align its behavior with the company’s ethical, legal and commercial principles.

But retrospective alignment does not remove capability; it simply limits what the AI image generator is allowed to output. Those limits are primarily a design and policy choice made by the company operating the chatbot, although these may also be shaped by legal or regulatory requirements imposed by governments, for example, requiring companies to disable or restrict certain features such as identity-preserving image generation.

Large, centrally hosted social media platforms could also play an important role here. All have the power to restrict the sharing of sexual imagery involving real people and to require explicit consent mechanisms from the people featured in such images. But to date, big tech companies have tended to drag their feet when it comes to labor-intensive moderation of their users’ content.

Research by Nana Nwachukwu, a PhD candidate at the SFI Centre for AI-Driven Digital Content Technology at Trinity College Dublin, highlighted the frequency of requests for sexualized images on Grok. Other research estimated that before the service went behind a paywall, up to 6,700 undressed images were being produced every hour.

This has prompted regulatory scrutiny in Europe and beyond. French officials described some outputs as manifestly illegal and referred them to prosecutors. The UK’s communications watchdog, Ofcom, has launched an investigation into X and xAI over the issue.

But this problem is not limited to one platform. In early 2024, nonconsensual AI-generated sexual images of pop star Taylor Swift, produced using publicly available tools, spread widely on X before they were removed due to a combination of legal risk, platform policy enforcement and reputational pressure.

Some platforms explicitly market minimal or no restrictions on content as a feature rather than a risk. It is simple enough to find websites promoting “unrestricted” image generation and privacy-focused use, relying largely on open-source models and offering far fewer moderation controls than mainstream providers. Furthermore, there is an even larger number of self-hosted image and video generation tools where safeguards can be removed entirely.

While precise figures are unavailable, independent estimates suggest tens of millions of AI-generated images are created daily across platforms, with video generation rapidly accelerating.

Another potential issue is that some AI chatbots, including Meta’s Llama and Google’s Gemma, can be downloaded to computers, even those with relatively light processing power. When run offline afterward, these models are completely free of oversight or moderation.

Even tightly controlled systems can be bypassed through “jailbreaking”: a way of constructing prompts to fool the generative AI system into breaking its own ethics filters. Jailbreaking exploits the fact that retrospective alignment systems depend on contextual judgment rather than absolute rules. Instead of directly asking for prohibited content, users reframe their prompts so the same underlying action appears to fall within an allowed category such as fiction, education, journalism or hypothetical analysis.

An early example was known as the “grandma hack” because it involved a recently deceased grandmother recounting experiences from her technical profession in chemical engineering, leading the model to generate step-by-step descriptions of prohibited activities.

The internet already contains an enormous quantity of illegal and nonconsensual sexual imagery, far beyond the capacity of authorities to remove. What generative AI systems change is the speed and scale at which new material can be produced. Law enforcement agencies have warned that this could lead to a dramatic increase in volume, overwhelming moderation and investigative resources.

Laws that may apply in one country can be ambiguous or unenforceable when services are hosted elsewhere. This mirrors long-standing challenges in policing child sexual abuse material and other illegal pornography, where content is frequently hosted offshore and rapidly redistributed. Once images spread, attribution and removal are slow and often ineffective.

By making countless millions more people aware of the possibility of sexualizing and “nudifiying” images, high-profile AI chatbots make it possible for large numbers of users to generate illegal and abusive sexual imagery through simple prompts in plain English. Estimates suggest Grok alone currently has anywhere from 35 million to 64 million monthly active users.

If companies can build systems capable of generating such imagery, they can also stop it being generated, at least in theory. In practice, however, the technology exists and there is a demand for it, so this capability can never be eliminated now.

---

The writer is a senior lecturer in computing and information systems at Cardiff Metropolitan University. This article is republished under a Creative Commons license.