Authorities in San Francisco have directed tech giants Apple and Google to remove a number of applications capable of generating non-consensual nude images, warning that such platforms enable digital sexual abuse.
The City Attorney, David Chiu, issued cease-and-desist notices to both companies, demanding the removal of 13 applications identified as tools for creating explicit deepfake content through face-swapping technology.
Chiu stated that the apps allow users to produce intimate images of individuals without consent, describing the practice as illegal, harmful and unacceptable.
The legal notices further accused the companies of benefiting financially from the apps through in-app purchase systems, urging them to immediately cut ties with developers involved and strengthen content moderation processes.
According to the city attorney’s office, California laws prohibit services that facilitate the creation or distribution of deepfake pornography, placing responsibility on platform providers to prevent such tools from being accessible.
Investigations revealed that the identified apps, some of which have recorded millions of downloads, often present themselves as standard image-editing or face-swapping tools while offering explicit content features after installation.
Experts say the growing use of artificial intelligence has made it easier to generate highly realistic images within seconds, raising concerns about the misuse of the technology, particularly against women and minors.
Reports have linked such tools to cases of online harassment, blackmail and psychological harm, with incidents recorded in multiple schools where manipulated images of students were circulated.
In response, Google said it has taken action by removing hundreds of apps that violated its policies and restricting related search terms on its platform, while maintaining that it does not allow sexually explicit or abusive content.
Apple had yet to issue a formal response at the time of reporting.
Despite existing policies prohibiting harmful content, watchdog groups and researchers have continued to identify similar applications across both companies’ app stores, highlighting gaps in enforcement.
Recent findings suggest that many of these tools operate as “dual-use” platforms, appearing harmless while retaining the capability to generate explicit material, thereby evading detection during review processes.
San Francisco authorities have indicated that further legal action may be pursued if the companies fail to comply, stressing the need for stronger safeguards to prevent the spread of exploitative AI technologies.
Source: WIRED
Leave a comment