The app store safety promise was always an illusion

A few years ago, the process of finding an app was simple. A user opened a digital storefront, typed a keyword into a search bar, and trusted that the results were safe because a multi-trillion-dollar corporation had vetted them. This was the core appeal of the walled garden. It offered a sanitized alternative to the wild, unpredictable open web. Today, that sense of security is gone. When a user searches for a simple photo editor, the algorithm often suggests tools designed to strip clothes from images without consent. This is the reality that prompted San Francisco City Attorney David Chiu to order Apple and Google to purge dozens of nudify apps from their platforms.

The shift from neutral pipes to active facilitators

Historically, tech giants operated under the protection of legal frameworks that treated them as neutral intermediaries. They were the pipes, not the water. If a user shared something illegal, the user was at fault, not the service provider. This paradigm has shifted. California law now criminalizes activity that knowingly facilitates or recklessly aids the creation of non-consensual deepfake pornography. A 2025 law further allows victims to pursue civil actions against the third-party platforms that host this material. The city of San Francisco now argues that Apple and Google are no longer passive hosts. By processing payments and suggesting these apps through search algorithms, these companies are active participants in a predatory economy.

Zooming out to the industry level, this legal pressure marks the end of the hands-off era for app moderation. In the past, companies relied on automated filters to catch prohibited content; today, those filters are failing against the sheer volume of generative AI tools. The Tech Transparency Project found that both companies were warned about these apps as early as January 2026. Despite these warnings, the apps remained available for months. This delay suggests that the infrastructure for app review is buckling under the weight of its own scale. When a platform hosts millions of pieces of software, manual review becomes a bottleneck that most corporations choose to ignore in favor of growth.

The economics of non-consensual code

There is a financial logic behind the presence of these apps. Apple and Google typically take a 30 percent commission on in-app purchases. When a nudify app charges a user for credits to generate an image, a portion of that money goes directly to the platform owner. David Chiu estimates that these companies have made millions of dollars from these transactions. This creates a conflict of interest between safety and revenue. In everyday terms, it is like a shopping mall that takes a cut from a store selling stolen goods. The mall has every incentive to keep the store open as long as the rent is paid and the authorities are not knocking on the door.

Technically speaking, the problem is not just the existence of the apps but how the stores promote them. The Tech Transparency Project noted that Google and Apple intentionally steered users toward these tools through search suggestions. This is a byproduct of algorithmic curation. The store's search engine is designed to show users what they are likely to download, not what is ethical to host. If thousands of people search for a specific type of exploitative tool, the algorithm prioritizes that tool to maximize the probability of a transaction. The code does not have a moral compass; it only has a conversion rate.

The breakdown of the algorithmic gatekeeper

Under the hood, nudify apps often use sophisticated methods to bypass the initial review process. Developers might submit a seemingly harmless photo filter app for approval. Once the app is live, it can pull down new code or access remote AI models that perform the actual image manipulation. This is a form of technical debt in the moderation stack. The review process happens at a single point in time, but software is dynamic. A developer can change the behavior of an app after it is already on a user's phone. This makes the one-time check at the gate an ineffective security measure.

Paradoxically, the same companies that market their devices as privacy-focused tools are the ones providing the distribution for privacy-violating software. Apple often uses its control over the ecosystem to restrict data sharing between legitimate apps, yet it struggled to identify software that uses AI to target individuals. This reflects a fragmented approach to platform safety. One part of the company builds robust encryption; another part of the company processes payments for apps that generate non-consensual intimate images. The two sides of the business do not always communicate.

Legacy systems in an AI world

Many of the tools used to monitor app stores are legacy systems. They were built for an era when apps were static packages of code. In the current landscape, an app is often just a thin wrapper around a web-based API. The heavy lifting of the AI processing happens on a remote server that the app store reviewers cannot see. This makes the app itself a black box. If the API is a restaurant waiter, the app store only sees the person taking the order. It has no way of knowing what is happening in the kitchen until the dish is already on the table.

Consequently, the responsibility for safety has shifted from the platform to the public. When Apple and Google claim they take swift action after violations are reported, they are admitting that their proactive systems failed. They are relying on users, journalists, and city attorneys to do the work of moderation for them. This creates a reactive environment where the harm is already done before the app is removed. For the victims of deepfake pornography, a suspension that happens months after an app has been downloaded thousands of times is a hollow victory.

Reclaiming control over the digital storefront

Ultimately, the order from San Francisco is a reminder that the software running our lives is not a lawless frontier. The code that powers an iPhone or an Android device is subject to the same societal rules as any other product. We have reached a point where digital literacy requires us to look past the intuitive interface of an app store and see the business rules behind it. The curated experience we were promised is a marketing construct. In practice, the storefront is an open marketplace where the primary goal is the transaction.

We should observe our own digital habits and ask why we trust these platforms so implicitly. The presence of an app on an official store is not a seal of quality or safety. It is a sign that the developer followed a set of technical specifications and agreed to share their revenue. As users, we must demand more than reactive moderation. We should expect the companies that control the infrastructure of our digital lives to be as vigilant about our safety as they are about their profit margins. The city of San Francisco has made it clear that the era of plausible deniability for tech giants is over.

Sources:
San Francisco City Attorney's Office official correspondence
Tech Transparency Project April 2026 report
California Civil Code Section 1708.85
TechCrunch and Wired investigative reports