Before Your AI App Edits a Real Person: A Consent Gate for Founders

A user uploads a portrait, asks your AI app to change the clothes and setting, and shares the result. That can be an ordinary creative workflow. It can also become impersonation, humiliation, sexual abuse, or exploitation when the uploader is not the person pictured, the subject is a child, the requested change is intimate, or the output leaves the private workspace without the subject's permission.

This guide is for founders and small teams shipping photo editors, avatar makers, video generators, virtual try-on tools, face animation, character apps, or any product that can transform a recognizable person. The core judgment is: an upload is not proof of consent, and a content filter is not a consent system. Before a real-person transformation can run, the product must classify who is depicted, what transformation is requested, where the result may travel, and what evidence and remedy the risk level requires.

You will leave with a real-person media consent gate, a concrete launch scenario, a capability matrix, a minimum consent receipt, an abuse test pack, an incident workflow, and explicit boundaries for cases a small team should not support. This is a product-risk framework, not legal advice and not a promise that any detector can determine age, identity, or consent perfectly. Jurisdiction, audience, distribution model, and product facts still require qualified legal and child-safety review.

The need is not hypothetical. The National Center for Missing & Exploited Children says its CyberTipline received more than 7,000 reports related to generative-AI child exploitation over two years, while emphasizing that known reports do not capture all incidents. The relevant founder question is not whether your app was built for abuse. It is whether the product's capabilities, defaults, and response path make predictable abuse difficult to perform, difficult to distribute, and possible for a harmed person to stop.

Define the objects before you ask for consent

Teams often add one checkbox—“I have permission to use this image”—and call the problem solved. That statement compresses several different permissions into one ambiguous click.

Use these terms separately:

• Uploader: the account sending the source media. The uploader may or may not be depicted.
• Subject: a recognizable person represented in the input or output.
• Source authorization: whether the uploader is allowed to possess and submit the source file.
• Transformation consent: whether the subject agreed to this kind of change, such as animation, age alteration, clothing replacement, or voice synchronization.
• Distribution consent: whether the subject agreed to the result being downloaded, posted publicly, used commercially, or shared with another person.
• Revocation and remedy: how the subject can withdraw permission where applicable, report abuse, and request removal or account action.
• Provenance: information about how the media was created or modified. Provenance can support transparency; it does not itself prove that the subject consented.

The product should therefore treat consent as a scoped decision bound to a subject, source, transformation class, destination, time, and policy version—not as a permanent property of an account.

Start with a capability map, not a moderation prompt

Before selecting a model or writing safety prompts, list what the product can actually do. Review each path from input to output, including features that seem secondary: importing from a URL, processing a shared album, remixing community posts, cloning a previous project, exporting without a watermark, generating a public link, and invoking the model through an API.

The NIST Generative AI Profile identifies obscene, degrading, or abusive content—including synthetic child sexual abuse material and nonconsensual intimate imagery—as a generative-AI risk, and recommends risk management across the lifecycle rather than at one model call. For a small product, that means the upload form, queue, model provider, object storage, gallery, support inbox, and deletion job are one safety system.

Create one row for every capability:

This is not a universal legal table. It is a conservative product starting point. The useful discipline is to decide which capabilities exist and who can access them before a clever prompt turns an undocumented edge case into a feature.

A concrete failure scenario: the harmless avatar tool

Imagine a two-person startup launching an “event avatar” app. A user uploads a photo, selects a theme, and receives a stylized image. The founders believe the app is low risk because it does not advertise nudity, the upstream model has safety filters, and every uploader accepts the terms.

The first version has five hidden gaps.

First, the upload screen says “your photo,” but does not prevent images of classmates, coworkers, former partners, or strangers. Second, free-form prompts can change clothing and body shape even though the preset menu looks benign. Third, a share link is public to anyone with the URL and is indexed by analytics and preview services. Fourth, original files and outputs remain in object storage after the user deletes the project. Fifth, the support form asks for an account email, so a depicted person who never opened an account cannot report the image.

An attacker uploads a school group photo, crops one face, requests escalating clothing changes through prompt variations, and shares the output. Individual generations sometimes fail, but retries eventually produce a harmful result. The subject discovers it through a third party. Support cannot locate the output because the reporter lacks the creator's project ID. The team removes one URL but retains the asset and does not check duplicate shares.

No single missing classifier caused this incident. The failure crossed product language, input authority, prompt controls, retry behavior, sharing defaults, storage, support identity, and removal scope. “Our provider filtered the prompt” would not answer the harmed subject's immediate needs.

A consent gate would have changed the path:

1. The app would classify the input as a real-person image and ask whether the uploader is depicted or has scoped authority.
2. A face-bearing input plus clothing/body transformation would enter a restricted path, not the general prompt queue.
3. The transformation policy would evaluate the prompt, source image, intermediate edits, and output—not text alone.
4. New real-person projects would remain private, with public sharing requiring a separate action and notice.
5. The system would retain a privacy-preserving receipt linking source hash, output IDs, policy version, and share surfaces.
6. A non-user subject could report by URL or file, receive a case number, and trigger containment across known copies.

Build a real-person media consent gate

Run the following seven checks before generation, not after the result has already been delivered.

Gate 1: Detect the real-person path conservatively

Use product context, structured user choices, and available media signals to decide whether a recognizable real person may be involved. Do not present an age or face classifier as truth. If the system cannot distinguish a real person from a photorealistic synthetic person, choose the safer path for sensitive transformations.

Avoid a purely hidden decision. Ask a plain question such as: “Does this image show you, another real person, or a fictional/synthetic character?” Explain why the answer changes available features. Treat contradictory signals as uncertainty, not permission.

Gate 2: Bind authorization to a purpose

For low-risk self-editing, a clear attestation may be proportionate. For another person's recognizable image, ask for a narrower statement: the uploader has permission for the named transformation and intended destination. Do not preselect it. Do not bury it in general terms.

An attestation is evidence of what the uploader claimed; it is not proof that the claim is true. Its value is to set a clear boundary, support enforcement, and interrupt casual misuse. Higher-risk workflows need stronger controls or should be unavailable.

Gate 3: Classify the transformation, not only the prompt

A harmless prompt can be paired with a sensitive source, and an oblique prompt can evade keyword rules. Classify the requested operation and the output: sexualization, simulated nudity, age change, body change, injury, humiliation, hate, political endorsement, medical condition, criminal conduct, and commercial impersonation deserve separate rules.

Google Play's official AI-generated content policy explicitly includes image-to-image apps and apps creating recordings of real people, and lists AI-generated nonconsensual sexual deepfakes among prohibited examples. Apple's App Review Guidelines require user-generated-content apps to filter objectionable material, provide reporting and blocking, respond to concerns, and disclose third-party AI data sharing with explicit permission. Passing an upstream model's filter is therefore not the same as meeting the product's distribution and response responsibilities.

Gate 4: Give minors a separate safety boundary

Do not treat “likely adult” as evidence that a person is an adult. Age estimation can be uncertain, biased, and manipulable. If a source depicts a known or possible minor, disable sensitive transformations and risky sharing paths. Do not ask a child to consent away a high-consequence product risk.

The UK's Children's Code calls for children's best interests, high privacy by default, data minimization, and protection from nudges that weaken privacy. The European Commission's DSA guidelines for protecting minors similarly use safety and privacy by design and recommend private defaults, easy blocking, and controls against unwanted distribution of intimate material. Applicability depends on the service and jurisdiction, but the product principle travels well: uncertainty about age increases caution; it does not lower it.

Gate 5: Separate creation permission from distribution permission

The safest useful default for real-person transformations is a private workspace. Download, public gallery, searchable profile, remix, and share-link creation are separate capabilities and should have separate decisions. Tell the user whether a link is public, guessable, expiring, downloadable, or visible to preview crawlers.

Do not rely on “unlisted” as a synonym for private. An unlisted URL can be forwarded, logged, screenshotted, or exposed through referrers. For high-risk media, consider disabling public links entirely. For lower-risk media, use expiration, revocation, authentication where appropriate, and an asset-level kill switch.

Gate 6: Preserve a minimal, privacy-aware receipt

Record enough to reproduce a safety decision and contain an incident, but not a new archive of biometric data. A receipt can store keyed hashes, internal asset IDs, policy/model versions, declared relationship, transformation class, sharing state, decision codes, and deletion deadlines. Restrict access and document retention.

The C2PA Content Credentials explainer is useful for understanding provenance: credentials can record origin, modifications, and AI involvement in a cryptographically bound structure. It also states an important limit: provenance does not judge whether the recorded facts are good or true. Use provenance as one transparency signal, not as a substitute for subject permission, abuse prevention, or removal.

Gate 7: Make remedy work for a person without an account

A depicted person may not know the uploader, project ID, or model prompt. Give reporters a path using a public URL, output file, or enough descriptive information to locate the asset. Do not require them to create an account or upload the abusive file repeatedly. Return a case number and state what happens next.

In the United States, the FTC says covered platforms subject to the TAKE IT DOWN Act must provide a clear process and remove covered nonconsensual intimate images and known identical copies within 48 hours of a valid request. The exact legal scope requires counsel, but every image product benefits from practicing the operational capabilities now: intake, verification appropriate to the request, containment, duplicate discovery, status tracking, escalation, and documented closure.

Use a decision matrix that cannot average away harm

Do not calculate one “safety score” in which a severe failure can be offset by good image quality or a benign theme. Use categorical outcomes.

Human review is not a button labeled “send to moderation.” Define reviewer training, visible evidence, prohibited data exposure, response time, decision options, and escalation. Reviewers should be able to return insufficient_evidence without forcing approval or rejection.

Store a consent and safety receipt

The receipt below is an implementation template, not a compliance certificate. Avoid storing the raw media inside the receipt, and adapt fields with privacy and legal review.

real_person_gate:
  receipt_id: rpm-2026-08-16-00192
  policy_version: real-person-v3
  owner: trust-and-safety
  source_asset_id: asset-internal-1842
  source_keyed_hash: hmac-sha256:...

subject_path:
  declared_subject: self | another_adult | possible_minor | synthetic | unknown
  machine_signal: real_person_possible
  resolution: restricted_real_person_path
  raw_biometric_template_stored: false

authorization:
  source_authority_attested: true
  transformation_scope: professional_avatar
  destination_scope: private_and_download
  attestation_text_version: consent-v2
  attested_at: 2026-08-16T02:40:00Z

safety_decision:
  prompt_class: ordinary_portrait_edit
  source_class: non_explicit
  output_class: non_explicit
  retry_risk: normal
  result: allow_restricted
  decision_codes: [REAL_PERSON, PRIVATE_DEFAULT, NO_PUBLIC_GALLERY]

distribution:
  public_share: false
  downloadable: true
  provenance_attached: true
  known_output_ids: [output-771]

retention:
  source_delete_after: 24h
  output_delete_after: account_policy
  receipt_delete_after: risk_and_legal_policy
  deletion_job_id: delete-551

remedy:
  subject_report_url: /safety/report
  disable_asset_capability: enabled
  duplicate_lookup_capability: enabled
  escalation_playbook: ncii-csam-v2

The most important fields are the constraints, not the claim that consent exists. The receipt should show what the uploader asserted, what the system inferred, which features remained unavailable, and how the team can act if the assertion is challenged.

Test the abuse journey, not just model refusal

A prelaunch test set should include complete journeys. At minimum, test these twelve cases:

1. A user honestly declares that another adult appears in the image.
2. A user falsely selects “this is me” for a third-party portrait.
3. A group photo is cropped to isolate one possible minor.
4. A benign first prompt is followed by progressively sensitive edits.
5. A blocked phrase is paraphrased, misspelled, translated, or represented with an image reference.
6. The same blocked source is retried across accounts, sessions, API keys, or crop variants.
7. A safe generation is edited into a harmful result through an inpainting or remix feature.
8. A private output becomes accessible through a share preview, analytics log, gallery, or copied URL.
9. The project is deleted, but source, thumbnail, CDN object, backup, or share record remains available.
10. A depicted non-user reports an output without knowing the creator or project ID.
11. A valid report identifies one file while duplicate shares and resized variants exist.
12. A provider, classifier, or moderation queue becomes unavailable during generation.

OpenAI describes a layered child-safety approach that includes pre-deployment and production safeguards, hash matching, reporting, and external partnership. A startup using a hosted model will not reproduce a large provider's systems, but the architecture lesson is applicable: provider refusal, application policy, storage controls, abuse monitoring, and incident response cover different failure points.

Design retention and deletion as safety controls

Real-person media is not ordinary generated text. Original uploads may reveal faces, homes, schools, badges, documents, locations, or relationships. Generated outputs can remain harmful even when technically synthetic.

Define retention by object:

• raw upload;
• normalized or cropped derivative;
• model-provider copy;
• prompt and negative prompt;
• embedding or biometric-like representation;
• generated output and thumbnail;
• public or unlisted share;
• moderation evidence;
• consent receipt;
• logs, backups, and CDN copies.

Hash-based matching can help find known copies without making a harmed person repeatedly distribute the image. StopNCII's process creates a digital fingerprint on the person's device and shares the hash with participating platforms, which look for matches under their policies. Exact hashes have limits: crops, re-encodes, screenshots, and edits may not match, while perceptual matching introduces its own error and governance questions. Use duplicate detection as a containment aid, not automatic proof of violation.

Prepare the incident path before accepting uploads

Write the incident runbook while the team is calm. A minimal path is:

1. Receive: accept reports from users and depicted non-users; issue a case ID.
2. Triage: identify possible minors, imminent threats, extortion, intimate imagery, and widespread sharing.
3. Contain: disable access and sharing when policy permits; stop queued variants; restrict the implicated account or API key.
4. Preserve minimally: retain only evidence required for investigation, reporting, appeal, or legal duty under controlled access.
5. Find known copies: check output IDs, share records, exact hashes, and cautiously governed similarity tools.
6. Escalate: route child-safety, law-enforcement, platform, provider, and legal questions to named owners.
7. Communicate: tell the reporter what was received, what can be disclosed, current status, and next step.
8. Remediate: remove confirmed violations, revoke links, address caches, apply account action, and verify completion.
9. Learn: convert the path into a regression test without retaining unnecessary harmful media.

Know what this gate does not solve

This framework reduces foreseeable product failures; it does not establish identity, age, authority, or consent with certainty. Checkboxes can be false. Identity documents can be stolen and create dangerous retention obligations. Face matching can misidentify people. Age estimation is probabilistic. A provenance credential can be removed or can faithfully record a workflow that was never authorized. Watermarks can be cropped. Private outputs can be photographed. Human reviewers can make mistakes or be harmed by repeated exposure.

The gate is also not a reason to collect maximum personal data. If a capability requires invasive verification that a small team cannot secure and govern, the correct launch decision may be to narrow or remove the capability. “We can build it” and “we can operate it responsibly” are different tests.

Some products need specialist review before launch: tools directed to children; sexual or romantic experiences; medical, legal, political, or employment impersonation; biometric identification; public-figure generation; open model hosting; public galleries; encrypted services where moderation claims exceed technical visibility; and products operating across conflicting jurisdictional duties.

Finally, the gate cannot transfer responsibility to the model provider. Provider controls may change, fail, or cover only the generation endpoint. Your app still chooses who can upload, how often they can retry, which edits can chain, what is stored, what becomes public, and whether a subject can obtain relief.

A two-day founder implementation plan

On day one, map capability and consequence. List every real-person input, transformation, export, share, remix, API, and deletion path. Mark critical blocks: sexualization or intimate fabrication involving real people, any exploitative treatment of minors, and transformations your team cannot review safely. Set real-person projects to private by default. Add distinct source, transformation, and distribution attestations where proportionate. Name one safety owner and one backup.

On day two, make the decision observable. Implement a versioned receipt without raw media, stable asset IDs, a non-user reporting route, an asset-level disable action, deletion jobs, and a small abuse test pack. Run the twelve journeys above. Verify the upstream provider's current rules and data retention rather than assuming them. Document what happens when a classifier, queue, or provider fails; sensitive paths should fail closed.

Do not launch the public gallery, batch API, or sensitive editing features simply because the ordinary avatar path passes. Each expands the threat model and needs its own gate. Start with the narrowest capability that delivers the user value, observe attempted misuse and legitimate friction, and reopen the review whenever models, policies, audiences, sharing, or markets change.

Final launch checklist

Before enabling real-person transformation, confirm:

• [ ] We distinguish uploader, subject, source authorization, transformation consent, and distribution consent.
• [ ] We know every real-person entry point, including URL import, remix, API, batch, and shared projects.
• [ ] Sensitive real-person and possible-minor transformations have categorical blocks, not average scores.
• [ ] Real-person workspaces are private by default, and sharing is a separate informed action.
• [ ] Prompt, source, output, retry pattern, and distribution state are evaluated at the appropriate layers.
• [ ] The consent/safety receipt records claims and constraints without pretending to prove consent.
• [ ] Raw uploads, derivatives, thumbnails, provider copies, logs, shares, and backups have explicit retention rules.
• [ ] A depicted non-user can report by URL or file and receive a case number.
• [ ] The team can disable an asset and known shares, search for known copies, and verify deletion.
• [ ] Child-safety, intimate-image, extortion, and imminent-harm cases have named escalation routes.
• [ ] Provider or classifier failure closes sensitive paths instead of silently allowing them.
• [ ] We rerun the gate after material changes to model, policy, audience, capability, or distribution.

References

• NCMEC: The Growing Concerns of Generative AI and Child Sexual Exploitation
• NIST: Artificial Intelligence Risk Management Framework—Generative AI Profile
• Google Play: Understanding the AI-Generated Content Policy
• Apple: App Review Guidelines
• UK ICO: Age Appropriate Design—Children's Code
• European Commission: Guidelines on the Protection of Minors under the DSA
• FTC: TAKE IT DOWN Act Enforcement—Business Guidance
• C2PA: Content Credentials Explainer
• StopNCII: How the Hash-Based Process Works
• OpenAI: Combating Online Child Sexual Exploitation and Abuse
• FBI: Sextortion
• Google Search Central: Creating Helpful, Reliable, People-First Content