Purpose and foundations
What governance decides
Privacy concerns the appropriate handling of information about people, including adverse consequences produced by planned and authorized processing. This is broader than confidentiality. Encrypting a dataset and excluding attackers can protect it from unauthorized access while leaving unresolved whether collecting it, inferring from it, or using it to make a consequential decision is appropriate.
Data governance assigns responsibility for decisions about information and ensures that those decisions are implemented throughout its lifecycle. Its scope includes personal information and confidential organizational information. Internal owners decide or escalate important uses; stewards administer definitions, access, retention, and quality; custodians implement storage, access, and disposal controls. These are accountability roles, not claims that an organization possesses unrestricted rights over the data.
In the European Union's General Data Protection Regulation (GDPR), personal data is information relating to an identified or identifiable person, called a data subject. Identification can be indirect, and processing includes collecting, storing, using, altering, disclosing, and erasing information. Inferences and linked records can therefore remain personal even when a direct identifier is absent. Other jurisdictions use different definitions, so an organization must determine which rules apply rather than treating this chapter as jurisdiction-free legal advice.
The handling decision
| Element | Question it answers |
|---|---|
| Information | What source, fields, representations, or inferred attributes are involved? |
| Purpose | What specific outcome justifies the processing? |
| Actor | Who requests, performs, or is affected by it? |
| Operation | Is the system reading, deriving, disclosing, changing, or deleting? |
| Recipient | Which person, team, service, or organization receives information? |
| Duration | How long does the continuing purpose justify each artifact? |
| Accountability | Who may approve the use and answer for its consequences? |
Security remains essential because attackers can bypass approved decisions, but it answers a complementary question: how are assets protected against hostile or unauthorized behavior? The AI Security chapter develops that boundary. Here the focus is the prior and continuing decision about what the system is allowed to do when it works as intended.
Specify the intended use
A purpose is the concrete outcome for which information is processed. Purpose limitation requires purposes to be specified early enough to constrain collection and later use; collection limitation asks whether the information gathered is necessary for that purpose. A proposed secondary use—a use beyond the original purpose—may require a new decision about authority, affected people, fields, recipients, and retention.
“Improve the product” is usually too broad to guide engineering. Answering a support request, measuring response quality, studying recurring defects, and training a model are different activities. Training changes learned model parameters using examples; ordinary inference uses an existing model to produce an output. The Machine Learning Fundamentals chapter explains that distinction in depth.
One conversation, different purposes
| Purpose | Potentially necessary information | Recipients and continuing need |
|---|---|---|
| Answer the request | Issue, product version, account context, reply channel | Support systems and assigned staff; retain according to the service and record purpose |
| Evaluate response quality | Issue and reply; direct contact fields may be unnecessary | Authorized reviewers or evaluation service; retain only the evaluation evidence needed |
| Train a model | Selected examples and target responses after a separate reuse decision | Training pipeline and any model supplier; dataset and resulting model require distinct lifecycle decisions |
Availability is not permission. Public webpages, customer uploads, purchased datasets, and employee records each require examination of their source, original purpose, applicable rights, notices, confidentiality, and proposed reuse. Permission is also different from fitness: authorized data can still be inaccurate, unrepresentative, stale, or unsuitable. Dataset fitness belongs in Data Quality and Curation.
Enduring privacy principles
Modern AI adds new representations and inferences, but its governance questions have older roots. Networked records increased organizations' capacity to combine information long before generative models. Three landmarks help explain why purpose, individual participation, and accountability remain central without suggesting that one framework replaced the others.
Enduring privacy principles
July 1973Records, Computers and the Rights of CitizensEstablished safeguards for openness, access, correction, secondary use, reliability, and protection against misuse in computerized record systems.
Contributors: U.S. Department of Health, Education, and Welfare advisory committee
What changed: Made the imbalance between record-keeping organizations and affected individuals an explicit systems problem, including the risks created by linking records across systems.
23 September 1980OECD Privacy GuidelinesConnected collection and purpose limits, individual participation, accountability, and transborder information flows.
Contributors: OECD Council and member-country representatives
What changed: Expressed privacy as a coordinated set of limits and responsibilities that remain relevant when information crosses organizational or national boundaries. The currently published recommendation contains revised text.
Developed during the 1990s; principles published 2009 and revised 2011Privacy by DesignMade preventive defaults, lifecycle protection, transparency, and privacy embedded in systems and organizational practices central design principles.
Contributors: Ann Cavoukian
What changed: Shifted attention toward building protective behavior into technology, infrastructure, and operations instead of depending primarily on operator restraint after deployment.
Selected foundations
| Date | Development | Contribution |
|---|---|---|
| July 1973 | Records, Computers and the Rights of Citizens | The US HEW committee described the imbalance created by computerized record systems and proposed openness, access, correction, limits on secondary use, reliability, and safeguards against misuse. |
| 23 September 1980 | OECD Privacy Guidelines | The original recommendation connected collection and purpose limits, individual participation, accountability, and transborder information flows. The currently published recommendation is revised text. |
| Developed during the 1990s; principles published 2009, revised 2011 | Privacy by Design | Ann Cavoukian argued for preventive defaults, lifecycle protection, transparency, and privacy embedded in systems, infrastructure, and organizational practices. |
These contributions solve different parts of the problem. Participation gives people ways to understand and challenge records. Purpose and use limits constrain organizational reuse. Preventive design asks engineers to make protective behavior the default instead of depending on operator restraint. AI makes all three more important because a system can synthesize new attributes, copy material into new services, and make a derived claim influential without reproducing any source sentence verbatim.
Information and responsibility
Inventory flows and copies
A useful inventory describes processing, not merely infrastructure. For every activity, record the source, affected people, purpose, fields, operation, recipient, storage location, responsible team, and retention rule. Then compare the declared design with requests, traces, configurations, supplier documentation, and observed storage. A diagram that names “database, app, model” can miss most of the governed artifacts.
A retrieval system illustrates the problem. Retrieval-augmented generation (RAG) supplies selected source material to a model while producing an answer; its full mechanism belongs in Retrieval-Augmented Generation. The source may also produce chunks, an index, provider requests, outputs, traces, feedback records, and evaluation exports. Each can differ in recipients, access controls, and lifetime.
An embedding is a numerical representation used for operations such as similarity search; Embeddings and Representation Learning explains how such representations work. Embeddings can reveal source information and belong in a sensitive-data inventory rather than being treated as anonymous. Model weights are learned parameters. They do not behave like ordinary rows: source deletion can remove a stored record without reversing its influence on trained parameters.
One answer, several governed artifacts
ExampleA single response can create independently governed copies across organizational boundaries.
Read the diagram as text
- Source record. Issue text, product version, account context, and source permissions.
- Search representation. Derived chunk and embedding linked to the source version.
- Application. Authenticates the requester, retrieves permitted context, and assembles a minimized request.
- Model provider request. Selected instructions and source facts cross an organizational boundary.
- Generated answer. Derived output whose recipient and content require disclosure authorization.
- Reduced trace. Identifiers, versions, timing, policy outcome, and status without full payload capture.
- Telemetry destination. A separate recipient receiving only approved trace fields.
- Source record → Search representation: data: derive chunk and embedding.
- Search representation → Application: data: permitted retrieved context.
- Application → Model provider request: data: minimized model request.
- Model provider request → Generated answer: data: generated response.
- Application → Reduced trace: data: selected event fields.
- Model provider request → Reduced trace: data: provider status and model ID.
- Reduced trace → Telemetry destination: data: filtered telemetry.
Artifacts commonly missed
| Artifact | Inventory questions |
|---|---|
| Uploads and connector copies | Which account supplied them, whose information appears, and where is a managed copy retained? |
| Indexes and embeddings | Which source/version produced them, which permissions apply, and how are updates or deletion propagated? |
| Prompts, outputs, and tool payloads | Which fields cross an organizational boundary, and can generated output disclose more than its destination permits? |
| Traces, caches, and feedback | Is content captured, who can inspect it, and what diagnostic purpose justifies retention? |
| Evaluation and training exports | Was reuse approved, are labels or responses generated, and which later artifacts depend on the dataset? |
| Unregistered applications | Can staff upload governed information outside approved identities, contracts, regions, or controls? |
Real products can create governed artifacts beyond the visible answer. Microsoft documents interaction records, retained versions of referenced files, uploads, and Copilot Pages under separate storage and retention behavior. That does not establish any tenant's configuration, but it demonstrates why feature-level inventory is necessary: the user-facing task alone does not enumerate the copies.
Assign accountable decisions
A processing activity needs explicit decision authority. A senior owner may approve major purposes; a steward may administer access and retention; a custodian may implement storage controls; an independent reviewer may examine whether the design meets its conditions. Separating these responsibilities prevents implementation convenience from silently deciding the purpose.
In the GDPR, a controller determines purposes and essential means, while a processor is a separate entity processing on a controller's behalf. Actual activities determine these roles, not contractual labels or internal titles. A supplier can therefore have different roles for different activities, and a controller remains responsible for choosing and overseeing processors within the applicable framework.
A privacy impact assessment examines the nature and purpose of processing, necessity, alternatives, affected people, foreseeable consequences, safeguards, and remaining risk. A Data Protection Impact Assessment (DPIA) is a named legal assessment in GDPR contexts for processing likely to create high risk; its trigger and required consultation are jurisdiction-specific. An assessment is useful only when its conditions influence scope, mitigation, approval, or rejection and are revisited when the processing changes.
Decision responsibility
| Decision | Accountable work |
|---|---|
| Add model-training reuse | Approve the purpose and authority, identify affected people, set conditions, and accept or reject remaining risk |
| Admit an external recipient | Verify the recipient, purpose, contractual role, onward sharing, region, and permitted fields |
| Implement restrictions | Configure identities, policy checks, routing, storage, retention, and deletion behavior |
| Verify operation | Inspect specifications and mechanisms, interview responsible operators, and test expected behavior |
| Handle a challenge | Restrict use when appropriate, inspect evidence, decide correction or other action, propagate the result, and communicate it |
A reviewable decision record
- Purpose and authority — The precise outcome, applicable organizational and external authority, and affected information.
- Conditions — Allowed actors, operations, recipients, regions, duration, safeguards, and prohibited reuse.
- Responsibility — The accountable decision maker, implementers, reviewers, and complaint or correction owner.
- Uncertainty — Unresolved risks, rejected alternatives, evidence limits, and assumptions that must remain true.
- Reconsideration — Exception expiry and triggers such as a new source, model, supplier path, recipient, incident, or purpose.
Distinguish people and actors
A digital identity distinguishes an actor within an online context. It need not establish a real-world identity, and one person can have several digital identities. Identity proofing validates evidence and attributes when real-world assurance is needed. Authentication verifies control of an authenticator bound to an account; authorization decides which operations that identity may perform on particular resources.
Do not collapse the actors
| Entity | Role in a request | Authority to establish |
|---|---|---|
| Person described by the record | The data subject or another affected person | Not implied by the requester's login; disclosure and correction can affect this person |
| Authenticated requester | The account initiating the operation | What this requester may do in the current tenant and context |
| Executing application or agent | The service principal performing calls | Its own powers and the delegated limits under which it acts |
| Resource service | The system holding the protected record | Which current policy and target version govern the operation |
| Recipient | The person, team, or service receiving output | Whether this destination may receive the specific derived information |
Delegation preserves the distinction between the represented party and the actor exercising delegated rights. OAuth token exchange, for example, can express subject and actor identities, but issuance and scope still depend on policy. The token's subject is not necessarily every person mentioned in the accessed records. The system must preserve tenant context and attributable actor identity through downstream calls instead of treating a broadly privileged service account as the user's authority.
A workshop on identity for AI agents demonstrates a useful boundary: signing into an agent establishes a user–agent relationship, while connecting an upstream account is a separate authorization step with additional scopes. Likewise, an agent's permitted task must remain bounded as explained in Agent Engineering. Neither step grants authority over every third party described in the resulting records.
Identity errors can invert governance outcomes. A mistaken record association may disclose another person's information, correct the wrong record, or delete the wrong person's material. Shared human accounts also erase attribution. Consequential requests therefore need proportionate requester verification, tenant-bound record matching, target validation, and a review path when identity remains uncertain.
Establish permitted uses
Permission is layered. Organizational approval answers whether the organization accepts a purpose and its risks. A legal ground may be required for personal-data processing. Contracts and licenses determine some acquisition, confidentiality, and reuse rights. Individual consent may authorize a specified activity where valid. A technical grant controls API access. Evidence for one layer does not establish the others.
Different permission questions
| Question | Possible evidence | What it does not establish |
|---|---|---|
| Did the organization approve the purpose? | Decision record and accountable approval | External legal, contractual, or individual authority |
| Is there an applicable legal ground? | Jurisdiction-specific analysis and required notices | Copyright permission, confidentiality rights, or technical access |
| Are acquisition and reuse authorized? | License, contract, source documentation, confidentiality terms, or applicable exception | Data quality or authority over third-party personal information |
| Was valid consent obtained where relied upon? | Contemporaneous notice, granular choice, actor, time, purpose, and withdrawal path | That consent was freely given merely because a checkbox was stored |
| Can this identity call the service now? | Current token, scope, resource policy, and target context | Permission for every processing purpose or downstream recipient |
Under GDPR guidance, valid consent must be freely given, specific, informed, and unambiguous. The controller must be able to demonstrate what the person was told and how and when the choice was made; withdrawal should be as easy as giving consent. Employment relationships can undermine voluntariness. CNIL illustrates purpose specificity with company-event photographs: agreement to use an image in communications does not establish agreement to use it for AI training.
Consent is not the only possible authority for processing, and withdrawal does not mean every record must always be erased. Different purposes may have independently established authority, while exceptions or preservation obligations may apply. Conversely, a broad interface notice, public availability, or a supplier's bare assurance cannot manufacture permission. CNIL recommends examining third-party dataset sources, collection conditions, notices, legal grounds, and sharing terms; a contractual statement that consent exists is not the underlying evidence.
Implementable permission record
- Purpose — The single processing purpose this record supports.
- Scope — Information, represented party, operations, recipients, and prohibited uses.
- Authority — The organizational decision and applicable legal, contractual, licensing, consent, or delegation basis.
- Conditions — Effective period, region, notice version, supplier restrictions, review triggers, and withdrawal handling.
- Evidence — References to supporting records without copying unnecessary personal information into the permission record itself.
Transformation and access
Minimize without losing the task
Data minimization asks for the least information needed for the approved purpose across several dimensions: fields, number of people, precision, recipients, and duration. Start with the task. If support triage needs an issue description and product version, a direct email address, full account history, and exact location may be unnecessary even if they are available.
Sensitivity depends on consequences and context. UK GDPR special categories include health, genetic information, political opinions, religious beliefs, and biometrics processed for unique identification. Financial or commercially confidential information can be highly sensitive without falling into those categories. Intentionally inferring a protected attribute can matter even when the input fields appear ordinary.
Transformations make different claims
| Transformation | What changes | What can remain |
|---|---|---|
| Redaction | Removes selected content | Other identifiers, surrounding context, and recoverable copies |
| Pseudonymization | Replaces direct associations with pseudonyms | Linkability across records and reidentification using separately held information |
| Aggregation | Produces group summaries rather than individual rows | Disclosure through small groups, repeated releases, or unusual combinations |
| Anonymization assessment | Evaluates whether identification remains reasonably possible in context | A conclusion limited to the release, auxiliary information, recipients, and threat assumptions assessed |
A quasi-identifier is an attribute that can identify someone when combined with other information. Removing names is therefore not enough. Narayanan and Shmatikov showed that auxiliary knowledge of a few movie ratings and approximate dates could help link a person to a Netflix Prize rating record, exposing additional ratings. The research used specific datasets and assumptions; it demonstrates a mechanism, not a population-wide reidentification rate.
Before and after minimization
| Version | Input | Assessment |
|---|---|---|
| Original | “Maya Chen, maya@example.test, enterprise tenant Northwind-Research, version 4.8: PDF exports omit table headers.” | Contains direct contact information and a distinctive tenant name beyond the stated triage need. |
| Reduced | “Enterprise tenant, version 4.8: PDF exports omit table headers.” | Preserves the issue and version; the tenant class may still narrow identity and should be retained only if routing or reproduction needs it. |
| Further reduced | “Version 4.8: PDF exports omit table headers.” | Reduces exposure further; usefulness must be tested if tenant-specific configuration affects diagnosis. |
Utility and privacy are separate tests. Research on privacy-preserving prompting found that acceptable reductions differed across tasks and models, while recovery tests showed that some transformed information remained inferable. Embeddings also require their own assessment: reconstruction research has recovered source text and names under particular encoder and access assumptions. Hashes, synthetic replacements, or vectors are transformations—not automatic anonymity guarantees.
Preserve origins and dependencies
Lineage records how identified information and artifacts originate and change. The W3C PROV model distinguishes entities, activities, and responsible agents. Derivation links a resulting entity to a source that affected it; attribution and association describe responsibility. These records are assertions, not proof that the source is true, the transformation was authorized, or the lineage is complete.
Version and granularity matter. A summary may depend on particular source records. An index entry may depend on one document version. A training dataset can be linked to a model version, but that coarse relationship does not show which training record caused a generated sentence. OpenLineage also distinguishes direct value transformations from indirect influences such as filtering or joining: a field can affect which output records exist without appearing in their values.
Lineage has different granularities
ExampleRecord-level derivatives can support targeted cleanup, while a dataset-to-model link is a coarser provenance claim.
Read the diagram as text
- Source S7. An identified and versioned source record with handling-policy references.
- Summarization activity. Creates a derived record while retaining the source dependency.
- Summary R2. Record-level lineage can identify S7 as a contributor.
- Indexing activity. Creates a search representation for S7.
- Search record I9. Stable source and chunk identities support targeted update or removal.
- Training dataset D3. Contains selected examples, including a version derived from S7.
- Training activity. Changes model parameters using the dataset.
- Model M4. Dataset-level provenance links M4 to D3, not each output to each record.
- Source S7 → Summarization activity: data: source input.
- Summarization activity → Summary R2: derivation: produces.
- Source S7 → Indexing activity: data: source input.
- Indexing activity → Search record I9: derivation: produces.
- Source S7 → Training dataset D3: selection: contributes example.
- Training dataset D3 → Training activity: data: training input.
- Training activity → Model M4: derivation: produces parameters.
Synthesized context makes this operational. A fact created from chat, documents, or business records may not appear verbatim in any source, and the sources may carry different authority. The provenance for LLM-built knowledge graphs talk describes retaining source episodes and linking derived facts back to them. When entities merge, their source relationships must survive; when a fact changes, the evidence responsible for invalidation should remain inspectable.
Lineage should carry purpose and handling references without becoming an unrestricted duplicate of source content. Keep stable source and artifact identifiers, versions, transformation identity, responsible actor, timestamps, and policy references. Retain source payloads only under their own access and retention rules. When sources are combined, preserve each source relationship so policy can evaluate the resulting artifact explicitly.
Enforce current authority
A policy affects behavior only when an enforcement mechanism controls the protected operation. Least privilege limits each actor to necessary authority. Complete mediation requires authorization checks for every access path, including recovery and maintenance. Fail-safe defaults deny access without explicit permission. A model instruction or tool description can express intent, but the model must not be able to bypass or modify the enforcement point.
Role-based access control assigns permissions through roles. Attribute-based access control (ABAC) evaluates attributes of the requester, resource, operation, and environment against policy. ABAC can express conditions such as tenant, record classification, purpose, action, destination, and time. Whatever vocabulary is used, authorization must reach copied indexes, caches, and downstream services; organizing documents into categories is not enforcement.
Authority is operation-specific. Permission to read a record does not imply permission to send it to an external recipient or modify it. Rich Authorization Requests can represent structured details such as operation, amount, recipient, or file path, but the resource server must enforce the approved details. A proposed record change also needs target validation and factual review. An HTTP If-Match precondition can reject a write if the reviewed version became stale; a matching version establishes neither authority nor correctness.
Long-running work creates a time-of-check problem. A task can retrieve material while permission is valid, continue computing after revocation, and later attempt a disclosure. Google Zanzibar's 2019 design illustrates coordination between content and authorization versions so checks can respect relevant permission changes. Its guarantee depends on participating clients following the consistency protocol; it does not automatically govern copied summaries or external recipients.
Authority can change before the effect
ExampleA permitted read does not establish continuing permission for a later disclosure.
The read enforcement point checks current policy and permits the source read. The application creates a working result.
Read the diagram as text
- Requester. Authenticated in tenant A.
- Protected source. A tenant A record.
- Policy service. Supplies current authorization state.
- Read enforcement point. Checks current authority before releasing the protected source.
- Disclosure enforcement point. Checks current authority before sending the working result.
- Read grant active. Historical state permitting the initial read.
- Working result. Computation derived from the permitted read.
- Disclosure permission revoked. New current state established while computation continues.
- External recipient. Proposed destination for the result.
- Disclosure denied. The effect is blocked under current policy.
- Requester → Read enforcement point: control: request read.
- Read grant active → Policy service: state: initial authority.
- Policy service → Read enforcement point: control: read decision.
- Read enforcement point → Protected source: data: authorized read.
- Protected source → Working result: data: derive result.
- Disclosure permission revoked → Policy service: state: current authority.
- Working result → Disclosure enforcement point: data: proposed disclosure.
- Policy service → Disclosure enforcement point: control: disclosure decision.
- Disclosure enforcement point → Disclosure denied: control: deny effect.
- Read under an active grant. The read enforcement point checks current policy and permits the source read. The application creates a working result. Active: Requester, Protected source, Policy service, Read enforcement point, Read grant active, Working result. New: Requester, Protected source, Policy service, Read enforcement point, Read grant active, Working result.
- Permission changes. The historical grant remains visible as the reason the read occurred, while a distinct revocation state becomes current. Active: Requester, Protected source, Policy service, Read enforcement point, Read grant active, Working result, Disclosure permission revoked. New: Disclosure permission revoked.
- Recheck before disclosure. The application submits the retained result to the disclosure enforcement point. Current policy produces an explicit denial, so no payload is sent. Active: Requester, Protected source, Policy service, Read enforcement point, Disclosure enforcement point, Read grant active, Working result, Disclosure permission revoked, External recipient, Disclosure denied. New: Disclosure enforcement point, External recipient, Disclosure denied.
The practical rule is simple: recheck current authority immediately before a consequential disclosure or mutation, using the actual requester, acting service, resource, action, recipient, and relevant policy state. The model may propose an operation, while Structured Outputs and Tool Calling explains validation at the proposal boundary. AI Security covers malicious attempts to redirect that proposal; governance still defines the allowed effect.
Control derived disclosures
An output can reproduce a fact, combine several facts, or infer a new attribute. Disclosure should therefore be evaluated using the output's content, purpose, audience, and destination—not merely the requester's ability to read each input. A permitted internal answer can become an impermissible external export when the recipient or downstream reuse changes.
Information-flow control constrains where information may travel as it is transformed. Myers and Liskov's October 1997 decentralized model attached owners and permitted readers to information. Combining two values preserved both restrictions: if one source allowed readers r1 and r2 while another allowed r2 and r3, their combined value could flow to r2, the common permitted reader. Relaxing an owner's restriction—declassification—required that owner's authority. Applying this formal model to AI requires an actual enforcement implementation; the paper does not prove that an LLM will obey labels.
Combined information preserves both source restrictions
A result derived from two sources can flow only to readers permitted by every contributing source policy.
Read the diagram as text
- Source A: {r1, r2}. Owner A permits readers r1 and r2.
- Source B: {r2, r3}. Owner B permits readers r2 and r3.
- Combined value: {r2}. The result preserves both policies, leaving their common permitted reader.
- Reader r2. The only reader permitted by both contributing sources.
- Reader r1 excluded. Permitted by Source A but not Source B, so the combined value cannot flow here.
- Reader r3 excluded. Permitted by Source B but not Source A, so the combined value cannot flow here.
- Owner-authorized declassification. A possible policy change requiring authority for the owner whose restriction would be relaxed; it is not exercised in this example.
- Source A: {r1, r2} → Combined value: {r2}: contributes value and policy.
- Source B: {r2, r3} → Combined value: {r2}: contributes value and policy.
- Combined value: {r2} → Reader r2: permits disclosure.
Implementation semantics can be subtle. Zep documents that a search result can match a metadata filter when at least one associated source episode satisfies it, while access policy evaluates effective metadata combined from associated episodes. A source filter therefore answers “can this artifact be found through this source?” rather than “do all contributing sources permit this recipient?” Those questions must not share an accidental boolean shortcut.
Sensitivity labels and output redaction can support decisions but do not replace authorization. Microsoft documents conditional label inheritance for supported Copilot content paths, while also documenting separately retained artifacts. The application must still identify the recipient, purpose, contributing sources, and permitted reuse. A personal assistant may choose a private channel instead of a group response, as one family-and-friends agent talk describes, but reliable routing requires verifiable audience and channel policy rather than model discretion alone.
Finally, keep a generated assertion distinct from an approved record change. An answer may be labeled as an inference with source links and uncertainty. Writing that claim into a customer, medical, or employment record can require stronger evidence, an authorized reviewer, a version precondition, and a correction path. Fluent text is not a change authorization.
Retention and changed rights
Set artifact lifetimes
Retention keeps information available over time. A retention rule should name the continuing purpose, clock-start event, review or expiry condition, authorized readers, disposition action, and owner of any exception. Indefinite retention needs an actual continuing justification rather than cheap storage or imagined future usefulness.
Records schedules distinguish active use, cutoff, a retention period, and eventual destruction or transfer. NARA's guidance illustrates event-based triggers such as case closure and age-based triggers measured from creation or receipt. It supplies a scheduling model, not durations for private AI systems. An applicable, separately established preservation duty can postpone the scheduled disposition.
Artifact-specific retention
| Artifact | Continuing purpose and trigger | Disposition questions |
|---|---|---|
| Source records | Operational, contractual, or record purpose; trigger may be case closure or relationship end | Delete, archive, restrict, or preserve under an applicable exception |
| Model requests and outputs | User-visible history, safety review, or dispute handling; trigger may be response completion | Can selected metadata replace full content? Are provider copies governed separately? |
| Traces and logs | Specific diagnostic, security, or accountability question; trigger may be event creation | Exclude secrets and unnecessary payloads; control access, exports, backups, and disposal |
| Indexes and memory | Search or continuity purpose; trigger follows source expiry, correction, or relationship end | Propagate permissions and deletion; prevent restoration from stale sources |
| Evaluation and training datasets | Approved evaluation or learning purpose; trigger may be experiment or model lifecycle | Track source dependencies, permitted reuse, versions, and resulting model artifacts |
| Provider application state | Feature operation under supplier terms; trigger and eligibility are feature-specific | Verify settings, exceptions, third parties, and deletion support |
| Backups | Recovery purpose; trigger follows backup generation and rotation policy | Place erased information beyond use while awaiting overwrite and prevent restoration to active processing |
| Model artifacts | Approved deployment and reproducibility purpose; trigger may be replacement or withdrawal | Govern learned influence, access, provenance, and retired checkpoints separately |
Auditability does not require indiscriminate payload retention. A decision record can preserve actor, operation, artifact and version reference, policy result, effect, time, and reviewer while sensitive evidence remains separately access-controlled and expires under its own rule. OWASP advises excluding credentials, tokens, sensitive personal data, and payment information from ordinary logs unless appropriately protected. Observability develops diagnostic signal design; Agent Memory covers retained information across interactions.
Backups require explicit communication. Under ICO erasure guidance, information may be removed from live systems while backup copies await scheduled overwrite, but those copies should remain beyond use and not be restored for another purpose. Exceptions can apply. “Deleted from the active database” and “physically absent from every backup” are therefore different completion claims.
Respond to changed authority
Changed authority is not one operation. Consent withdrawal stops processing that depended on that consent, subject to applicable distinctions and exceptions. Access revocation disables a credential or permission. Restriction limits processing while a dispute or objection is resolved. Rectification addresses inaccurate information. Erasure addresses qualifying records. Each has different triggers, targets, and completion evidence.
Do not collapse unlike requests
| Change | Immediate question | Possible continuing state |
|---|---|---|
| Consent withdrawal | Which purposes relied on this consent, and how is future processing stopped? | Records may remain for another independently established purpose or applicable obligation |
| Credential or access revocation | Which tokens, sessions, API keys, jobs, and downstream grants remain active? | Existing records may remain while future access is blocked |
| Restriction or objection | Which processing must pause while grounds or accuracy are examined? | Storage can continue under limited conditions |
| Rectification | What does the record assert, what evidence bears on accuracy, and who decides? | An accurate history of the former error or dispute can remain alongside correction |
| Erasure | Which active, derived, supplier, and backup copies are covered, and which exceptions apply? | Backup disposal or learned influence may remain unresolved and must be described accurately |
A disputed inference needs careful classification. ICO guidance distinguishes incorrect or misleading factual information from a record clearly identifying an opinion and, where appropriate, its author. That does not make every AI inference an exempt opinion. The organization should examine what the record asserts, the consequences of continued use, the person's evidence, and whether processing should be restricted during review.
Revoking single sign-on may leave independent access paths. In a cross-app access talk, incident response invalidated network and Okta sessions while separately configured MCP tokens and API keys still required inventory and revocation. Token survival depends on each service's behavior, but the example shows why one identity lifecycle cannot be assumed to cover every credential.
Response record
- Requester and authority — How identity and any representative authority were verified proportionately.
- Targets — The records, purposes, credentials, derived artifacts, recipients, and pending operations affected.
- Interim control — Any restriction needed while accuracy, authority, or scope is investigated.
- Decision — Responsible reviewer, evidence, rationale, exceptions, and resulting disposition.
- Propagation and communication — Downstream actions, unresolved states, and the outcome communicated to the affected person.
Propagate correction and deletion
Propagation begins with stable identities and observable change records. A deletion marker, or tombstone, lets asynchronous consumers learn that a source was removed. The system must keep the marker long enough for delayed indexers and failed jobs, prevent stale events from recreating the record, and verify derived cleanup before purging the evidence needed to coordinate it.
Source deletion is not automatically index deletion. Azure AI Search documents that physical blob deletion alone does not remove the corresponding search document. Deletion detection must be configured, and a soft-deletion marker must remain observable long enough for the indexer to process it. Adding a policy after missed deletions cannot reconstruct those events. Stable source and chunk identities are essential, especially when one source produces several derived documents.
Even a graph-aware deletion can leave derived information. Zep documents that deleting an episode removes nodes and edges only under specified association conditions, while shared names and summaries are not regenerated and may retain information from the deleted episode. A successful episode-deletion response therefore establishes a bounded operation, not removal of all derived content.
Deletion has several completion states
ExampleFuture use can be blocked before every active derivative and backup copy has been disposed of.
A durable deletion marker updates current policy before asynchronous cleanup is complete.
Read the diagram as text
- Source record. The active record targeted by an approved deletion decision.
- Search derivative. An index entry linked to the source identity.
- Backup copy. A recovery copy governed by rotation and beyond-use controls.
- Policy service. Blocks future processing and restoration under current state.
- Deletion marker active. Makes the changed decision observable to asynchronous consumers.
- Future use blocked. Reads, jobs, and exports are denied while cleanup continues.
- Active copies removed. The source and linked search derivative are deleted and checked.
- Backup awaiting disposal. The copy is beyond use and remains pending scheduled overwrite.
- Deletion marker active → Policy service: state: deletion required.
- Policy service → Future use blocked: control: deny future use.
- Source record → Active copies removed: disposition: delete active source.
- Search derivative → Active copies removed: disposition: remove linked derivative.
- Backup copy → Backup awaiting disposal: disposition: place beyond use.
- Deletion marker active → Backup awaiting disposal: control: prevent restoration.
- Block future use. A durable deletion marker updates current policy before asynchronous cleanup is complete. Active: Source record, Search derivative, Backup copy, Policy service, Deletion marker active, Future use blocked. New: Source record, Search derivative, Backup copy, Policy service, Deletion marker active, Future use blocked.
- Remove active copies. The source and linked search derivative are deleted and their active absence is verified. The backup still exists. Active: Source record, Search derivative, Backup copy, Policy service, Deletion marker active, Future use blocked, Active copies removed. New: Active copies removed.
- Track backup disposition. The backup is placed beyond use and cannot restore the deleted record, but physical disposal remains pending under the rotation schedule. Active: Source record, Search derivative, Backup copy, Policy service, Deletion marker active, Future use blocked, Active copies removed, Backup awaiting disposal. New: Backup awaiting disposal.
Propagation obligations
- Block future use — Apply current restriction at reads, jobs, exports, and writes before cleanup finishes.
- Stop or fence pending work — Prevent queued ingestion, delayed retries, and long-running jobs from recreating obsolete state.
- Remove or rebuild derivatives — Delete affected index entries, caches, summaries, and evaluation copies, or recompute them from permitted sources.
- Coordinate suppliers — Request deletion or return and track the scoped result and any unresolved exceptions.
- Protect restoration — Keep backup copies beyond use and replay current tombstones or policy state before restored data becomes active.
- Communicate scope — Report what is blocked, removed, awaiting disposal, excepted, or unresolved rather than returning one ambiguous success flag.
Search visibility is also separate from deletion acknowledgement. Some stores expose deletion only after a refresh, while retained replay windows can allow late events to recreate state. Verify the user-visible search result, the underlying active record, each derived copy, and the restoration path independently. The next section treats model parameters separately because they are not deletable rows.
Address learned influence
Training changes learned parameters, so removing a training record does not reverse its contribution. Machine unlearning refers to attempts to remove selected training influence from a learned model. The claim is stronger than deleting the dataset or suppressing a few outputs and needs its own evidence.
Cao and Yang's 2015 work made the distinction explicit: retraining after removing records is conceptually straightforward but can be expensive, while their evaluated alternative reorganized suitable algorithms around stored sums so selected contributions could be subtracted. Their experiments included recommendation, spam, and malware tasks. This does not establish an unlearning method for arbitrary large language models; training mechanics belong in Pretraining and Midtraining.
Different responses make different claims
| Response | What changes | Evidence still needed |
|---|---|---|
| Delete training record | Stored dataset and its ordinary derivatives | Proof of propagation; no claim that learned influence changed |
| Retrain with excluded records | Parameters produced by a new training run | Verified exclusion, baseline and pipeline provenance, evaluation, and deployment replacement |
| Apply an unlearning method | Selected parameter state under the method's assumptions | Method-specific removal and utility tests, threat model, and resistance to misleading verification |
| Replace the model | Deployed artifact and future request path | Approval of the replacement's data, supplier, behavior, and handling conditions |
| Restrict deployment | Who can use the existing artifact and for what purpose | Enforced scope, monitoring, review trigger, and accurate statement of unresolved influence |
Verification is fragile. Research on image-classification datasets demonstrated methods by which a dishonest provider could pass studied unlearning checks while preserving information associated with supposedly removed data. The result is bounded to its attacks and experiments, but it establishes an important governance rule: a passing check supports only the removal property, model family, and adversary conditions assessed.
A model trained on personal information is not automatically anonymous. The European Data Protection Board calls for examining extraction and identification through queries, release context, auxiliary information, and reasonably available means. When complete removal of influence is not established, governance may require retraining, replacement, narrower deployment, or an explicit unresolved limitation rather than a claim that the model has forgotten.
Suppliers and assurance
Approve actual service paths
Supplier approval attaches to an actual path: product, endpoint, feature, model version, settings, deployment region, organizational recipients, contractual role, human-access rules, and additional suppliers. “We use provider X” is not precise enough because different features can have different storage requirements and eligibility for restrictive controls.
A data processing agreement (DPA) defines processing obligations where that relationship applies. A subprocessor is another processor engaged in the processing chain. Review its actual activity, location, safeguards, change-notification terms, audit support, and exit behavior. Contracts allocate obligations but do not prove operational fulfillment.
Separate the questions vendors often compress. “Not used for training” does not mean “not stored”: logs, application state, files, caches, and human review can follow different rules. Data residency concerns storage location, while regional processing concerns computation location. Verify both for the selected feature and deployment path.
Review each request path separately
- Use and storage — Verify training use, feature-required state, retention triggers, deletion support, and documented exceptions.
- Recipients and location — Identify the provider, infrastructure route, support access, subprocessors, and storage and processing regions.
- Exit and change — Define export, deletion or return, replacement behavior, and renewed review when a model, feature, fallback, setting, or supplier changes.
Apple's Private Cloud Compute (PCC) provides a bounded architecture example. Its June 10, 2024 introduction stated requirements to use personal inputs only to fulfill the request, retain none after returning the response, and prevent privileged operators from bypassing those limits. The design removed general-purpose administrative paths and described direct encryption to validated nodes and deletion on completion. These are Apple's architecture assertions, not a universal property of cloud inference or an independent audit.
On June 8, 2026, Apple announced extending PCC onto Google Cloud infrastructure with NVIDIA GPUs while retaining its declared handling requirements and Apple-controlled software trust. The announcement described a preview that would gradually reach the complete protections. It illustrates why an infrastructure or supplier change requires renewed review even when the intended product capability is unchanged; it does not establish that every promised protection or research resource was later completed.
Self-hosting changes responsibility rather than eliminating it. Operators take on more patching, capacity, availability, inference-server, and storage work while remaining responsible for application authorization and appropriate data use. Managed services transfer some operational work under service-specific terms. Deployment choices belong in Local and On-Device AI, while dispatch and fallback policy belongs in Model Routing and LLM Gateways.
Verify and revisit decisions
Governance becomes assurance when each material requirement has an owner, implementation point, assessment method, expected outcome, and response to failure. NIST control-assessment guidance distinguishes examination of specifications and mechanisms, interviews about operation, and tests that exercise controls under stated conditions. Evidence supports only the scope and conditions assessed.
Requirement-to-check matrix
| Requirement | Enforcement and owner | Assessment and expected outcome | What remains separate |
|---|---|---|---|
| Unauthorized recipients cannot retrieve or receive restricted material | Read and disclosure policy; application owner | Negative tests across tenants and destinations are denied and recorded | Supplier internal access and undiscovered bypass paths |
| Permission changes affect later operations | Current authorization check; identity and service owners | Revoke during a long-running task; later disclosure or write is denied | Already authorized copies and external recipients |
| Training reuse is disabled | Dataset export and provider project controls; data owner | Inspect configuration and attempt prohibited export; no training dataset or eligible provider path is created | Provider internal behavior, requiring contractual or supplier evidence |
| Deletion reaches derived state | Tombstone, lineage, index cleanup, supplier workflow; storage owner | Delete a source, verify search absence, summary regeneration, supplier status, and restore behavior | Learned influence and lawful preservation exceptions |
| Record changes require correct authority and target version | Write policy, review gate, and version precondition; record owner | Unauthorized, stale, and wrong-target writes fail without mutation | Truth of an approved new value |
| Telemetry excludes unnecessary sensitive content | Instrumentation and collector rules; observability owner | Inspect emitted records and send seeded sensitive fields; fields are absent or transformed as specified | Unknown fields and downstream retention or access |
Different evidence establishes different facts. An application test can show routing behavior but not a provider's undisclosed internal retention. A contract can establish an obligation but not a tenant's configuration. A configuration screenshot cannot prove runtime enforcement. Execution records help reconstruct effects but may be incomplete or overly sensitive. Supplier attestations support their stated scope rather than universal compliance.
Telemetry is itself governed information. OpenTelemetry advises collecting attributes that serve a diagnostic purpose and using processors to remove fields, filter records, transform values, or enforce allowlists. The OpenLLMetry talk describes placing such preprocessing in a self-deployed collector before export. Installing a collector proves nothing about its rules; seeded tests and inspection of actual exported records are still required.
Reviews should be triggered by new purposes, sources, recipients, models, features, supplier paths, retention terms, legal conditions, incidents, or evidence that a control failed. When a violation occurs, stop or contain affected processing, preserve only necessary evidence with provenance and restricted access, identify affected records and people, correct or delete state as appropriate, communicate through the accountable process, and require renewed approval before resuming. AI Security covers incident mechanics; Evals and Benchmarks covers broader behavioral evaluation.
The resulting discipline is continuous but not vague: decide a specific purpose, inventory the real path, identify actors, establish authority, minimize inputs, preserve dependencies, enforce current policy, set artifact lifetimes, propagate changes, verify scoped outcomes, and reconsider the decision when its assumptions change. Governance succeeds when an approved handling decision remains true in the running system—and when the organization can detect, explain, and repair the cases where it does not.
Open questions
How can organizations verify removal of learned influence at useful scale? Retraining can be expensive, and studied unlearning checks can be fooled under bounded attack models. Progress would require model-appropriate removal definitions, reproducible adversarial verification, and honest deployment decisions when influence remains uncertain.
How should mixed-source permissions compose in real AI applications? Operational metadata systems can apply different semantics to source filtering and access-policy evaluation. Progress would require tested lineage and policy behavior for summaries, inferred facts, exports, and permission changes.
How can privacy-minimal evidence support strong assurance? Complete prompts and tool results aid investigation but create another sensitive dataset; sparse logs can omit the cause of failure. Progress would define question-driven evidence schemas, protected linkage to separately retained source material, and tests showing that investigators can reconstruct consequential decisions without indiscriminate payload capture.
How should governance adapt when supplier infrastructure changes beneath a stable product interface? A model, fallback, region, feature, or subprocessor can change handling boundaries without changing the visible task. Progress would include machine-readable service-path declarations, configuration-linked change notices, and blocking when a path no longer satisfies approved conditions.



























