The short answer
If you would need a reason to give personal information to a contractor, you need the same reason to give it to an AI tool. The use has to be within the purpose the information was collected for. Sending it to a provider is a disclosure, so the provider’s account terms and location matter. You remain responsible for accuracy, for security, and for not keeping it longer than needed, and the tool does not take that responsibility off you. Where an AI system affects a decision about a person, the person should be able to understand and challenge it. None of this stops you using AI. All of it has to be designed in.
Purpose: the question to ask before every new use
Information collected to deliver a service is collected for that purpose. Using it to train a model, to profile customers for marketing they did not agree to, or to answer questions unrelated to why it was collected is a new use, and a new use needs a basis: the original purpose, a directly related one, or the person’s authorisation. The practical test for an AI use case is a sentence: “we collected this to do X; the AI will use it to do Y; Y is X, or directly related, or we asked.” If the sentence cannot be written, the use case needs redesigning.
Disclosure: the provider is a third party
When personal information is pasted into a tool, or sent to a model through an API, it has been disclosed to the provider. Whether that disclosure is permitted depends on why, on what the provider may do with it, and on where it goes. Business accounts that contractually prevent the provider training on your data, that store it in a known location, and that delete it on a schedule are the difference between a permitted disclosure to a service provider and something you cannot explain. If information leaves New Zealand, the Act’s rules on overseas disclosure apply, and the provider’s terms and location need to satisfy them. Personal and free accounts rarely do.
Accuracy: a confident model is not a correct one
You must take reasonable steps to ensure personal information is accurate before using it, and a summary or profile generated by a model is information you are now using. Models produce plausible text; plausible is not accurate. The design answer is a person checking any AI output about an individual before it is relied on or recorded, retrieval from your actual records rather than the model’s memory, and a record of the source so a wrong statement can be traced and corrected. An AI that writes a note into a customer record without review is an accuracy problem waiting for a complaint.
Security: the assistant sees what the account sees
The obligation to protect personal information against loss and unauthorised access does not change; the exposure does. An assistant connected to your files can surface, in one answer, personal information from a folder nobody remembered was shared. An agent acts with the permissions of its account. Reasonable security now means checking permissions before switching on any tool that reads documents, multi-factor authentication on every account an AI system uses, and limiting each system to the information its purpose needs. The breach notification rules apply to AI-caused disclosures exactly as to any other.
Retention: the conversation is a record
Personal information may not be kept longer than the purpose requires. AI adds new copies: the prompt history in the tool, the logs of an agent, the index of a knowledge base, the provider’s retained conversations. Each is a place personal information now lives, and each needs a retention decision. The design should say where copies exist, how long each is kept, and how deletion works, including at the provider. A retention schedule that covers the CRM and forgets the chat history is not a retention schedule.
Access and correction: can you find it?
People are entitled to ask what personal information you hold about them and to request correction. If some of it now sits in an AI system’s logs or index, you need to be able to find it there too. The record-keeping the architecture builds for its own reasons (what the AI saw, produced and did) is also what makes access requests answerable. Design it once, for both.
Automated decisions and fairness
Where an AI system informs or makes a decision that affects a person, the decision has to be explainable and challengeable in practice, and the information used has to be relevant and accurate. The design answer is human review for decisions with a consequence, explanations written in plain language alongside any score, and a record of the factors. Even where the law does not strictly require a person in the loop, a decision you cannot explain is one you cannot defend.
What to do this month
- List every AI tool and system in use, the account type, and what personal information each can reach.
- Move work involving personal information to business accounts with training opt-outs, known storage locations and deletion controls.
- Check permissions before any assistant reads your documents; turn on multi-factor authentication everywhere.
- Write the purpose sentence for each AI use case that touches personal information.
- Add the AI copies to your retention schedule, and confirm how deletion works at the provider.
- Put human review in front of any AI output about an individual that will be relied on or recorded.
Where to go from here
The AI Governance service on this site writes the policy, the purpose assessments, the retention schedule and the review process for your AI use, aligned to ISO/IEC 42001 and the NIST AI RMF, and works alongside your privacy adviser rather than instead of one. The AI Opportunity Score below flags, in five minutes, whether shadow AI, permissions or the absence of a policy is the most urgent gap.
Published 12 September 2026 · Be AI