People often focus on whether an AI output is safe to trust. The mirror-image question is whether the input was safe to share.
Prompts, uploaded documents, pasted code and connected mailboxes can contain personal, confidential or commercially sensitive information. Treating an AI tool like a disposable search box can therefore create data risk even when nothing is technically “hacked”.
The workplace lesson
Samsung restricted employee use of ChatGPT and similar tools in 2023 after staff pasted confidential source code into them. The problem was not a breach of the company network; it was employees sending sensitive material to a tool that had not been approved for that use. Forbes and CNBC.
Product terms differ
Consumer chat products, business tiers and APIs can have different training, retention and human-review policies even when they come from the same company. OpenAI, for example, publishes separate data controls for its API, including retention and training arrangements. OpenAI API data controls.
Deletion is not always symmetrical with disclosure
European regulatory guidance and research have raised the possibility that personal data used in training can remain represented in model parameters and may be extractable in some circumstances. See the Paul Weiss summary of EDPB and ICO guidance, Google Research and related research.
Connected assistants create another risk: prompt injection
When an AI assistant can read web pages, documents, emails or connected services, instructions hidden in that content can influence its behaviour. OWASP treats prompt injection as a major risk because language models cannot always reliably separate instructions from data. OWASP GenAI — Prompt Injection and OWASP AI Agent Security Cheat Sheet.
What to do with this
- Classify information before you paste or upload it.
- Use approved tools for personal, confidential, health, financial, client or NDA-protected information.
- Check the exact product and subscription tier you are using.
- Minimise what you share: remove names and unnecessary identifiers.
- Use excerpts instead of whole files when possible.
- Scope connected-app permissions carefully.
- Treat unexpected instructions originating from external content as suspicious.
Rule to remember: disclosure is a decision. Make it deliberately.
