Introduction
Which files can an AI assistant access when summarising internal documents? Can a system analysing a security alert disable a user account on its own? Security in AI systems requires answering these questions about data access and action permissions during design.
Alongside model accuracy, organisations need to consider the information a system uses, how it processes external content and what it can do through connected systems. This article focuses on generative AI, large language models (LLMs), enterprise knowledge assistants and agents that can use tools.
My recommendation is to define acceptable error boundaries and security responsibilities alongside the project's business value. The NIST AI Risk Management Framework provides a framework for addressing risk throughout design, development, use and evaluation.
Main Section
1. Define the purpose and decision boundaries
Summarising a document and approving a financial transaction have different consequences. Security design should reflect the system's task and the business impact of an incorrect output.
Before the project begins, document the answers to these questions:
- What will the system do, and which tasks are outside its scope?
- Which data sources will it access?
- Will it only provide recommendations, or can it take action?
- What would be the impact of a wrong decision?
- Who owns the system and accepts its risks?
For example, a system classifying help desk tickets and an agent changing production configurations cannot be assessed against identical success criteria. The latter requires particular attention to the consequences of an incorrect action and the ability to intervene.
2. Preserve the user's data access boundaries
Retrieval-augmented generation (RAG), commonly used in enterprise knowledge assistants, retrieves relevant material from an external knowledge source to support the model's response. Search and document access in this architecture should be constrained by the user's permissions.
OWASP's guidance on vector and embedding weaknesses addresses access and data separation issues in this area. A document that an employee cannot open directly should not become accessible indirectly through an assistant's response.
For example, an employee asking a general policy question should not receive confidential contract details belonging to another department. Authorisation should be enforced before content is sent to the model.
For external model services, examine where information is processed, how long it is retained and whether it is used for training. Hosting a model internally still leaves the organisation responsible for access controls, logging and maintenance.
3. Account for external content influencing model behaviour
A web page, document or email can contain instructions intended to divert a model from its task. Attempts to redirect the model through external content are known as indirect prompt injection.
OWASP notes that RAG and fine-tuning do not fully eliminate this risk. Writing “follow the rules” in a system prompt therefore does not establish a sufficient security boundary.
The application should treat external content as untrusted data and enforce data access and action permissions independently of the model's interpretation. Input filters and protective layers can help, but do not provide a complete protection guarantee on their own.
4. Limit an agent's authority to its task
For an agent that uses tools, the available actions should be explicitly defined. OWASP's excessive agency guidance recommends limiting unnecessary functionality and permissions, with human approval for high-impact actions.
Consider a SOC assistant. It may investigate a suspicious session and provide a recommendation to an analyst. Granting it permission to isolate a production server requires additional design around action scope, approval, reversal and revocation of authority.
An example starting model could work as follows:
- The assistant can read incident information and prepare a recommendation.
- An authorised analyst can review the target and proposed action.
- Approval is bound to a specific action, target and validity period.
- Backend systems check authorisation again when the action is executed.
The model saying “this action has been approved” should not count as valid authorisation evidence for the application.
5. Include the use of outputs in the security design
A fluent response does not guarantee accuracy. Additional checks are needed when model output becomes a command, query or automated decision.
For example, before applying a firewall rule suggested by an assistant, validate its source, destination, service and business justification. Test its effect on existing access.
For generated code and queries, retain application security controls such as input validation, parameterised queries and secure execution environments. These risk areas are also addressed in the OWASP LLM Top 10.
6. Track the model and data supply chain
Models, datasets, libraries and connectors are dependencies. OWASP's supply chain guidance recommends assessing the security of these components.
An organisation should know which model and component versions are in production. A model file of uncertain origin or unverified data should not enter production without controlled review.
Permission to add documents to a knowledge base also matters. Changes should have identifiable owners and a traceable history, with deliberate corruption of trusted information considered as a risk.
7. Test before release and monitor afterwards
Include boundary cases alongside normal usage in the test plan:
- What happens when a user requests a document outside their permissions?
- How does the system respond when an external document tries to redirect its task?
- Can a person detect and stop an incorrect recommendation?
- Do time and cost limits work when an agent makes repeated calls?
- Do existing controls remain effective after the model or knowledge source changes?
In production, model versions, accessed sources, tool calls and approval decisions should be traceable. Since logs may themselves contain sensitive information, establish access and retention rules for them.
Prepare in advance how automation will be stopped, permissions revoked and a previous version restored following a security incident or an unacceptable error.
Conclusion
Security in AI systems requires designing data access, model behaviour, connected tools' permissions and the human role in decision-making together. An organisation should be able to explain which actions the system is technically prevented from taking, alongside its capabilities.
To begin, select one use case and document four elements: the data accessed, permitted actions, decisions requiring human approval and the method for stopping the system. Then verify those boundaries in a test environment.
This provides a concrete starting point for connecting the expected benefits of AI with risks the organisation can accept.