An LLM response may look like ordinary text, but applications increasingly use that text as an instruction. A generated value can become HTML in a browser, a query sent to a database, an argument passed to an API, a file path, an email template, or a command executed by an automation tool. The moment the response crosses that boundary, a language problem becomes an application security problem.

OWASP describes this risk as LLM05:2025 Improper Output Handling. It occurs when model output is passed to downstream components without sufficient validation, sanitization, encoding, or control. The potential consequences include cross site scripting, server side request forgery, privilege escalation, unintended data changes, and remote code execution.

The central rule is straightforward: treat the model as an untrusted source. A system prompt, safety policy, or structured response format can improve consistency, but none of them converts probabilistic output into trusted code. Deterministic controls must decide whether a value is permitted in the exact context where the application intends to use it.

Applications often connect models to useful capabilities. A support assistant may generate a refund request. A data assistant may create a query. A coding assistant may propose a shell command. A browser agent may create a URL or render Markdown. These connections create value because the model can translate natural language into structured actions.

The same translation creates risk. A user controlled prompt can influence the model response, and the application may then execute that response with privileges the user does not possess directly. In effect, the model becomes an indirect path to a database, browser, file system, internal service, or administrative function. If the next component trusts the response, the model can accidentally or deliberately expand the user’s reach.

Indirect content can have the same effect. A model may read a webpage, document, email, ticket, or retrieved passage that contains manipulative instructions. Even when the user is not malicious, the generated output can carry attacker controlled data toward a downstream component. Output handling therefore needs to assume that both the user prompt and external context may influence the result.

Browser rendering is one important boundary. Generated HTML, Markdown, links, or scripts can create cross site scripting, unsafe redirects, or data exfiltration when rendered without context appropriate encoding. The application should not assume that a model will consistently refuse to produce dangerous markup. It should encode or sanitize the output according to the browser context.

Database and backend execution create a larger blast radius. Generated SQL should not be executed as a raw string, and generated file paths should not be accepted without canonicalization and allow lists. Shell commands, code fragments, expressions, and configuration values need strict controls. A model response should never move directly into functions such as execution or evaluation without a constrained interpreter and explicit validation.

Tool calls create another failure mode. A syntactically valid JSON object may still request an unauthorized action, excessive amount, unexpected destination, or destructive parameter. Schema validation confirms shape, not business permission. The application must separately enforce authorization, limits, ownership, state, and policy before any tool performs the requested operation.

The safest output contract is narrow. Instead of asking the model to produce arbitrary code or prose that another component interprets, the application should define a small set of allowed actions and fields. Enumerations, length limits, value ranges, destination allow lists, and required identifiers reduce ambiguity and make deterministic validation practical.

Structured output is useful when it is combined with enforcement. The application can reject unknown fields, additional properties, invalid types, unsupported actions, and values outside approved limits. It should also bind important values on the server. For example, the authenticated account identifier should come from the session, not from a model generated argument.

Context appropriate handling remains necessary after schema validation. Text intended for HTML needs encoding. Database values should use parameterized queries. URLs require scheme, host, path, and redirect checks. File paths need canonicalization and directory restrictions. Each sink has different security requirements, so one generic output filter cannot protect every destination.

A model can recommend an action without receiving authority to perform it. This separation is especially valuable for refunds, account recovery, profile changes, financial actions, access grants, data deletion, or external communication. The interface can show the proposed action, supporting evidence, and expected effect before a person or deterministic policy approves it.

Human review should be designed for understanding rather than ceremony. Reviewers need to see what the model read, what it proposed, which account and resources are affected, and why the action is considered safe. A confirmation button that hides the destination or impact creates the appearance of control without giving the reviewer enough information to make a decision.

Lower risk actions can use policy approval rather than manual review. The application may permit a small, reversible action when the session is trusted, the amount is within limits, and the request matches expected behavior. Higher risk or unusual actions can be paused. This approach preserves useful automation while keeping authority in deterministic systems.

Output validation answers whether the proposed value is safe for a destination. It does not answer whether the person or automation requesting the action should be trusted. A perfectly valid tool call can still be abusive when it originates from a compromised account, scripted campaign, or manipulated customer journey.

Runtime signals can change the enforcement decision. An account requesting recovery from a familiar device with consistent behavior may receive a different path than the same account arriving through unfamiliar infrastructure and immediately attempting sensitive changes. The action schema remains identical, but the surrounding evidence changes the risk.

CrossClassify can provide this external context through device, network, behavioral, automation, and account signals. The CrossClassify AI agent security solution helps applications classify activity around AI enabled actions. It does not validate generated code or replace secure programming controls, but it can support risk based decisions before sensitive actions proceed.

Teams should log the original request, relevant external context, model version, generated output, validation results, authorization decision, tool arguments, action result, and reviewer identity when applicable. Sensitive values should be redacted or protected within those logs. The objective is to reconstruct what happened without creating another source of exposure.

Testing should include malicious prompts, manipulated documents, unexpected encodings, extra JSON fields, boundary values, unsupported tools, revoked permissions, and output intended for different sinks. A response that is safe as plain text may be dangerous as HTML or SQL. Tests must follow the data into the component that finally interprets it.

Security ownership should also be clear. Model teams can define expected formats and evaluation sets. Application teams must validate and encode output. Platform teams control tool credentials and network reach. Fraud and security teams monitor abuse patterns. No group should assume the model provider has secured the complete downstream workflow.

Improper output handling is not primarily a question of whether the model gives a correct answer. It is a question of what the application does with that answer. Generated content becomes dangerous when it is interpreted by a browser, database, operating system, API, or business workflow without the controls normally applied to untrusted input.

The practical response is to narrow the output contract, validate structure and meaning, encode for the destination, parameterize database operations, restrict tools, separate recommendation from execution, and require approval for consequential actions. These controls should be deterministic and should remain effective even when the model behaves unexpectedly.

Runtime trust adds the final context. The application should evaluate not only whether the output is technically valid, but also whether the account, device, behavior, and session surrounding the action appear trustworthy. This layered design lets organizations use LLM automation without giving generated text an unchecked path into business systems.