Streaming LLM guardrails: you can't unsay a token

Streaming LLM guardrails: a token can't be unsent. How NeMo, Guardrails AI and Bedrock trade speed for safety, and how a sentence gate checks each claim.

2026-09-17

A token an LLM has streamed cannot be recalled. Once text reaches the reader, a guardrail that objects to it can only stop what comes next. So a guardrail on a stream makes one real decision: check before release and make the reader wait, or release first and accept that a blocked answer has already been read. There are three workable shapes: check after release, gate fixed chunks, or gate whole sentences. The assistant on this site, which I built, gates sentences. The cost is one sentence of waiting. The limit is that it checks whether a claim is supported, not whether it is true.

This article covers the last check before a grounded answer is released. The layers before it, from refusing out-of-scope questions to a retrieval-score floor, are in RAG hallucination guardrails: a five-layer design.

Why you can't unsay a streamed token

A post-hoc validator sees the full answer only after the reader has read it. If the third sentence invents a date, deleting it afterwards changes the record, not what was read. A recent arXiv preprint by Christopher M. Frost states the boundary exactly: "a control that detects a prohibited pattern after releasing its completing chunk cannot recall it".

Vendors document the same limit. Azure OpenAI's opt-in asynchronous filter returns text immediately and guarantees its violation signal only within about 1,000 characters of the violating content. Its troubleshooting advice is to "implement content redaction in your application to remove already-displayed content". OWASP's LLM09:2025 entry recommends tools and processes "to automatically validate key outputs". The open question is when validation runs relative to release.

Three ways to guard a stream: post-hoc, chunk-gated, sentence-gated

Approach	Reader sees text	A blocked answer	Examples
Post-hoc (release first)	Before its check (NeMo: each chunk once buffered; Azure and Bedrock async: as generated)	Partly read before the stop	NeMo with `stream_first` on (its default); Azure and Bedrock asynchronous modes
Chunk-gated	After each chunk passes	Blocked chunk never shown	Azure and Bedrock defaults; NeMo with `stream_first` off
Sentence-gated	After each sentence passes	Blocked sentence never shown	This site's assistant

What NeMo Guardrails' stream_first trades off

Output-rail streaming is off by default in NeMo Guardrails. Once enabled, it runs the output rails, NeMo's checks on generated text, on chunks of 200 tokens by default, carrying 50 tokens of context into the next chunk. The stream_first flag defaults to True: "The client receives each chunk of tokens before output rails process that chunk. This provides faster time-to-first-token, but if a rail blocks the content, the user has already received the tokens." The stream then ends with a JSON error.

With False, "the user never sees blocked content, but time-to-first-token increases by the rail execution time per chunk." That is the trade: time to first token (TTFT) against blocked text reaching the reader. Once output-rail streaming is on, NeMo releases first by default; the two cloud filters below check first. If a rail exists to stop something that must never be seen, change the default.

What Guardrails AI's sentence validators trade off

Guardrails AI chooses the sentence: "By default, validators wait until they have accumulated a sentence's worth of content from the LLM before running validation." Validating single sentences rather than the whole response "helps save on costs for validators that require expensive inference".

Not all on_fail types are supported with streaming, and fixes from several validators wait "for all validators to accumulate enough chunks" before they are merged. The docs do not say plainly whether raw tokens reach the user before validation, so check that in your own integration.

Buffered by default: Azure OpenAI and Amazon Bedrock

Azure OpenAI buffers completion text by default, filters the buffer, and returns "content chunks" rather than single tokens. Amazon Bedrock Guardrails scans chunks before sending them in its default synchronous mode, which "introduces some latency" but "provides better accuracy". In Bedrock's asynchronous mode, chunks "may contain inappropriate content until guardrails scan completes", and sensitive information cannot be masked.

How a sentence gate validates a stream

The assistant on this site answers only from the site's own content, and a question that is not about me and that no passage supports never reaches the model. It speaks about me in the third person. On a site that carries my name, the worst failure is an invented credential, and a credential is a claim. A sentence is the smallest unit that carries one, so the gate buffers to a sentence boundary, checks, and only then releases. The lab's RAG vs raw demo calls the same streaming endpoint.

Timeline of a sentence gate: tokens collect in a buffer until a sentence ends; the first sentence passes four checks and is shown without its marker; the second contains a value missing from the sources, so it is withheld and the stream stops.

Where a sentence ends while tokens are still arriving

Terminators: . ! ? …, plus the Arabic-script question mark ؟ and full stop ۔, and, for Chinese and Hindi, 。 ！ ？, the danda । and the double danda ॥. A run such as ?! is one boundary. The Arabic-script semicolon ؛ joins clauses, so it is left out on purpose.
Whitespace must follow. That one rule keeps 3.14 and example.com whole. The exception is the Chinese full stop, exclamation mark and question mark, which are followed by no space. Inside a quotation they end the sentence where the quotation closes, and not at all when a comma follows it.
Not every full stop ends a sentence. One followed by a lowercase letter does not (et al. and), and neither does the full stop of an abbreviation on a short list, such as e.g. or the German bzw., or the one after the day in a German date (am 17. September).
The next character decides. A boundary is final only when a character arrives that is neither a space nor a citation marker, because the markers after a full stop belong to the sentence before it: all of them, [S1][S2] or [S1] [S2], not only the first.
A list marker after a boundary is not a sentence. Otherwise a 1. opening the answer or a new sentence would be judged as an uncited claim and stop a grounded answer at its first bullet.
A cap. Past a thousand characters without a boundary, the buffer is cut at the last space and checked anyway.
A blank line survives. A sentence that followed a blank line is released after one, so a two-paragraph answer stays two paragraphs.

Waiting for the next character usually costs one character of lag, a little more when markers follow the full stop. The last sentence is checked when the stream ends. Comparisons fold Persian digits and separators (۳٫۱۴ matches 3.14) and Arabic letterforms, drop Arabic short-vowel marks, and treat the zero-width non-joiner as a space: the same variants that hybrid search for Persian RAG handles on the retrieval side.

The four deterministic checks: citations, values and voice

Citation range. A marker such as [S7] when six sources were supplied names an invented source.
Citation required. A sentence of eight words or more, or with any number, URL or email address, needs a marker. Short connective sentences pass.
Verbatim support. Every number, URL and email address must appear in a source the sentence cites. The match respects digit boundaries, so a year in a source cannot vouch for a percentage that shares its last digits, and decimals that differ only in trailing zeros are one value: 20.00 is 20. A comma that does not group thousands is a decimal comma: 2,5 is 2.5, not 25.
Voice. A first-person word stops the stream. The English list is short and fixed: first-person pronouns and their contractions. The Persian list is curated: pronouns plus a few high-risk verb forms, such as the Persian for "I am" and "I have". Persian marks person in verb endings that ordinary nouns can share, so a rule broad enough to catch every ending would catch nouns too. The lists for the site's other languages hold pronouns and possessives, and the Spanish, Portuguese, Russian, Arabic and Hindi lists add a few verbs, because those languages usually say "I" with the verb alone; Chinese is checked for 我 anywhere in the sentence except in a compound of 自我, "self-", such as 自我修正 ("self-correction"). Which lists apply depends on the language the answer was requested in. English, Persian, Arabic, Russian, Hindi and Chinese apply to every answer. English is the language of the model's instructions, so it is the one an answer is most likely to drift into; the other five use scripts that no Latin name or acronym can spell. The exception is an Arabic answer, which is not checked for the Persian "I" and "we", because in Arabic the same words mean "from" and "what". The German, French, Spanish and Portuguese lists apply only to answers requested in that language: their short words also spell names and other languages' ordinary words, like the "Ma" of "Ma et al.". Acronyms in capitals, a surname before "et al." and the numeral in "Annex I" are set aside before any list is read.

What happens when a sentence fails

Every sentence that passed before the failure is released, including earlier ones in the same batch.
The gate stops and stays stopped: one unsupported sentence discredits the rest.
A guard event carries the violation reasons and fixed, localised copy saying the rest was withheld, and the final done event records the stop. The blocked sentence is never sent; the server logs only its length.
The provider stream is closed, which is how the design cancels the upstream request. What was generated before the stop is billed all the same, so it is recorded against the monthly spend cap. Both provider protocols report output usage in a final event that a stream cut short never reaches, so the service keeps the text it received and fills in whatever the provider left unreported at one token per UTF-8 byte, a count that errs high. Tokens a reasoning model spends thinking never arrive as text, so that estimate leaves them out. The five-layer article covers the cap.

The chat panel marks the turn as withheld and keeps its released text in the conversation history, because that text is supported up to the stop.

Truncation is a different failure, and an earlier version of this assistant confused the two. When the model ran out of output room, the gate judged the cut-off fragment as a finished sentence and withheld it as uncited. The reader was told that an answer which had only been interrupted could not be supported by the sources. When no text came back at all, the turn vanished without a message. Now the unfinished fragment is dropped unjudged and a cut-off notice is shown. An interrupted sentence is not an unsupported one. If both happen, the guard stop wins.

Citation markers are internal

The prompt asks for a [Sn] marker on every sentence that states a fact, and the gate enforces the citation rule on every substantive sentence. Once a sentence passes, its markers are stripped and the spacing repaired, including before Persian punctuation, so readers see prose, not markers. Dropping markers from the prompt instead would fail every substantive sentence on the citation rule.

When the model has nothing to say

The prompt asks the model for a fixed token, not a sentence, when the sources do not cover the question. A refusal written by the model would be a claim like any other: an uncited one, which the gate would stop with a notice that blames the sources for a question they were never meant to answer.

The gate never releases the token. While the text so far could still become the token, it is held; the first character that rules the token out sends the text down the usual path, so an ordinary answer waits for nothing extra. When an answer opens with the token, the service replaces it with the site's fixed refusal, the same copy any other refusal shows. When the token follows sentences already released, those sentences stand and the answer ends there.

Checking "supported" is not checking "true"

In a Stanford-led study, Magesh and colleagues ran what they describe as the first preregistered evaluation of AI legal research tools. The AI research tools from LexisNexis and Thomson Reuters each hallucinated between 17% and 33% of the time. An answer counted as hallucinated if it was "either incorrect or misgrounded", and misgrounded means "key factual propositions are cited but misinterpret the source or reference an inapplicable source". Misgrounded answers carry citations. What that means for legal AI products is covered in a separate article.

A string gate cannot see misgrounding, and this one has known blind spots:

A sentence that cites two sources can take a value from either. The check is per sentence, not per clause.
A decimal point counts as a boundary. 14.52 in a source can vouch for 52% or 14, and a thousands separator does the same: 1,200 vouches for 200.
Words are not checked. A number-free sentence that swaps a role or an organisation passes with any in-range marker.
Short uncited sentences pass. A sentence under eight words with no number, URL or address needs no marker. Chinese puts no spaces between words, so its characters are counted instead, three to every two words: a sentence of Chinese characters alone needs no marker below twelve.
Fixed word lists have gaps. In English, we'd and let's pass, and so does a pronoun written all in capitals, such as WE or US, which is read as an acronym. Persian first-person coverage leans on the prompt: a first-person verb that is not on the curated list passes. The other languages' lists are curated too, and German, French, Spanish or Portuguese first person in an answer requested in another language is left to the prompt.
Closing punctuation can merge sentences. After a quote that closes on a terminator, as in ?» or .", only a capital letter or a line break ends the sentence; after a bracket, as in .), only a line break or a citation marker does. So two sentences can be judged as one after .) before a capital, after ؟» in Arabic, Persian and Hindi, which have no capital letters, and after the German „…?“ and »…?«, which the splitter does not read as closing quotes at all. The Chinese marks keep their own rule, above.

The error also runs the other way. An abbreviation that is not on the list, followed by a capital, ends a "sentence" early, as Mr. does before a name, and so does a quoted question followed by a name, or, in German, by any noun. If that fragment has eight words or more and no marker, a grounded answer is withheld. A list introduced by a line ending in a colon has no boundary before its first marker. The splitter cuts right after 1., and that fragment fails the citation rule. In both cases the gate fails closed: it withholds a grounded answer rather than release an unchecked one.

The corpus moves too. Published articles are indexed for the assistant, so their figures, this article's 17% and 33% included, become values the gate will vouch for. Supported means "somewhere in what the sentence cites", not "about this".

The gate is the last check before release, not the only one. Joren and colleagues found that larger models "often output incorrect answers instead of abstaining" when the context is not sufficient, and Google Research notes that while RAG generally improves overall performance, it "paradoxically reduces the model's ability to abstain from answering when appropriate". So refusal and a score floor come first, as the five-layer design sets out. Frost's preprint states the limit of its own result in words that fit this gate as well: "policy conformance for a deterministic backstop, not evidence of semantic safety or policy completeness".

SSE for LLM streaming: fetch, proxies and cancellation

A sentence gate only helps if each released sentence reaches the reader at once. Four transport details decide that.

Why fetch and ReadableStream, not EventSource

The WHATWG specification gives the EventSource constructor a URL and one option, withCredentials: no method and no body. A question with its locale and history is a body, so the endpoint is a POST that returns Server-Sent Events. The browser reads it with fetch, a ReadableStream reader and an AbortController, whose abort(), in MDN's words, "is able to abort fetch requests, consumption of any response bodies, and streams".

A hand-written parser needs these rules:

Buffer across reads. A chunk is not a message.
Dispatch on a blank line. Until one arrives, the event is incomplete.
Ignore comment lines, the ones that start with :.
Join multi-line data: all data: lines in one event join with a line feed.
Accept CRLF, LF and CR as line endings.
Strip one space after the colon, and only one.

Decode with TextDecoder and {stream: true}, the setting MDN gives "if processing the data in chunks". Otherwise a Persian character split across two chunks decodes as replacement characters.

Proxy buffering: nginx and Caddy

nginx buffers proxied responses by default, and an X-Accel-Buffering response header can switch that off. The assistant's stream sends X-Accel-Buffering: no and Cache-Control: no-cache, no-store, no-transform; no-transform is there because a proxy that recompresses the body can coalesce chunks. Caddy flushes text/event-stream responses immediately, and the site's Caddy configuration also sets flush_interval -1 on the chat route. The service worker registers no route for the stream, because a caching strategy would hold the response until it ended.

Keep-alives and errors inside an open stream

The SSE specification suggests a comment line "every 15 seconds or so" to protect against legacy proxies that drop connections after a short timeout, and the server pings on that interval. The status is already 200 when the first byte leaves, so failures travel as an error event inside the stream.

Cancellation that also stops the bill

Stopping in the browser aborts the fetch. The server checks for a disconnect as each token arrives. When it finds one, it breaks out of the loop and closes the provider stream, which in this design cancels the upstream request. Two things deserve a test rather than an assumption. First, that every proxy and CDN between the browser and your API passes a client disconnect through. Test that path; do not trust the configuration. Second, when your provider reports usage: if output usage arrives only with the final chunk, a stream you close early never receives it, and a spend cap that counts reported usage will under-count what was generated.

How to test a streaming guardrail

The tests cover the gate at three levels:

Gate unit tests on fixed sources. Fabricated values, out-of-range markers, uncited claims and first-person words (in each of the site's languages) stop the stream; supported values and short connective sentences pass; earlier good sentences survive a later bad one.
Splitter tests. Decimals and domains are not boundaries, trailing markers stay with their sentence, and ؟ ends a sentence. Chinese text splits into sentences without spaces. The one worth copying: text fed in three-character pieces must split exactly like the whole text.
Stream tests with a scripted provider replaying an answer in small pieces: a fabricated value never reaches the wire, a guard stop or a disconnect closes the upstream stream, and truncation is reported as truncation.

A sketch of the two central tests, written for this article rather than taken from the site:

# Sketch only: SentenceSplitter and SentenceGate stand for your own classes.
import pytest

ANSWER = "The reading room opens early [S1]. Is the café open? Yes, on weekdays [S2]."

def pieces(text, size):
    return [text[i:i + size] for i in range(0, len(text), size)]

def split_streamed(text, size):
    splitter = SentenceSplitter()
    sentences = [s for piece in pieces(text, size) for s in splitter.feed(piece)]
    tail = splitter.flush()
    return sentences + ([tail] if tail else [])

@pytest.mark.parametrize("size", [1, 2, 3, 7])
def test_piece_size_does_not_change_sentences(size):
    assert split_streamed(ANSWER, size) == split_streamed(ANSWER, len(ANSWER))

def test_unsupported_value_never_reaches_the_reader():
    gate = SentenceGate(sources=["The library opens at nine."])
    released = ""
    for piece in pieces("It opens at nine [S1]. Late fees ended in 2019 [S1]. More", 7):
        outcome = gate.feed(piece)
        released += outcome.text
        if outcome.stopped:
            break
    assert "2019" not in released and gate.stopped

None of this red-teams a live model. What I have measured is the gate's own mistakes: answers recorded on production and replayed offline, to find where it stopped a grounded answer. That counts false stops for one model and one set of questions, not how often a real model fabricates, so I am not publishing it as a rate.

Checklist: shipping a guardrail on a streamed answer

Decide what must never be seen. That sets check-before or release-first, whatever your framework's default.
Pick the unit and name the wait: a fixed chunk or a sentence.
Use deterministic checks where strings suffice, folding digits and letterforms first. A model-based rail adds its execution time to every unit.
Call a boundary certain only when the next character arrives, and cap the buffer.
On a violation, release what passed, stop, show fixed copy, never echo the blocked text, and close the upstream stream.
Keep truncation separate from violation, and count what was generated, not what was shown.
Transport: POST with fetch, a stream reader and AbortController; a streaming decoder; comment pings; errors as events.
Proxies: X-Accel-Buffering: no, no-transform, event-stream flushing, a tested disconnect path, and no service worker on the stream.
Tests: replay answers in small pieces and assert that a blocked value never reaches the wire.

If you are designing a grounded assistant and want a second pair of eyes on its release path, the diagnostics page is where that conversation starts.

FAQ

Can you validate streamed LLM output without giving up streaming?

Yes. Buffer to a unit you can check, and release each unit as it passes. The reader sees text arrive sentence by sentence or chunk by chunk, rather than token by token.

Does buffering to a sentence hurt time to first token?

It delays visible output by one sentence: text appears once the first sentence is complete and checked. String checks are cheap next to generation, but I have not measured the difference on this site, so I give no number.

Should I turn off stream_first in NeMo Guardrails?

Turn it off if a blocked chunk must never be seen. NeMo documents the cost: time to first token grows by the rail execution time per chunk. Keep the default only if your product can accept a reader seeing tokens that a rail later blocks.

Why not use EventSource for LLM streaming?

EventSource takes a URL and a credentials flag, so it cannot send a request body. A chat request with history is a POST, so use fetch with a stream reader, which also gives you cancellation through AbortController.

Does a citation check stop hallucinations?

No. It stops invented source numbers, uncited substantive sentences and values missing from the sources a sentence cites. It cannot tell whether a cited source supports the claim, which is the misgrounded failure. Treat it as a backstop behind refusal and a retrieval floor.