Streaming LLM responses in Next.js without the jank

A chat interface that waits eight seconds and then dumps a wall of text feels broken, even when the answer is good. Streaming fixes the perception problem, but a naive implementation brings new ones: dropped chunks, runaway requests after the user navigates away, and a UI that re-renders on every token.

Here is the setup I reach for in the App Router.

#Stream from a route handler

A route handler can return a ReadableStream directly. The key detail is forwarding the request's abort signal, so closing the tab also stops the model call you are paying for.

// app/api/chat/route.ts
export async function POST(req: Request) {
  const { messages } = await req.json();

  const upstream = await fetch("https://api.example.com/v1/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages, stream: true }),
    signal: req.signal, // cancel upstream when the client disconnects
  });

  return new Response(upstream.body, {
    headers: { "Content-Type": "text/event-stream", "Cache-Control": "no-cache" },
  });
}

If you need to transform the provider's events into plain text, pipe the body through a TransformStream instead of buffering it.

#Read the stream on the client

On the client, read chunks with a reader and decode them as they arrive. Keep an AbortController in a ref so a "Stop" button and unmounting both cancel the request.

const controller = new AbortController();
const res = await fetch("/api/chat", {
  method: "POST",
  body: JSON.stringify({ messages }),
  signal: controller.signal,
});

const reader = res.body!.getReader();
const decoder = new TextDecoder();
let text = "";

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  text += decoder.decode(value, { stream: true });
  scheduleRender(text);
}

Note { stream: true } on the decoder. Without it, a multi-byte character split across two chunks turns into garbage.

#Batch renders to the frame

Setting React state on every chunk can mean hundreds of renders per second on a fast model. Batch updates to the next animation frame instead:

let pending = "";
let frame = 0;

function scheduleRender(next: string) {
  pending = next;
  if (frame) return;
  frame = requestAnimationFrame(() => {
    frame = 0;
    setAnswer(pending);
  });
}

The text still appears to flow continuously, and the main thread stays free for scrolling and typing.

#Details that make it feel finished

• Keep the scroll pinned to the bottom only while the user is already at the bottom. If they scroll up to reread, stop following.
• Render Markdown incrementally but tolerate unclosed code fences mid-stream; close them virtually until the real fence arrives.
• Show a real stop state. When cancelled, keep the partial answer and mark it as stopped rather than erasing it.
• Handle errors in-line. A dropped connection mid-answer should say so under the partial text, with a retry that resends the same messages.

Streaming is mostly plumbing, but it is the plumbing users feel most directly. Get cancellation and render batching right and the rest of the interface gets to shine.