Streaming Responses

Table of Contents

1. Why Stream at All
2. What a Stream Looks Like on the Wire
3. Reading the Body Chunk by Chunk
4. The Naive Parser and Why It Breaks
5. A Parser That Buffers: sseData
6. chatStream: Tokens In, Text Out
7. Printing Tokens as They Arrive
8. Usage Arrives at the End
9. Stopping a Stream: AbortController
10. Streaming With a Conversation
11. Recipe: The stream Module
12. Try It Yourself
13. Quick Review: Common Questions

1. Why Stream at All

A model writes its answer one token at a time, as you saw in How LLMs Work. A long answer can take many seconds to finish. With the normal API call from Your First API Call, your program waits in silence, and then the full text appears at once. The user stares at a blank screen and wonders if the app is broken.

Streaming asks the server to send each piece as soon as it is written. The user sees the first words almost immediately and reads while the rest is still being generated. This is why chat apps type out the answer.

Think of two restaurants. In one, the cook sends out the whole meal when the last dish is ready. In the other, each dish arrives as soon as it is cooked. The meal takes the same total time, but the second one feels faster, because you are eating from the start.

Two numbers describe this:

• Time to first token: how long until the first piece arrives. Streaming makes this short.
• Total time: how long until the reply is complete. Streaming does not make this shorter. The model writes at the same speed.

So streaming improves how fast the app feels, not how fast it is. It also lets the user press Stop when the answer is going the wrong way, which saves money (section 9). The price is more code: the reply is no longer one JSON object, so you must parse a stream. This chapter builds that parser.

Honest note about the mock. The mock server answers the normal (non-stream) call instantly, because it has no real writing to do. So with the mock you will not see a big difference in "normal" time. In stream mode it deliberately pauses about 40 ms between words, so you can see tokens arrive one by one. Against a real model, the normal call would wait for the entire answer.

2. What a Stream Looks Like on the Wire

To turn streaming on, add "stream": true to the request body. The requestInit() function in src/client.js already does this when you pass stream: true. Try it with curl; the -N flag turns off curl's output buffering so you can watch events arrive:

curl -sN http://localhost:8787/v1/chat/completions \
  -H "authorization: Bearer test-key" \
  -H "content-type: application/json" \
  -d '{"model":"mock-1","stream":true,"messages":[{"role":"user","content":"hello"}]}'
# output
# data: {"id":"mock-13","choices":[{"index":0,"delta":{"role":"assistant"}}]}
#
# data: {"id":"mock-13","choices":[{"index":0,"delta":{"content":"Hello! "}}]}
#
# data: {"id":"mock-13","choices":[{"index":0,"delta":{"content":"How "}}]}
#
# data: {"id":"mock-13","choices":[{"index":0,"delta":{"content":"can "}}]}
#
# ... (more pieces) ...
#
# data: {"id":"mock-13","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{...}}
#
# data: [DONE]

(The output above is shortened: the pieces "I ", "help ", "you " and "today?" are left out, and the usage object is abbreviated. The id mock-13 is a request counter and varies; the words are stable.)

This format is called server-sent events, or SSE. It is a very simple text format:

• The response has the header content-type: text/event-stream, and the connection stays open while events are written.
• Each event is one or more lines. Our lines start with data: followed by a payload, here JSON.
• A blank line (\n\n) ends an event. That blank line is the separator your parser must look for.
• The special payload [DONE] means "the stream is over". It is not JSON, so you must check for it before calling JSON.parse.

Each JSON payload has a delta, the new piece, not the whole text so far. The first event only announces the role. The following ones carry a content string such as "Hello! ". Notice that the pieces include the trailing space, so you can simply glue them together. The last JSON event has an empty delta, a finish_reason of "stop", and the usage numbers for the whole request.

Real providers using this style send the same kind of stream. Details differ (extra fields, keep-alive comment lines starting with a colon), so read your provider's documentation. The ideas stay the same.

3. Reading the Body Chunk by Chunk

When you fetch() a streaming response, res.body is a stream of bytes. In modern Node.js you can read it with for await, which gives you one chunk at a time as it arrives. Let us look at the raw chunks:

// demo-chunks.js
import { baseUrl } from "./src/config.js";
import { requestInit } from "./src/client.js";

const messages = [{ role: "user", content: "hello" }];
const res = await fetch(`${baseUrl}/v1/chat/completions`, requestInit({ messages, stream: true }));
console.log(res.status, res.headers.get("content-type"));

const decoder = new TextDecoder();
let n = 0;
for await (const chunk of res.body) {
  n += 1;
  console.log(`--- chunk ${n} (${chunk.length} bytes)`);
  console.log(decoder.decode(chunk, { stream: true }));
}
// output (shortened: chunks 3 to 7 are left out)
// 200 text/event-stream
// --- chunk 1 (76 bytes)
// data: {"id":"mock-5","choices":[{"index":0,"delta":{"role":"assistant"}}]}
//
// --- chunk 2 (77 bytes)
// data: {"id":"mock-5","choices":[{"index":0,"delta":{"content":"Hello! "}}]}
//
// ...
// --- chunk 8 (239 bytes)
// data: {"id":"mock-5","choices":[{"index":0,"delta":{"content":"today?"}}]}
//
// data: {"id":"mock-5","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"usage":{"prompt_tokens":2,"completion_tokens":8,"total_tokens":10}}
//
// data: [DONE]

The chunk sizes vary, and so do the ids and the number of chunks. Here the first chunks carry one event each, but the last chunk (239 bytes) carries three events at once, because the final writes happened close together and the network delivered them together.

That is the key lesson: chunk boundaries are not event boundaries. A chunk can contain:

• exactly one event,
• several events glued together,
• half of an event,
• or even half of a character (Khmer, Chinese and emoji take several bytes each, and a chunk can end in the middle of one).

TCP, the protocol under HTTP, delivers a stream of bytes. It makes no promise about where the pieces are cut. On localhost the cuts often line up nicely, which is dangerous: your code works on your laptop and breaks in production.

Also notice the decoder. A TextDecoder turns bytes into text. With the option { stream: true } it remembers an unfinished character at the end of one chunk and completes it when the next chunk arrives. Without it, a half character becomes the replacement symbol and your Khmer text turns into garbage.

4. The Naive Parser and Why It Breaks

Here is the parser most people write first: for each chunk, split into lines, and parse every data: line.

To show the failure on demand, we use a simulated network: the same SSE text, cut into chunks at awkward places. (A real network can produce the same cuts, but not on request, so we fake it for a reliable demo.)

// demo-naive.js
// A SIMULATED network: the same SSE text, cut into chunks at awkward places.
const sse =
  'data: {"choices":[{"delta":{"content":"Hello! "}}]}\n\n' +
  'data: {"choices":[{"delta":{"content":"World"}}]}\n\n' +
  "data: [DONE]\n\n";
const enc = new TextEncoder();
const cuts = [0, 60, 95, sse.length]; // not aligned with event boundaries
async function* fakeBody() {
  for (let i = 0; i < cuts.length - 1; i++) yield enc.encode(sse.slice(cuts[i], cuts[i + 1]));
}

// NAIVE parser: treat every chunk as if it held whole events.
const decoder = new TextDecoder();
let text = "";
for await (const chunk of fakeBody()) {
  for (const line of decoder.decode(chunk).split("\n")) {
    if (!line.startsWith("data:")) continue;
    const data = line.slice(5).trim();
    if (data === "[DONE]") continue;
    try {
      text += JSON.parse(data).choices[0].delta.content;
    } catch (err) {
      console.log("broken piece:", JSON.stringify(data));
    }
  }
}
console.log("text so far:", JSON.stringify(text));
// output
// broken piece: "{"
// text so far: "Hello! "

The expected text is "Hello! World". We got "Hello! ". The second word is lost. Why?

• Chunk 1 holds the whole first event plus the first 7 characters of the second event (data: {). The naive code parses the first event correctly, then chokes on the lonely {.
• Chunk 2 holds the rest of the second event, "choices":[.... This line does not start with data:, so the code silently skips it. The word "World" disappears.

The worst part is that nothing crashes. You get a reply with random missing words, and only sometimes. Such bugs are painful, because they depend on network timing. The fix is a standard one.

5. A Parser That Buffers: sseData

The fix: do not parse chunks. Collect text in a buffer, and cut out only complete events. An event is complete when the buffer contains the blank line \n\n. Whatever is left after the last \n\n stays in the buffer, waiting for the next chunk to finish it.

Create src/stream.js with the parser first:

// src/stream.js (part 1)

// Turn a stream of byte chunks into complete SSE "data:" payloads.
export async function* sseData(body) {
  const decoder = new TextDecoder();
  let buffer = "";
  for await (const chunk of body) {
    buffer += decoder.decode(chunk, { stream: true });
    let end;
    while ((end = buffer.indexOf("\n\n")) !== -1) {
      const event = buffer.slice(0, end);
      buffer = buffer.slice(end + 2);
      for (const line of event.split("\n")) {
        if (line.startsWith("data:")) yield line.slice(5).trim();
      }
    }
  }
}

The function name has a *: it is an async generator. Instead of returning one value, it yields many values over time, and the caller reads them with for await. Here is what each part does:

1. buffer += decoder.decode(chunk, { stream: true }) adds the new text to what is left over from before.
2. while ((end = buffer.indexOf("\n\n")) !== -1) finds each blank line. There may be zero, one, or several events in the buffer.
3. buffer.slice(0, end) is one complete event; the rest goes back into buffer.
4. Lines that start with data: give a payload that we yield. Other lines, such as comments that start with a colon, are ignored.

Test it with the same awkward cuts that broke the naive parser, and with a Khmer word cut in the middle of a character:

// demo-sse.js
import { sseData } from "./src/stream.js";

const sse =
  'data: {"choices":[{"delta":{"content":"Hello! "}}]}\n\n' +
  'data: {"choices":[{"delta":{"content":"World"}}]}\n\n' +
  "data: [DONE]\n\n";
const enc = new TextEncoder();
const cuts = [0, 60, 95, sse.length]; // the same awkward cuts as before
async function* fakeBody() {
  for (let i = 0; i < cuts.length - 1; i++) yield enc.encode(sse.slice(cuts[i], cuts[i + 1]));
}

let text = "";
for await (const data of sseData(fakeBody())) {
  if (data === "[DONE]") break;
  text += JSON.parse(data).choices[0].delta.content;
}
console.log(JSON.stringify(text));

// Harder: cut in the middle of a multi-byte character (Khmer text).
const khmer = 'data: {"t":"សួស្តី"}\n\n';
const bytes = enc.encode(khmer);
async function* splitBytes() {
  yield bytes.slice(0, 14); // ends inside a character
  yield bytes.slice(14);
}
for await (const data of sseData(splitBytes())) console.log(JSON.parse(data).t);
// output
// "Hello! World"
// សួស្តី

Both outputs are stable (no network involved). The full text is back, and the Khmer word survived being cut in the middle of a character, thanks to { stream: true }.

Limits of this small parser:

• It looks for \n\n. The SSE standard also allows \r\n\r\n line endings. Providers in this style normally use \n, but if you meet one that uses \r\n, normalize the buffer first with buffer.replace(/\r\n/g, "\n").
• An event with several data: lines is supposed to be joined into one payload with newlines. We yield each line separately, which is fine for JSON-per-line streams like these.
• It ignores event: and id: lines. Our API does not use them.

6. chatStream: Tokens In, Text Out

Now combine the pieces. chatStream(options, onToken) sends the request with streaming on, parses events with sseData, calls onToken(piece) for every new piece of text, and returns the full text and the usage.

Add this to src/stream.js, with two imports at the very top of the file:

// src/stream.js (part 2)
import { baseUrl } from "./config.js";
import { requestInit, readError } from "./client.js";

// ... sseData from part 1 goes here ...

export async function chatStream(options, onToken = () => {}) {
  const res = await fetch(`${baseUrl}/v1/chat/completions`, requestInit({ ...options, stream: true }));
  if (!res.ok) throw await readError(res);
  let text = "";
  let usage = null;
  for await (const data of sseData(res.body)) {
    if (data === "[DONE]") break;
    const event = JSON.parse(data);
    const piece = event.choices?.[0]?.delta?.content;
    if (piece) { text += piece; onToken(piece); }
    if (event.usage) usage = event.usage;
  }
  return { text, usage };
}

Walk through it:

• It reuses requestInit and readError from client.js. The headers, key and JSON body are built in one place. If the server replies with an error status (401, 400, 500), we get the same ApiError as the normal call, before any streaming starts. Errors arrive as ordinary JSON responses, not as events.
• onToken has a default of a function that does nothing, so you can call chatStream() without a callback and just wait for the result.
• event.choices?.[0]?.delta?.content uses optional chaining. The first event (the role announcement) and the last (finish and usage) have no content, so we skip them with if (piece).
• text += piece collects the whole answer. We need the full text to store in the conversation history afterwards.
• if (data === "[DONE]") break ends the loop. Leaving a for await loop early also closes the response body for you.

7. Printing Tokens as They Arrive

console.log always adds a newline, which would put every word on its own line. To print on the same line, use process.stdout.write. It prints exactly what you give it, with no newline.

// demo-stream.js
import { chatStream } from "./src/stream.js";
import { Conversation } from "./src/conversation.js";

const convo = new Conversation().user("Hello there");
process.stdout.write("Bot: ");
const { text, usage } = await chatStream({ messages: convo.messages() }, (piece) => {
  process.stdout.write(piece);
});
process.stdout.write("\n");
console.log("full text:", JSON.stringify(text));
console.log("usage:", usage);
// output
// Bot: Hello! How can I help you today?
// full text: "Hello! How can I help you today?"
// usage: { prompt_tokens: 10, completion_tokens: 8, total_tokens: 18 }

In a terminal you would see the words appear one after another, roughly 40 ms apart. A book cannot show animation, but the printed result is stable: the same text and the same usage every time, however the network cuts the chunks. Only the timing changes.

Let us also measure that timing, to see the "first token" effect. We record when each piece arrives:

// demo-timeline.js
import { chatStream } from "./src/stream.js";

const t0 = performance.now();
const pieces = [];
await chatStream(
  { messages: [{ role: "user", content: "Tell me about Phnom Penh" }] },
  (piece) => pieces.push({ ms: Math.round(performance.now() - t0), piece }),
);
console.log("pieces:", pieces.length);
console.log("first piece after", pieces[0].ms, "ms:", JSON.stringify(pieces[0].piece));
console.log("last piece after ", pieces.at(-1).ms, "ms:", JSON.stringify(pieces.at(-1).piece));
// output (the milliseconds VARY on every run; the words are stable)
// pieces: 7
// first piece after 62 ms: "Phnom "
// last piece after  310 ms: "Cambodia."

The first word arrived at about 60 ms, while the whole answer needed about 310 ms. A non-streaming call would show nothing until the 310 ms mark. With a real model, where long answers take many seconds, that gap is the difference between "it feels alive" and "is it frozen?".

8. Usage Arrives at the End

Why does chatStream return usage? The model only knows how many tokens it wrote after it finishes, so the numbers come in the last JSON event, the one with finish_reason: "stop". You saw it in the raw chunks in section 3 and in the demo above: { prompt_tokens: 10, completion_tokens: 8, total_tokens: 18 }.

This matters in three ways:

• You can track cost even when streaming. Add the numbers to a running tally, as chapter 5 does.
• If the stream is cut short, usage can be null. If the user presses Stop or the connection drops, the final event never arrives. The function above then throws instead of returning (section 9), so keep your own count of the pieces you received.
• Some providers only include usage if you ask for it with an extra request option. Our mock always sends it. Check your provider's documentation, and if usage is null, do not crash. Treat it as "unknown".

9. Stopping a Stream: AbortController

A user may press Stop because the answer is wrong, too long, or boring. A good app really stops the request, instead of hiding the text while the model keeps writing (and billing) in the background.

JavaScript's tool for this is the AbortController. You create one, give its signal to fetch, and call controller.abort() when you want to cancel. Our requestInit already passes signal through to fetch, so chatStream supports it without any change:

// demo-abort.js
import { chatStream } from "./src/stream.js";

const controller = new AbortController();
let partial = "";
let count = 0;

try {
  await chatStream(
    {
      messages: [{ role: "user", content: "hello" }],
      signal: controller.signal,
    },
    (piece) => {
      partial += piece;
      count += 1;
      if (count === 3) controller.abort(); // the user pressed Stop
    },
  );
  console.log("finished normally");
} catch (err) {
  if (err.name !== "AbortError") throw err;
  console.log("stopped by the user");
  console.log("partial text:", JSON.stringify(partial));
}
// output
// stopped by the user
// partial text: "Hello! How can "

We aborted after the third piece. Output is stable here, because we stop by counting pieces, not by time. What happens inside:

1. controller.abort() tells fetch to give up. It closes the connection, so the server learns that nobody is listening.
2. The for await loop inside chatStream stops with an error whose name is "AbortError".
3. Our catch block recognizes that name and treats it as a normal outcome. Any other error is re-thrown, so real bugs are not hidden.

Notice that chatStream throws and does not return the partial text. That is why the demo collected partial itself inside the callback. In a real chat app, you decide what to do with the half answer. Showing it and keeping it in the history (perhaps with a "(stopped)" note) is common. Throwing it away is also fine. Either way, do not store a user message without any assistant reply, or the next request will have two user messages in a row.

Aborting is also how you build timeouts. AbortSignal.timeout(ms) creates a signal that aborts by itself:

// demo-timeout.js
import { chatStream } from "./src/stream.js";

try {
  await chatStream({
    messages: [{ role: "user", content: "Tell me about Phnom Penh" }],
    signal: AbortSignal.timeout(100), // give up after 100 ms
  });
} catch (err) {
  console.log(err.name, "-", err.message);
}
// output
// TimeoutError - The operation was aborted due to timeout

Note the error name: a timeout gives TimeoutError, while a manual abort() gives AbortError. In a UI you may want to show "stopped" for one and "the server took too long" for the other. Chapter 7 uses this for retries and slow servers.

If the server rejects the request, the failure happens before the first event, so you get the usual ApiError (for example status 500 for the model mock-error, or 401 for a wrong key). But a stream can also fail halfway, for example if Wi-Fi drops. Then for await throws a network error after you already printed some words. Your UI should keep the text it has and show a clear message such as "connection lost, try again".

10. Streaming With a Conversation

Streaming and memory fit together. The only rule: store the full reply in the history after the stream ends, not each piece.

// demo-stream-chat.js
import { chatStream } from "./src/stream.js";
import { Conversation } from "./src/conversation.js";

const convo = new Conversation();

async function say(text) {
  convo.user(text);
  process.stdout.write(`You: ${text}\nBot: `);
  const { text: reply, usage } = await chatStream({ messages: convo.messages() }, (p) =>
    process.stdout.write(p),
  );
  convo.assistant(reply); // store the FULL reply once the stream ends
  process.stdout.write(`\n    [${usage.prompt_tokens} in, ${usage.completion_tokens} out]\n`);
}

await say("My name is Dara");
await say("What is my name?");
// output
// You: My name is Dara
// Bot: You said: My name is Dara
//     [11 in, 7 out]
// You: What is my name?
// Bot: Your name is Dara.
//     [22 in, 5 out]

The second request sends the earlier turns, so its input count (22) is larger than the first (11), just as in Conversation and Memory. Everything printed here is stable.

This is the core of the command-line chatbot in the next chapter: a Conversation, a chatStream call that prints tokens with process.stdout.write, and a usage tally.

11. Recipe: The stream Module

Here is the full final src/stream.js. It imports from config.js and client.js, and has no other dependencies:

// src/stream.js
import { baseUrl } from "./config.js";
import { requestInit, readError } from "./client.js";

// Turn a stream of byte chunks into complete SSE "data:" payloads.
export async function* sseData(body) {
  const decoder = new TextDecoder();
  let buffer = "";
  for await (const chunk of body) {
    buffer += decoder.decode(chunk, { stream: true });
    let end;
    while ((end = buffer.indexOf("\n\n")) !== -1) {
      const event = buffer.slice(0, end);
      buffer = buffer.slice(end + 2);
      for (const line of event.split("\n")) {
        if (line.startsWith("data:")) yield line.slice(5).trim();
      }
    }
  }
}

export async function chatStream(options, onToken = () => {}) {
  const res = await fetch(`${baseUrl}/v1/chat/completions`, requestInit({ ...options, stream: true }));
  if (!res.ok) throw await readError(res);
  let text = "";
  let usage = null;
  for await (const data of sseData(res.body)) {
    if (data === "[DONE]") break;
    const event = JSON.parse(data);
    const piece = event.choices?.[0]?.delta?.content;
    if (piece) { text += piece; onToken(piece); }
    if (event.usage) usage = event.usage;
  }
  return { text, usage };
}

Options for chatStream are the same as for chat(): messages, model, maxTokens, temperature and signal. usage is null if the server never sent it.

sseData is separate from chatStream on purpose. It knows nothing about chat. It only turns bytes into events, so you can test it with a fake body (as we did) and reuse it with any server that speaks SSE.

12. Try It Yourself

1. Watch the raw stream. Run the curl -N command from section 2 with a different message, for example "what is my name?". Count how many data: events you get. Which events carry no text, and what do they carry instead?
2. Break it on purpose. Change cuts in demo-sse.js to cut the text at every single byte (cuts from 0 to sse.length, step 1). Does sseData still return "Hello! World"? Then try the naive parser with the same cuts. What does it print?
3. Handle CRLF. Write a variant sseDataCRLF that works when the server separates events with \r\n\r\n. Test it with a hand-made fake body.
4. A typing speed meter. Using onToken, count the pieces per second and print "N pieces in M ms" at the end. Why would a real model give a different number from the mock?
5. Stop on a word. Abort the stream as soon as the text received so far contains the word "help". Print the partial text. What does the usage look like after an abort, and why?
6. Keep the partial. In demo-stream-chat.js, abort the second answer after two pieces and still store what you received as an assistant message ending in " (stopped)". Print convo.messages() and check that the roles still alternate.

13. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You now have the three core pieces of a chat program: a client, a conversation, and a streaming reader. In the next chapter, A Command-Line Chatbot, you put them together into a real interactive program with commands like /reset and /usage, and you learn to test it by piping text into it.