Conversation and Memory

Table of Contents

1. Prove the API Forgets
2. Memory Is an Array You Resend
3. A Conversation Class
4. Using It in a Loop
5. The System Prompt: Give the Bot a Persona
6. Watching the Cost Grow
7. Trimming Old Turns
8. What Trimming Loses
9. Saving and Loading History
10. Recipe: The conversation Module
11. Try It Yourself
12. Quick Review: Common Questions

1. Prove the API Forgets

In Your First API Call you built src/client.js, with a chat() function that sends messages and returns the reply. Every call you made was a single question and a single answer. Real chatbots hold a whole conversation. How?

First, let us prove the surprising fact, which you may already know from Context Windows and Memory: the API remembers nothing between calls. Start the mock server in one terminal (node mock-llm.js) and use another terminal for the demos. If the mock is on a different port, set LLM_BASE_URL first.

// demo-memory.js
import { chat } from "./src/client.js";

// Call 1: tell the model our name.
const first = await chat({
  messages: [{ role: "user", content: "Hi, my name is Dara." }],
});
console.log("1:", first.text);

// Call 2: a brand-new request. We send ONLY the new question.
const second = await chat({
  messages: [{ role: "user", content: "What is my name?" }],
});
console.log("2:", second.text);
// output
// 1: Hello! How can I help you today?
// 2: I do not know your name yet. You have not told me.

Two calls, one after the other, to the same server, with the same key. The second call has no idea what the first one said. Each request is a separate world.

Two notes on the output. Both lines are stable, because the mock has no randomness. And the first reply is only a greeting because the mock is a canned stand-in: it sees the word "Hi" and answers with a greeting. It is not intelligent. A real model would reply more usefully, and it would also forget your name in call 2, for the same reason.

Honest reminder. The mock server only pretends to be a model. It recalls a name by searching your earlier messages for "my name is X". That is enough to show how memory works, which is the point of this chapter. The HTTP shape and the code you write are the same for a real provider.

2. Memory Is an Array You Resend

If the server remembers nothing, who remembers? Your code does. The trick is simple: keep every message in an array, and send the whole array each time. Think of a teacher who has forgotten the whole class over the weekend. Every Monday you hand them the full diary of what happened so far, and they carry on as if they had been there.

Here it is with a plain array and no helper classes:

// demo-history.js
import { chat } from "./src/client.js";

const history = [{ role: "system", content: "You are a helpful assistant." }];

async function say(text) {
  history.push({ role: "user", content: text });
  const { text: reply } = await chat({ messages: history });
  history.push({ role: "assistant", content: reply });
  console.log("You:", text);
  console.log("Bot:", reply);
}

await say("Hi, my name is Dara.");
await say("What is my name?");
console.log("messages in history:", history.length);
// output
// You: Hi, my name is Dara.
// Bot: Hello! How can I help you today?
// You: What is my name?
// Bot: Your name is Dara.
// messages in history: 5

Same question as before, but now the answer is right. The only change: the second call carried the earlier messages with it. After two turns the array holds five messages: one system message, two user messages, and two assistant replies.

The order matters. The array is a transcript, oldest first, and the roles alternate: user, assistant, user, assistant. The system message sits at the very start. This is exactly the messages field you met in What an LLM API Is.

There are two easy mistakes here:

• Forgetting to push the assistant reply. Then the next request has your questions but none of the answers, and the model gets confused about what it already said.
• Pushing your user message and then crashing on the call. Now the history ends with a user message that never got an answer. When you retry, you must not push the same message again.

Managing this array by hand in every program gets messy. So we wrap it.

3. A Conversation Class

A class is a small box that holds data and the functions that work on it. Our Conversation box holds the system prompt and the turns, and gives us four verbs: user(), assistant(), messages() and reset(). There is a fifth, trim(), which we use in section 7.

Create src/conversation.js:

// src/conversation.js
export class Conversation {
  constructor(system = "You are a helpful assistant.") {
    this.system = system;
    this.turns = []; // { role: "user" | "assistant", content }
  }
  user(content) { this.turns.push({ role: "user", content }); return this; }
  assistant(content) { this.turns.push({ role: "assistant", content }); return this; }
  messages() { return [{ role: "system", content: this.system }, ...this.turns]; }
  // Keep only the newest `maxTurns` messages (the system prompt is always kept).
  trim(maxTurns) {
    if (this.turns.length > maxTurns) this.turns = this.turns.slice(-maxTurns);
    return this;
  }
  reset() { this.turns = []; return this; }
}

Some design choices worth noticing:

• The system prompt is stored apart from the turns. messages() builds the final array by putting the system message first and the turns after it. That way trim() can never delete the system prompt by accident.
• user() and assistant() return this. That lets you chain calls: convo.user("hi").assistant("hello").user("what now?").
• messages() returns a new array each time. You can pass it straight to chat(). Changing that array later will not change the conversation, though the message objects inside are shared.
• The class does not call the API. It only manages data. That keeps it easy to test and to reuse with streaming in the next chapter, and with the command-line bot after that.

4. Using It in a Loop

Now the chat loop is short. Add the user message, call the API, store the reply. We also print the token usage that comes back with each response, for a reason you will see in section 6.

// demo-convo.js
import { chat } from "./src/client.js";
import { Conversation } from "./src/conversation.js";

const convo = new Conversation();

async function say(text) {
  convo.user(text);
  const { text: reply, usage } = await chat({ messages: convo.messages() });
  convo.assistant(reply);
  console.log(`You: ${text}`);
  console.log(`Bot: ${reply}`);
  console.log(`     prompt_tokens=${usage.prompt_tokens} completion_tokens=${usage.completion_tokens}`);
}

await say("Hello there");
await say("My name is Sokha");
await say("What is my name?");
await say("What is the capital? Phnom Penh, right?");
console.log("messages sent next time:", convo.messages().length);
// output
// You: Hello there
// Bot: Hello! How can I help you today?
//      prompt_tokens=10 completion_tokens=8
// You: My name is Sokha
// Bot: You said: My name is Sokha
//      prompt_tokens=23 completion_tokens=7
// You: What is my name?
// Bot: Your name is Sokha.
//      prompt_tokens=34 completion_tokens=5
// You: What is the capital? Phnom Penh, right?
// Bot: Phnom Penh is the capital of Cambodia.
//      prompt_tokens=49 completion_tokens=10
// messages sent next time: 9

Everything here is stable: the mock is deterministic, and its token counts are a simple estimate (about one token per four characters). Real providers count with their own tokenizer, so real numbers will differ, but they will follow the same pattern.

Look at the second reply, "You said: My name is Sokha". The mock only has a few canned replies, and that is the fallback. A real model would say something friendly. Do not worry about the wording; watch the mechanics. By turn 3 the mock found "my name is Sokha" in the history and answered correctly. By the last turn we are about to send nine messages: the system message and four full exchanges.

5. The System Prompt: Give the Bot a Persona

The system message is the instruction from you, the developer, that sits before the conversation. It sets the bot's role, tone and rules: "You are a polite librarian", "Answer in two sentences", "Never give medical advice". The user does not see it, but it travels in the messages array with every request.

Because Conversation takes the system prompt in its constructor, a persona costs one line. The mock has exactly one persona trick: if the system prompt contains the word "pirate", it starts every reply with "Arr!".

// demo-persona.js
import { chat } from "./src/client.js";
import { Conversation } from "./src/conversation.js";

for (const system of ["You are a helpful assistant.", "You are a pirate. Stay in character."]) {
  const convo = new Conversation(system).user("Tell me about Phnom Penh");
  const { text } = await chat({ messages: convo.messages() });
  console.log(`[${system}]\n  ${text}`);
}
// output
// [You are a helpful assistant.]
//   Phnom Penh is the capital of Cambodia.
// [You are a pirate. Stay in character.]
//   Arr! Phnom Penh is the capital of Cambodia.

Same user question, different system prompt, different answer. A real model follows a persona much more fully: it would change its vocabulary and style, not only add one word. The mechanism is identical. For more on writing good system prompts, and on why they are not a security boundary, see Prompts, Roles and Structured Output.

A few practical rules:

• Keep the system prompt short and specific. You pay for it on every single request, because it is resent every time (next section).
• Do not put secrets in it. A user can often trick a model into repeating its instructions. Never place passwords, keys or private business rules there.
• Treat it as a request, not a lock. The model usually follows it, but it is not guaranteed. Check outputs that matter.
• Change it between chats, not mid-chat. You can set convo.system at any time, but old replies were written under the old rules, so the history may look inconsistent.

6. Watching the Cost Grow

Each request carries the entire history. So the input you send, and usually pay for, grows with every turn. The usage.prompt_tokens field makes this visible. Let us send six similar messages and keep a running total:

// demo-cost.js
import { chat } from "./src/client.js";
import { Conversation } from "./src/conversation.js";

const convo = new Conversation();
let total = 0;
for (let i = 1; i <= 6; i++) {
  convo.user(`Message number ${i}, please answer politely.`);
  const { text, usage } = await chat({ messages: convo.messages() });
  convo.assistant(text);
  total += usage.prompt_tokens;
  console.log(`turn ${i}: prompt_tokens=${usage.prompt_tokens}  running total=${total}`);
}
// output
// turn 1: prompt_tokens=18  running total=18
// turn 2: prompt_tokens=41  running total=59
// turn 3: prompt_tokens=65  running total=124
// turn 4: prompt_tokens=88  running total=212
// turn 5: prompt_tokens=112  running total=324
// turn 6: prompt_tokens=135  running total=459

Each turn adds roughly 23 tokens (one question plus one reply), but the total you have been billed for grows much faster. At turn 1 you sent 18 tokens. At turn 6 you sent 135 tokens in one request, and 459 over the whole chat. If the numbers were ten times larger, it would still be the same shape: the cost of a long chat grows roughly with the square of its length, because every new turn pays again for all the old ones.

That is why long chats get expensive, and why a good app limits how much history it sends. Chapter 7 turns this into a budget you can check in code. For the theory, see the cost section of Context Windows and Memory.

The context window also has a hard limit. When the history no longer fits, a real API returns an error (often status 400) instead of an answer. You must shorten the history before that happens.

7. Trimming Old Turns

The simplest fix is to keep only the newest messages. That is what trim(maxTurns) does: if there are more than maxTurns turns, drop the oldest ones. The system prompt is stored separately, so it is always kept.

// demo-trim.js
import { chat } from "./src/client.js";
import { Conversation } from "./src/conversation.js";

const convo = new Conversation();
convo.user("My name is Dara").assistant("Nice to meet you.");
for (let i = 1; i <= 4; i++) convo.user(`filler ${i}`).assistant(`You said: filler ${i}`);
console.log("before trim:", convo.turns.length, "turns");

convo.trim(4);
console.log("after trim: ", convo.turns.length, "turns");
console.log("first kept: ", convo.turns[0]);

convo.user("What is my name?");
const { text } = await chat({ messages: convo.messages() });
console.log("Bot:", text);
// output
// before trim: 10 turns
// after trim:  4 turns
// first kept:  { role: 'user', content: 'filler 3' }
// Bot: I do not know your name yet. You have not told me.

The history shrank from 10 turns to 4, which saves money and avoids the window limit. But look at the last line: the bot forgot the name. That is the price of trimming, covered next.

There is also a sharp edge in trim(). It counts messages, not pairs, so with an even maxTurns it can cut in the middle of an exchange and leave an assistant message at the front. Look:

// demo-trim-edge.js
import { Conversation } from "./src/conversation.js";

const c = new Conversation();
for (let i = 1; i <= 3; i++) c.user(`q${i}`).assistant(`a${i}`);
c.user("q4");
c.trim(4);
console.log(c.turns.map((t) => `${t.role}:${t.content}`));
// output
// [ 'assistant:a2', 'user:q3', 'assistant:a3', 'user:q4' ]

The history now starts with an answer to a question that is gone. Some providers accept this and some complain. The simplest habit is to drop one more message whenever the first turn is an assistant, or to trim whole pairs (see the budget helper below). Exercise 3 asks you to fix it properly.

Counting messages is a crude measure, because a message can be 3 tokens or 3,000. A better rule drops the oldest pair until the estimated size fits a budget. Here is a small helper you can keep beside your demos (it is not part of Conversation, to keep that class tiny):

// demo-budget.js
import { Conversation } from "./src/conversation.js";

const estimate = (text) => Math.ceil(text.length / 4);
const size = (convo) =>
  convo.messages().reduce((sum, m) => sum + estimate(m.content) + 4, 0);

// Drop the oldest user+assistant PAIR until the chat fits the budget.
function trimToBudget(convo, maxTokens) {
  while (size(convo) > maxTokens && convo.turns.length > 1) {
    convo.turns.splice(0, 2);
  }
  return convo;
}

const convo = new Conversation("Be brief.");
for (let i = 1; i <= 6; i++) convo.user(`question number ${i}`).assistant(`You said: question number ${i}`);
console.log("before:", convo.turns.length, "turns, ~" + size(convo) + " tokens");
trimToBudget(convo, 60);
console.log("after: ", convo.turns.length, "turns, ~" + size(convo) + " tokens");
console.log("starts with:", convo.turns[0].role, "-", convo.turns[0].content);
// output
// before: 12 turns, ~127 tokens
// after:  4 turns, ~47 tokens
// starts with: user - question number 5

Removing two at a time keeps the history starting on a user turn, because the turns always alternate. The + 4 is a made-up allowance for the wrapper each message has (role and markers), and / 4 is the usual rough estimate of four characters per token. Both are estimates: a real tokenizer will give different counts, and the real window limit is measured in the provider's tokens. Leave a safety margin.

8. What Trimming Loses

Trimming makes the bot forget. In section 7 the name "Dara" lived in one of the oldest messages, so it fell off the end. Nothing is wrong with the code; this is the trade you chose. Here are the common ways to cope, from simplest to most advanced:

• Pin important facts in the system prompt. If the app knows the user's name, preferred language or account type, put it in the system message ("The user's name is Dara"), which trim() never touches. Our mock is too simple to read the system message, so it cannot show this, but a real model uses it.
• Keep the first message too. The first user message often states the task. Some apps keep it and drop the middle.
• Summarize old turns. Ask the model to compress the old part into a short note, then keep the note plus recent turns. This costs one extra call and loses detail, but keeps the gist.
• Store facts outside the chat. Save things like names and preferences in a database, and add them to the prompt when needed. This is how "memory" features in big products tend to work.
• Start a new chat. A new topic often deserves a clean history. It is cheaper and gives sharper answers. A /reset command is a good idea. Our reset() method does exactly that.

There is no perfect answer. Pick what fits your app, and tell your users honestly that the bot has limited memory.

9. Saving and Loading History

If the program ends, the array in memory is gone. For a command-line bot or a small project, saving the history to a JSON file is the simplest persistence there is. We write a tiny script that loads the history if a file exists, answers one question, and saves again:

// save-load.js
import { readFile, writeFile } from "node:fs/promises";
import { chat } from "./src/client.js";
import { Conversation } from "./src/conversation.js";

const FILE = "history.json";

async function load() {
  const convo = new Conversation();
  try {
    const saved = JSON.parse(await readFile(FILE, "utf8"));
    convo.system = saved.system;
    convo.turns = saved.turns;
  } catch (err) {
    if (err.code !== "ENOENT") throw err; // no file yet is fine
  }
  return convo;
}

async function save(convo) {
  const data = { system: convo.system, turns: convo.turns };
  await writeFile(FILE, JSON.stringify(data, null, 2));
}

const convo = await load();
console.log("loaded turns:", convo.turns.length);

const question = process.argv[2] ?? "What is my name?";
convo.user(question);
const { text } = await chat({ messages: convo.messages() });
convo.assistant(text);
console.log("Bot:", text);

await save(convo);
console.log("saved turns:", convo.turns.length);

Run it three times. Each run is a separate process, so the only memory between runs is the file:

node save-load.js "My name is Chenda"
node save-load.js
node save-load.js "what is my name?"
# output
# loaded turns: 0
# Bot: You said: My name is Chenda
# saved turns: 2
# loaded turns: 2
# Bot: Your name is Chenda.
# saved turns: 4
# loaded turns: 4
# Bot: Your name is Chenda.
# saved turns: 6

Each run starts with more turns than the last. The second run used the default question, "What is my name?", and the third asked it again, now in lowercase. Open history.json and you will see the system prompt and the list of turns as plain text:

{
  "system": "You are a helpful assistant.",
  "turns": [
    {
      "role": "user",
      "content": "My name is Chenda"
    },
    {
      "role": "assistant",
      "content": "You said: My name is Chenda"
    }
  ]
}

(That is the file after the first run; later runs add more turns.)

Some cautions about files:

• A chat history can hold private data. Names, phone numbers and whatever else a user typed. Do not commit history.json to Git. Add it to .gitignore.
• A file is for one user. A web app with many users needs a database, with one conversation per user, not one shared file.
• A crash while writing can corrupt the file. For serious use, write to a temporary file and rename it, or use a database.
• Old chats still cost tokens when you resend them. Saving is not a reason to keep an unlimited history. Trim after loading.

10. Recipe: The conversation Module

Your toolkit now has one new file next to config.js, client.js. Here is the full final src/conversation.js:

// src/conversation.js
export class Conversation {
  constructor(system = "You are a helpful assistant.") {
    this.system = system;
    this.turns = []; // { role: "user" | "assistant", content }
  }
  user(content) { this.turns.push({ role: "user", content }); return this; }
  assistant(content) { this.turns.push({ role: "assistant", content }); return this; }
  messages() { return [{ role: "system", content: this.system }, ...this.turns]; }
  // Keep only the newest `maxTurns` messages (the system prompt is always kept).
  trim(maxTurns) {
    if (this.turns.length > maxTurns) this.turns = this.turns.slice(-maxTurns);
    return this;
  }
  reset() { this.turns = []; return this; }
}

The file has no imports and no dependencies. The save and load code from section 9 stays in your demo script on purpose: where and how to store history depends on your app. A command-line bot saves to a file, a web app uses a database, and a test saves nothing.

This module did not change config.js or client.js, and nothing in the HTTP layer needed to know about conversations. That is a good sign that the pieces are separated well.

11. Try It Yourself

1. Prove it twice. Write a script that calls chat() with the history ["My name is Dara", "What is my name?"] as two separate calls, then as one call with both messages in the array. Explain in one sentence what the server saw each time.
2. Persona switch. Create a conversation with the system prompt "You are a pirate." Ask a question, then set convo.system = "You are a helpful assistant." and ask again. Look at the full messages() array. Why do the old replies still say "Arr!"?
3. Fix the leading assistant. Change your own copy of trim (or write trimSafe in a demo file) so that after trimming, if the first turn is from assistant, it is dropped too. Test it with the q1..q4 example from section 7.
4. Count the bill. Using the pattern from section 6, run a 12-turn chat and print the running total of prompt_tokens. Then call trim(4) after every turn and run it again. How much smaller is the total?
5. Pin a fact. Write a remember(convo) helper that scans the user turns for "my name is X", and rebuilds the system prompt as "You are a helpful assistant. The user's name is X.". Then trim the history hard. Does the mock still answer? Explain why or why not (hint: what does the mock read?).
6. Reset command. Extend save-load.js so that the question /reset clears the history file instead of sending a message. What should happen to the system prompt?

12. Quick Review: Common Questions

Questions beginners often ask about this chapter. Try answering each one yourself first, then click the question to open the answer.

Where to go next. You can now hold a conversation, but you only see the reply after the model has finished writing all of it. For long answers that feels slow. The next chapter, Streaming Responses, shows how to receive and print the reply word by word as it is generated, and how to stop it midway.