# Cost and latency

> What decides the tokens and the response time of each turn, the options that reduce them and what each costs in model calls, and how to measure your app.

- URL: https://docs.mentoragent.net/configure/cost-and-latency
- Verified with: MentorAgent 1.2.0
- Last updated: 2026-11-08
- Packages: Novior.MentorAgent

Use this page when you plan an agent with many actions, or when it already costs more or answers more slowly than you expected. It explains what decides the tokens and the time of a turn, and which options reduce them. For each option, it says what it costs in extra model calls. Then it shows how to read the numbers and how to measure your own app.

> **Note:**
> These pages give no measured token counts yet. A measurement with 10, 30 and 50 actions, with and without tool filtering, is planned. Until it is published here, the effect of each option is described by what it sends and by the model calls it adds. To get numbers for your app, [measure it](https://docs.mentoragent.net/configure/cost-and-latency#measure-your-app): your actions, instructions and model decide them.

## Where the tokens and the time go

Each turn is one or more calls to the model. A turn that answers from the conversation is one call. A turn that calls an action is at least two: one call asks for the action, and one writes the reply from its result. Each call sends:

- **the instructions**, with your application's description and the page the user is on;
- **the conversation so far**, with the results of earlier actions;
- **the schema of every tool offered**: the name, the description and the parameters of each action. With many actions, this is usually the largest part of every call;
- **the user's message**.

Most of the time of a turn is the time of these calls, one after the other. A larger call takes longer to read, and each extra call adds its own time before the reply.

MentorAgent logs the usage of every model call at Information level, so you can check the effect of each option. The line looks like this one, shown here on two lines:

```text
[MentorAgent] Tokens — model: gpt-4.1, in: 740, out: 95, call total: 835,
  1180ms | session: 740+95=835 over 1 call(s)
```

## The options

| Option | What it reduces | What it adds | What it needs |
|---|---|---|---|
| [Tool filtering](https://docs.mentoragent.net/configure/cost-and-latency#send-fewer-tools) | The tool schemas in every call | One embedding of the message per turn | An `EmbeddingGenerator` |
| [Specialists](https://docs.mentoragent.net/configure/cost-and-latency#move-actions-to-specialists) | The tools of the coordinator | At least two model calls on the turns that they handle | `ChatClient` |
| [Model routing](https://docs.mentoragent.net/configure/cost-and-latency#use-the-strong-model-only-when-needed) | The turns on the expensive model | One embedding, or one small call, per turn | `StrongChatClient` |
| [Compaction](https://docs.mentoragent.net/configure/cost-and-latency#send-a-shorter-history) | The history in long conversations | Nothing: no model call | A history kept by your application |
| [Stable instructions](https://docs.mentoragent.net/configure/cost-and-latency#keep-the-instructions-the-same) | What a caching model reads again | Nothing | A history kept by your application |
| [A small classifier model](https://docs.mentoragent.net/configure/cost-and-latency#mentoragents-own-checks) | The cost and time of MentorAgent's own checks | A second model to configure | `ClassifierChatClient` |

They work together. The [complete configuration](https://docs.mentoragent.net/configure/cost-and-latency#complete-configuration) at the end turns them all on.

### Send fewer tools

[`EnableToolFiltering`](https://docs.mentoragent.net/reference/options#enabletoolfiltering) compares the user's message with your actions, by their meaning, and sends the model only those that match. The model still chooses freely among them. MentorAgent's own tools are always sent: navigation, memory, the handoff to specialists and teams, skills and UI actions.

- **It needs an embedding model**, an [`EmbeddingGenerator`](https://docs.mentoragent.net/reference/options#embeddinggenerator). Each turn embeds the message once; the embeddings of your actions are computed once and kept. Without an embedding model, every tool is sent and a warning is logged.
- **`ToolFilterMaxTools`**, 12 by default, caps the matched actions sent in addition to MentorAgent's own tools.
- **A message that matches nothing still gets three actions**: the three best-ranked that need no confirmation and no role. That costs about 80 input tokens per call on such turns, greetings included. A greeting can score higher than a question that needs data, so the filter does not trust the score alone.
- **Only the actions that the user may run are ranked.** An action whose roles the user lacks never travels.

The filter decides which actions reach the model, so an action that changes data and asks for no confirmation can reach it on any message. See [Tool filtering and confirmation](https://docs.mentoragent.net/concepts/governance#tool-filtering-and-confirmation).

### Move actions to specialists

A [specialist](https://docs.mentoragent.net/concepts/agent-levels) holds its own actions. The coordinator carries one tool, `route_to_specialist`, in place of all of them, so the turns that the coordinator handles alone send fewer schemas.

A turn handed to a specialist costs more:

- one call to the router, a small agent with a prompt of about 400 tokens and no actions, which picks the specialist;
- at least one call to the specialist, and one more for each round of its actions;
- the coordinator's own calls, which ask for the handoff and write the reply.

Choose specialists for an area with its own set of actions and its own rules, not to save tokens on every turn. A [team](https://docs.mentoragent.net/concepts/agent-levels#cost-and-latency) costs at least one call per member's turn, up to `MaxIterations` turns: it is the most expensive level. The router, the specialists and the team members all run on your `ChatClient`, and their calls appear on the dashboard.

### Use the strong model only when needed

Set [`StrongChatClient`](https://docs.mentoragent.net/reference/options#strongchatclient) to a stronger, more expensive model, and keep `ChatClient` on a cheaper one. [`RoutingStrategy`](https://docs.mentoragent.net/reference/options#routingstrategy) decides, turn by turn, which one answers:

| Strategy | How it decides | Extra cost per turn |
|---|---|---|
| `Semantic` | Compares the message with examples of hard requests, `RoutingComplexExemplars`, from `RoutingThreshold` up | One embedding |
| `Classifier` | A small model call labels the turn simple or complex | One small model call |
| `Cascade` | Answers on the cheap model, and runs the turn again on the strong one when the answer falls short | Only on calls that do not stream. Chat turns always stream, so there it decides like `Classifier` |
| `Custom`, the default | Your function, `UseStrongModelAsync` | Whatever your function does |

- **Point the label call elsewhere.** `Classifier` and `Cascade` use [`RoutingClassifierClient`](https://docs.mentoragent.net/reference/options#routingclassifierclient) when you set it, and `ChatClient` otherwise.
- **Write your own examples for `Semantic`.** Your list replaces the built-in one; write the examples in the languages your users type, and describe the kind of work rather than the topic.
- **A strategy that lacks what it needs does nothing.** MentorAgent logs a warning and every turn stays on `ChatClient`: see [Production checklist](https://docs.mentoragent.net/configure/production#common-errors).

### Send a shorter history

The conversation is sent again with every call, so a long conversation makes every call larger. [`EnableCompaction`](https://docs.mentoragent.net/reference/options#enablecompaction) shrinks it before each call, without a model call:

- it collapses the results of old actions;
- it keeps the last [`CompactionMaxTurns`](https://docs.mentoragent.net/reference/options#compactionmaxturns) turns, 8 by default, intact;
- if the history is still larger than [`CompactionTokenThreshold`](https://docs.mentoragent.net/reference/options#compactiontokenthreshold), 4000 tokens by default, it cuts it to that size.

It changes what is sent, never what is stored. It works with the built-in history and with your own history provider. With [`UseServiceManagedHistory`](https://docs.mentoragent.net/reference/options#useservicemanagedhistory), where the service keeps the conversation, it is not installed.

### Keep the instructions the same

Many providers bill a repeated beginning of the prompt at a lower price, or read it faster: this is prompt caching. MentorAgent keeps the fixed part of the instructions first, and the parts that change last, so the fixed part can be cached without any option.

The page the user is on, its data, its address and the time still close the instructions, and they change with every page and every minute. A model that caches the whole request by its beginning, a local model above all, then reads the whole conversation again. [`SessionContextPlacement`](https://docs.mentoragent.net/reference/options#sessioncontextplacement) set to `LastUserMessage` moves these lines to the start of the user's last message, in the request sent to the model only. The instructions then stay the same for the whole session.

- **The stored conversation does not change.** It keeps the message as the user wrote it.
- **A turn with RAG documents still changes the instructions**, because the documents stay there.
- **It needs a history kept by your application.** With `UseServiceManagedHistory` or a prebuilt `Agent`, it has no effect, and a warning at startup says so.

### MentorAgent's own checks

Some options add a small model call to each turn, before the reply: the safety check, the off-topic check of `RefuseOutOfScope`, and the checks behind [`RequireLookupForRecordQuestions`](https://docs.mentoragent.net/reference/options#requirelookupforrecordquestions) and [`RequireActionForChangeRequests`](https://docs.mentoragent.net/reference/options#requireactionforchangerequests). When memory captures facts by itself, with `MemoryAutoCapture`, one more call after the turn extracts them. These calls send a few hundred tokens and read back one word.

- **Give them a small, fast model**, [`ClassifierChatClient`](https://docs.mentoragent.net/reference/options#classifierchatclient). Without it they run on `ChatClient`. The safety check runs before the reply that the user is waiting for, so its time adds to every turn.
- **A check on the reply holds the reply.** With `EnableOutputSafetyCheck` or an `OutputGuardrail`, the answer appears all at once, after the check, instead of word by word.
- **[`WarmUpAtStartup`](https://docs.mentoragent.net/reference/options#warmupatstartup)** does the shared setup work once the host has started: the embeddings of your actions, the MCP connections and the cards of remote agents. Without it the first user waits for it.

## Read the numbers

MentorAgent counts the tokens, the calls and the time of every model call, per model: `ChatClient`, `StrongChatClient`, the embedding model and `ClassifierChatClient` when it is a different deployment. The calls of the router, the specialists and the team members are counted too. Three ways read the same numbers:

- **the dashboard**, the `MentorDashboard` component, on a page that you protect: see [Production checklist](https://docs.mentoragent.net/configure/production#authentication-and-endpoints);
- **`GET /mentor/admin/metrics`** of `Novior.MentorAgent.Server`, for the `DashboardRole` role;
- **`IMentorMetrics.GetSnapshot()`** in your own code, as in [Measure your app](https://docs.mentoragent.net/configure/cost-and-latency#measure-your-app).

The snapshot holds the totals and one row per model:

| Member | What it holds |
|---|---|
| `TotalInputTokens`, `TotalOutputTokens`, `RequestCount` | The tokens and the model calls, for every model together |
| `TurnCount`, `ServedCount`, `ErrorCount` | The turns started, answered and failed |
| `Models` | One `MentorModelUsage` per model: `ModelId`, `Kind` (`Cheap`, `Strong`, `Embedding` or `Other`), tokens, `RequestCount`, `AvgLatencyMs` and `AvgTimeToFirstByteMs` |
| `RequestsWithoutUsage`, on each model | Calls that the provider billed without reporting tokens, usually a turn that the user stopped. They count as calls, and add nothing to the tokens |
| `TotalCost`, and the costs on each model | `null`, not 0, for a model without a price |
| `TopActions` | The actions called most |

- **Costs need your prices.** MentorAgent ships no prices. Set [`ModelPricing`](https://docs.mentoragent.net/reference/options#modelpricing) with your prices per million tokens, under the model id that the dashboard shows. On Azure OpenAI that id is your deployment name.
- **The numbers live in memory**, for one process, and a restart clears them. To keep them, register an `IMentorMetricsStore` before `AddMentorAgent()`. For several instances, read the totals from your OpenTelemetry backend: see [Run on several instances](https://docs.mentoragent.net/configure/production#run-on-several-instances).

## Measure your app

The number of actions, their descriptions, your instructions and your model decide the cost, so measure on your app. Measure each configuration with the same questions:

1. Write 10 to 20 questions that your users ask, in their words: reads, writes, greetings and follow-ups.
2. Run them with one configuration, then change one option and run them again.
3. Compare the model calls, the tokens per model and the time of a turn.

This method sends each question in a new conversation and cancels every confirmation box, so nothing changes in your data. It reads the calls and the tokens from the snapshot, and measures the time of each turn:

```csharp title="CostMeasurement.cs"
using System.Diagnostics;
using Novior.MentorAgent.Abstractions.Interfaces;   // IMentorMetrics and others

public static class CostMeasurement
{
    // Sends each question in a new conversation and reports, per model,
    // the calls and tokens it took, and the median time of a turn.
    public static async Task<string> MeasureAsync(
        IServiceProvider services, IReadOnlyList<string> questions)
    {
        var metrics = services.GetRequiredService<IMentorMetrics>();
        metrics.Reset();   // a fresh window: run this on a test instance
        var times = new List<long>();

        foreach (var question in questions)
        {
            await using var scope = services.CreateAsyncScope();
            var state = scope.ServiceProvider
                .GetRequiredService<IMentorStateService>();
            // Measure without changing data: every confirmation is cancelled
            state.OnConfirmationRequired += box => state.Cancel(box.ActionId);

            var clock = Stopwatch.StartNew();
            await scope.ServiceProvider.GetRequiredService<IMentorOrchestrator>()
                .SendMessageAsync(question);
            times.Add(clock.ElapsedMilliseconds);
        }

        var snapshot = metrics.GetSnapshot();
        times.Sort();
        var lines = new List<string>
        {
            $"{questions.Count} questions, {snapshot.RequestCount} model calls, " +
            $"median turn {times[times.Count / 2]} ms",
        };
        foreach (var m in snapshot.Models)
        {
            lines.Add($"{m.ModelId} ({m.Kind}): {m.RequestCount} calls, " +
                $"{m.InputTokens} in, {m.OutputTokens} out");
        }
        return string.Join('\n', lines);
    }
}
```

- **It resets the counters** with `Reset()`, so run it on a test instance of your app, never in production: the dashboard starts again from zero.
- **It calls your real model**, so it costs money and needs the key. Count the calls before you run it: with the default options, a question that calls one action is at least two.
- **The model id comes from the provider.** A client that reports none shows `unknown`.
- **To test your actions without a model**, use the fake model of [Test without a model](https://docs.mentoragent.net/configure/test-without-model). It reports no tokens, so it cannot measure cost.

## Complete configuration

The options of this page in one registration:

```csharp title="Program.cs"
using Microsoft.Extensions.AI;         // AsIChatClient, AsIEmbeddingGenerator
using Novior.MentorAgent.Extensions;   // AddMentorAgent
using Novior.MentorAgent.Options;      // MentorRoutingStrategy and others
using OpenAI;                          // OpenAIClient
```

```csharp title="Program.cs"
var openAI = new OpenAIClient(builder.Configuration["OpenAI:Key"]
    ?? throw new InvalidOperationException("Set OpenAI:Key in a secret store."));

builder.Services.AddMentorAgent(options =>
{
    options.AppName = "Case management";
    options.ChatClient = openAI.GetChatClient("gpt-4o-mini").AsIChatClient();
    options.ScanAssemblies = [typeof(Program).Assembly];

    // Fewer tool schemas per call: only the actions that match the message
    options.EmbeddingGenerator = openAI
        .GetEmbeddingClient("text-embedding-3-small")
        .AsIEmbeddingGenerator();
    options.EnableToolFiltering = true;
    options.ToolFilterMaxTools = 12;

    // MentorAgent's own short decisions on a small, fast model
    options.ClassifierChatClient =
        openAI.GetChatClient("gpt-4.1-nano").AsIChatClient();

    // The stronger model only for the turns that need it
    options.StrongChatClient = openAI.GetChatClient("gpt-4o").AsIChatClient();
    options.RoutingStrategy = MentorRoutingStrategy.Semantic;

    // A shorter history in each call; the stored conversation is unchanged
    options.EnableCompaction = true;
    options.CompactionTokenThreshold = 4000;
    options.CompactionMaxTurns = 8;

    // Instructions that stay the same for the whole session
    options.SessionContextPlacement =
        MentorSessionContextPlacement.LastUserMessage;

    // The shared setup work before the first user arrives
    options.WarmUpAtStartup = true;
});
```

The values of `ToolFilterMaxTools`, `CompactionTokenThreshold` and `CompactionMaxTurns` are the defaults, written out to show where you would change them. The [Production checklist](https://docs.mentoragent.net/configure/production#cost-and-model-routing) goes through the same options with the rest of a deployment.

## Common errors

- **Every tool is still sent.** No `EmbeddingGenerator` is set, or `ChatClient` is not: the log has a warning.
- **The strong model is never used.** The strategy lacks what it needs: `UseStrongModelAsync` for `Custom`, an `EmbeddingGenerator` for `Semantic`.
- **The dashboard shows tokens but no cost.** The model has no entry in `ModelPricing`, or the entry uses another id: on Azure OpenAI, use the deployment name.
- **The first answer after a start is slow.** The shared setup work runs on the first message: set `WarmUpAtStartup`.
- **A local model takes many seconds before the first word on every new page.** It reads the whole conversation again: set `SessionContextPlacement` to `LastUserMessage`.

---

- Previous: [Test without a model](https://docs.mentoragent.net/configure/test-without-model.md)
- Next: [Production checklist](https://docs.mentoragent.net/configure/production.md)