Cost and latency

Use this page when you plan an agent with many actions, or when it already costs more or answers more slowly than you expected. It explains what decides the tokens and the time of a turn, and which options reduce them. For each option, it says what it costs in extra model calls. Then it shows how to read the numbers and how to measure your own app.

Each turn is one or more calls to the model. A turn that answers from the conversation is one call. A turn that calls an action is at least two: one call asks for the action, and one writes the reply from its result. Each call sends:

• the instructions, with your application's description and the page the user is on;
• the conversation so far, with the results of earlier actions;
• the schema of every tool offered: the name, the description and the parameters of each action. With many actions, this is usually the largest part of every call;
• the user's message.

Most of the time of a turn is the time of these calls, one after the other. A larger call takes longer to read, and each extra call adds its own time before the reply.

MentorAgent logs the usage of every model call at Information level, so you can check the effect of each option. The line looks like this one, shown here on two lines:

[MentorAgent] Tokens — model: gpt-4.1, in: 740, out: 95, call total: 835,
  1180ms | session: 740+95=835 over 1 call(s)

They work together. The complete configuration at the end turns them all on.

EnableToolFiltering compares the user's message with your actions, by their meaning, and sends the model only those that match. The model still chooses freely among them. MentorAgent's own tools are always sent: navigation, memory, the handoff to specialists and teams, skills and UI actions.

• It needs an embedding model, an EmbeddingGenerator. Each turn embeds the message once; the embeddings of your actions are computed once and kept. Without an embedding model, every tool is sent and a warning is logged.
• ToolFilterMaxTools, 12 by default, caps the matched actions sent in addition to MentorAgent's own tools.
• A message that matches nothing still gets three actions: the three best-ranked that need no confirmation and no role. That costs about 80 input tokens per call on such turns, greetings included. A greeting can score higher than a question that needs data, so the filter does not trust the score alone.
• Only the actions that the user may run are ranked. An action whose roles the user lacks never travels.

The filter decides which actions reach the model, so an action that changes data and asks for no confirmation can reach it on any message. See Tool filtering and confirmation.

A specialist holds its own actions. The coordinator carries one tool, route_to_specialist, in place of all of them, so the turns that the coordinator handles alone send fewer schemas.

A turn handed to a specialist costs more:

• one call to the router, a small agent with a prompt of about 400 tokens and no actions, which picks the specialist;
• at least one call to the specialist, and one more for each round of its actions;
• the coordinator's own calls, which ask for the handoff and write the reply.

Choose specialists for an area with its own set of actions and its own rules, not to save tokens on every turn. A team costs at least one call per member's turn, up to MaxIterations turns: it is the most expensive level. The router, the specialists and the team members all run on your ChatClient, and their calls appear on the dashboard.

Set StrongChatClient to a stronger, more expensive model, and keep ChatClient on a cheaper one. RoutingStrategy decides, turn by turn, which one answers:

• Point the label call elsewhere. Classifier and Cascade use RoutingClassifierClient when you set it, and ChatClient otherwise.
• Write your own examples for Semantic. Your list replaces the built-in one; write the examples in the languages your users type, and describe the kind of work rather than the topic.
• A strategy that lacks what it needs does nothing. MentorAgent logs a warning and every turn stays on ChatClient: see Production checklist.

The conversation is sent again with every call, so a long conversation makes every call larger. EnableCompaction shrinks it before each call, without a model call:

• it collapses the results of old actions;
• it keeps the last CompactionMaxTurns turns, 8 by default, intact;
• if the history is still larger than CompactionTokenThreshold, 4000 tokens by default, it cuts it to that size.

It changes what is sent, never what is stored. It works with the built-in history and with your own history provider. With UseServiceManagedHistory, where the service keeps the conversation, it is not installed.

Many providers bill a repeated beginning of the prompt at a lower price, or read it faster: this is prompt caching. MentorAgent keeps the fixed part of the instructions first, and the parts that change last, so the fixed part can be cached without any option.

The page the user is on, its data, its address and the time still close the instructions, and they change with every page and every minute. A model that caches the whole request by its beginning, a local model above all, then reads the whole conversation again. SessionContextPlacement set to LastUserMessage moves these lines to the start of the user's last message, in the request sent to the model only. The instructions then stay the same for the whole session.

• The stored conversation does not change. It keeps the message as the user wrote it.
• A turn with RAG documents still changes the instructions, because the documents stay there.
• It needs a history kept by your application. With UseServiceManagedHistory or a prebuilt Agent, it has no effect, and a warning at startup says so.

Some options add a small model call to each turn, before the reply: the safety check, the off-topic check of RefuseOutOfScope, and the checks behind RequireLookupForRecordQuestions and RequireActionForChangeRequests. When memory captures facts by itself, with MemoryAutoCapture, one more call after the turn extracts them. These calls send a few hundred tokens and read back one word.

• Give them a small, fast model, ClassifierChatClient. Without it they run on ChatClient. The safety check runs before the reply that the user is waiting for, so its time adds to every turn.
• A check on the reply holds the reply. With EnableOutputSafetyCheck or an OutputGuardrail, the answer appears all at once, after the check, instead of word by word.
• WarmUpAtStartup does the shared setup work once the host has started: the embeddings of your actions, the MCP connections and the cards of remote agents. Without it the first user waits for it.

MentorAgent counts the tokens, the calls and the time of every model call, per model: ChatClient, StrongChatClient, the embedding model and ClassifierChatClient when it is a different deployment. The calls of the router, the specialists and the team members are counted too. Three ways read the same numbers:

• the dashboard, the MentorDashboard component, on a page that you protect: see Production checklist;
• GET /mentor/admin/metrics of Novior.MentorAgent.Server, for the DashboardRole role;
• IMentorMetrics.GetSnapshot() in your own code, as in Measure your app.

The snapshot holds the totals and one row per model:

• Costs need your prices. MentorAgent ships no prices. Set ModelPricing with your prices per million tokens, under the model id that the dashboard shows. On Azure OpenAI that id is your deployment name.
• The numbers live in memory, for one process, and a restart clears them. To keep them, register an IMentorMetricsStore before AddMentorAgent(). For several instances, read the totals from your OpenTelemetry backend: see Run on several instances.

The number of actions, their descriptions, your instructions and your model decide the cost, so measure on your app. Measure each configuration with the same questions:

1. Write 10 to 20 questions that your users ask, in their words: reads, writes, greetings and follow-ups.
2. Run them with one configuration, then change one option and run them again.
3. Compare the model calls, the tokens per model and the time of a turn.

This method sends each question in a new conversation and cancels every confirmation box, so nothing changes in your data. It reads the calls and the tokens from the snapshot, and measures the time of each turn:

using System.Diagnostics;
using Novior.MentorAgent.Abstractions.Interfaces;   // IMentorMetrics and others

public static class CostMeasurement
{
    // Sends each question in a new conversation and reports, per model,
    // the calls and tokens it took, and the median time of a turn.
    public static async Task<string> MeasureAsync(
        IServiceProvider services, IReadOnlyList<string> questions)
    {
        var metrics = services.GetRequiredService<IMentorMetrics>();
        metrics.Reset();   // a fresh window: run this on a test instance
        var times = new List<long>();

        foreach (var question in questions)
        {
            await using var scope = services.CreateAsyncScope();
            var state = scope.ServiceProvider
                .GetRequiredService<IMentorStateService>();
            // Measure without changing data: every confirmation is cancelled
            state.OnConfirmationRequired += box => state.Cancel(box.ActionId);

            var clock = Stopwatch.StartNew();
            await scope.ServiceProvider.GetRequiredService<IMentorOrchestrator>()
                .SendMessageAsync(question);
            times.Add(clock.ElapsedMilliseconds);
        }

        var snapshot = metrics.GetSnapshot();
        times.Sort();
        var lines = new List<string>
        {
            $"{questions.Count} questions, {snapshot.RequestCount} model calls, " +
            $"median turn {times[times.Count / 2]} ms",
        };
        foreach (var m in snapshot.Models)
        {
            lines.Add($"{m.ModelId} ({m.Kind}): {m.RequestCount} calls, " +
                $"{m.InputTokens} in, {m.OutputTokens} out");
        }
        return string.Join('\n', lines);
    }
}

• It resets the counters with Reset(), so run it on a test instance of your app, never in production: the dashboard starts again from zero.
• It calls your real model, so it costs money and needs the key. Count the calls before you run it: with the default options, a question that calls one action is at least two.
• The model id comes from the provider. A client that reports none shows unknown.
• To test your actions without a model, use the fake model of Test without a model. It reports no tokens, so it cannot measure cost.

The options of this page in one registration:

using Microsoft.Extensions.AI;         // AsIChatClient, AsIEmbeddingGenerator
using Novior.MentorAgent.Extensions;   // AddMentorAgent
using Novior.MentorAgent.Options;      // MentorRoutingStrategy and others
using OpenAI;                          // OpenAIClient

var openAI = new OpenAIClient(builder.Configuration["OpenAI:Key"]
    ?? throw new InvalidOperationException("Set OpenAI:Key in a secret store."));

builder.Services.AddMentorAgent(options =>
{
    options.AppName = "Case management";
    options.ChatClient = openAI.GetChatClient("gpt-4o-mini").AsIChatClient();
    options.ScanAssemblies = [typeof(Program).Assembly];

    // Fewer tool schemas per call: only the actions that match the message
    options.EmbeddingGenerator = openAI
        .GetEmbeddingClient("text-embedding-3-small")
        .AsIEmbeddingGenerator();
    options.EnableToolFiltering = true;
    options.ToolFilterMaxTools = 12;

    // MentorAgent's own short decisions on a small, fast model
    options.ClassifierChatClient =
        openAI.GetChatClient("gpt-4.1-nano").AsIChatClient();

    // The stronger model only for the turns that need it
    options.StrongChatClient = openAI.GetChatClient("gpt-4o").AsIChatClient();
    options.RoutingStrategy = MentorRoutingStrategy.Semantic;

    // A shorter history in each call; the stored conversation is unchanged
    options.EnableCompaction = true;
    options.CompactionTokenThreshold = 4000;
    options.CompactionMaxTurns = 8;

    // Instructions that stay the same for the whole session
    options.SessionContextPlacement =
        MentorSessionContextPlacement.LastUserMessage;

    // The shared setup work before the first user arrives
    options.WarmUpAtStartup = true;
});

The values of ToolFilterMaxTools, CompactionTokenThreshold and CompactionMaxTurns are the defaults, written out to show where you would change them. The Production checklist goes through the same options with the rest of a deployment.

• Every tool is still sent. No EmbeddingGenerator is set, or ChatClient is not: the log has a warning.
• The strong model is never used. The strategy lacks what it needs: UseStrongModelAsync for Custom, an EmbeddingGenerator for Semantic.
• The dashboard shows tokens but no cost. The model has no entry in ModelPricing, or the entry uses another id: on Azure OpenAI, use the deployment name.
• The first answer after a start is slow. The shared setup work runs on the first message: set WarmUpAtStartup.
• A local model takes many seconds before the first word on every new page. It reads the whole conversation again: set SessionContextPlacement to LastUserMessage.