MentorAgent Docs
Menu

How-to guideNovior.MentorAgent

Test without a model

Test your actions, roles and confirmation boxes in automated tests with a fake IChatClient, develop offline with a local model, and catch regressions.

View as Markdown
On this page

Use this page to test your actions, their roles and their confirmation boxes in an automated test, without calling a model. A fake model plays the part of the real one: it asks MentorAgent to call one of your actions, and MentorAgent runs that call as in production. The test needs no key, no network and no money, and it gives the same result every time.

When to choose it

Choose it for the rules that your code decides, which a test must check on every build:

  • Everything but the model is real. MentorAgent finds your actions, offers the model only those that the user may run, checks the roles, shows the confirmation box, and runs your method. Your action policies and observers run too.
  • The model's choices are not tested. The fake model always asks for the tool that you give it. Whether a real model picks the right action for a message is a question for a test with the real model: see Catch regressions with the real model.

Write a fake model

MentorAgent talks to the model through an IChatClient of Microsoft.Extensions.AI, so a fake model is a class that implements it. This one asks for one tool on the first call, with the arguments that you give it. On the next call it gets the result of the tool, and answers "Done.":

ScriptedChatClient.cs
using System.Runtime.CompilerServices;
using Microsoft.Extensions.AI;

// A model that asks for one tool, then answers "Done." No network, no key.
public sealed class ScriptedChatClient(
    string tool, Dictionary<string, object?> arguments) : IChatClient
{
    const string CallId = "call-1";

    // The tools that MentorAgent offered, and what the tool call returned
    public List<string> OfferedTools { get; } = [];
    public string? ToolResult { get; private set; }

    public Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        CancellationToken cancellationToken = default)
        => Task.FromResult(new ChatResponse(Next(messages, options)));

    public async IAsyncEnumerable<ChatResponseUpdate> GetStreamingResponseAsync(
        IEnumerable<ChatMessage> messages, ChatOptions? options = null,
        [EnumeratorCancellation] CancellationToken cancellationToken = default)
    {
        await Task.Yield();
        var reply = Next(messages, options);
        yield return new ChatResponseUpdate(reply.Role, reply.Contents)
        {
            MessageId = Guid.NewGuid().ToString("N"),
        };
    }

    ChatMessage Next(IEnumerable<ChatMessage> messages, ChatOptions? options)
    {
        var result = messages.SelectMany(m => m.Contents)
            .OfType<FunctionResultContent>()
            .FirstOrDefault(r => r.CallId == CallId);
        if (result is not null)
        {
            // Second call: the tool ran, or was refused. Keep what it returned
            ToolResult = result.Result?.ToString();
            return new ChatMessage(ChatRole.Assistant, "Done.");
        }
        if (options?.Tools is not { Count: > 0 } tools)
            return new ChatMessage(ChatRole.Assistant, "Done.");

        // First call: ask for the tool even if it is not offered,
        // like a model that makes up a tool name
        OfferedTools.AddRange(tools.Select(t => t.Name));
        return new ChatMessage(ChatRole.Assistant,
            [new FunctionCallContent(CallId, tool, arguments)]);
    }

    public object? GetService(Type serviceType, object? serviceKey = null) => null;

    public void Dispose() { }
}
  • It asks for the tool even when the tool is not offered, like a model that makes up a tool name. That is how a test proves that a call made anyway is refused.
  • OfferedTools lists the tools that MentorAgent offered on the first call. An action that the user may not run is not in the list.
  • ToolResult is what the tool call returned to the model: the value of your method, or the reason why the call was refused.
  • A call without tools gets "Done." at once. That covers the calls that are not a turn of the chat, such as an evaluation.

Build the app under test

The test builds the same services as your app, with the fake model in ChatClient. These are the actions of the example: a write that needs the Supervisors role and a confirmation, and a read.

CaseMergeActions.cs
public class CaseMergeActions
{
    // What the actions did: in your app, your own service and its store
    public List<string> Merged { get; } = [];

    [MentorAction(
        Description = "Merges a duplicate support case into another case",
        RequiredRoles = ["Supervisors"],
        RequiresConfirmation = true)]
    public string MergeCases(string duplicate, string into)
    {
        Merged.Add($"{duplicate} into {into}");
        return $"Case {duplicate} merged into {into}.";
    }

    [MentorAction(
        Description = "Gets the owner of a support case",
        ReadOnly = true)]
    public string GetCaseOwner(string caseNumber) => "Alex";
}

In the example the class records what it did, so that the test can read it. In your app, register your real service with a fake store, or with a test database.

The test host registers the actions, signs a user in and adds MentorAgent with the fake model. Then it sends one message, as the widget does, and answers every confirmation box:

MentorTestHost.cs
using System.Security.Claims;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.AI;
using Novior.MentorAgent.Abstractions.Interfaces;   // IMentorOrchestrator
using Novior.MentorAgent.Extensions;                // AddMentorAgent

public static class MentorTestHost
{
    // Your services, MentorAgent with the fake model, and a signed-in user
    public static WebApplication Build(IChatClient model, ClaimsPrincipal user)
    {
        var builder = WebApplication.CreateBuilder();
        builder.Services.AddSingleton<CaseMergeActions>();
        builder.Services.AddScoped<AuthenticationStateProvider>(
            _ => new SignedIn(user));
        builder.Services.AddMentorAgent(options =>
        {
            options.AppName = "Case management";
            options.ChatClient = model;
            options.ScanAssemblies = [typeof(CaseMergeActions).Assembly];
        });
        return builder.Build();
    }

    // One turn: the message, and the user's answer to every confirmation
    // box. Returns how many boxes the user saw.
    public static async Task<int> SendAsync(
        WebApplication app, string message, bool approve)
    {
        await using var scope = app.Services.CreateAsyncScope();
        var services = scope.ServiceProvider;
        var state = services.GetRequiredService<IMentorStateService>();
        var boxes = 0;
        state.OnConfirmationRequired += request =>
        {
            Interlocked.Increment(ref boxes);
            if (approve) _ = state.ConfirmAsync(request.ActionId);
            else state.Cancel(request.ActionId);
        };
        await services.GetRequiredService<IMentorOrchestrator>()
            .SendMessageAsync(message);
        return boxes;
    }

    sealed class SignedIn(ClaimsPrincipal user) : AuthenticationStateProvider
    {
        public override Task<AuthenticationState> GetAuthenticationStateAsync()
            => Task.FromResult(new AuthenticationState(user));
    }
}
  • The user comes from the AuthenticationStateProvider, as in a Blazor app. MentorAgent reads the roles from the user's role claims, so the test gives each user the roles to test.
  • The confirmation box reaches the test through IMentorStateService.OnConfirmationRequired. The test answers it with ConfirmAsync or Cancel, as the user would with the buttons of the box.
  • No web server starts. The test builds the app and calls IMentorOrchestrator, the service behind the widget, from a scope of its own.

The test project references the ASP.NET Core shared framework, because the host is a WebApplication. In a project with the Microsoft.NET.Sdk SDK, add it, with the namespaces that an ASP.NET Core project imports by itself:

XML
<ItemGroup>
  <FrameworkReference Include="Microsoft.AspNetCore.App" />
  <Using Include="Microsoft.AspNetCore.Builder" />
  <Using Include="Microsoft.Extensions.DependencyInjection" />
</ItemGroup>

Test actions, roles and confirmation

Each test picks a user and the tool that the fake model asks for, sends one message, and checks what happened:

CaseMergeTests.cs
using System.Security.Claims;

public static class CaseMergeTests
{
    static readonly ClaimsPrincipal Supervisor = SignedInUser("Supervisors");
    static readonly ClaimsPrincipal Agent = SignedInUser("Agents");
    const string Ask = "Merge C-2 into C-1";

    // A supervisor confirms: one box, then the cases are merged
    public static async Task SupervisorMergesAfterConfirming()
    {
        var model = new ScriptedChatClient("merge_cases",
            new() { ["duplicate"] = "C-2", ["into"] = "C-1" });
        await using var app = MentorTestHost.Build(model, Supervisor);

        var boxes = await MentorTestHost.SendAsync(app, Ask, approve: true);

        var cases = app.Services.GetRequiredService<CaseMergeActions>();
        Check(boxes == 1, "one confirmation box");
        Check(cases.Merged.SequenceEqual(["C-2 into C-1"]), "the cases merged");
    }

    // The supervisor cancels: the box is shown, and nothing changes
    public static async Task NothingChangesWhenTheUserCancels()
    {
        var model = new ScriptedChatClient("merge_cases",
            new() { ["duplicate"] = "C-2", ["into"] = "C-1" });
        await using var app = MentorTestHost.Build(model, Supervisor);

        var boxes = await MentorTestHost.SendAsync(app, Ask, approve: false);

        var cases = app.Services.GetRequiredService<CaseMergeActions>();
        Check(boxes == 1, "one confirmation box");
        Check(cases.Merged.Count == 0, "nothing merged");
    }

    // An agent lacks the role: not offered, no box, refused if called anyway
    public static async Task AnAgentCannotMerge()
    {
        var model = new ScriptedChatClient("merge_cases",
            new() { ["duplicate"] = "C-2", ["into"] = "C-1" });
        await using var app = MentorTestHost.Build(model, Agent);

        var boxes = await MentorTestHost.SendAsync(app, Ask, approve: true);

        var cases = app.Services.GetRequiredService<CaseMergeActions>();
        Check(!model.OfferedTools.Contains("merge_cases"), "not offered");
        Check(model.ToolResult?.Contains("Access denied") == true, "refused");
        Check(boxes == 0, "no confirmation box");
        Check(cases.Merged.Count == 0, "nothing merged");
    }

    static ClaimsPrincipal SignedInUser(string role) => new(new ClaimsIdentity(
        [new Claim(ClaimTypes.Name, "alex"), new Claim(ClaimTypes.Role, role)],
        authenticationType: "Cookies"));

    // In your tests, use the assertions of your framework, such as Assert.True
    static void Check(bool condition, string expected)
    {
        if (!condition)
            throw new InvalidOperationException($"Expected: {expected}");
    }
}
  • A supervisor confirms. One box is shown, and the method runs after the answer.
  • The supervisor cancels. The box is shown, and the method never runs.
  • An agent lacks the role. merge_cases is not offered to the model. The call that the fake model makes anyway is refused before any box appears, and the model gets the refusal as the result of the tool. See Roles, confirmation and governance.

The examples check their results with a small Check method, so they compile without a test framework. In your tests, use the assertions of your framework, and mark each method as a test, for example with [Fact] in xUnit.

To test an action policy, register it in the test host as in your app. The call of the fake model goes through it. When the policy stops the call, ToolResult holds what the model reads: that a rule stopped the call, followed by the reason of your policy.

Develop offline with a local model

To try the chat by hand without a key or a network, run a model on your machine. Ollama exposes an endpoint that the OpenAI client can call, so only the ChatClient line changes. Pick a model that supports tool calling: a model without it answers, but never runs your actions.

A local model is slower than a hosted one, and it can choose differently. Use it to develop, and keep the tests on the fake model, which gives the same result every time.

Catch regressions with the real model

A change to your instructions or to your model can make the answers worse or longer without breaking any test. MentorEvaluator sends a list of messages to your ChatClient, measures the tokens of each answer, and fails when an answer is empty or the tokens go over the budget:

EvaluationTests.cs
using Novior.MentorAgent.Evaluation;   // MentorEvaluator, EvalCase

var evaluator = services.GetRequiredService<MentorEvaluator>();
var report = await evaluator.RunAsync(
    [
        new EvalCase("Who owns case C-1?", MaxTokens: 1500),
        new EvalCase("What can you do for me?", MaxTokens: 1500),
    ],
    new MentorEvalOptions
    {
        SystemInstructions = "You are the assistant of a case desk.",
        MaxTotalTokens = 3000,   // fails when a change makes answers longer
    });
report.ThrowIfFailed();   // fails the test on a regression
  • It calls the real model, so it costs money and needs the key. Run it on its own schedule, not on every build.
  • It tests the answers, not your actions. Each message goes straight to the ChatClient, with the instructions in SystemInstructions: no tools, no roles and no confirmation. To test your actions, use the fake model above.
  • With Judge = true and an expectation in each EvalCase, a model scores how well each answer meets the expectation, and MinQuality sets the lowest score that passes.

Common errors

  • The test waits and never ends. A confirmation box is waiting for an answer. Subscribe to OnConfirmationRequired before you send the message, in the same scope.
  • No confirmation box, and the method never runs. The user does not hold the role of the action. Check the claims of the test user: MentorAgent reads them with IsInRole.
  • The fake model asks for the tool, but nothing runs. The tool name is not the one of your action. A tool name is the method name in snake case, with a trailing Async dropped first: MergeCasesAsync is merge_cases.
  • The test throws InvalidOperationException: Configure ChatClient or Agent. The fake model is not in ChatClient. MentorAgent checks its options the first time it reads them, here at the first message. Set the fake model in the AddMentorAgent callback, as in the test host.

See also

  • Roles, confirmation and governance — Role checks and human-in-the-loop approval for AI agent tools in .NET, enforced in code on every channel and not in the prompt, and what stays in your code.
  • Model providers — Connect your .NET AI agent to OpenAI, Azure OpenAI, Ollama and other local models, or to your own Agent Framework AIAgent, and choose models with tool calling.
  • Production checklist — Settings to review before production: governance, safety check, rate limits, cost and routing, history, several instances, observability and outgoing data.
  • Attributes — The parameters of MentorAction, MentorAgent, MentorTeam, TeamMember, MentorPage and MentorSkill, with their defaults and rules.

On this page