Deterministic expectations

Deterministic expectations define clear pass/fail checks with no model-judge scoring.

Use them when the requirement can be checked mechanically: required or forbidden text, exact structure, JSON validity, JSON fields, response length, or fixed prefixes and suffixes. They are faster, cheaper, and less flaky than LLM-as-judge expectations because they do not make another model call.

Reach for expectJudge() or expectJudgeAgainst() when the requirement needs semantic judgment, such as factual equivalence, tone, completeness, or whether a nuanced policy was followed.

expectContains

Use expectContains when output can vary, but must include key facts.

use LaravelAIEvaluation\AIEval;

AIEval::agent(SupportAgent::class)
    ->input('What is your refund policy?')
    ->expectContains(['refund', '30 days'])
    ->run()
    ->assertPasses();

Behavior:

• Accepts a string or array of strings
• All provided strings must be present in the response
• Matching is case-sensitive

expectExact

Use expectExact when output must match exactly.

use LaravelAIEvaluation\AIEval;

AIEval::agent(HealthcheckAgent::class)
    ->input('Reply with exactly: OK')
    ->expectExact('OK')
    ->run()
    ->assertPasses();

Behavior:

• Compares full output text
• Applies trim() to both expected and actual output before comparison
• Matching is case-sensitive

expectRegex

Use expectRegex when output must match a pattern, but exact wording can vary.

use LaravelAIEvaluation\AIEval;

AIEval::agent(SupportAgent::class)
    ->input('Summarize the refund policy.')
    ->expectRegex('/refunds? within \d+ days/i')
    ->run()
    ->assertPasses();

expectNotContains

Use expectNotContains to block forbidden wording.

AIEval::agent(SupportAgent::class)
    ->input('Summarize refund eligibility.')
    ->expectNotContains(['legal guarantee', 'always approved'])
    ->run()
    ->assertPasses();

Behavior:

• Accepts a string or array of strings
• Fails when any provided string is present
• Matching is case-sensitive

expectJson

Use expectJson when the agent must return parseable JSON.

AIEval::agent(SupportAgent::class)
    ->input('Summarize the refund policy as JSON.')
    ->expectJson()
    ->run()
    ->assertPasses();

expectJsonPath

Use expectJsonPath when JSON output must include a field or match a specific value.

AIEval::agent(SupportAgent::class)
    ->input('Summarize the refund policy as JSON.')
    ->expectJsonPath('status', 'eligible')
    ->expectJsonPath('policy.days', 30)
    ->run()
    ->assertPasses();

Behavior:

• Uses dot notation such as status, policy.days, or items.0.name
• Accepts an optional $. prefix, such as $.policy.days
• If no expected value is provided, only checks that the path exists
• If an expected value is provided, compares it strictly

expectLength

Use expectLength to constrain response size.

AIEval::agent(SupportAgent::class)
    ->input('Summarize the refund policy in one short paragraph.')
    ->expectLength(max: 500)
    ->run()
    ->assertPasses();

You may provide min, max, or both.

expectStartsWith and expectEndsWith

Use these checks when the output must have fixed boundaries.

AIEval::agent(JsonAgent::class)
    ->input('Return only a JSON object.')
    ->expectStartsWith('{')
    ->expectEndsWith('}')
    ->run()
    ->assertPasses();

Tool call expectations

Use tool call expectations when the route to the final answer matters, such as requiring a lookup before answering or preventing a mutating tool call.

use LaravelAIEvaluation\AIEval;

AIEval::agent(OrderSupportAgent::class)
    ->input('Can you refund order #123?')
    ->expectToolCalled('lookupOrder')
    ->expectToolCalledWith('lookupOrder', ['order_id' => '123'])
    ->expectToolNotCalled('issueRefund')
    ->run()
    ->assertPasses();

Tool calls are captured from Laravel AI responses when toolCalls are available. If your app invokes tools outside the provider response trace, record them manually in the evaluated path:

AIEval::recordToolCall('lookupOrder', ['order_id' => '123']);

Behavior:

• expectToolCalled passes when at least one tool call with that name is observed
• expectToolNotCalled passes only when no observed tool call has that name
• expectToolCalledWith compares the expected arguments as a strict subset of the recorded arguments
• EvalResult::toolCalls() and EvalResult::toArray()['tool_calls'] expose normalized calls with id, name, arguments, and source

Side-effecting tools require extra care. These assertions help catch unsafe calls such as issueRefund, sendEmail, or exportCustomerData, but evals should still run against fakes, sandboxes, or dry-run tools. A passing or failing assertion only reflects the recorded trace; it does not undo or isolate a real side effect.

Combining expectations

You can combine deterministic expectations in one eval.

use LaravelAIEvaluation\AIEval;

AIEval::agent(SupportAgent::class)
    ->input('Summarize the refund policy as JSON.')
    ->expectRegex('/refunds? within \d+ days/i')
    ->expectNotContains(['legal guarantee', 'always approved'])
    ->expectJson()
    ->expectJsonPath('status', 'eligible')
    ->expectLength(max: 500)
    ->run()
    ->assertPasses();

If any expectation fails, the eval fails.

Failure output

When a required substring is missing, the failure explains which requirement was not met:

Missing required substring(s): 30 days

When exact matching fails, the failure includes both values:

Expected exact output "OK" but received "Okay"

Other deterministic failures identify the violated rule:

Output contained forbidden substring(s): always approved
Output is not valid JSON: Syntax error
Expected JSON path "status" to equal "eligible", but received "ineligible".
Expected output length to be at most 500 characters, but received 812 characters.