Inconsistent AI answers: compare the inputs before changing the prompt

One assistant sends a duplicate-charge ticket to Billing. Another sends it to Bug Support. A third uses different words but also chooses Billing. Counting three different answers misses the useful distinction: the first and third agree on what to do. Start by naming the decision that must stay consistent, then compare what each run actually received.

Write the expected decision before running the comparison

For a support-routing task, the label matters more than whether an answer says “send” or “route”. For brainstorming, different suggestions may be the point. Choose the part of your own task that must stay stable before treating all variation as failure.

OpenAI’s evaluation guidance notes that generative systems can vary on the same input and recommends task-specific evaluations with explicit success criteria. The exercise below uses that principle to separate three things: different wording, a decision unsupported by the supplied rules, and a change in the supplied rules themselves.

The entire example is fictional, dated 12 October 2026. The packets and answers below are constructed to explain the diagnosis; they are not measured runs, a benchmark or results from HeyBrain.

Use this complete fixed input

Copy all four paragraphs below together for each trial:

Task: Classify the ticket using only the routing rules in this packet. Return one label and one sentence explaining it. Allowed labels: Billing, Access, Bug, Undetermined. If the supplied rules do not support a label, choose Undetermined. Do not browse, use external knowledge or infer additional company rules.

Routing rules, full packet R1: Billing handles duplicate charges, invoice corrections and refund requests. Access handles inability to sign in or reset a password. Bug handles a product function that fails after the customer has signed in. When more than one issue is present, choose Billing if a duplicate charge is reported. These are the complete rules for this exercise.

Ticket T1, received 12 October 2026: “My card was charged twice for the same subscription invoice. I can still sign in and use the app. Please help me with the second charge.”

Output format: Label — one-sentence reason.

The expected label is Billing: the ticket explicitly reports a duplicate charge and says access works. Nothing says a product function failed. Keep this expectation beside the input, rather than asking whichever model you are comparing to decide its own passing answer.

Start a new conversation for every trial

Use one assistant and the same exposed model choice for the first comparison. Open a new conversation for trial A, paste the complete input once and save the response. Open another new conversation for trial B and repeat the identical input. Do the same for trial C. Do not ask “try again” inside the earlier conversation: its previous answer becomes another part of the history.

Use a mode without browsing or connected-source retrieval where available; the source packet is already in the prompt. Record the app, displayed model label and time, and retain the exact input and each output. If using an API, also retain the complete messages, exposed model version and generation settings. Do not change model, prompt and source packet together.

A new conversation removes the earlier visible exchange, but it does not prove that an application has identical hidden instructions, saved memory or internal routing on every run. Disable optional personalisation for the exercise if available and record what remains unknown. Describe this as a comparison with matched visible inputs unless you can establish stronger control.

Read three possible answers without mistaking them for a result

These are illustrative outputs, not observations:

Trial A: “Billing — the customer reports a duplicate charge for one subscription invoice.”

Trial B: “Billing — duplicate charges belong to Billing under R1.”

Trial C: “Bug — the payment system appears to have malfunctioned.”

A and B both satisfy the task. C invents a reason to choose Bug and ignores the explicit routing rule. Its explanation may sound plausible, but diagnosing the payment system was not the assignment.

If your actual fixed-packet trials produced this pattern, you would have observed a decision failure with the same visible evidence. That supports checking instruction following, model settings and the surrounding app. It would not, on its own, prove which hidden mechanism caused the failure or estimate the rate at which customers experience it.

Now change only the source packet

For a second exercise, keep T1, the task, output format and chosen assistant unchanged. Start another fresh conversation. Replace only the routing-rules paragraph with this deliberately incomplete packet:

Routing rules, partial packet R2: Access handles inability to sign in or reset a password. Bug handles a product function that fails after the customer has signed in. These are the only routing rules supplied for this exercise.

The expected answer is now “Undetermined — the supplied rules do not say where to route a duplicate charge.” Billing is still an allowed output label, but the instruction requires support from the supplied rules, and R2 gives none. The correct response changed because the evidence changed.

This does not test a real retrieval system. It makes the difference visible. In your application, inspect the passages that retrieval actually supplied before blaming generation. A document existing in the company library does not show that its Billing paragraph reached this particular answer. If the app exposes no retrieval trace, record that limit; a citation alone may not reveal the entire input.

Make the next change match the finding

Use the result to choose a small next investigation, then repeat the same saved case after a change. Keep the case that previously failed alongside a normal case; a prettier single rerun does not establish a dependable repair.

For a real support workflow, add examples from the kinds of tickets you handle, with permitted data and agreed expected routes. Three illustrative answers to one ticket would not establish production reliability. The purpose here is to stop changing everything at once.

If the source arrived but the answer stretches what it says, use the worked citation check. If you need to assemble the initial packet, the complete briefing example shows that separate job.

Prepared by HeyBrain Editorial Team with AI assistance. The linked primary guidance was checked on 12 October 2026. Routing rules, ticket and trial answers are constructed teaching examples. No model experiment, customer incident or product performance measurement is claimed.