Issues & Diagnosis

A signal describes a behavior you want to detect in your agent’s work. Start with one concrete failure: an agent claiming a submission succeeded when the linked tool rejected it, for example.

​

Start with a Run

Open Telemetry and inspect a completed session. Check the user’s request, the agent’s response, the relevant tool result and any later retry or recovery. The SDK’s success field describes the instrumented execution. It does not prove that a claim was submitted, a shipment was created or a document review was complete. Preserve the actual tool output so a signal can evaluate that distinction. If you haven’t connected an agent yet, follow the quickstart or inspect the recorded insurance example.

​

Define a Signal

In Signals → Define a signal, name the behavior and describe when it counts as a failure. Include a counterexample and explain when the evidence is insufficient. For a false-submission signal: Save and prepare the policy, then capture evidence from a relevant run. Preparing a definition does not itself score every session. Configure the capture and evaluation options for the workflow, and check the report for skipped or failed work.

​

Review Decisions

Open Signals → Review decisions to see captured items and saved predictions. The list includes items without a prediction so they aren’t silently counted as successful evaluations. Open an item and follow its citations to the captured messages and tool results. Check delivery gaps, truncation and the evidence boundary before deciding whether the prediction is correct. “No failure” is specific to the signal; it is not a general guarantee that the agent did its job. Record your review or correction on the item. Earlier predictions and reviews remain available, so a correction does not rewrite what the model originally said. Include missed failures and a sample of unflagged runs when checking quality.

​

Investigate and Compare

Use Compare decisions and the policy’s Evaluation reports to inspect saved runs, their coverage and current reviews. Check which cases were evaluated, which were skipped, and whether a change introduces false alarms or misses failures. For code changes, Code Integration can provide repository context, suggestions and a pull request for review. Test the proposed change against both the failing case and healthy cases before merging it.

​

Processing and Coverage

Session processing is asynchronous. Availability depends on captured content, workspace limits, enabled processing and provider responses. Custom policy capture and evaluation have their own configuration; receiving telemetry does not mean a prediction has been produced. When a result is missing, start with the session’s delivery and processing status. The CLI Doctor checks trace delivery without running a model. It does not certify detector accuracy. Model judgments and diagnosis suggestions still need validation against your workflow’s actual outcomes.

​

Next Steps