Operating an AI agent: monitoring, changes and recovery

Assign roles matching the work

Name a process owner, source owner and technical operator. The first defines the supported task, the second maintains content and the third investigates software and integrations. One person can cover several roles, but the responsibilities should remain distinguishable. Otherwise a content problem may be treated indefinitely as a technical outage, with neither team able to resolve the cause.

Define decision authority too. Who can add a source, disable an operation or pause the application? Provide a substitute and a discoverable contact route. A list of names is insufficient when nobody knows which access is required. During handover, demonstrate that the designated people can actually carry out their duties using the rights, interfaces and instructions available to them.

Monitor three different kinds of outcome

Availability shows whether the application responds. Content quality concerns accurate information and appropriate boundaries. Business outcomes concern useful processing, reduced handling or better preparation. These can diverge: an agent may answer quickly while selecting the wrong customer, or produce accurate drafts employees never use because nobody understands which queue owns the next step.

Create a limited dashboard with meaningful signals. These might include failed connections, questions lacking sources, handovers, outstanding suggestions and completed tasks. Explain each signal and assign somebody to respond. Numerous charts without decision rules offer little practical control. Begin with alerts that prompt real action, then add measurements only when they help understand or improve an identified operating issue.

Bound usage and unintended load

Limit steps, repeated attempts and concurrent processing. A recurring failure must not generate unlimited model requests or write operations. Decide what can wait and when a request should go to an employee. Appropriate boundaries depend on the process: an internal summary can generally wait differently from a time-sensitive customer enquiry involving an existing commitment or an approaching deadline.

Track actual usage by task or group where appropriate. A total charge does not reveal which application consumes unexpected resources. Investigate sudden growth before merely increasing the budget. Volume may genuinely have risen, but the cause could also be a loop, an incorrect input route or a changed source producing longer responses. Usage controls therefore belong in the functional design, alongside cost monitoring.

The agent belongs to a process that needs to remain manageable when a model cannot provide a useful answer: Maintaining n8n workflows: ownership, checks and controlled change.

Keep logs useful and proportionate

Retain events that explain what happened: task reference, relevant version, proposed operation, approval and outcome. Do not automatically duplicate complete conversations and documents into every logging service. Decide what detail investigation requires and who may inspect it. The information inside a log can be more sensitive than the technical event that caused the record to be written.

Make events across systems relatable. A task reference can connect the original request, agent steps and CRM outcome. Without that relationship, an operator may see a successful model request alongside a failed customer operation without recognising that they belong together. Use a fictional incident to check whether the available evidence is sufficient to establish the actual business result rather than only server activity.

Treat meaningful changes as small releases

New instructions can change behaviour even when application code is unchanged. The same applies to revised sources, a different model or another tool. Record the reason, version and owner for each meaningful change. Describe the expected improvement and which existing route could be affected. This makes it possible to understand later why the application began behaving differently.

Run relevant regression checks against retained examples. OpenAI describes evaluation as an ongoing part of development. Operationally, a change therefore needs fresh evidence for the affected task. Do not demonstrate only the new capability; exercise earlier boundaries and handover as well. Keep a way back when the modified behaviour proves unsuitable, and explain which state and data that return path preserves.

OpenAI: Evaluation best practices ↗

Illustrative incident: a correct reply but no task

A fictional service assistant says an enquiry has been prepared, but expired access prevents its task reaching the CRM. An availability check of the chat window remains green. Only inspection of actual processing exposes the problem. The operator pauses writing and routes new enquiries through the ordinary process while the failed connection and affected requests are investigated.

Enquiry references identify incomplete cases. Employees check whether a task was created before entering anything again. Once access is restored, the relevant cases resume under controlled review. The team also changes the user message to distinguish preparation from processing more clearly. The incident becomes a new regression case rather than disappearing into a resolved technical support ticket with no lasting improvement.

Practise pausing and controlled resumption

Explain what pausing actually does. Does it stop only new requests, interrupt scheduled jobs or allow approved operations to continue? Employees need to understand what might still be in flight. Keep outstanding work discoverable and prevent a restart from replaying everything without review. Uncertain outcomes require investigation before further writes, particularly where repeating a task has external consequences.

Exercise recovery with fictional cases. Ask somebody to pause the route, establish what remains pending, use the ordinary process and deliberately resume later. Assess whether the instructions are sufficient. A backup or export helps only when its contents and use are understood. Match the recovery design to the disruption and uncertainty the business can realistically tolerate in the supported workflow.

Reassess when business work changes

Hold periodic substantive reviews: does the agent still support the same task, are the sources correct and do employees use its results? A new service, reorganisation or revised contract can invalidate earlier assumptions. NIST frames AI risk management across the lifecycle; for a small application this chiefly means that ownership and assessment continue after initial development rather than ending at launch.

Decide explicitly whether to expand, revise or retire the application. Low usage does not automatically justify making it a larger project. Investigate whether the underlying problem still exists and whether ordinary software now provides a better route. Effective operation protects more than technical uptime: it keeps the application appropriate for real work, organisational responsibilities and the capacity available to support it.

NIST: AI Risk Management Framework ↗

Turn the explanation into a project decision

Record your assumptions, unknowns and acceptance checks in the matching tool. Use the service guides to discuss delivery and scope.

• AI agent development ↗
• AI chatbot development ↗

Questions about this topic

Continue with a related question