How to keep an always-on AI agent from breaking when you update it

An always-on agent never gets a quiet moment to be fixed. It answers messages at 3 a.m., runs jobs on a timer and keeps notes from one day to the next. So the riskiest minute in its life is rarely a hard task. It is an update.

This guide is a routine for making that minute boring. It works for one agent on one server, and it holds up as you add more. Keep a way back, change one thing, let the machine restart itself within limits, check health before you call it done, and roll back without waiting for anyone.

Why updates break agents

Most breakage after an update falls into three groups. A dependency moved and the agent no longer starts. The agent starts but has lost something it needs, such as a login, a setting or its memory files. Or it runs, and quietly does the wrong thing because a default changed.

The first group is loud and easy to spot. The other two are quiet, and quiet is what hurts an agent that people rely on while they sleep.

Keep a way back before you start

Before any change, take a snapshot you can return to in one step: the program version, its settings and the files it writes to. A copy that takes an hour to restore is not a way back. It is a project.

Then write down what working means for this agent. One message it must answer, one job it must finish, one file it must still read. You will need that list later.

Change one thing at a time

Bundle five changes, one of them breaks, and you have five suspects. Ship them one by one, on one machine first, and give each some time before the next. A test machine that looks like the real one is worth more than a long checklist.

Let it restart, within limits

Crashes happen. What matters is what the machine does next. On Linux, the service manager can bring the agent back by itself, and stop trying when it keeps failing, so a broken update does not spin forever. A generic example:

# agent.service: a generic example, not a real setup
[Unit]
Description=Always-on agent
After=network-online.target
Wants=network-online.target
# Give up after 5 failed starts in 5 minutes
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
ExecStart=/usr/local/bin/my-agent
# Come back after a crash, but not after a clean stop
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

The limits matter as much as the restart. Five tries in five minutes, then stop, is a better default than trying forever.

Check health before you call it done

An agent that started is not an agent that works. Run the list you wrote before the update, every time, in the same order.

1. The service is running and has not restarted in the last few minutes.
2. It answers a test message, start to finish.
3. It can still sign in to every account it uses.
4. Its memory files are there, and it can read them.
5. The next scheduled job runs on time.

If a check fails, roll back first and look for the cause second. The people who use the agent need it working tonight. The reason can wait until morning.

Roll back without waiting for anyone

Decide in advance who may roll back and when, so nobody waits for permission at 3 a.m. The best rollback is automatic: a check fails, the old version comes back, and a person gets a short note saying what happened.

That completes the routine. Snapshot, change, restart within limits, check, roll back. Done the same way every time, an update stops being an event.