End-to-end deployment monitoring in Umbraco

I’ve argued before that giving agents the tools and access to monitor and verify their own work, end to end, is one of the keys to agentic engineering that works.

Intelligence isn’t the bottleneck anymore, and the gap between frontier models matters less than it used to. What makes the difference now is how you tie the pieces together, how you steer, and how much autonomy you give the agent.

I maintain the umbraco.com website, and one piece was missing from my own workflow there: the production deploy. Whenever I deployed and something went wrong, I had to explain it to the agent myself and point it at whatever I suspected was the culprit. That took time, time the agent could have spent investigating a problem it already knew was happening. So I started looking into letting the agent watch the deploy with me, and giving it eyes on what was going out and what happened while the deploy ran.

That’s how deploy-watch was born.

On Umbraco Cloud, Live is promoted from the portal. Git knows nothing about it, CI knows nothing about it, and nothing tells you a deploy has started, or that it failed. So instead of asking the pipeline, the agent watches the environment. A deploy causes changes nothing else can: a new process, a document type GUID that didn’t exist a minute ago, a class name in the rendered markup. Watch for those and the environment becomes the status page. It’s the only one that tells the truth anyway.

That took three pieces: a bookmark in git, the Umbraco CLI to read the environment, and a skill that turns what it reads into a verdict.

Git knows what is going out

The first question before any deploy is “what is live right now?”, and on Cloud main can’t answer it. Everything merged to main lands on dev automatically. Live is a manual promotion the repo never hears about.

The fix is an annotated git tag, deploy/live/<date>-<time>, on the exact commit Live runs. It moves only after a deploy is verified, never before. A small script lists every PR merged since the tag, splits them into site changes and repo-only changes such as CI and docs, and flags any PR that changes package versions, because Umbraco packages run their own migrations on first boot. That list is the deploy’s scope, and everything the watch checks is derived from it.

The CLI reads the environment

None of this works without a way to read, and carefully write, the environment from the terminal. This is where the Umbraco CLI comes into play. Every step that touches Umbraco goes through it, against a named profile per environment:

• deploy status compares the repo’s schema artifacts with the environment and reports drift. Deploy’s schema pass skips artifacts that are in sync and processes the ones that have drifted, and processing is where things break. So the drift list is the risk list, in one command.
• deploy watch owns the phase machine. It baselines the process id, the health paths and the search indexes, then reports baseline, restarting, app-alive, serving, landed, settling, and finally verified, failed or timeout. It only says verified once the environment has stayed healthy for a full settle window. One passing sample is not verification, because a deploy often isn’t finished when the site first comes back: on Cloud it can restart again, or keep processing in the background. My favourite signal is the earliest one: deploy watch knocks on the management API’s token endpoint without credentials. While the app is down it gets a 503. The moment the new process is up, it gets a 4xx, before public pages come back, and on Cloud often before a second restart. Any 4xx means something is running and answering, which is exactly what you want to hear.
• deploy watch --logs carries the deploy’s log entries in the same stream, with the known noise filtered out, because you can’t investigate afterwards what you never watched. logs search digs further when something does go wrong.
• deploy watch --uda-dir checks the schema tripwires itself. It notes which schema files differ from the environment before the deploy, then confirms each one arrived after Deploy’s schema pass. indexer list and health cover the post-landing checks.
• document update --dry-run and document publish handle the content a schema change sometimes needs afterwards, with a backup taken first and the diff read before anything is applied.

deploy watch touches no pipeline or portal API, so it works the same on Umbraco Cloud and on-prem.

How a watch runs, step by step

The CLI lets the agent see the environment, and the deploy-watch skill decides what it looks at and when it’s allowed to call the deploy done. It’s a short set of rules and a procedure the agent follows, step by step.

Before the deploy:

• Scope. Read the bookmark and list what is going out, in plain terms.
• Tripwires. For each kind of change, pick one observable delta only this deploy can cause: a new document type’s GUID, a class name in the rendered markup, a response only the new C# code produces. Each one is checked absent on the target first, because a tripwire that’s already true is not a signal.
• Landmines. Run deploy status and cross-reference the drift with known failure classes, so the risk is stated while the deploy can still be held: expected clean, or a named risk with a fallback ready.

Then it arms the watch, one command that follows the environment and the deploy’s logs together, tells me it’s waiting, and I press deploy. The rules that matter from there:

• Silence must only mean “still deploying”. The watch reports every terminal state, failure as well as success, and if the timeout runs out before anything is verified, it tells me so instead of assuming all went well. With --heartbeat it keeps printing while it waits, so I know it hasn’t just gone quiet.
• Never write to an environment without an explicit go. The agent may propose a fix; a human approves it.
• The agent that armed the watch runs it. Handing the watch to a sub-agent puts a layer between the first failure signal and the context that has to act on it. That doesn’t mean the agent has to fix everything itself: once it knows what broke, handing the fix to a sub-agent is fine. It’s the watching that stays with the agent that armed it.
• Verified ends the job. Everything is disarmed, the bookmark moves, and the watch has no mandate past a clean deploy. Ongoing production monitoring is a separate decision, not the residue of a deploy watch.

A real run

You start it by typing /deploy-watch, or by telling the agent “I’m deploying to live, stand by”. Here’s what a real run against Live looks like, unedited. This run predates CLI 0.4.26, which folded the log stream into the watch itself.

Once it’s verified, the agent goes through the tripwires, the logs and the health checks and reports back. If the schema change needs follow-up content, it prepares that too through the Umbraco CLI, shows me the dry run, and only applies it when I say go.

Try it

• Install the CLI: brew install --cask albanistrefi/tap/umbraco-cli, then set up a profile per environment.
• Grab the deploy-watch skill and run /deploy-watch before you press deploy.