Many Pies

Many Pies

Tuesday, November 17, 2026

Why you need a watchdog for your cloud stuff

BBC Micro watchdog

In the last millennium I used to work for a small electronics company doing embedded software. One of the chips we had on our circuit boards was called a watchdog. It needed to be sent a signal every second, and if it didn't get that signal it would reset the whole board, so the system would start up again.

It's designed so that if the worst happens and your system freezes then it won't be long before its up and running again after a reboot. What happens if the watchdog fails? Well, that is a problem, but as it's such a simple device, its very unlikely that it will.

I implemented something similar in my current job a while back, except in software. An overnight job would look for the presence of certain files which should be changing every day or week, as some routine process would update them. For example, it would make sure that every day we had a new file that we'd downloaded from the bank. The same question applies - how would you know if that watchdog process had failed? The answer is that you wouldn't. However it was designed to be simple, so there was less of a chance of failure.

As it checked bank files (allowing for weekends) every bank holiday it would give a warning that the expected file wasn't there, but that was a reassurance that it was up and running.

There's a great article here, the database manager's incident response plan. However, that's dealing with what you do after an incident's happened. What can you put in place before it goes wrong? You could ask yourself - what are the critical things that a) have been reliable and b) could fail silently. 

A watchdog overnight job like the one that I built doesn't help as things become more cloudy and log files are no longer available on network drives. I can't think of a general solution - can you?