I've been pondering for a while the best way to see if the various websites that I'm responsible for are up and running. When I researched this a while back, there were any number of companies that would test them for you, for a fee. However I couldn't find any tools.
Today I came across Selenium which seems to be a pretty powerful set of tools. There's an IDE for writing tests, a component for testing on your webserver (Selenium core), a component for testing on a given remote machine (Selenium RC) and a component for handing out tests to arbitrary remote machines (Selenium Grid). You need to have machines with your target browser on it somewhere, so if you want to test your site in IE, you need IE installed on a machine.
The software is at various 0.x versions, so doesn't look that mature, but on the other hand it has been adopted by Google: Google Open Source Blog: Selenium Users Event Coming Up, so that's a good recommendation.
Time to go and write some test scripts...