Wisdom Multimedia Newsletter, Issue 2, July 2003

WELCOME
This is a special edition containing full details of the recent downtime and the work undertaken to resolve the problems.
12 June 2003, 8 a.m.
The server hosting Wisdom Multimedia accounts crashed causing two major problems:
- e-mails could not be delivered to any Wisdom Multimedia addresses
- web sites hosted on the server could not be accessed.
Staff at the ISP started work on, firstly, identifying the problem and then on re-building the server. I contacted all customers whose accounts are hosted by that ISP via a backup e-mail provider.
12 June 2003, 8 p.m.
ISP staff had created a new working server but they were unable to access backup disks due to the nature of the problem. I started preparing a restore strategy for web sites based on the backups I keep. Backups were retrieved from their off-site location. This also allowed me to identify and remove some redundant files which I did whilst waiting for the go ahead to start restoring.
12 June 2003, Midnight
ISP staff started to re-create accounts on the server and were working with one customer to test their strategies.
13 June 2003, 1 a.m.
I became involved with the testing.
13 June 2003, 2 a.m.
ISP staff identified a method to retrieve data from a combination of discs and backups. They started the restore in the order that customers had signed up which meant that we were near the front of the queue. I checked the files as they were being restored and identified those where it was necessary to restore from my backup.
13 June 2003, 7 a.m.
All my restores were complete and web sites were available.
13 June 2003, 7:30 a.m.
Go ahead given to re-create E-mail accounts.
13 June 2003, 9 a.m.
All my e-mail accounts had been recreated and I left for my day job.
13 June 2003, 7 p.m.
Work started re-creating and populating mailing lists.
13 June 2003, 8 p.m.
Work started re-creating e-mail forwarders and auto responders.
13 June 2003, 9:15 p.m.
Websites counters were recreated. If customers were able to provide statistics the counters were set to a known total otherwise the date was set and a new count started.
13 June 2003, 10:15 p.m.
There was still a problem with the data files for the counters and this was resolved in consultation with the helpdesk staff.
Countdowns to specific dates could not be restored and were showing random information. The problem was logged as a low priority since they were still restoring other sites.
13 June 2003, 11:30 p.m.
All problems were resolved. All customers were e-mailed and I finally went to sleep [litres of black coffee only work for a certain length of time at my age].
SUMMARY
The ISP's Managing Director, who was available on the support forum throughout the problem, assures me that the problem was not virus or hacker related and was caused by a hardware failure. The motherboard developed a critical fault during a backup procedure resulting in the corruption of both the live disc and the backup disc. I have no reason to doubt this and some of their staff worked continuously from 8 a.m. 12 June until well after my problems were resolved. I was impressed with both their level of communication and their commitment to providing a restore from the corrupt discs. Their conditions of use make it clear that clients are responsible for their own backups. I keep full on-site and off-site copies of all files. The ISP staff worked well beyond their legal responsibilities. They are going to review their backup procedures with clients and I have already talked with them about this. If any customers feel they would like to review the position and move to another ISP I would advise against it but will be happy to discuss alternatives.
Wisdom Multimedia was formed in 1988 by Colin Edwards as a Sole Trader
12 Rose Terrace, Greenside, RYTON, Tyne and Wear, NE40 4SA