About Integralvision Devon Web Design

28/06/12 - Serious Integralvision Server Outage

Please note: These reports are only updated when an important event occurs. Rest assured that nothing else will take prority until this situation has been rectified. We are literally working round the clock to get things up and running again.

20:30 Thursday night: The alarm sound on my iPhone to say there is a problem with the Number 2 server. Rushed to the office and spend some hours trying to reboot the serve without success.

22:30 Contact our data centre where the server is housed and put in a support request.

01:30 Friday morning: We are informed that the problem is hard drive failure which requires an engineer to fix. The engineers do not work through the night and we had to wait until 8am on Friday for them to arrive.

09:00 Friday morning: The engineers arrrive and they say it is not a hard drive failure. The problems appears to be with the operating system installed on the server which is preventing it booting up. We have a Server Side Engineer (SSE) on hand working on getting the system back up and running as soon as possible. Rest assured that we are doing everything we can to restore services to our clients.

10:40: The fault has been diagnosed as a "Kernel panic" on the server operating system. The work has been escalated to our data centres Linux specialists.

12:20: After further investigation by our data centre's Linux specialists the fault has proved more serious than we thought. It has been recommend that we replace the server completely. We have therefore ordered a new server. While the process of setting this up will be fairly quick, restoring all our client's websites, databases and email will take some time.

13:23: We have been informed that the data centre operations team will be setting up our new server next. This normally takes up to 24 hours but they have promised to do it in 2 hours.

15:29: The data centre operations team are installing the Operating System now so the server will be available very soon. We can then set about installing all the packages required to manage the server. Once everything is up and running we will begin restoring our websites, databases and email. Work will be prioritised in order of importance, for examlpe to restore core elements first.

17:20 Friday night: The operating System is now installed. We have started installing all the packages required to manage the server, email, databases, etc.. This is fairly time consuming as things such as security are very imporant when setting up servers.

01:00 Saturday morning: All the packages required to manage the server, email, databases, etc.. have been installed and the server is operational. We are now looking at accessing the external USB drive that contains all the backups from the previous server.

01:30 Saturday morning: Due to importance of this stage of the proceedings, the decision has been to leave dealing with accessing the backup drive until the morning.

09:00 Saturday morning: Some setbacks mounting the backup drive due to the operating system on the new server being more recent than the one on previous server. As the server is 200 miles away, all this is being done remotely using the equivalent of a text editor!

17:29 Saturday Afternoon: At last solved the issue of mounting the backup drive. All website files are safe and sound you will be glad to hear (this is the first time we were 100% certain of this, which has meant a stressful 24 hours). Unfortunately, accessing the files took the whole day which we had expected to spend restoring client websites.

20:05 Saturday Evening: Have File Transfer (FTP) access restored. Moving websites files across alphabetically.

02:30 Sunday Morning: All client website files moved to the new server. Starting on moving the databases for each site.

07:43 Sunday Morning: Back on moving the databases for each site to new server. We are taking this oportunity to perform a local backup as well of databases. While this is taking longer, when dealing with client data remotely in this way, having a copy stored here makes sense.

15:03 Sunday Afternoon: Some domains have been repointed at the new server. However, as the databases are not yet running this will result in an error page.

18:06 Sunday Evening: Successfully installed and configured database administration tools. We are about to start uploading client databases.

22:00 Sunday Evening: All databases uploaded. A fairly quick job. However, due to the new server running the latest version of all the software packages (PHP, MySQL, etc.) there have been lots of incompatabilities to sort out. It has taken 5 hours to get a single site working. However, the solutions apply to other database driven sites as well.

23:49 Sunday Evening: We are currently repointing client domain names to the new server as we cannot test sites are working correctly unless they are live. Updating our Domain Name Server that handles this aspect of making your websites work is pretty slow as each record has to be accessed and amended individually. Once the domains are all updated, we will get on to adding back in the ownership and permissions on the databases, as unfortunately, this is not one of the things that gets stored in the back ups.

02:56 Monday Morning: Client websites are coming back up pretty quickly now. All sites that we have repointed the domains to the new server should be showing up by now.

Ordinary html websites (i.e. without databases or content management systems) should display without problems. Database driven sites may show blank pages or error code. However, these can be fixed in 1 or 2 minutes. We have a list of these and are working through them as quickly as we can. If your site does not show up at all (i.e. times out), then the domain name has not yet been moved. We are doing these in batches and should have the remaining few domains repointed shortly.

The last job to do will be resetting up the email accounts. This wil be done on Monday. The first thing we need to do is install the email, spam and antivirus software. We won't know until then if we will need to manually reset up each email account, or if the system will simply recognise the original data. We will put up an announcement here how we get on.

Report 08:05 Monday Morning: Clients who domain names we do not manage will have to update the DNS settings themselves, or get their IT people to do it. If you have any problems, email us the log in details for your domain registrar and we will do it for you.

Our new IP address is: 37.220.22.18

If you manage your own email, you will only have to change the section for your website which will look something like:

@ 80.84.53.234
www 80.84.53.234

and you need to change this to:
@ 37.220.22.18
www 37.220.22.18

Remember this is only for clients who registered and manage their own domain names. If we invoice you for your domains, then you do not have to do anything.

09:22 Monday Morning: We are installing the software package that manages email. Unfortunatley this is not Outlook Express and it will take some time to install. We hope to have it up and running in 2 or 3 hours. Once it is working and tested. We will move all backups of email accounts into place. Email accounts should start working straight away. However, if you have been using our email systems as an IMAP account, where you stored the emails on our server, these will take longer to appear in your email program simply because we will have to copy all the backed up emails to the new server. In some cases this runs into gigabyes of email.

11:02 Monday Morning: Had a problem with some databases on sites. The database software on the new server is a more recent version. Although 95% of our client databases handled this fine, a few were not correctly imported and had to be amended by hand. These have all been updated and we are back on installing the email software.

Please note: If your website is displaying a white page, or there is some data missing, please email admin@integralvision.co.uk and let us know. If your website is not resolving on the Internet (i.e. it fails to load at all), please email the same address. Both these problems can be fixed quickly.

14:29 Monday: Websites displaying a white page issue should now be fixed. If your website is not resolving on the Internet (i.e. it fails to load at all), please email integralvision@ me.com as we might have missed repointing your domain at the new server.

Back on installing the email software package. Making good progress. Should be ready for testing by 3ish. If it all works OK, then we will restore the client email accounts from backups.

16:09: A few delays due to having to install some missing components. Not at the testing stage yet but getting close.

23:43: Sorry to not have posted before. Had out heads down trying to solve bugs in the email package that has not been updated to be compatible with the more recent software that came with the new server. These have put us extremely behind with getting email working. We were thinking of reverting to Plan B at midnight tonight but, as we are making good progress now, we will stick with Plan A.

02:30: We managed to get all the email software installed only to find yet more incompatibility with the latest os and database versions on the new server. Bascially we switched it all on, and everytime we tried to add an email account, got errors. We feel that we have no choice now but to switch to Plan B and move client emails to an older server that will not suffer these problems. This will mean repointing the mail record for client domains and then setting up email accounts from scratch. We expect to start this from 8am on Tuesday morning. The actual process of emailing working again will be quick, but it will take time to physically set up the email accounts as they will have to be done individually.

08:25 Tuesday: Informed by client that their email had come through this morning. There were one of the first we had set up on a different mail server. However this proved to be short term and the email delivery was not maintained. Began to have doubts about Plan B being viable.

12:08 Tuesday: We have been trying to book a server system administrator since Monday morning to hep with these problems, but they have all been booked up. We now have one booked and expect them to be looking at the issue around 1pm. He will try to fix the issue on our new server while we carry on trying to set up a temporary solution on Server 3.

16:12 Tuesday: The server system administrator has been hard at it all afternoon. He seems pretty confident he is making progress.

17:52 Tuesday: The server system administrator has just requested an email user account to test the system.

00:04 Wednesday: The server system administrator has been working through things so quickly, solving incompatibilities, etc. that there are too many to put up here.

In the meantime, we have managed to set up a temporary solution on Server 3 and are adding email accounts as quickly as we can. This may also be used as a fallback solution in case of any future issues.
At present this is for POP3 email accounts only where you download email to your computer. If you have an IMAP account where you leave your messages on the server, please disable this and set up a duplicate account as POP3.

13:22 Wednesday: The server system administrator worked until 3am and made more progress and this morning has most of the email system working. Until we are 100% sure of its reliabilty, we will not be resorting to Server 2 for email just yet but will continue with email on our Server 3.

We repointed all emails for client domains to Server 3 last night in preparation and this morning have been setting up the pop accounts and passwords, etc., on that. We are getting through these as fast as we can.

23:20 Wednesday: We have set up 130 email accounts on Server 3. About 50 left to go. Mostly gone well but a few problems due to the change to the IP address. This can take a while to update over the internet so links between systems get confused. If you have any problems sending or receiving, or are still without email please let us know.

10:27 Thursday: As the new server has higher levels of security you might be having log in problems with email. Please look in your email settings and turn on Password Authentication.

14:40 Thursday: 30 email account left to transfer. Expect to have them all done by 5pm. Spending quite a bit of time helping clients sort out minor niggles with email.

01:49 Friday: The new server came complete with all the latest versions of software needed to run the server and host websites. While this is a good thing from the point of view of bug fixes and security, the downside is that some features will have been Deprecated. This is causing some elements on some client's websites to stop working. For example, the sending of contact forms and access to content management systems. You might even have error messages displaying on your site in places. This only applies to sites built using the scripting language PHP.

09:00 Friday morning: We have a programmer looking at what the incompatibilities are (they will likely be the same across all sites), and will be producing fixes that can be applied as soon as possible.

14:05 Friday afternoon: We have successfully applied patches to the majority of client sites and the functionality of contact forms and access to content management system has been restored. We are only aware of one site that still has a contact and booking form problem and are still working on this.

14:30 Friday: We have discovered that the new server does not like the text encoding on the web pages we are hosting. For example, text for inverted commas, copyright, and pound signs may be affected. While this is an easy fix, it has to be applied to every web page of every site. We have made a start but I expect this will take the rest of the weekend.
If your website has odd symbols appearing in place of inverted commas, copyright, pound signs and even dashes, please let us know and we can fix this within minutes.

14:30 Friday: As far as we are aware, the final few email accounts are now back up and running. If this is not the case for you please telephone us on 0845 8387088 and we can usuallly sort out any problems over the phone. Please note that we are very busy dealing with this situation so, if we are unavailable, please leave a message with your phone number and we will look at your email from here and contact you with an update.
The spam filters still need some work so we apologise for any increase in spam. This will improve when we move email to the new server.

15:00 Friday afternoon: I have just been informed that our new server now has email working again. This will be tested over the weekend. Rest assured that we will not be moving clients emails again unless we can guarantee no downtime or loss of emails.

09:00 Monday: After testing with a number of our own email accounts, we are satisfied that Server 2 is dealing with email robustly. We have started moving clients email back, one account at a time. We will NOT move your email without informing you first. Please note that Server 2 has higher levels of security for sending and receiving emails. You will need to turn on passsword authorisation for both sending and receiving. This should just be a case of checking a box and on the receiving option, retyping your username and password. You will not have to do this until we inform you that we are about to move your email back

09:00 Monday:We are currently working on fixing problems with database driven websites caused by the upgrades to PHP (scripting language) and MySQL (database software). If you notice any odd behaviour from your database driven site please email us with details.
Please note that we would have been replacing our server and upgrading all software later this year. This would have been done over a period of months. Fixes to client websites under such circumstances would have been chargeable. This work is now being carried out for free.