The Road Ahead

Happy New Year and best wishes for a prosperous 2019 to all.

Merry Christmas to all our customers and visitors.

Over the last week or so another phishing spam message purporting to be from HM Revenue and Customs has appeared (see screenshot below). We have received it several times and we expect many of our customers have as well. The current version of this message originates from the address donotreply@hmrc-payert-gov.uk

We consider this message to be a phishing or scam message and is malicious. We strongly recommend all customers that receive this message delete it immediately and do not attempt to open or download any attachments.

We haven’t disappeared, we’re still here. The computer gods have just not being favouring us for the last couple of days.

One of our web servers, NENE, has been offline for nearly 48 hours due to events that are beyond our control. As this issue is not likely to be fixed before Friday (at the earliest), we have temporarily moved all affected websites including our own to a different server.

Our standard mail service have not been affected and are working as normal. We intentionally split our Web hosting and mail services exactly for this kind of situation.

So, what happened?

Over the past few weeks, our NENE web server has been misbehaving for no apparent reason. This behaviour was not apparent to our customers or visitors, but noticeable to us via our daily checking and monitoring.

As any investigation could cause service interruption, we held off until last weekend. Unfortunately, despite many hours of work we couldn’t trace the problem and we suspected that Windows itself may have been corrupted beyond easy repair.

As NENE is actually a virtual server, the quickest and easiest solution was to replace it with a brand new virtual machine (we had backed up all data prior to taking the server offline). With virtual servers, the process of installing and configuration is typically quicker than with physical hardware so we aimed only to be offline for a couple of hours.

So we provisioned a new virtual machine, a process that typically takes a few minutes. It didn’t. It took over six hours for our service provider to provision a new server. Once available, the new virtual server was far from healthy, with one of its’ more apparent problems being its’ inability to restart without hanging/crashing.

We’re currently sitting in a very frustrating limbo as our service provider attempts to find a fix. They’ve managed to independently replicate the problem but have yet to discover a cure.

Update 17:00, 4^th September

The latest update we have from our service provider is that they have managed to replicate the problem in their test environment and are currently investigating the underlying cause.

Update 15:30, 5^th September

Our service provider has traced the issue. However, as the fix requires some changes to their infrastructure that are unlikely to be completed before Friday (7^th September).

We will shortly begin migrating Hosted Exchange 2010 mailboxes to Hosted Exchange 2016. In the next week or so, we will be contacting individual Hosted Exchange 2010 customers to discuss and arrange a suitable date for migration.

Migration will occur on an agreed date between 20:00 and 06:00 BST and will require some client and DNS settings to be changed. In the case of the former, any device that currently connects to an Exchange 2010 mailbox may need to have one or two server addresses changed. If we are responsible for managing a customer's domain (i.e. it is registered with us), we will make the necessary DNS changes unless agreed otherwise. If a domain is outside of our control, we will distribute details of the required DNS changes.