The Road Ahead

We will be performing some routine server maintenance from around 20:00 tonight (UK or GMT time). There may be a slight impairment of our services whilst this work is taking place.

As always and updates or changes will be posted here and on our Twitter feed.

Update 11:16, 23rd January 2018

All maintenance work has been successfully completed.

We haven’t disappeared, we’re still here. The computer gods have just not being favouring us for the last couple of days.

One of our web servers, NENE, has been offline for nearly 48 hours due to events that are beyond our control. As this issue is not likely to be fixed before Friday (at the earliest), we have temporarily moved all affected websites including our own to a different server.

Our standard mail service have not been affected and are working as normal. We intentionally split our Web hosting and mail services exactly for this kind of situation.

So, what happened?

Over the past few weeks, our NENE web server has been misbehaving for no apparent reason. This behaviour was not apparent to our customers or visitors, but noticeable to us via our daily checking and monitoring.

As any investigation could cause service interruption, we held off until last weekend. Unfortunately, despite many hours of work we couldn’t trace the problem and we suspected that Windows itself may have been corrupted beyond easy repair.

As NENE is actually a virtual server, the quickest and easiest solution was to replace it with a brand new virtual machine (we had backed up all data prior to taking the server offline). With virtual servers, the process of installing and configuration is typically quicker than with physical hardware so we aimed only to be offline for a couple of hours.

So we provisioned a new virtual machine, a process that typically takes a few minutes. It didn’t. It took over six hours for our service provider to provision a new server. Once available, the new virtual server was far from healthy, with one of its’ more apparent problems being its’ inability to restart without hanging/crashing.

We’re currently sitting in a very frustrating limbo as our service provider attempts to find a fix. They’ve managed to independently replicate the problem but have yet to discover a cure.

Update 17:00, 4^th September

The latest update we have from our service provider is that they have managed to replicate the problem in their test environment and are currently investigating the underlying cause.

Update 15:30, 5^th September

Our service provider has traced the issue. However, as the fix requires some changes to their infrastructure that are unlikely to be completed before Friday (7^th September).

Some of our customers may experience a slightly impaired mail service this morning due to an ongoing issue with our data centre provider.

At midday today (23rd June 2018), our data centre provider informed us that an issue has been detected with the data storage system used by our standard mail services. They are investigating this issue and will be working to fix it as soon as possible.

At the time of writing, our standard mail services are working as expected. We continually monitor all our services and you may view our live service status at https://status.calzadamedia.com

Update 13:00

The latest updated we've received from our provider is that they've detected a drop in performance which they are investigating. We have been warned that performance may be degraded on a sporadic basis.

Our own monitoring system have yet to detect any impairment to our standard mail systems and they are working normally.

Update 14:48

Problem has been traced to a faulty disk that will be replaced. There is no impact to ongoing service performance and there is no indication to loss of data.

We will be doing some general housekeeping work on our servers tonight from around 19:00 to 21:00. During this period, normal service may be slightly impaired. To avoid any potential conflicts or issues, we request that hosting customers refrain from uploading files or content during this period.

This work only impacts our standard mail and hosting services. Hosted Exchange services will not be affected.

You may track the current status of all our services via our status website at https://status.calzadamedia.com/

Once again, Happy New Year to all our customers and visitors. We wish you all a happy and prosperous new year.

We will be doing some general maintenance on our servers tonight (28th November 2017) between 22:00 to 23:00. As is almost inevitable, we will have to restart each server at least once to apply some updates and other software upgrades.