Back to overview
Maintenance

London core router upgrade (LON2)

Aug 01 at 09:00am BST  –  Aug 03 at 08:00pm BST
Affected services
API
London, United Kingdom
Metal Provisioning
Resource Cache

Resolved
Aug 03 at 03:00pm BST

From 10:54AM BST we confirmed the transfer over to the new LON2 distribution infrastructure was completed.
We expect there to be no further action required for clients across the network, we will provide a full report of what has improved with this maintenance later this evening.

We have extended the maintenance window to indicate our ongoing effort to monitor traffic to ensure that it is as expected on the new infrastructure.

  • We are currently observing issues with some IPv6 prefixes as well as v6 stability across some upstreams/internal routes. We are investigating the cause of this currently. [FIXED]
  • We have identified a latency issue on DDoS protected transit for any clients that ingress via Amsterdam, we're working with NeoProtect on a resolution for this currently.

Updated
Aug 03 at 11:40am BST

We identified an misconfiguration with unprotected traffic on a client switch. This affected a small subset of users but has been resolved.

We are still investigating an ipv6 issue.

Updated
Aug 03 at 11:15am BST

Issues affecting a select protected block should be resolved.
We are working to resolve some final edge case routing issues with customer BGP services.

Updated
Aug 03 at 11:06am BST

We're aware of the sustained outage on ddos protected transit, this is due to a routing loop in the deployment of the new distribution layer, we are investigating and attempting to resolve.

Updated
Aug 03 at 10:49am BST

We are just doing some final fibre shifts then we will have to sort the route back. This will cause one more drop (probably for 3-5 minutes). After this we should be good to go.

Updated
Aug 03 at 10:06am BST

We're now starting to shift traffic to our new distribution layer in LON2.
There may be some brief blips across transit services while we move.

Updated
Aug 03 at 07:53am BST

We will proceed with these works again shortly. All traffic disruptive work will be completed by 12pm BST at the latest.

Updated
Aug 02 at 04:36pm BST

We have completed the maintenance for today. We will proceed with this again tomorrow at 8AM BST in low traffic hours.

Updated
Aug 02 at 01:20pm BST

We have identified the fault and we are starting to see services recover.

Updated
Aug 02 at 01:10pm BST

We are aware this situation is ongoing, we are currently trying to resolve a bug causing connectivity issues between our two core routers in LON1.

Updated
Aug 02 at 12:12pm BST

An unexpected issue during this maintenance has resulted in one of our core switches unresponsive. We are currently investigating the outage.

Updated
Aug 02 at 08:01am BST

This routing maintence will proceed shortly.

Updated
Aug 01 at 08:53pm BST

Tomorrow at approx 8AM BST we will be making adjustments to our routing policies in LON1 to prepare for the new LON2 setup and allow us to shift traffic.

This is likely to cause a brief interruption (timeout) to customers. Once this is complete we do not expect there to be any further disruption in this maintenance window.

Created
Aug 01 at 09:00am BST

EDIT: We have rescheduled this maintenance to be earlier due to site reliability issues pending this upgrade.

We will be doing some upgrades and work on our LON2 core routing stack to continue to improve our transit capacity as well as resiliency within the site.

Part of this maintenance will include us deploying two new core routers (similar to last years LON1 works).

As the site is fed by independent feeds we expect to be able to do this with no/minimal downtime.
We will provide more exact timeframes as well as updates closer to the time as they become available.