London core router upgrade (LON2)
Resolved
Aug 03 at 03:00pm BST
From 10:54AM BST we confirmed the transfer over to the new LON2 distribution infrastructure was completed.
We expect there to be no further action required for clients across the network, we will provide a full report of what has improved with this maintenance later this evening.
We have extended the maintenance window to indicate our ongoing effort to monitor traffic to ensure that it is as expected on the new infrastructure.
- We are currently observing issues with some IPv6 prefixes as well as v6 stability across some upstreams/internal routes. We are investigating the cause of this currently. [FIXED]
- We have identified a latency issue on DDoS protected transit for any clients that ingress via Amsterdam, we're working with NeoProtect on a resolution for this currently.
Affected services
Updated
Aug 03 at 11:40am BST
We identified an misconfiguration with unprotected traffic on a client switch. This affected a small subset of users but has been resolved.
We are still investigating an ipv6 issue.
Affected services
Updated
Aug 03 at 11:15am BST
Issues affecting a select protected block should be resolved.
We are working to resolve some final edge case routing issues with customer BGP services.
Affected services
Updated
Aug 03 at 11:06am BST
We're aware of the sustained outage on ddos protected transit, this is due to a routing loop in the deployment of the new distribution layer, we are investigating and attempting to resolve.
Affected services
Updated
Aug 03 at 10:49am BST
We are just doing some final fibre shifts then we will have to sort the route back. This will cause one more drop (probably for 3-5 minutes). After this we should be good to go.
Affected services
Updated
Aug 03 at 10:06am BST
We're now starting to shift traffic to our new distribution layer in LON2.
There may be some brief blips across transit services while we move.
Affected services
Updated
Aug 03 at 07:53am BST
We will proceed with these works again shortly. All traffic disruptive work will be completed by 12pm BST at the latest.
Affected services
Updated
Aug 02 at 04:36pm BST
We have completed the maintenance for today. We will proceed with this again tomorrow at 8AM BST in low traffic hours.
Affected services
Updated
Aug 02 at 01:20pm BST
We have identified the fault and we are starting to see services recover.
Affected services
Updated
Aug 02 at 01:10pm BST
We are aware this situation is ongoing, we are currently trying to resolve a bug causing connectivity issues between our two core routers in LON1.
Affected services
Updated
Aug 02 at 12:12pm BST
An unexpected issue during this maintenance has resulted in one of our core switches unresponsive. We are currently investigating the outage.
Affected services
Updated
Aug 02 at 08:01am BST
This routing maintence will proceed shortly.
Affected services
Updated
Aug 01 at 08:53pm BST
Tomorrow at approx 8AM BST we will be making adjustments to our routing policies in LON1 to prepare for the new LON2 setup and allow us to shift traffic.
This is likely to cause a brief interruption (timeout) to customers. Once this is complete we do not expect there to be any further disruption in this maintenance window.
Affected services
Created
Aug 01 at 09:00am BST
EDIT: We have rescheduled this maintenance to be earlier due to site reliability issues pending this upgrade.
We will be doing some upgrades and work on our LON2 core routing stack to continue to improve our transit capacity as well as resiliency within the site.
Part of this maintenance will include us deploying two new core routers (similar to last years LON1 works).
As the site is fed by independent feeds we expect to be able to do this with no/minimal downtime.
We will provide more exact timeframes as well as updates closer to the time as they become available.
Affected services