GTT connectivity issue in London
Resolved
Jul 14 at 10:11pm BST
We have confirmed this situation has been resolved and is not expected to reoccur. See the reason-for-outage below:
GTT (London, Poplar):
2025-07-14 21:08:47 BST - detected a short isolation with devices in London, UK, due to a reboot of a line card. Connectivity restored without intervention. Issue to be checked additionally with equipment vendor.
Scaleblade:
2025-07-14 21:05:47 - GTT network fault identified and equipment automatically failed over to RETN as our secondary site upstream. We identified an almost immediate recovery of regular transit blend services after a suspected delayed route purge from GTT's line-card reboot.
2025-07-14 21:06:57 - DDoS Protected services recovered slightly later as they rely on a different GTT based delivery with different failover mechanisms.
2025-07-14 21:11:50 - Observed GTT port-state become active again and system attempted to fail-back. Send (tx) traffic recovered, monitored reduced inbound (rx) traffic with restricted BGP prefix visibility.
2025-07-14 21:33:40 - GTT connectivity is confirmed restored and service is returned to normal/full operational.
What did we learn, what changes do we aim to make?
Its apparent while our active-passive failover solution is effective for an optic failure or "instant" outage, when a reboot or purge issue (as observed today) is present our system takes too long to fail-over.
In future we would like to entirely remove the requirement for egress failover with split routing between upstreams. We aim to implement this as part of our upcoming maintenance to better replicate what we have achieved in USA.
We are not happy with our automated recovery, in this case it did not cause us any further outages due to the type of fault observed on GTT's end, however the worry is with the existing setup we would/could fail-back before a service is confirmed resolved. We will investigate ways to continue our automated failover, but require manual confirmation before fall-back takes place.
What went well?
Fault identification was efficient, and we were able to report the fault to our customers as well as GTT prior to their NOC team issuing notice or identifying the fault their end.
Automated fail-over detection proved to be a valid approach but requires further refinement in order to be complete.
Any questions or concerns reach out to noc@scaleblade.com
Affected services
London, United Kingdom
Updated
Jul 14 at 09:14pm BST
GTT has recovered its state and we are starting to see full visibility across announced prefixes again.
Affected services
London, United Kingdom
Updated
Jul 14 at 09:08pm BST
GTT has acknowledged the major incident as a node isolation case, we are currently awaiting further information on what caused the routing issues.
Affected services
London, United Kingdom
Created
Jul 14 at 09:05pm BST
We have monitored an issue with our London GTT connectivity. Services automatically failed over and we are awaiting further updates from GTT NOC team.
We are currently seeing limited visibility for routes via GTT.
Affected services
London, United Kingdom