The mysterious case of the CDN routing drop

Sometimes, it’s the relative small things that draw your attention to the enormous power of AI. Recently, I was alerted to an issue with my website by my monitoring systems. According to the alert my website was down, and when I checked it, it did indeed appear to be unreachable. When I dug deeper however, the site itself still seemed to be serving - analytics showed hits on the site, and I could access it fine on my mobile (which happened to be connected to a VPN at the time).

Time to break out Claude for a debugging session. A few traceroute calls later and we had our first clue - access to my CDN providers network was dropping at the boundary between them and BT (my ISP). Oddly, everything worked fine if I VPN’d from outside the UK however. I duly raised an issue with the CDN provider and received a pretty fast response along the lines of “thanks for raising the ticket, we are aware of a few UK ISPs transiently blocking access to our network” (my emphasis).

Feeding that support response back to Claude, I got an immediate “ah ha!”. Claude had a theory, “but I need to validate it first - hang on”. The game was afoot! Soon enough Claude returned with an explanation and some data to back it up. We had monitoring showing some similar incidents over the past week or so, each event lasting for around 130-135 minutes. Claude’s insight was that the timing of these events precisely aligned to Premier League football match kick-offs. UK ISPs are required, by a court order obtained by The Football Association Premier League Limited, to block certain IP addresses streaming the matches illegally, for the duration of matches. It seemed our CDN provider had some dodgy customers and we were caught up in the blast radius.

As it was Saturday afternoon while we were debug the issue, Claude suggested that we could test the hypothesis. The current match was near finishing and another match was due to start around 15 mins after that, if our conclusion was correct, then we should see the block removed at the end of the match and happen again when the new match started. And that is precisely what we saw - the block released and restarted within a couple of minutes of the schedule stop/start times of the matches. The game was up.

I pinged my CDN provider a curtesy note to let them know what I found (knowing, full well, that they would already be well aware of why they were being blocked). 45 minutes later and my CDN provider had been taken out of the loop in serving my site, and I’ve had no problems since.

Reflecting on this after it was resolved, I wondered how long it might have taken to identify this issue without AI in the loop. The CDN provider definitely were not going to tell me. Sure, it might have been feasible to recognise the timings and form a theory about the potential alignment for football fixtures (although I am not a football fan so it would not have occurred to me naturally). Certainly, if I’d made that connection, finding out about the ISP blocking mandate would have been a trivial Google search. But, I think the reality is it would have taken several days of investigation and fruitless searching, rather than the couple of hours it took from flagging the event to resolving it, with Claude’s help.