I’ve had an interesting problem for the last few days. I can’t get to the Hack in the Box site, HITB.org, or the HITB NL site from my home near London. Turns out I can’t get to the THC.org site or rokabear.com either. That makes four hacking conferences who’s sites I can’t get to. And I’m not the only one, since apparently a number of people who are using Virgin Media in the UK as their ISP can’t get to these sites, while other people on other ISP’s in Britain can get to all four of these sites. I can even get to them if I log into my corporate VPN, just not while the traffic is flowing out through my home network. I’m not going to accuse Virgin Media of blocking these sites, but I’m also not ruling chicanery on their part out as a cause either. I also make no claims that I poses the network kung-fu to verify that any of my testing is more than scratching the surface of this problem.
So here’s how this all started: Yesterday morning I decided I saw a tweet that the early bird sign up for Hack in the Box Amsterdam was going to end soon. I know some of the organizers of the event, I’ve wanted to go for a long time, so I decided to get my ticket early and save the company a few bucks. I opened up a new tab in Chrome, typed in haxpo.nl and … nothing, the request timed out. Hmm. Ping gave me an IP, so the DNS records were resolving, but the site itself was timing out. I switched to the work computer, to find the same thing was happening. The I logged into the corporate VPN and tried again, suddenly everything worked. Curious.
At first I thought this might be a stupid DNS trick played at the ISP, so I changed my DNS resolvers to a pair of servers I have relative certainty aren’t going to play tricks, Google’s 22.214.171.124 and the DNS server from my old ISP back in the US, Sonic.net (who I highly recommend, BTW). This didn’t change anything, I still couldn’t get to HITB. I had to get working, so I did what any smart security professional does, I threw up a couple of tweets to see if anyone else was experiencing similar issues. And it turns out there were a number of people, all using Virgin Media, who had the identical problem. This is how I found out that THC and Rokabear are also not accessible for us.
As yesterday went by, I got more and more confirmations that none of these hacking sites are available for those of us on Virgin Media. At first I thought it might simply be VM blackholing the sites, but VM’s social media person sent me a link to review who was being blocked by court order by Virgin Media. I didn’t find any of the hacking sites listed in this, besides which Virgin Media actually throws up a warning banner page when they block a page, they don’t simply blackhole the traffic. They will limit your internet access if they feel you’re downloading too many big files during peak usage hours, but that’s a discussion for another day.
The next step was tracert. I a little chagrined to admit I didn’t think of tracert earlier in the process, but to be honest, I haven’t really needed to use it in a while. What I found was a bit interesting (and no, you don’t get the first two hops in my network chain, you have no need to know what my router’s IP is).
Tracing route to www.hitb.org [126.96.36.199]
3 9 ms 7 ms 7 ms glfd-core-2b-ae3-2352.network.virginmedia.net [188.8.131.52]
4 11 ms 7 ms 7 ms popl-bb-1b-ae3-0.network.virginmedia.net [184.108.40.206]
5 10 ms 11 ms 10 ms nrth-bb-1b-et-700-0.network.virginmedia.net [220.127.116.11]
6 11 ms 15 ms 14 ms tele-ic-4-ae0-0.network.virginmedia.net [18.104.22.168]
7 13 ms 16 ms 14 ms be3000.ccr21.lon02.atlas.cogentco.com [22.214.171.124]
8 16 ms 14 ms 16 ms be2328.ccr21.lon01.atlas.cogentco.com [126.96.36.199]
9 17 ms 15 ms 16 ms be2317.mpd22.lon13.atlas.cogentco.com [188.8.131.52]
10 88 ms 102 ms 103 ms be2350.mpd22.jfk02.atlas.cogentco.com [184.108.40.206]
11 99 ms 100 ms 91 ms be2150.mpd21.dca01.atlas.cogentco.com [220.127.116.11]
12 97 ms 94 ms 96 ms be2177.ccr41.iad02.atlas.cogentco.com [18.104.22.168]
13 102 ms 100 ms 105 ms te2-1.ccr01.iad01.atlas.cogentco.com [154.54.31..62]
14 101 ms 210 ms 211 ms te4-1.ccr01.iad06.atlas.cogentco.com [22.214.171.124]
15 90 ms 91 ms 99 ms edge03-iad-ge0.lionlink.net [126.96.36.199]
16 90 ms 94 ms 98 ms 188.8.131.52
17 nlayer.lionlink.net [184.108.40.206] reports: Destination net unreachable.
Rather than doing what I thought would be the logical thing and simply hoping across the channel and hitting Amsterdam fairly directly, my traffic leaves the VM network through Cogent Networks, hits a few systems in the US owned by a company called Lionlink Networks LLC and dies. So my traffic leaves the UK, travels to Switzerland, then to the US, over to Washington DC and then dies. And this happens with four separate hacker conference sites, but doesn’t appear to happen anywhere else. Oh, and all four hacking sites take the same basic route and all die shortly after hitting LionLink. Hmmmm.
I know I’m a professional paranoid. I know how BGP works and that it’s not unusual for traffic to bounce around the internet and go way, way, way, out of what a human would consider a direct route, but the fact that all four EU hacking sites all route back to the US and that they all die when they hit Lionlink is more than a little suspicious to me. It’s almost like someone is routing the traffic through Switzerland and the US so it can be monitored for hacker activity, since both countries have laws that allow for the capture of traffic that transgresses their borders. But of course, that would just be paranoid. Or it would have been in a pre-Snowden world. In a post-Snowden world, I have to assume most of my traffic is being monitored for anomalous behavior and that the only reason I noticed is because someone at Lionlink screwed up a routing table, exposing the subterfuge. But that would just be my paranoia speaking, wouldn’t it?
I’m hoping someone with deeper understanding of the dark magiks of the Internets can dig into this and share their findings with me. It’s interesting that this routing problem is only happening to people on Virgin Media and it’s interesting that the traffic is being routed through Switzerland and the US. What I have isn’t conclusive proof of anything; it’s just an interesting traffic pattern at this point in time. I’m hoping there’s a less sinister explanation for what’s going on than the one I’m positing. If you look into this, please share your findings with me. I might just be looking at things all wrong but I want to learn from this experience whether I’m right or not.
Thanks to @gsuberland, @clappymonkey, @sawaba @tomaszmiklas, @module0x90 and others who helped verify some of my testing on twitter last night. And special thanks to @l33tdawg for snooping and making sure I got signed up for HITB.
Update – And here it is, a much more believable explanation than spying, route leakage. So much for my pre-dawn ramblings.
From Hacker News on Ycombinator:
This is a route leak, plain and simple. Don’t forget to apply Occam’s Razor. All of those sites which are “coincidentally” misbehaving are located in the same /24.
This is what is actually happening. Virgin Media peers with Cogent. Virgin prefers routes from peers over transit. Cogent is turrible at provisioning and filtering, and is a large international transit provider.
Let’s look at the route from Cogent’s perspective:
BGP routing table entry for 220.127.116.11/24, version 2031309347
Paths: (1 available, best #1, table Default-IP-Routing-Table)
54098 11557 4436 40015 54876
18.104.22.168 (metric 10105011) from 22.214.171.124 (126.96.36.199)
Origin incomplete, metric 0, localpref 130, valid, internal, best
Community: 174:3092 174:10031 174:20999 174:21001 174:22013
If Cogent was competent at filtering, they’d never learn a route transiting 4436 via a customer port in the first place, but most likely someone at Lionlink (54098) is leaking from one of their transit providers (Sidera, 11557) to another (Cogent, 174).
Also, traffic passing through Switzerland is a red herring — the poster is using a geoip database to look up where a Cogent router is. GeoIP databases are typically populated by user activity, e.g., mobile devices phoning home to get wifi-based location, credit card txns, etc. None of this traffic comes from a ptp interface address on a core router. GeoIP databases tend to have a resolution of about a /24, whereas infrastructure netblocks tend to be chopped up into /30s or /31s for ptp links and /32s for loopbacks, so two adjacent /32s could physically be located in wildly different parts of the world. More than likely, that IP address was previously assigned to a customer. The more accurate source of information would be the router’s hostname, which clearly indicates that it is in London. The handoff between Virgin and Cogent almost certainly happens at Telehouse in the Docklands.
If someone were, in fact, trying to intercept your traffic, they could almost certainly do so without you noticing (at least at layer 3.)