Sky Cable Philippines’ broken internet connection

I’ll open with this; chances are if you are reading this from a Sky Cable Philippines Broadband connection then it probably took well over a minute to load the page, that’s if the page even opened at all! The reason for this is that this website is hosted in the UK and to connect from your Sky Cable connection to this server you would have likely been routed via ae-13.r01.tkokhk01.hk.bb.gin.ntt.net (129.250.5.31) which amongst a growing number of other hops seems to grind to a halt. You would be forgiven in thinking “If this is happening outside of the Philippines then why are you singling out Sky Cable?”, the answer is because this ONLY happens with Sky Cable and not any other Philippines ISP, and this issue is starting to affect many other websites including Apple’s download servers, Amazon S3 Hosting, and even Sky Cable’s own website hosted in Singapore. All of these sites when served through certain CDNs in Hong Kong or Singapore will not be able to serve you traffic above 20~100KB/sec, or in the case of Apple a slightly better max speed of about 300 KB/sec. That’ll take you well over 5 hours to download an iOS update!

Let me go into more details, and let me get some things out the way first, because I am sure there will be things you are thinking before we get to the end of this post.

    My current line speed with Sky Cable in the Philippines is 55Mbps which works out at around 6.875MB/sec when downloading files from the internet. If you do not know the difference between Mbps and MB/Sec then please go read this.
    I also have a 5Mbps PLDT line which maxes out with a file download speed of about 600 KB/Sec (Just over half a megabyte a second).
    That means the PLDT line is 11 times slower than the Sky Cable line, yet as you will see later on, the PLDT line can reach download speeds on problematic servers well above what Sky Cable can.
    All my tests are fully repeatable at ANY time of the day, there is zero argument for network traffic being the cause of these issues.
    These issues have been confirmed by two other people using Sky Cable lines.
    I had been trying to get Sky Cable to acknowledge this issue for nearly a year without anyone understanding what I’m talking about.
    Finally a month ago I started speaking with one of the head engineers for Sky Broadband via Facebook who initially acknowledged and confirmed the issues. I’ll be referencing some of my conversations with him later in the post.
    NOTHING has been done to fix these issues and I am sure they are affecting the majority of Sky Broadband customers in some way, the only reason why this has gone unnoticed for so long is because the median speed of Philippines internet users is so slow that everyone is used to things taking an age to download.
    I’m certain that if this was a UK or USA ISP that it would be all over Social Media, and in the USA you’d probably see a class action lawsuit or something due to the lack of coming anywhere close to providing a service to these servers that could be deemed reasonable.

Where did all this start? Why are Sky Broadband downloads from Apple so slow?

About a year ago I had noticed that downloads from Apple servers were going really slow at about 50 KB/Sec, I took to Twitter to see if anyone else had the same issue and sure enough there were a couple of posts talking about the same problem, so I responded…

Side note: Let me get one thing off my chest now. Sky Cable’s technical support is diabolical. They are very nice, they take the time to respond but they literally have no technical training whatsoever. It doesn’t stop at the support guys on Twitter though, even if you call them and have them send out a network technician you will end up with a guy coming round your house who has had about 2 hours on-boarding training and has been told that if things don’t work to “reset the router”. Reset the router will forever be burned into my eyes and ears as it is Sky Cable’s only solution to ANY problem. Having relationship issues? Reset the router. Got diarrhea? Reset the router. The devil has appeared in front of you and is about to cast you down to the depths of hell? Reset the router my friend 😉

Right, so back to Apple. As I mentioned I saw that the issue was only happening on my Sky Cable line but it was happening to ANY Apple download server. So that would be my Macbook updating, my iPhone updating whilst connected to my home Wifi, etc. Obviously when I took my iPhone off the home network and just using Globe’s LTE the downloads would fly down, how odd.

From there I thought that perhaps Sky Cable were being sneaky and were throttling traffic for some reason, so I loaded up my VPN and connected to a LA server. Sure enough the download speed from Apple servers ON THE VERY SAME LINE jumped up from around 50KB/Sec to about 3MB/Sec – over 60 times faster!

There is also a YouTube video that shows this very thing happening. Slow speeds to Apple, VPN on, Slow speeds go away. (https://youtu.be/Wl5t7J8YBuo)

I showed this to Sky Cable, I showed this to the technicians they sent to my house, and we went round and round in circles for about another month before I just gave up and hoped that somehow it would get fixed. How wrong I was, the issue was just going to get worse.

More websites start getting slower and slower.

So between late last year and now I noticed that more and more websites were getting slower, not just page loads but actual downloads. I posted in the Philippines Internet Facebook group but unfortunately my post was met with confusion and the majority of discussions are currently around Mobile Broadband, however it did catch the eye of a Sky Cable head Engineer (who we’ll just call “*Joe”) and we got talking about the problem and trying to troubleshoot it. The next thing that I found out about the problem was pretty confusing for me, it would seems that changing the DNS server used would have a direct impact on increasing download speeds, but some of these DNS servers would work for a little while and then stop working. I decided to test these DNS servers from my Sky Cable line and also test them against my PLDT line, which if you remember has a max speed which is 11 times slow than my Sky Line.

*Just a note about Joe, I’ve got to say that the guy has been more than nice in entertaining me when asking these questions. He could have easily just kept quite but he is at least reaching out to aide in some way. So if you ever read this, Joe, I do say thank you for your help despite my frustrations with Sky Cable as a company.
SkyCable Speed Tests

Click this image to see a full size version (https://i0.wp.com/www.chewie.co.uk/wp-content/uploads/2017/08/Screen-Shot-2017-08-19-at-21.43.54.png?resize=1000%2C597)

Now the image above is pretty in your face, but it’s actually quite easy to understand, a few points…

    I tested the same set of servers on both my SkyCable line and my PLDT line 10 times
    Each test set was tested with the DNS settings changed. I used the 10 fastest DNS servers from my location as per a DNS tester.
    These test were done late at night around midnight, network traffic congestion at this time is almost none existent.
    The servers I tested were…
        Fast.com (This was a test control server to show full speed line capability)
        NVIDIA Driver download (This was a test control server to show full speed line capability)
        Apple update server (which seems to be hosted in Singapore)
        Put.io Hong Kong (a simple hop from the Philippines)
    I also included a couple of lines of the TraceRoute to see if there was anything obvious that was going on.

So what does this table actually show us? Well there are two things that stand out straight away.

    With the problematic servers my PLDT line (11 times slower than the Sky Cable line remember) almost ALWAYS maxed out at around 500 KB/Sec whilst the Sky Cable line could barely get above 90KB/Sec in some cases.
    Simply changing the DNS server used on the Sky Cable line can result in much faster downloads if you manage to get the right DNS server. However I can tell you from my own experience that this solution doesn’t tend to last very long.
    The DNS Server switch might just be a red herring for Apple servers. I think there is perhaps two separate issues which is everything Apple (~200KB/Sec), and then everything not Apple (< 20KB/Sec)

So with the above data I went back to Joe (the Sky Cable engineer) and showed him what was going on. I asked a couple of questions…

Why are Apple Downloads so slow with Sky Cable? and How come PLDT aren’t having the same issues? This was his response…

    We’ve escalated the issue with Apple. we can get more bandwidth through multiple connection. seems that they are limiting our connection.

I followed up on this and asked for some clarification and was told…

    For Apple, seems that they have limited the speed per stream. And the updates are http. We can go as high as 2.5mbps per tcp connection. Or about 10mbps for 4 tcp connections. We also think that PLDT has a transparent caching for http traffic that’s why you can have the full bw. For other sites we are looking at direct peering with thier respective CDNs in HK or SG.

Ok, so a response at least and Sky are looking into it. However here is a question, why on earth would Apple limit the download speed to a national ISP? Especially when Apple make billions of dollars through their App store and iTunes stores. What possible sense does that make?

Secondly, if PLDT have this so called transparent cache then why don’t Globe and Smart have the same issue that Sky Cable have? The issue is only with Sky Cable, not with any other Philippines ISP.

When asked about the DNS, i was told…

    Pointing to our DNS ensures that the CDN will point to the nearest node. if that node is congested, then we may experience low bandwidth.

Unfortunately my further questions went unanswered so I decide to conduct some more tests and collect a little more data to more websites.

Click for a larger version (https://i0.wp.com/www.chewie.co.uk/wp-content/uploads/2017/08/Slow-Servers.png?resize=1000%2C274)

Now what I have done here is expand the test to 13 total servers, and as I mentioned in the intro we have some heavy hitters here like Amazon S3, Reddit, and Sourceforge. I’ve not bothered to test these against the PLDT line because you can take my word on it and see from everything I have written above that there is no issue with any other ISP.

So what does the above tell us?

    My two control servers, Fast.com and Nvidia are still pumping out max download speeds.
    More and more websites are being pulled into this Sky Cable issue.
    The average download speed from these servers is 91.27 KB/Sec or a line speed of 0.72MBPS, these speeds are reminiscent of dial up internet from 25 years ago!
    The slowest connection which is 5.5 KB/Sec is 1,250 times slower than the line is capable of, and must be hundreds of times slower than the average delivery speed described in Sky Cables’ contract.
    That 5.5KB/Sec speed means it takes 30 minutes to download a 10.8MB file. It’d take you 3x longer to download three Spotify songs than it would to listen to them!
    A 73MB file (probably about the average iOS app store App) takes over 2 hours!
    At this speed it would take you around 101 hours to download an iOS update from Apple.
    Doing traceroutes to each of these servers shows that they are CDNs serving from Hong Kong or Singapore.

Whilst not in the list Sky Cables own website is hosted in Singapore and can be affected by this issue, let’s have a look at the Traceroute for their own site from their own line…

    traceroute to mysky.com.ph (54.251.113.26), 64 hops max, 52 byte packets
    1 192.168.0.1 (192.168.0.1) 3.056 ms 2.510 ms 2.353 ms
    2 130.105.48.1 (130.105.48.1) 14.490 ms 18.795 ms 19.823 ms
    3 ge-1-5.sj1.skybroadband.com.ph (114.108.192.145) 18.360 ms 15.155 ms 23.655 ms
    4 130.105.0.30 (130.105.0.30) 8.130 ms
    130.105.0.28 (130.105.0.28) 18.952 ms
    130.105.0.82 (130.105.0.82) 14.599 ms
    5 130.105.0.48 (130.105.0.48) 34.346 ms 15.106 ms 16.821 ms
    6 ix-xe-2-1-0-3-0.e453.net (116.0.93.217) 43.291 ms 41.665 ms 42.000 ms
    7 if-ae-16-2.tcore2.svw-singapore.as6453.net (116.0.93.142) 72.434 ms
    if-ae-32-2.tcore2.svw-singapore.as6453.net (180.87.15.80) 92.321 ms 78.678 ms
    8 180.87.15.206 (180.87.15.206) 77.894 ms 74.093 ms 72.200 ms
    9 * * *
    10 * * *
    11 203.83.223.15 (203.83.223.15) 65.758 ms 100.979 ms
    203.83.223.74 (203.83.223.74) 70.910 ms
    12 * * *

It all looks to be ok with hops times that should be fine, but I suspect the issue is around ix-xe-2-1-0-3-0.tcore2.hk2-hong-kong.as6453.net (116.0.93.217) because it seems that tcore1 or tcore2 for HK or SG always causes issues. For Singapore I have also found that Equinix also has major issues, 13335.sgw.equinix.com (27.111.228.132) 41.833 ms 54.802 ms 49.804 ms.

So I gave all the info to Joe, and basically what I got back was this…

    We are now correlating the destination IP. there must be a common choke point somewhere.

    We will also be connecting to a major IX in the US and EU this 3rd Qtr. Lets see how will this improve our connectivity.

Yes Joe there must be a choke point somewhere, why have Sky been unable to fix this despite it being an issue for over a year? Are we really just supposed to cross our fingers and hope that some new IX will solve this issue? Why do you not know what is causing it right now?

Joe also had this to say about Equinix…

    We are looking at our Equinix connection. we just activated our 2nd 10G in Singapore early this year and it’s not balanced.

Why has this not been addressed yet? It surely couldn’t have been connected correctly in the first place?

After writing this entire post I did ask Joe again about the ix-xe-2-1-0-3-0.tcore2.hk2-hong-kong.as6453.net (116.0.93.217) hop which is NTT owned.

    Our NTT link are down since last July, most of the traffic are routed to pccw now.

So we are starting to get somewhere now, it is all still a bit of a mystery but here are some facts that have been confirmed by Joe and are obviously having an impact on Sky Cable’s ability to deliver international network traffic into the Philippines…

    Sky cable know there is an issue connecting to Apple servers but have been unable to resolve it.
    They have an issue with a line to SG Equinix “earlier this year”, but again have been unable to resolve it.
    They have a completely down line link to NTT since July and claim that traffic to Hong Kong is being routed to PCCW, but obviously this isn’t the case else the hops to this very website wouldn’t be showing in the traceroute. Again, they have been unable to resolve this.

Over to you for your help.
I have tried to get this issue resolved and I have tried to get other people to understand that there is an issue but I have hit a brick wall and I need your help.

    If you are using Sky Cable could you please open this Google Spreadsheet (https://docs.google.com/spreadsheets/d/1tXBk_BsYwXctHkhPgUaEKYJBZYC3HHQyQO2gBD4QQdo/edit#gid=1471083352)
    Enter your name on a new line, then click each of the links in the header row.
    These links will open downloads, save the downloads and then open your download manager which is usually CTRL(CMD)+J for Firefox or CTRL(CMD)+Shift+J on Chrome
    You can then see the download speed that is shown in the download manager. Simply enter this into the spreadsheet.

Hopefully with enough data we can get some exposure on this. Maybe it turns out that it is just my line that has an issue, maybe it is just a set of customers in a certain area, but maybe it is an issue which is affecting a lot of people and instead of us all moaning about Philippines internet speed, we might be able to do something to fix an issue that a 8.7billion revenue per year company just hasn’t been bothered with.

Feel free to comment below, and if you have knowledge in the network field then I am happy to try any other tests. I appreciate you reading such a long post