(Note: this issue is already resolved, I’m posting about it only as an interesting happening.)
tl;dr – Many users with AT&T as their home ISP users couldn’t access ModMyi.com for 6 days, turned out they were blocking CloudFlare IPs, and CloudFlare moved serving us to another IP, fixing the issue.
ModMyi.com gets over a million page views a day when there is a jailbreak out, and we use CloudFlare as a CDN (distributes our content across the world so people can access our site quicker) and to cache the site (save on bandwidth). Recently, I started getting many messages through Twitter, Facebook, and our site of people who simply couldn’t get the site to load at all.
After troubleshooting with all the users, we were able to discover each and every person having the issue was using AT&T as their ISP. Traceroutes were similar to below – nothing made it past the first or second hop (AT&T):
C:\>tracert modmyi.com Tracing route to modmyi.com [126.96.36.199] over a maximum of 30 hops: 1 1 ms 1 ms 1 ms homeportal [192.168.1.254] 2 23 ms 44 ms 32 ms 172-6-208-3.lightspeed.snantx.sbcglobal.net [172 .6.208.3] 3 * * 24 ms 188.8.131.52 4 * 24 ms 21 ms 184.108.40.206 5 * * * Request timed out. 6 21 ms 21 ms 20 ms 220.127.116.11 7 ded3-g4-3-0.sfldmi.ameritech.net [18.104.22.168] reports: Destination hos t unreachable. Trace complete. C:\>
I reached out to AT&T through their @ATTCustomerCare Twitter account, and was surprised to reach a person with a real conversational tone rather quickly (within an hour). She assured me they were “looking into it,” and did in fact reach out to some of the users who’d reported issues. She forwarded it on to higher tiers, who were given incorrect contact information for me – 6 days went by with me messaging her daily (direct message through Twitter) and her assuring me “they were on it.”
Finally, I reached out to CloudFlare support to see if perhaps they were the issue (they’d assured me before they were not, but we were getting nowhere). They did a bit more sleuthing, and discovered for whatever reason AT&T was blocking a couple of CloudFlare’s IP addresses – one of which happened to be the one CloudFlare was serving ModMyi.com from. They moved serving ModMyi.com to another of their IPs, and the problem was instantly resolved.
Oddly, within a few hours of that fix, someone from “the President’s office at AT&T” (perhaps just a title – someone from the same place called me in January when this issue ALSO occurred) called me to check on the issue. I let them know CloudFlare had discovered AT&T was blocking their IPs and moved us, so it was resolved for us but not them. He thanked me, and did actually give me a direct number to reach him if this happened again. It was a good call, although 6 days late – he even mentioned “I know we’re a monolithic company and it can be hard to reach someone.” Very cool to have a direct contact for fixing this issue should it occur again.
An interesting bit of troubleshooting for a strange issue, figured I’d share.