Update on DigitalOcean’s connectivity issue with 4.2.2.2

This is the followup post of the following report: Seems that DigitalOcean haven’t fixed anything yet. 8 days since my last post. This page https://status.digitalocean.com/ show no information regarding this issue. Other people also reported similar behavior. Come on, do something DigitalOcean! Here is the latest benchmarks from my server to 3 different DNS provider: 4.2.2.2 (Level3), 8.8.8.8 (Google) and 208.67.222.222 (OpenDNS). I issued 10 dig queries for google.com, each of them 10 seconds apart. 6/10 queries sent to 4.2.2.2 are timed out. None of them happen for 8.8.8.8 and 208.67.222.222 tuananh@codepie:~$ for i in {1..10}; do dig google.com @4.2.2.2 | grep ‘connection timed out’; sleep 10; done; ;; connection timed out; no servers could be reached ;; connection timed out; no servers could be reached ;; connection timed out; no servers could be reached ;; connection timed out; no servers could be reached ;; connection timed out; no servers could be reached ;; connection timed out; no servers could be reached tuananh@codepie:~$ for i in {1..10}; do dig google.com @8.8.8.8 | grep ‘connection timed out’; sleep 10; done; tuananh@codepie:~$ for i in {1..10}; do dig google.com @208.67.222.222 | grep ‘connection timed out’; sleep 10; done;

Speedtest for your Linux server

Have you ever wonder how to test network speed (Internet specifically) of your server? Well, with GUI you can use something like speedtest.net, but how about CLI server, where you only have command-line interface? There are indeed several option: 1. Speedtest for CLI: https://pypi.python.org/pypi/speedtest-cli Install: easy_install speedtest-cli Use: speedtest 2. wget You first need to find some “big” files. My favorite is Ubuntu image: http://mirror.anl.gov/pub/ubuntu-iso/DVDs/ubuntu/14.04/release/ubuntu-14.04-server-amd64+mac.iso Use: wget -O /dev/null your_link It will actually not save anything on your system, so you don’t have to deal with clean up stuffs after you’ve done.

DigitalOcean droplets (at least for NYC2 region) are having trouble connecting to 4.2.2.2

I noticed a noticeable degrade in network performance in my droplets. It took forever to open a connection. It happened from last week I guess. Restart server does not help. I though it’s just temporary. However today I noticed that, DigitalOcean by default assign 2 DNS servers for every droplet in NYC2 region: nameserver 4.2.2.2 nameserver 8.8.8.8 Here is the result for ping from my droplet to both servers: tuananh@codepie:~$ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_req=1 ttl=46 time=13.7 ms 64 bytes from 8.8.8.8: icmp_req=2 ttl=46 time=13.8 ms 64 bytes from 8.8.8.8: icmp_req=3 ttl=46 time=13.8 ms 64 bytes from 8.8.8.8: icmp_req=4 ttl=46 time=13.8 ms 64 bytes from 8.8.8.8: icmp_req=5 ttl=46 time=13.7 ms 64 bytes from 8.8.8.8: icmp_req=6 ttl=46 time=13.7 ms 64 bytes from 8.8.8.8: icmp_req=7 ttl=46 time=13.7 ms 64 bytes from 8.8.8.8: icmp_req=8 ttl=46 time=13.7 ms 64 bytes from 8.8.8.8: icmp_req=9 ttl=46 time=13.7 ms 64 bytes from 8.8.8.8: icmp_req=10 ttl=46 time=13.7 ms ^C — 8.8.8.8 ping statistics — 10 packets transmitted, 10 received, 0% packet loss, time 9014ms rtt min/avg/max/mdev = 13.705/13.774/13.883/0.147 ms tuananh@codepie:~$ ping 4.2.2.2 PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data. ^C — 4.2.2.2 ping statistics — 167 packets transmitted, 0 received, 100% packet loss, time 167318ms Performing dig returns similar problem: tuananh@codepie:~$ dig google.com @4.2.2.2 ; < > DiG 9.8.1-P1 < > google.com @4.2.2.2 ;; global options: +cmd ;; connection timed out; no servers could be reached As you can see, somehow my droplet won’t be able to connect to 4.2.2.2. A simple switch to 8.8.8.8 as main DNS resolver and thing’s back to normal.

MySQL bug prevents you from connecting to custom port on MySQL server

It took me a great deal of time and effort to figure out this. In MySQL-client you can specify hostname and port to connect to a different MySQL instance on a different machine and/or different port rather than default localhost instance on your machine. For example, I have 2 MySQL instances running on two different machine, and one of them is behind firewall. Therefore, I need to use SSH tunnel to forward requests to port 3306 of the machine behind firewall. Things got little complicated when I tried to connect using –port or -P. Since I used the same password for both MySQL server (which I shouldn’t), it took me a while to figure out I still connect to the localhost instance. The reason is that, when you specify -P only, mysql will switch to socket mode instead of TCP mode. Here is what you need to do: mysql -P port –protocol TCP Adding –protocol TCP will force mysql to use TCP connection, thus will connect to the remote instance instead. Hope that helps!

Dropbox-like synchronization for Linux

One of the requirement for load-balancing servers is server file need to be synchronized. Otherwise, part of your visitors can see your new WordPress post but won’t be able to see the attached photos. rsync can’t do the job properly, because any synchronization tool need to look at the previous state of files in order to determine if new files have been added or any file has been changed or deleted. Fortunately there are several tool: BittorrentSync: This is a fully automatic solution, and as close to Dropbox as possible. You just need to download <code>btsync</code>, generate directory private key and input it into another instance in the other server. Done. Unison: This program run on top of rsync as default (but you can change it), but you need a little trick in order to run it properly. My favorite command is: unison -batch -prefer newer -silent -owner -group -times -perms 777 //dir1 //dir2 The meaning of this command is: -batch: Run in batch mode without asking confirmation -prefer newer: Prefer newer file if conflict -owner: Preserve owner information -group: Preserve group information -times: Preserve time -perms 777: Preserve permission That’s it. If you know any other tool, let me know in the comment section. Happy sync’ing!

My perfect setup (hint: CloudFlare, DigitalOcean, StartSSL, nginx, apache and private servers)

My situation is a little bit complicated: I have a powerful server completely under firewall (no inbound connection from outside) I want to run several websites (mostly blogs) I want to support SSL At the beginning, DigitalOcean is the best choice. I will have my own server, host unlimited websites, have full control and DigitalOcean is blazingly fast. I selected the smallest plan with 20G SSD and 512MB RAM. It would be more than enough for my blogs. I installed my own LAMP stack, get my own SSL certificate from StartSSL (you should get your own, too. It’s free!) Everything is fine until after several week. My server crashed every few hours. There are lots of requests coming for wp-comment-post.php, xmlrpc.php and wp-login.php. Unfortunately I can’t disabled them. Apache’s mod_security and mod_qos couldn’t help much. I have to write a temporary cron script to restart apache2 daemon whenever server load bigger than 20. Doesn’t improved much. My server still crash. There come nginx. Not work. Then CloudFlare. The same Until I decided to use my dedicated server to handle requests. Then it works!, not perfectly but we will be there later. In short, my configuration is like this: INTERNET < -> CLOUDFLARE < -> NGINX (DIGITALOCEAN) < -> APACHE (MY DEDICATED SERVER) There are several technical challenges that need to be solved: How can I forward requests to my dedicated server (completely under firewall) How can my end point (apache on my dedicated server) recognize IP from visitors correctly (since there are several layers in between? The solution for my first challenge is actually very simple: SSH Tunnel. There is one catch: Each website in my dedicated server will have to use its own port. And here is why: Assume I have 2 websites, example.com and codepie.org. I assigned port

Replace tab with space in Vim

I’m a fan of spaces. I hate tabs, actually. It does not look good on vim, git, … you name it. Unfortunately lots of Linux config files are still using tab, like apache2 and so on. I’m like, why don’t they switch to space completely. The reason is that, under standard screen (80×24), each tab costs 8. After several indentation, you hardly can see anything or almost every line of config will be break into two lines. Doesn’t look good. Here is a trick to replace tab with space, and ensure vim will use space by default. If you want to use space as default in your system, put it at the end of /etc/vim/vimrc, otherwise at the end of ~/.vimrc set expandtab set tabstop=4 set shiftwidth=4 Of course you can replace 4 by any number of your choice. For me 4 is perfect. 2 is too small and 8 is definitely too big. But what if you want to replace tab with space in existing files? Open vim and type this command: :retab Voila! Happy vim-ing. Source: http://stackoverflow.com/questions/426963/replace-tab-with-spaces-in-vim

Simple tool for load-testing HTTP server

Today I came across a very simple tool for load-testing. It’s called AB and come with Apache HTTP Server. The command line arguments are very simple: ab -n 10000 -c 10 http://your_web_site/url.html Where -n is the number of successful requests and -c is the number of concurrence connections. If you want to have more control over the URL you want to test, and much more, I recommend JMeter (also from Apache) Source: http://serverfault.com/questions/2107/tools-for-load-testing-http-servers

Quick and dirty way to prevent XML-RPC Pingback Attacks

XML-RPC Pingback Attacks are really annoying. You often see lots of POST /xmlrpc.php requests from different IP address from time to time. Sometimes, the number of requests is big enough to crash a server (yes, my server is a tiny angel). But I can’t disable WordPress XML-RPC because I need that for Jetpack to work (post by email, how cool!). Therefore, I figure out a quick and dirty way to prevent this by only allow Jetpack IP address to call XML-RPC. These IP are not public by WordPress (I don’t know why) as they stated that: “We aren’t able to provide any IP addresses for Jetpack as they fluctuate. You could try whitelisting *.wordpress.com for both inbound and outbound traffic, as a workaround.” However, by looking at my server log, I see 2 potential IP address ranges. Quick lookup confirms my suspicion. Here they are (Update: Ben (in the comment below) provided me with a list of IP addresses he found in his server log. I double checked and updated them here.): 66.135.32.0/19 66.155.0.0/18 69.174.240.0/20 72.232.0.0/17 76.74.248.0/21 192.0.64.0/18 198.181.116.0/22 207.198.64.0/18 209.15.0.0/16 216.151.208.0/20 And here is sample configuration in nginx server { location ~ xmlrpc\.php { deny all; allow 127.0.0.0/24; allow ::1/128; allow 66.135.32.0/19; allow 66.155.0.0/18; allow 69.174.240.0/20; allow 72.232.0.0/17; allow 76.74.248.0/21; allow 192.0.64.0/18; allow 198.181.116.0/22; allow 207.198.64.0/18; allow 209.15.0.0/16; allow 216.151.208.0/20; } } If you know any other IP ranges, let me know and I will update my post. Happy blogging!