Downtime 12th/13th January 2015

twisted1919

Administrator
Staff member
As you might have noticed, mailwizz.com website has been down a few times these two days.
Seems that having a higher number of visitors than usual and getting somehow of a DDOS attack has made the server unresponsive for a while and required manual intervention to put it back online.

I called it "somehow a DDOS attack" because i am really not sure what it is at this point, basically we're getting hundred requests per second for something that looks like a torrent tracker announcement file.
This could be on purpose or because maybe a tracker was hosted on same ip address mailwizz.com is now, again, not sure.

Anyway, this is how our apache access logs look like, maybe somebody else has had same issue :
Code:
# tail -f /var/log/httpd/access_log
61.172.7.240 - - [13/Jan/2015:23:45:49 +0000] "GET /announce?info_hash=%D7%80r%FC%F5C%5C%B3%96%7D%A3%CB%DA%E7%7E%3E%1F%C5T%B8&peer_id=%2DUT3000%2D%22%2A%86%C3%26%E7%5FP%E0%E3%3F%13&ip=192.168.2.102&port=1080&uploaded=0&downloaded=0&left=289742100&numwant=200&key=1810226630&compact=1&event=started HTTP/1.0" 403 210 "-" "Bittorrent"
218.78.210.115 - - [13/Jan/2015:23:45:50 +0000] "GET /announce?info_hash=%F1%14%D1%E2%C6%CD%19%D5S%FEAjZ%21%F3%EF%BE%85i%19&peer_id=%2DSD0100%2D%5FlW%5B%40%85i%A7S%DC9%FA&ip=172.25.156.1&port=8741&uploaded=1074266112&downloaded=1074266112&left=0&numwant=200&key=26233&compact=1 HTTP/1.0" 403 210 "-" "Bittorrent"
42.95.127.12 - - [13/Jan/2015:23:45:50 +0000] "GET /announce.php?info_hash=%0F%CFj%CA%0C0S%2B%1D%D0%B6G%5E%CF%94LN%EF%AC%D1&peer_id=%2DSD0100%2D%5B%05%B8%E0%C8%D08%F0%7D%A3m%0A&ip=42.95.127.12&port=10667&uploaded=1420627878&downloaded=1420627878&left=0&numwant=200&key=15529&compact=1 HTTP/1.0" 403 214 "-" "Bittorrent"
221.217.31.115 - - [13/Jan/2015:23:45:50 +0000] "GET /announce?info_hash=P%01%A3%96%B6%FD7%9C%B0%AFJ%00%F4%A3%85%F4%A3%85%E4%FF&peer_id=%2DSD0100%2DH%2A%F8GK%F16%22%1A%19%1C%D3&ip=221.217.31.115&port=9577&uploaded=7300042559&downloaded=7300042559&left=0&numwant=200&key=27905&compact=1 HTTP/1.0" 403 210 "-" "Bittorrent"
42.92.39.181 - - [13/Jan/2015:23:45:51 +0000] "GET /announce?info_hash=%D9%F0%B3N%07%5E%96e%E3%81%BD%9FEB%E7%97%B2%27%25%3A&peer_id=%2DSD0100%2D%B9%D1%F0%28%E8%D6rtT%97%93O&ip=10.2.207.48&port=9957&uploaded=220907662&downloaded=220907662&left=689700864&numwant=200&key=5957&compact=1 HTTP/1.0" 403 210 "-" "Bittorrent"
218.85.148.170 - - [13/Jan/2015:23:45:52 +0000] "GET /announce?info_hash=%28NW%60%22%40v%BC%90s%25%F4S%27%A3%86%2A%3E%1D%05&peer_id=%2DSD0100%2D%E2%5E%C1%84%87%DEw%10%DB%60iS&ip=10.0.0.71&port=14290&uploaded=3978360352&downloaded=3978360352&left=703531488&numwant=200&key=6870&compact=1 HTTP/1.0" 403 210 "-" "Bittorrent"
218.82.235.147 - - [13/Jan/2015:23:45:52 +0000] "GET /announce.php?info_hash=%80%84%3D%D8UE%B7%3B%90%97%88%07%2D%E9BXC%19%BE%B8&peer_id=%2DSD0100%2D%3C%1D%CF%82%E1%D3k%B6%85%CD%939&ip=192.168.1.3&port=8500&uploaded=626691031&downloaded=626691031&left=1127947721&numwant=200&key=7254&compact=1 HTTP/1.0" 403 214 "-" "Bittorrent"
101.85.224.36 - - [13/Jan/2015:23:45:53 +0000] "GET /announce?info_hash=g%03%29%02%A1%26%90R%8D%3C%C4%B1%B7%B1%12s%175u%B7&peer_id=%2DSD0100%2D%B3N%80%26%B6%BF%F5%A4%9D%2BXU&ip=192.168.3.100&port=9509&uploaded=928776192&downloaded=928776192&left=0&numwant=200&key=2752&compact=1 HTTP/1.0" 403 210 "-" "Bittorrent"
60.164.168.122 - - [13/Jan/2015:23:45:53 +0000] "GET /announce.php?info_hash=%F6%87%9D%E1%7F%40%F6%0C%1E%88%0F%1Ds%8C%AF%0FP%89%B1%86&peer_id=%2DSD0100%2D%7E%5B%E4%AC%DEBK%18%85%C3%3Bj&ip=192.168.1.104&port=11288&uploaded=1140850688&downloaded=1140850688&left=0&numwant=200&key=17982&compact=1 HTTP/1.0" 403 214 "-" "Bittorrent"
116.237.7.60 - - [13/Jan/2015:23:45:53 +0000] "GET /announce.php?info_hash=%8B%8C%09d%21%DDo%3FN%11%F9%C4%9C%14%7C%FB%0E%90%CD%98&peer_id=%2DSD0100%2D%9D%FEjT%2C%9AI%95%B7%9D%0E%B4&ip=116.237.7.60&port=16738&uploaded=854188696&downloaded=854188696&left=117440512&numwant=200&key=19622&compact=1 HTTP/1.0" 403 214 "-" "Bittorrent"
14.18.243.10 - - [13/Jan/2015:23:45:53 +0000] "GET /announce/ ?info_hash=%A7%B3%C0%E2K%9A%BDoH%CCs%AF%8A2%9D%DD%95%2C%9A%5C&peer_id=%2DSD0100%2D%96%DC%B6r%96%D11Z%94%22%AF%21&ip=118.196.132.211&port=18689&uploaded=4194304&downloaded=4194304&left=917766144&numwant=200&key=24248&compact=1 HTTP/1.0" 403 211 "-" "Bittorrent"
::1 - - [13/Jan/2015:23:45:53 +0000] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.4.16 (internal dummy connection)"
210.21.68.2 - - [13/Jan/2015:23:45:54 +0000] "GET /announce?info_hash=%23%3C%97%F72O%8Frs%21%19L%DA%C4%3B1%DAc%9A%D7&peer_id=%2DSD0100%2DHA%C8%8A%AEd%0F5%20%B0%0A%D5&ip=203.88.204.241&port=9086&uploaded=4344856&downloaded=4344856&left=16401675240&numwant=200&key=20313&compact=1 HTTP/1.0" 403 210 "-" "Bittorrent"
117.146.18.196 - - [13/Jan/2015:23:45:55 +0000] "GET /announce?info_hash=%00%28%14%aegPT%14%f5%fd%ec%15p%cd%25%1b%9fv%2d%03&peer_id=%2dSP3605%96%16%2e%5e%af%8c%5e%8e%14%80%b8%c89&port=27155&uploaded=134813463&downloaded=384822568&left=71827456&key=D8685E95&compact=1&numwant=200&no_peer_id=1&ipv6=2001%3a0%3a9d38%3a6ab8%3a1420%3add15%3a8a6d%3aed3b HTTP/1.0" 403 210 "-" "BTSP/3605"

Initially the entire website was running on a 1GB linode with a single core, which is something that can be brought in it's knees very easily, no wonder it went down at the first serious load.
Bumping to a 4GB RAM / 4 Core linode and adding some simple apache deny rules seems to do the trick for going over the bump right now.
I guess if the issue persists we'll add a firewall in front of the server but for now i don't think it's the case.

Thanks for understanding (the downtime) and sorry for any inconvenience.
 
Back
Top