Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.
By SimonVlc on
Hi all,
5 days ago my server started to have several load peaks. From a normal load of 0.2-0.4, it went to 25-35 during periods of 30-60 seconds, every 15-20 minutes (except during night hours where the load maintains itself stable).
On every peak, my open connections and apache processes went up abnormally too.
netstat -n | grep :80| wc -l ~100 (normal is 25-30)
ps -ax | grep httpd | wc -l ~200 (normal is 15-25)
And this is a server-status report from a peak, where we can can see a lot of open POST connections from a single ip (that changes on every peak):
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 16778 0/51/4262 W 10.99 2 0 0.0 0.09 12.75 92.80.149.246 example.org POST /comment/reply/2826 HTTP/1.0
1-0 14091 0/432/4044 _ 60.92 0 5578 0.0 1.27 12.09 92.80.149.246 example.org POST /comment/reply/1061 HTTP/1.0
2-0 16763 0/54/4211 _ 12.75 0 3629 0.0 0.11 12.36 92.80.149.246 example.org POST /comment/reply/2775 HTTP/1.0
3-0 16756 0/72/3241 _ 14.76 0 6196 0.0 0.10 8.37 92.80.149.246 example.org POST /comment/reply/827 HTTP/1.0
4-0 16522 0/133/3052 W 17.41 2 0 0.0 0.22 8.71 92.80.149.246 example.org POST /comment/reply/2773 HTTP/1.0
5-0 16779 1/46/3375 C 10.83 0 6109 0.0 0.07 9.23 92.80.149.246 example.org POST /comment/reply/368 HTTP/1.0
6-0 16757 0/31/3672 _ 14.71 0 3823 0.0 0.08 10.80 92.80.149.246 example.org POST /comment/reply/2385 HTTP/1.0
7-0 16531 0/69/3053 W 18.72 0 0 0.0 0.08 8.02 92.80.149.246 example.org POST /comment/reply/3115 HTTP/1.0
8-0 16532 0/79/3420 W 13.44 2 0 0.0 0.14 9.37 92.80.149.246 example.org POST /comment/reply/2489 HTTP/1.0
9-0 14697 0/341/2989 W 24.19 2 0 0.0 0.87 8.68 92.80.149.246 example.org POST /comment/reply/722 HTTP/1.0
10-0 16765 0/26/2742 W 15.26 0 0 0.0 0.03 7.66 92.80.149.246 example.org POST /comment/reply/3068 HTTP/1.0
11-0 14939 0/306/3042 W 19.53 1 0 0.0 0.91 8.15 92.80.149.246 example.org POST /comment/reply/2233 HTTP/1.0
12-0 16534 0/64/3268 _ 14.62 0 5388 0.0 0.06 9.47 92.80.149.246 example.org POST /comment/reply/3068 HTTP/1.0
13-0 14098 0/426/2757 W 48.80 1 0 0.0 1.18 6.87 77.224.61.174 example.org GET /foros/poquer-general/ HTTP/1.1
14-0 16780 0/15/2492 W 10.76 3 0 0.0 0.00 6.43 92.80.149.246 example.org POST /comment/reply/3073 HTTP/1.0
15-0 14941 0/263/2326 W 29.19 4 0 0.0 0.63 6.04 92.80.149.246 example.org POST /comment/reply/2516 HTTP/1.0
16-0 16781 1/17/2386 C 11.62 1 8525 0.0 0.00 6.89 92.80.149.246 example.org POST /comment/reply/3037 HTTP/1.0
17-0 16535 0/65/2190 _ 16.37 0 5378 0.0 0.10 5.63 92.80.149.246 example.org POST /comment/reply/597 HTTP/1.0
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 16778 0/66/4277 W 20.88 2 0 0.0 0.09 12.75 76.22.156.89 example.org POST /comment/reply/294 HTTP/1.0
1-0 14091 0/432/4044 W 60.92 131 0 0.0 1.27 12.09 200.126.247.239 example.org POST /foros/newthread.php?do=postthread&f=39 HTTP/1.1
2-0 - 0/0/4228 . 21.15 8 0 0.0 0.00 12.40 ::1 nuevo.example.org GET / HTTP/1.0
3-0 17341 0/5/3252 W 3.60 0 0 0.0 0.00 8.37 92.80.149.246 example.org POST /comment/reply/1609 HTTP/1.0
4-0 16522 0/158/3077 _ 27.07 4 6079 0.0 0.22 8.71 92.80.149.246 example.org POST /comment/reply/3110 HTTP/1.0
5-0 16779 0/97/3426 W 19.17 2 0 0.0 0.16 9.32 76.22.156.89 example.org POST /comment/reply/291 HTTP/1.0
6-0 16757 0/51/3692 W 23.72 5 0 0.0 0.14 10.87 92.80.149.246 example.org POST /comment/reply/2217 HTTP/1.0
7-0 16531 4/89/3073 K 25.04 2 1 9.9 0.10 8.05 163.178.124.135 example.org GET /foros/customavatars/avatar1105_1.gif HTTP/1.1
8-0 16532 0/97/3438 _ 25.48 0 4694 0.0 0.15 9.38 92.80.149.246 example.org POST /comment/reply/2436 HTTP/1.0
9-0 14697 0/359/3007 W 35.61 2 0 0.0 0.87 8.68 76.22.156.89 example.org POST /comment/reply/306 HTTP/1.0
10-0 16765 0/43/2759 W 25.18 2 0 0.0 0.03 7.66 76.22.156.89 example.org POST /comment/reply/287 HTTP/1.0
11-0 14939 0/323/3059 W 30.92 0 0 0.0 0.91 8.15 92.80.149.246 example.org POST /comment/reply/1859 HTTP/1.0
12-0 16534 0/85/3289 _ 26.74 1 4725 0.0 0.07 9.48 92.80.149.246 example.org POST /comment/reply/2629 HTTP/1.0
13-0 - 0/0/2784 . 58.15 27 9765 0.0 0.00 6.88 92.80.149.246 example.org POST /comment/reply/622 HTTP/1.0
14-0 16780 0/38/2515 W 21.53 0 0 0.0 0.01 6.44 127.0.0.1 nuevo.example.org GET /server-status HTTP/1.0
15-0 14941 0/294/2357 _ 38.69 2 6880 0.0 0.66 6.06 92.80.149.246 example.org POST /comment/reply/3095 HTTP/1.0
16-0 - 0/0/2405 . 21.96 4 0 0.0 0.00 6.90 ::1 nuevo.example.org GET / HTTP/1.0
17-0 16535 1/92/2217 C 26.41 0 4165 0.0 0.14 5.67 92.80.149.246 example.org POST /comment/reply/469 HTTP/1.0
18-0 14943 0/302/2642 _ 38.85 4 2821 0.0 0.62 7.06 74.6.25.19 example.org GET /popular/siempre?sort=asc&order=comentarios&page=16%2C17 HT
19-0 16782 0/38/2582 _ 21.64 0 6528 0.0 0.06 7.24 92.80.149.246 example.org POST /comment/reply/1849 HTTP/1.0
20-0 16767 0/45/2286 W 16.95 85 0 0.0 0.07 5.09 200.126.247.239 example.org POST /foros/newthread.php?do=postthread&f=39 HTTP/1.1
21-0 14945 0/248/2414 W 39.51 4 0 0.0 0.64 6.34 92.80.149.246 example.org POST /comment/reply/1631 HTTP/1.0
22-0 16783 0/17/1482 W 9.50 2 0 0.0 0.06 4.10 76.22.156.89 example.org POST /comment/reply/272 HTTP/1.0
23-0 16798 0/6/3276 W 3.91 139 0 0.0 0.00 8.99 200.126.247.239 example.org POST /foros/newthread.php?do=postthread&f=39 HTTP/1.1
24-0 14948 0/185/1951 W 34.47 3 0 0.0 0.49 5.18 92.80.149.246 example.org POST /comment/reply/1336 HTTP/1.0
25-0 16784 0/36/1628 W 17.61 3 0 0.0 0.01 4.30 92.80.149.246 example.org POST /comment/reply/1154 HTTP/1.0
26-0 16785 0/30/1893 W 17.12 3 0 0.0 0.01 4.87 92.80.149.246 example.org POST /comment/reply/1830 HTTP/1.0
27-0 16786 0/36/2154 _ 21.08 3 5704 0.0 0.03 5.85 92.80.149.246 example.org POST /comment/reply/2442 HTTP/1.0
28-0 16787 0/32/1183 _ 17.39 2 8603 0.0 0.04 2.73 92.80.149.246 example.org POST /comment/reply/2340 HTTP/1.0
29-0 16788 0/46/1341 _ 20.31 0 3839 0.0 0.03 3.11 92.80.149.246 example.org POST /comment/reply/1127 HTTP/1.0
30-0 - 0/0/1785 . 20.15 19 2917 0.0 0.00 4.44 122.162.44.219 example.org POST /comment/reply/1096 HTTP/1.0
31-0 16790 0/64/2168 _ 19.60 3 5370 0.0 0.13 5.79 92.80.149.246 example.org POST /comment/reply/1336 HTTP/1.0
32-0 17342 0/23/1810 _ 2.41 2 3510 0.0 0.04 5.21 92.80.149.246 example.org POST /comment/reply/744 HTTP/1.0
33-0 - 0/0/1806 . 53.74 16 0 0.0 0.00 4.57 ::1 nuevo.example.org GET / HTTP/1.0
34-0 16792 0/46/2173 _ 17.05 1 4726 0.0 0.06 5.70 92.80.149.246 example.org POST /comment/reply/991 HTTP/1.0
35-0 16793 0/45/1843 W 17.85 3 0 0.0 0.07 5.27 76.22.156.89 example.org POST /comment/reply/302 HTTP/1.0
36-0 16794 0/31/1105 W 19.58 3 0 0.0 0.00 2.55 92.80.149.246 example.org POST /comment/reply/3062 HTTP/1.0
37-0 16795 0/12/964 W 14.49 3 0 0.0 0.00 2.47 92.80.149.246 example.org POST /comment/reply/1974 HTTP/1.0
38-0 - 0/0/1176 . 15.88 13 0 0.0 0.00 3.73 ::1 nuevo.example.org GET / HTTP/1.0
39-0 16816 0/52/1153 W 11.50 3 0 0.0 0.09 2.95 76.22.156.89 example.org POST /comment/reply/298 HTTP/1.0
40-0 16820 0/34/1587 W 14.50 0 0 0.0 0.01 4.40 76.22.156.89 example.org POST /comment/reply/285 HTTP/1.0
Could this be an attack? Could it be a drupal bug? Please, take a look and give me your opinion.
Thanks in advance, Simon.
Comments
Looks like you have been discovered by a spambot network...
Hi Simon,
Since the requests in question on you list are comment posts and you don't mention spam comments on your site I would think it's a spam bot network targeting your site but Drupals build-in form protection is doing it's stuff and discarding their posts. The bots think the post was accepted and just keep on posting.
Unfortunately this also means that each post is causing a full bootstrap of Drupal just to discard input from a bot. This eat resources on the server.
Have a look at the web server access log for the same time period.
Look for the request pattern ... direct post without referrer? Strange or missing user-agent?
If you find a pattern you could use htaccess to give the bots an "403 Access Denied" while still leaving the access open from the remote IPs as they most likely are infected home computers.
By deflecting the spam attack in htaccess you should see the resource use drop to near normal levels as Drupal no longer gets bootstrapped to handle the spam bots. The server is still using resources to pass a request from a spam bot through htaccess before deciding whether or not to pass handing on to PHP and Drupal.
Since spambots have found you you should consider joining Distributed Spam Harvester Tracking Network | Project Honey Pot and add a trap on your site or a link to one elsewhere (see the description on the site). This would also give you a key to use with the http:BL module.
Hope this helps...
Regards,
Christian Larsen
A few more ideas...
Hi again,
I mentioned a few more ideas in a comment on another post (How do I stop spam from 202.83.212.236?) which you might find useful...
Regards,
Christian Larsen
Thanks a lot Christian, I
Thanks a lot Christian,
I will try with your solutions.
Cheers, Simon.
Just now I got another peak.
Just now I got another peak. This afternoon closed anonymous comments and that helped to reduce the load.
This is the access log from an attacker ip:
87.120.23.219 - - [22/Nov/2007:23:24:13 +0100] "POST /comment/reply/1522 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:16 +0100] "POST /comment/reply/2006 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:15 +0100] "POST /comment/reply/1526 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:16 +0100] "POST /comment/reply/1692 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:17 +0100] "POST /comment/reply/1945 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:17 +0100] "POST /comment/reply/1541 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:15 +0100] "POST /comment/reply/1336 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:17 +0100] "POST /comment/reply/1634 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:18 +0100] "POST /comment/reply/2990 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:12 +0100] "POST /comment/reply/2629 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
87.120.23.219 - - [22/Nov/2007:23:24:11 +0100] "POST /comment/reply/1040 HTTP/1.0" 302 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
No pattern there, true? Cheers, Simon.
=-=
googling the IP shows that it is definetly a spam bot, thus block the IP
The problem is that the ip
The problem is that the ip change with every "attack", so blocking it is just a small improvement (the attacks lasts only 60-120 seconds).
htaccess code?
Sorry, I know it's been a while since this post but I'm experiencing similar hits from lots of IP addresses - any advice on 403 deflecting code to add to htaccess? I haven't set up url rewrite so my urls look like ?q=comment/reply/xxx etc
5 days of problems solved
5 days of problems solved with mod_security.
Thanks all for your help!!!
Great to hear :-)
Hi Simon,
it is great to hear that you found a solution :-).
There is a pattern in the log snippet above.
Assuming the "-" part is the referrer you have posts of a comments from nowhere. A normal pattern would be a POST to "comment/reply/###" with the same URL in the referrer.
You could simply compare POST URL with the referrer and discard any that does not match.
One drawback though is that this would also apply to people how have disabled sending referrer in their browser or are using a proxy that strips the referrer information from requests. Investigating the use of "HTTP/1.0" instead of "HTTP/1.1" might lessen the impact of this problem.
You can find other patterns if you keep an eye on your logs. You can also search the web to find sample code to block loads of misbehaving bots, spambots, harvesters, site copiers, worms etc. and use this to block them upfront before they actually visit you and turn up in the logs ;-)
Remember to have more than one way of stopping ze bad bots like the spam module, http:BL module, etc. ... in other words ... have defense in layers.
Regards,
Christian Larsen