Apache 和其他服务挂起答案

【问题标题】：Apache and Other Services HungApache 和其他服务挂起
【发布时间】：2014-03-19 05:14:16
【问题描述】：

我在 Gentoo 上有一个生产 LAMP 服务器，它一直在运行个人 Intranet 网站。从昨天下午开始，它在任何连接后不久就挂起；相对较小的登录页面加载正常，但在登录时涉及大量 MySQL 查询和数据，它会无限期挂起。

SSH 仍然能够连接，但奇怪的是，在传输了几页的字符后，它也突然挂起。因此，要获得以下信息，我必须重新登录。我尝试了 /etc/init.d/apache2 restart 和 /etc/init.d/mysql restart，然后完全重新启动了系统；然而，问题仍然存在。详情如下。

顶部：

top - 12:23:52 up  1:34,  2 users,  load average: 0.16, 0.09, 0.06 Tasks:  81 total,   1 running,  80 sleeping,   0 stopped,   0 zombie Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st Mem:   3920788k total,   123476k used,  3797312k free,     4676k buffers Swap:  1227772k total,        0k used,  1227772k free,   48524k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND   447 root      20   0     0    0    0 S    0  0.0   0:00.16 khubd
    1 root      20   0  2020  640  568 S    0  0.0   0:00.51 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0
    5 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/u:0
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1
    9 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1
   10 root      20   0     0    0    0 S    0  0.0   0:00.69 kworker/0:1

Apache 日志显示了常见的黑客攻击尝试：

# tail -50 /var/log/apache2/error_log
[Mon Mar 17 19:03:48 2014] [error] [client 116.58.240.169] File does not exist: /var/www/mysite/pma
[Mon Mar 17 19:03:48 2014] [error] [client 116.58.240.169] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 05:58:42 2014] [error] [client 202.53.8.82] File does not exist: /var/www/mysite/admin.cgi
[Tue Mar 18 07:19:42 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/phpTest
[Tue Mar 18 07:19:43 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/phpMyAdmin
[Tue Mar 18 07:19:43 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/pma
[Tue Mar 18 07:19:44 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 08:24:16 2014] [error] [client 222.5.204.73] invalid request-URI \xcc\\\xa4/\x83\x8f\x90:\x84\x90\x0f\xc4\x8dfe\xecb\x94v\x1f[\xd7Z\x95$X\xaby\x13k\x88\xf2\xeb\xf7\x1b\xfc\xe8a\xff
[Tue Mar 18 08:29:49 2014] [error] [client 76.3.191.245] invalid request-URI
[Tue Mar 18 08:38:00 2014] [error] [client 35.2.240.149] invalid request-URI
[Tue Mar 18 08:50:52 2014] [error] [client 173.26.148.34] invalid request-URI
[Tue Mar 18 10:57:48 2014] [error] [client 110.175.79.216] invalid request-URI
[Tue Mar 18 10:57:53 2014] [error] [client 110.248.140.59] invalid request-URI D\xe8\x91a\xbc\xe5WZ\xd0C]\x9f~\xb5\x89\bd\x9e"[w,\xc6\xd9\xde\x8b]#JJ\xbf\x12
[Tue Mar 18 14:24:54 2014] [error] [client 108.14.2.113] invalid request-URI
[Tue Mar 18 14:40:08 2014] [error] [client 86.217.136.41] invalid request-URI \x94FI-\x02;4JVOV\x0f\xba\b
[Tue Mar 18 14:45:42 2014] [error] [client 98.119.127.76] invalid request-URI
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-120x120-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-120x120.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon.png
[Tue Mar 18 16:20:45 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/phpTest
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/phpMyAdmin
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/pma
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 16:40:58 2014] [error] [client 122.170.93.35] invalid request-URI
[Tue Mar 18 16:57:54 2014] [error] [client 124.107.151.190] invalid request-URI
[Tue Mar 18 17:36:17 2014] [error] [client 68.147.250.90] invalid request-URI \x1d\x1e;&\x9e\xd2\xa8\xc2GNQ\\
[Tue Mar 18 23:38:20 2014] [error] [client 92.240.68.153] request failed: error reading the headers
[Wed Mar 19 02:52:43 2014] [error] [client 162.213.24.36] File does not exist: /var/www/mysite/CFIDE
[Wed Mar 19 06:26:06 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:07 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:07 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:09 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:15 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 07:48:28 2014] [error] [client 201.161.37.93] File does not exist: /var/www/crownware/manager
[Wed Mar 19 09:27:08 2014] [error] [client 113.184.228.73] invalid request-URI \xad_X\xdf\x9aIM6x\x01ti\xf6Ko\xebi
[Wed Mar 19 09:36:06 2014] [error] [client 162.213.24.36] File does not exist: /var/www/crownware/CFIDE
[Wed Mar 19 10:28:15 2014] [notice] caught SIGTERM, shutting down
[Wed Mar 19 10:28:17 2014] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0j PHP/5.4.6--pl0-gentoo configured -- resuming normal operations
[Wed Mar 19 10:43:31 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:31 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:35 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:35 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:36 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:47:16 2014] [notice] caught SIGTERM, shutting down
[Wed Mar 19 10:49:32 2014] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0j PHP/5.4.6--pl0-gentoo configured -- resuming normal operations
[Wed Mar 19 10:53:45 2014] [error] [client 65.60.209.141] Invalid URI in request \x13\xe0\x94\xc4\xa4o\xd1\xd3*\xe0\xe7\x1a\xce\xd9\xe8\t\xca\xc3k\x9f\xb0\x06\x13\xbcE\x17\xbb\x02\x9c:\xffD\x8d\x1f\x85Wv\x14\xfd\x8f\xe3k\xc6\xfe\xf7\x1bu
[Wed Mar 19 12:20:07 2014] [error] [client 173.24.52.209] invalid request-URI

来自 /var/log/mysql/mysqld.err 的最后一条感兴趣的消息（5 天前）：

140314  9:56:02  InnoDB: ERROR: the age of the last checkpoint is 9448765,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

版本：

# uname -a
Linux myhost 3.3.8-gentoo #1 SMP Fri Sep 28 09:34:42 MYT 2012 i686 Intel(R) Xeon(R) CPU E31220 @ 3.10GHz GenuineIntel GNU/Linux

# mysqld -V
140319 12:37:13 [Warning] '--default-character-set' is deprecated and will be removed in a future release. Please use '--character-set-server' instead.
140319 12:37:13 [Warning] '--default-collation' is deprecated and will be removed in a future release. Please use '--collation-server' instead.
mysqld  Ver 5.1.62-log for pc-linux-gnu on i686 (Gentoo Linux mysql-5.1.62-r1)

# apache2 -V
Server version: Apache/2.2.23 (Unix)
Server built:   Oct 27 2012 19:17:52
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture:   32-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"

# php -v
PHP 5.4.6--pl0-gentoo (cli) (built: Oct 27 2012 18:42:24)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies

磁盘似乎还有足够的空间：

# df
Filesystem     1K-blocks     Used Available Use% Mounted on
rootfs         960125048 84604800 826748732  10% /
udev               10240        0     10240   0% /dev
/dev/sda3      960125048 84604800 826748732  10% /
tmpfs            1960392      220   1960172   1% /run
rc-svcdir           1024       64       960   7% /lib/rc/init.d
cgroup_root        10240        0     10240   0% /sys/fs/cgroup
shm              1960392        0   1960392   0% /dev/shm

Apache 进程：

# ps -ef|grep -i apache
root      2060     1  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2062  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2066  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2067  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2068  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2069  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2070  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2123  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2124  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2125  2060  0 10:49 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2148  2060  0 10:50 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache    2149  2060  0 10:50 ?        00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start

sracing父（根）进程反复显示这个，不知道是否正常：

# strace -p 2060
Process 2060 attached
select(0, NULL, NULL, NULL, {0, 669445}) = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)

SSH 在几千字节后也挂起的事实表明我应该比 Apache 更宽。下一步如何诊断？

【问题讨论】：

标签： linux apache2 lamp freeze

【解决方案1】：

从 Apache 日志中可以明显看出，您的网站是典型漏洞利用脚本的目标，这些漏洞利用脚本只是试图通过请求已知应用程序来轰炸服务器以寻找漏洞。

这可能导致了妥协 - 这部分很难说，因为您没有详细说明您的机器上正在运行哪些其他脚本。

我建议在您的服务器上运行 rootkit 分析器或类似工具。

此外，这类问题更适合serverfault.com，因为它与编程无关（stackoverflow 是关于什么），而是系统管理/服务器管理。

为防止此类请求访问您的服务器，建议使用 WAF（Web 应用程序防火墙）或其他代理，它们会在此类请求到达您的计算机之前对其进行限制和阻止。

naxsi 是 nginx 的一个模块，提供了一个开源的 WAF。

【讨论】：