Hoge CPU load VPS door meerdere -DSSL processen

[roelvde]

Eerder deze week kreeg ik al mails van de VPS dat de serverload te hoog was 11.6 etc. Hierna ging ik dit controleren en zag ik niks vreemds.

Maar sinds gisteravond/vannacht staat de server onder een zware CPU load met 1x een piek naar 140%.

11856

Nu komt steeds het proces: /usr/sbin/httpd -k start -DSSL naar boven, dit vaak ook meerdere keren (1 tot 10) tegelijk, met een cpuload rond de 60-70% of lager.


top - 12:25:22 up 1 day, 12:04, 0 users, load average: 0.60, 0.79, 0.85
Tasks: 159 total, 3 running, 156 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.6%us, 4.4%sy, 0.6%ni, 83.5%id, 1.5%wa, 0.0%hi, 0.0%si, 0.3%st
Mem: 3922748k total, 2453076k used, 1469672k free, 302324k buffers
Swap: 1048568k total, 86812k used, 961756k free, 775904k cached


Page: 1 2 3 4 Advanced Search
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31420 admin 20 0 274m 76m 5588 R 98.3 2.0 0:47.21 /usr/sbin/httpd -k start -DSSL
31744 admin 20 0 265m 67m 5288 R 41.3 1.8 0:39.23 /usr/sbin/httpd -k start -DSSL
6003 mysql 20 0 1316m 245m 5444 S 3.9 6.4 25:11.16 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/bladewebsites.com-online.identiteit.err --pid-file=/var/lib/mysql/bladewebsites.com-online.identiteit.pid
1123 root 20 0 243m 1084 324 S 2.0 0.0 1:18.88 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
1 root 20 0 19356 504 292 S 0.0 0.0 0:00.72 /sbin/init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kthreadd]
3 root RT 0 0 0 0 S 0.0 0.0 0:01.57 [migration/0]
4 root 20 0 0 0 0 S 0.0 0.0 0:02.64 [ksoftirqd/0]
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 [migration/0]
6 root RT 0 0 0 0 S 0.0 0.0 0:00.41 [watchdog/0]
7 root RT 0 0 0 0 S 0.0 0.0 0:00.77 [migration/1]
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 [migration/1]
9 root 20 0 0 0 0 S 0.0 0.0 0:02.90 [ksoftirqd/1]
10 root RT 0 0 0 0 S 0.0 0.0 0:00.33 [watchdog/1]

Dit is de error log van apache /var/log/httpd/error_log:


[Thu Oct 23 00:11:03 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 00:11:03 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 00:11:03 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 00:11:03 2014] [notice] mod_ruid2/0.9.7 enabled
[Thu Oct 23 00:11:03 2014] [notice] Apache/2.2.26 (Unix) mod_ssl/2.2.26 OpenSSL/1.0.1e-fips DAV/2 PHP/5.3.28 configured -- resuming normal operations
[Thu Oct 23 01:00:17 2014] [error] [client 127.0.0.1] client denied by server configuration: /var/www/html/server-status
[Thu Oct 23 01:00:17 2014] [error] [client 127.0.0.1] File does not exist: /var/www/html/403.shtml
[Thu Oct 23 05:24:26 2014] [error] [client 172.243.178.100] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /tmUnblock.cgi
[Thu Oct 23 10:29:19 2014] [error] [client 146.185.239.100] File does not exist: /var/www/html/400.shtml
[Thu Oct 23 10:29:19 2014] [emerg] (13)Permission denied: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [alert] Child 19302 returned a Fatal error... Apache is exiting!
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:21 2014] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Thu Oct 23 10:29:21 2014] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Thu Oct 23 10:29:21 2014] [emerg] (22)Invalid argument: couldn't release the accept mutex
[Thu Oct 23 10:29:22 2014] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Thu Oct 23 10:29:24 2014] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Thu Oct 23 10:30:03 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 10:30:03 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 10:30:03 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 10:30:03 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Oct 23 10:30:04 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 10:30:04 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 10:30:04 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 10:30:04 2014] [notice] mod_ruid2/0.9.7 enabled
[Thu Oct 23 10:30:04 2014] [warn] pid file /var/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Oct 23 10:30:04 2014] [notice] Apache/2.2.26 (Unix) mod_ssl/2.2.26 OpenSSL/1.0.1e-fips DAV/2 PHP/5.3.28 configured -- resuming normal operations
[Thu Oct 23 10:42:34 2014] [error] [client 222.76.242.219] File does not exist: /var/www/html/manager
[Thu Oct 23 10:42:34 2014] [error] [client 222.76.242.219] File does not exist: /var/www/html/404.shtml
[Thu Oct 23 11:48:58 2014] [error] [client 101.226.169.216] File does not exist: /var/www/html/robots.txt, referer: http://bloempaal.com/robots.txt
[Thu Oct 23 11:48:58 2014] [error] [client 101.226.169.216] File does not exist: /var/www/html/404.shtml, referer: http://bloempaal.com/robots.txt
zend_mm_heap corrupted
zend_mm_heap corrupted
[Thu Oct 23 11:58:07 2014] [notice] caught SIGTERM, shutting down
[Thu Oct 23 11:58:08 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 11:58:08 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 11:58:08 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 11:58:08 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Oct 23 11:58:09 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 11:58:09 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 11:58:09 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 11:58:09 2014] [notice] mod_ruid2/0.9.7 enabled
[Thu Oct 23 11:58:09 2014] [notice] Apache/2.2.26 (Unix) mod_ssl/2.2.26 OpenSSL/1.0.1e-fips DAV/2 PHP/5.3.28 configured -- resuming normal operations

om 00:11 en 11:58 heb ik de server een keer restart.

Hier haal ik uit dat er iets mis is met het SSL certificaat voor een van onze sites. Hier heb ik eerder nog geen problemen mee gehad.
Nu heb ik wel sinds 2 dagen wordpress draaien voor een kleine mobiele site /m/. Hier hangen ook een paar security plugins in (wordfence en iThemes Security). Ook is CSF geïnstalleerd in directadmin.

Komt de hoge load van -DSSL hier vandaan of zijn er andere dingen aan de hand met mijn VPS? Zijn hier simpele oplossingen voor om dit probleem te verhelpen en in de toekomst te voorkomen?
Zijn er tools om dit soort processen makkelijk te monitoren en erachter te komen wat dit precies veroorzaakt?

[MarkKapitein]

Wat je kan doen,

Met csf kan je apachestatus laten mailen bij hoge load.
Stel dat even goed in.
Dan zie je ook of het mogelijk je WP installatie is die de hoge load veroorzaakt.

[roelvde]

De load fluctueert enorm, van een paar % (wat bij ons normaal is) naar 100+% met soms meerdere DSSL processen op 50 60 80%.

[cyrano]

Is je Wordfence up-to-date?

https://vexatioustendencies.com/wordfence-v5-2-3-2-stored-xss-insufficient-logging-throttle-bypass-exploit-detection-bypass/

icm met de problemen bij Openpeering recent zie ik daar wel een paar mogelijkheden in om de load van een server de hoogte in te jagen...

[systemdeveloper]

Een -DSSL betekent alleen maar dat apache gestart is met ssl ondersteuning, dus in principe zal de load niks met je ssl te maken hebben (buiten het feit dat ssl verbindingen wel iets meer load hebben dan gewone http verbindingen vanwege de encryptie, maar dat is alleen al er ook daadwerkelijk veel ssl pages worden opgevraagd natuurlijk).

Als je apache 2.4 hebt moet je eens

Mutex posixsem

in je httpd.conf zetten en apache herstarten.

Een hoge kan komen omdat je apache processen crashen en het main httpd process doorlopens bezig is met nieuwe deamons starten. Maar een grotere kans is vaak dat ergens een cronjob hangt of dat een plugin van een site (of site zelf) gewoon ergens een rotte sql query heeft.

[roelvde]

Dit weekend geprobeerd het probleem op te lossen, maar nog steeds zonder succes. Misschien dat iemand ons kan helpen die hier wel de juiste kennis van heeft?

[roelvde]

als ik een strace doe van een DSSL proces kom ik steeds dit tegen. Kan ik hier uit opmaken dat er iets mis is gegaan met een joomla website? Deze draait al een tijdje en eerder geen problemen mee gehad.


lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries/idna_convert", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/admin/domains", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/admin", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries/idna_convert/idna_convert.class.php", {st_mode=S_IFREG|0644, st_size=94834, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries/idna_convert", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/admin/domains", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/admin", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/admin", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
open("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries/idna_convert/idna_convert.class.php", O_RDONLY) = 49
fstat(49, {st_mode=S_IFREG|0644, st_size=94834, ...}) = 0
fstat(49, {st_mode=S_IFREG|0644, st_size=94834, ...}) = 0
fstat(49, {st_mode=S_IFREG|0644, st_size=94834, ...}) = 0
mmap(NULL, 94834, PROT_READ, MAP_SHARED, 49, 0) = 0x7f74d7407000
munmap(0x7f74d7407000, 94834) = 0
close(49) = 0
poll([{fd=48, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(48, "o\0\0\0\3SELECT COUNT(*) FROM jos_us"..., 115) = 115
read(48, "\1\0\0\1\1\36\0\0\2\3def\0\0\0\10COUNT(*)\0\f?\0\2 5\0\0"..., 16384) = 63
poll([{fd=48, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(48, "m\2\0\0\3INSERT INTO `jos_k2_comment"..., 625) = 625
read(48, "\n\0\0\1\0\1\375\224O\5\2\0\1\0", 16384) = 14
chdir("/") = 0
gettimeofday({1414350858, 588564}, NULL) = 0
poll([{fd=48, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(48, "R\6\0\0\3UPDATE `jos_session`\nSET `d"..., 1622) = 1622
read(48, "0\0\0\1\0\1\0\2\0\0\0(Rows matched: 1 Cha"..., 16384) = 52
write(48, "\1\0\0\0\1", 5) = 5
shutdown(48, 2 /* send and receive */) = 0
close(48) = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
writev(47, [{"HTTP/1.1 200 OK\r\nDate: Sun, 26 O"..., 222}, {"\37\213\10\0\0\0\0\0\0\3", 10}, {"\253V\312M-.NLOU\262R\nJML.\311LU(\311OMO-\3\22)\212\na"..., 64}, {"\334\35\303\v@\0\0\0", 8}], 4) = 304
gettimeofday({1414350858, 600839}, NULL) = 0
write(27, "304 1112\n", 9) = 9