Onderwerp: Hoge CPU load VPS door meerdere -DSSL processen

Eerder deze week kreeg ik al mails van de VPS dat de serverload te hoog was 11.6 etc. Hierna ging ik dit controleren en zag ik niks vreemds.

Maar sinds gisteravond/vannacht staat de server onder een zware CPU load met 1x een piek naar 140%.



Nu komt steeds het proces: /usr/sbin/httpd -k start -DSSL naar boven, dit vaak ook meerdere keren (1 tot 10) tegelijk, met een cpuload rond de 60-70% of lager.

Code:

top - 12:25:22 up 1 day, 12:04, 0 users, load average: 0.60, 0.79, 0.85
Tasks: 159 total, 3 running, 156 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.6%us, 4.4%sy, 0.6%ni, 83.5%id, 1.5%wa, 0.0%hi, 0.0%si, 0.3%st
Mem: 3922748k total, 2453076k used, 1469672k free, 302324k buffers
Swap: 1048568k total, 86812k used, 961756k free, 775904k cached


Page: 1 2 3 4    Advanced Search
PID	USER	PR	NI	VIRT	RES	SHR	S	%CPU	%MEM	TIME+	COMMAND
31420	admin	20	0	274m	76m	5588	R	98.3	2.0	0:47.21	/usr/sbin/httpd -k start -DSSL
31744	admin	20	0	265m	67m	5288	R	41.3	1.8	0:39.23	/usr/sbin/httpd -k start -DSSL
6003	mysql	20	0	1316m	245m	5444	S	3.9	6.4	25:11.16	/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/bladewebsites.com-online.identiteit.err --pid-file=/var/lib/mysql/bladewebsites.com-online.identiteit.pid
1123	root	20	0	243m	1084	324	S	2.0	0.0	1:18.88	/sbin/rsyslogd -i /var/run/syslogd.pid -c 5
1	root	20	0	19356	504	292	S	0.0	0.0	0:00.72	/sbin/init
2	root	20	0	0	0	0	S	0.0	0.0	0:00.00	[kthreadd]
3	root	RT	0	0	0	0	S	0.0	0.0	0:01.57	[migration/0]
4	root	20	0	0	0	0	S	0.0	0.0	0:02.64	[ksoftirqd/0]
5	root	RT	0	0	0	0	S	0.0	0.0	0:00.00	[migration/0]
6	root	RT	0	0	0	0	S	0.0	0.0	0:00.41	[watchdog/0]
7	root	RT	0	0	0	0	S	0.0	0.0	0:00.77	[migration/1]
8	root	RT	0	0	0	0	S	0.0	0.0	0:00.00	[migration/1]
9	root	20	0	0	0	0	S	0.0	0.0	0:02.90	[ksoftirqd/1]
10	root	RT	0	0	0	0	S	0.0	0.0	0:00.33	[watchdog/1]

Dit is de error log van apache /var/log/httpd/error_log:

Code:

[Thu Oct 23 00:11:03 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 00:11:03 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 00:11:03 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 00:11:03 2014] [notice] mod_ruid2/0.9.7 enabled
[Thu Oct 23 00:11:03 2014] [notice] Apache/2.2.26 (Unix) mod_ssl/2.2.26 OpenSSL/1.0.1e-fips DAV/2 PHP/5.3.28 configured -- resuming normal operations
[Thu Oct 23 01:00:17 2014] [error] [client 127.0.0.1] client denied by server configuration: /var/www/html/server-status
[Thu Oct 23 01:00:17 2014] [error] [client 127.0.0.1] File does not exist: /var/www/html/403.shtml
[Thu Oct 23 05:24:26 2014] [error] [client 172.243.178.100] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /tmUnblock.cgi
[Thu Oct 23 10:29:19 2014] [error] [client 146.185.239.100] File does not exist: /var/www/html/400.shtml
[Thu Oct 23 10:29:19 2014] [emerg] (13)Permission denied: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [alert] Child 19302 returned a Fatal error... Apache is exiting!
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:20 2014] [emerg] (43)Identifier removed: couldn't grab the accept mutex
[Thu Oct 23 10:29:21 2014] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Thu Oct 23 10:29:21 2014] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Thu Oct 23 10:29:21 2014] [emerg] (22)Invalid argument: couldn't release the accept mutex
[Thu Oct 23 10:29:22 2014] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Thu Oct 23 10:29:24 2014] [emerg] (22)Invalid argument: couldn't grab the accept mutex
[Thu Oct 23 10:30:03 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 10:30:03 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 10:30:03 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 10:30:03 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Oct 23 10:30:04 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 10:30:04 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 10:30:04 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 10:30:04 2014] [notice] mod_ruid2/0.9.7 enabled
[Thu Oct 23 10:30:04 2014] [warn] pid file /var/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Oct 23 10:30:04 2014] [notice] Apache/2.2.26 (Unix) mod_ssl/2.2.26 OpenSSL/1.0.1e-fips DAV/2 PHP/5.3.28 configured -- resuming normal operations
[Thu Oct 23 10:42:34 2014] [error] [client 222.76.242.219] File does not exist: /var/www/html/manager
[Thu Oct 23 10:42:34 2014] [error] [client 222.76.242.219] File does not exist: /var/www/html/404.shtml
[Thu Oct 23 11:48:58 2014] [error] [client 101.226.169.216] File does not exist: /var/www/html/robots.txt, referer: http://bloempaal.com/robots.txt
[Thu Oct 23 11:48:58 2014] [error] [client 101.226.169.216] File does not exist: /var/www/html/404.shtml, referer: http://bloempaal.com/robots.txt
zend_mm_heap corrupted
zend_mm_heap corrupted
[Thu Oct 23 11:58:07 2014] [notice] caught SIGTERM, shutting down
[Thu Oct 23 11:58:08 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 11:58:08 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 11:58:08 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 11:58:08 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Oct 23 11:58:09 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 11:58:09 2014] [warn] RSA server certificate CommonName (CN) `www.alexsysteembouw.nl' does NOT match server name!?
[Thu Oct 23 11:58:09 2014] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Oct 23 11:58:09 2014] [notice] mod_ruid2/0.9.7 enabled
[Thu Oct 23 11:58:09 2014] [notice] Apache/2.2.26 (Unix) mod_ssl/2.2.26 OpenSSL/1.0.1e-fips DAV/2 PHP/5.3.28 configured -- resuming normal operations

om 00:11 en 11:58 heb ik de server een keer restart.

Hier haal ik uit dat er iets mis is met het SSL certificaat voor een van onze sites. Hier heb ik eerder nog geen problemen mee gehad.
Nu heb ik wel sinds 2 dagen wordpress draaien voor een kleine mobiele site /m/. Hier hangen ook een paar security plugins in (wordfence en iThemes Security). Ook is CSF geïnstalleerd in directadmin.

Komt de hoge load van -DSSL hier vandaan of zijn er andere dingen aan de hand met mijn VPS? Zijn hier simpele oplossingen voor om dit probleem te verhelpen en in de toekomst te voorkomen?
Zijn er tools om dit soort processen makkelijk te monitoren en erachter te komen wat dit precies veroorzaakt?

Wat je kan doen,

Met csf kan je apachestatus laten mailen bij hoge load.
Stel dat even goed in.
Dan zie je ook of het mogelijk je WP installatie is die de hoge load veroorzaakt.

De load fluctueert enorm, van een paar % (wat bij ons normaal is) naar 100+% met soms meerdere DSSL processen op 50 60 80%.

Is je Wordfence up-to-date?

https://vexatioustendencies.com/word...ection-bypass/

icm met de problemen bij Openpeering recent zie ik daar wel een paar mogelijkheden in om de load van een server de hoogte in te jagen...

Een -DSSL betekent alleen maar dat apache gestart is met ssl ondersteuning, dus in principe zal de load niks met je ssl te maken hebben (buiten het feit dat ssl verbindingen wel iets meer load hebben dan gewone http verbindingen vanwege de encryptie, maar dat is alleen al er ook daadwerkelijk veel ssl pages worden opgevraagd natuurlijk).

Als je apache 2.4 hebt moet je eens

Mutex posixsem

in je httpd.conf zetten en apache herstarten.

Een hoge kan komen omdat je apache processen crashen en het main httpd process doorlopens bezig is met nieuwe deamons starten. Maar een grotere kans is vaak dat ergens een cronjob hangt of dat een plugin van een site (of site zelf) gewoon ergens een rotte sql query heeft.

Dit weekend geprobeerd het probleem op te lossen, maar nog steeds zonder succes. Misschien dat iemand ons kan helpen die hier wel de juiste kennis van heeft?

als ik een strace doe van een DSSL proces kom ik steeds dit tegen. Kan ik hier uit opmaken dat er iets mis is gegaan met een joomla website? Deze draait al een tijdje en eerder geen problemen mee gehad.

Code:

lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries/idna_convert", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/admin/domains", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/admin", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries/idna_convert/idna_convert.class.php", {st_mode=S_IFREG|0644, st_size=94834, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries/idna_convert", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl/public_html", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/admin/domains/schoonheidssalongeno.nl", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/admin/domains", {st_mode=S_IFDIR|0750, st_size=4096, ...}) = 0
lstat("/home/admin", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home/admin", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0711, st_size=4096, ...}) = 0
open("/home/admin/domains/schoonheidssalongeno.nl/public_html/libraries/idna_convert/idna_convert.class.php", O_RDONLY) = 49
fstat(49, {st_mode=S_IFREG|0644, st_size=94834, ...}) = 0
fstat(49, {st_mode=S_IFREG|0644, st_size=94834, ...}) = 0
fstat(49, {st_mode=S_IFREG|0644, st_size=94834, ...}) = 0
mmap(NULL, 94834, PROT_READ, MAP_SHARED, 49, 0) = 0x7f74d7407000
munmap(0x7f74d7407000, 94834)           = 0
close(49)                               = 0
poll([{fd=48, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(48, "o\0\0\0\3SELECT COUNT(*) FROM jos_us"..., 115) = 115
read(48, "\1\0\0\1\1\36\0\0\2\3def\0\0\0\10COUNT(*)\0\f?\0\25\0\0"..., 16384) = 63
poll([{fd=48, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(48, "m\2\0\0\3INSERT INTO `jos_k2_comment"..., 625) = 625
read(48, "\n\0\0\1\0\1\375\224O\5\2\0\1\0", 16384) = 14
chdir("/")                              = 0
gettimeofday({1414350858, 588564}, NULL) = 0
poll([{fd=48, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
write(48, "R\6\0\0\3UPDATE `jos_session`\nSET `d"..., 1622) = 1622
read(48, "0\0\0\1\0\1\0\2\0\0\0(Rows matched: 1  Cha"..., 16384) = 52
write(48, "\1\0\0\0\1", 5)              = 5
shutdown(48, 2 /* send and receive */)  = 0
close(48)                               = 0
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0
writev(47, [{"HTTP/1.1 200 OK\r\nDate: Sun, 26 O"..., 222}, {"\37\213\10\0\0\0\0\0\0\3", 10}, {"\253V\312M-.NLOU\262R\nJML.\311LU(\311OMO-\3\22)\212\na"..., 64}, {"\334\35\303\v@\0\0\0", 8}], 4) = 304
gettimeofday({1414350858, 600839}, NULL) = 0
write(27, "304 1112\n", 9)              = 9