Steve Grubb
18/04/03, 19:35
BACKGROUND
-----------
Xinetd is a popular inetd replacement. Shortly after the 2.3.9 release in
September 2002, it was realized that xinetd was leaking file descriptors.
That problem turned out to be that file descriptors were not always being
closed whenever a connection was rejected. 2.3.10 was released with this
fixup among others in January.
Sometime in February, a machine that I admin was hit by an ftp worm. It
created > 5000 connections in 1 second. Xinetd promptly keeled over.
Xinetd had been running for over a month with no downtime. The machine has
next to no ftp traffic and only from 2 sources, so it was configured to be
run via xinetd rejecting connections via tcp_wrappers. The machine had
weathered worm attacks in the past, so this puzzled me.
TESTING
-------
Eventually, I started looking at xinetd with valgrind. I used the
following commandline:
valgrind --leak-check=yes --leak-resolution=med --num-callers=8 \
--logfile-fd=9 /usr/sbin/xinetd -d -pidfile /var/run/xinetd.pid \
-stayalive 9> out.txt
Depending on your setup, you may need to use something higher than 9.
Xinetd was tested on connections that succeed and connections that are
rejected due to configuration settings. The easiest way to test this is to
use the following setup for chargen:
service chargen
{
type = INTERNAL
user = root
protocol = tcp
wait = no
access_times = 2:00-3:00
# only_from = 192.168.1.3/24
# no_access = 192.168.1.3/24
}
The point is to set it up in a way that the connection is guaranteed to be
rejected. Then do a:
telnet localhost chargen
After a couple seconds "ctl-] quit"
Then, /etc/rc.d/init.d/xinetd stop
Valgrind reports the following:
==18939== 144 bytes in 1 blocks are definitely lost in loss record 36 of 45
==18939== at 0x40160DB8: malloc (vg_clientfuncs.c:103)
==18939== by 0x804FE22: (within /usr/sbin/xinetd)
==18939== by 0x805A496: (within /usr/sbin/xinetd)
==18939== by 0x8053611: (within /usr/sbin/xinetd)
==18939== by 0x805340D: (within /usr/sbin/xinetd)
==18939== by 0x40294A46: __libc_start_main (in /lib/libc-2.3.2.so)
==18939== by 0x804A310: (within /usr/sbin/xinetd)
==18939==
THE PROBLEM
-----------
Using objdump -S /usr/sbin/xinetd, the block of code in question comes
from service.c:
void svc_request( struct service *sp )
{
connection_s *cp ;
status_e ret_code;
cp = conn_new( sp ) ;
if ( cp == CONN_NULL )
return ;
if (sp->svc_not_generic)
ret_code = spec_service_handler(sp, cp);
else
ret_code = svc_generic_handler(sp, cp);
if ( ret_code != OK )
{
if ( SVC_LOGS_USERID_ON_FAILURE( sp ) )
if( spec_service_handler( LOG_SERVICE( ps ), cp ) == FAILED ) {
conn_free( cp, 1 );
return;
}
CONN_CLOSE(cp);
}
}
The above code has several problems. One background piece of information
is that the sigchld handler in xinetd (child_exit->server_end->
svc_postmortem) normally frees the connection's data. If the ret_code is
not OK, the connection was only closed. This is little more than close(cp-
>co_descriptor); This does not free cp since sigchld will not be called.
It was only if the log service call failed that the connection was freed.
The above code also did not take into account ret_code == OK if the
service was no_wait or special. In both of those cases, the sigchld
handler is not invoked so the memory pointed to by cp is lost when the
call returns.
CONSEQUENCES
------------
The memory area pointed to by cp is 144 bytes. Since the variable goes out
of scope, it is permanently lost with no way of finding it again. The
memory losses are cumulative, too. It would take little more than
while true; do telnet localhost chargen < /dev/null; done;
to DOS the services provided by xinetd if you could identify a machine
that uses xinetd to reject connections. Xinetd provides a rich set of
options for rejecting connections, this includes: tcp_wrappers, only_from,
no_access, sensors, access_times, cps, load_avg, etc.
It should also be noted that if you DO NOT have any statements in the
xinetd.conf file that would cause xinetd to reject a connection, then you
are free from this problem.
SOLUTION
--------
Xinetd 2.3.11 fixes the memory leaks as well as other problems discovered
since 2.3.10 was released. All users of xinetd 2.3.10 are strongly urged
to upgrade ASAP to avoid DOS conditions. Anyone running 2.3.9 is also
strongly urged to upgrade since they are leaking file descriptors.
Your xinetd version can be determined by typing "xinetd -version" (that's
version with 1 dash).
The new tarball is: www.xinetd.org/xinetd-2.3.11.tar.gz
This problem has been assigned CAN-2003-0211 to track the bug.
This bug was also reported here:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88537
If you are affected, see if your vendor has an updated xinetd for you.
-Steve Grubb
-----------
Xinetd is a popular inetd replacement. Shortly after the 2.3.9 release in
September 2002, it was realized that xinetd was leaking file descriptors.
That problem turned out to be that file descriptors were not always being
closed whenever a connection was rejected. 2.3.10 was released with this
fixup among others in January.
Sometime in February, a machine that I admin was hit by an ftp worm. It
created > 5000 connections in 1 second. Xinetd promptly keeled over.
Xinetd had been running for over a month with no downtime. The machine has
next to no ftp traffic and only from 2 sources, so it was configured to be
run via xinetd rejecting connections via tcp_wrappers. The machine had
weathered worm attacks in the past, so this puzzled me.
TESTING
-------
Eventually, I started looking at xinetd with valgrind. I used the
following commandline:
valgrind --leak-check=yes --leak-resolution=med --num-callers=8 \
--logfile-fd=9 /usr/sbin/xinetd -d -pidfile /var/run/xinetd.pid \
-stayalive 9> out.txt
Depending on your setup, you may need to use something higher than 9.
Xinetd was tested on connections that succeed and connections that are
rejected due to configuration settings. The easiest way to test this is to
use the following setup for chargen:
service chargen
{
type = INTERNAL
user = root
protocol = tcp
wait = no
access_times = 2:00-3:00
# only_from = 192.168.1.3/24
# no_access = 192.168.1.3/24
}
The point is to set it up in a way that the connection is guaranteed to be
rejected. Then do a:
telnet localhost chargen
After a couple seconds "ctl-] quit"
Then, /etc/rc.d/init.d/xinetd stop
Valgrind reports the following:
==18939== 144 bytes in 1 blocks are definitely lost in loss record 36 of 45
==18939== at 0x40160DB8: malloc (vg_clientfuncs.c:103)
==18939== by 0x804FE22: (within /usr/sbin/xinetd)
==18939== by 0x805A496: (within /usr/sbin/xinetd)
==18939== by 0x8053611: (within /usr/sbin/xinetd)
==18939== by 0x805340D: (within /usr/sbin/xinetd)
==18939== by 0x40294A46: __libc_start_main (in /lib/libc-2.3.2.so)
==18939== by 0x804A310: (within /usr/sbin/xinetd)
==18939==
THE PROBLEM
-----------
Using objdump -S /usr/sbin/xinetd, the block of code in question comes
from service.c:
void svc_request( struct service *sp )
{
connection_s *cp ;
status_e ret_code;
cp = conn_new( sp ) ;
if ( cp == CONN_NULL )
return ;
if (sp->svc_not_generic)
ret_code = spec_service_handler(sp, cp);
else
ret_code = svc_generic_handler(sp, cp);
if ( ret_code != OK )
{
if ( SVC_LOGS_USERID_ON_FAILURE( sp ) )
if( spec_service_handler( LOG_SERVICE( ps ), cp ) == FAILED ) {
conn_free( cp, 1 );
return;
}
CONN_CLOSE(cp);
}
}
The above code has several problems. One background piece of information
is that the sigchld handler in xinetd (child_exit->server_end->
svc_postmortem) normally frees the connection's data. If the ret_code is
not OK, the connection was only closed. This is little more than close(cp-
>co_descriptor); This does not free cp since sigchld will not be called.
It was only if the log service call failed that the connection was freed.
The above code also did not take into account ret_code == OK if the
service was no_wait or special. In both of those cases, the sigchld
handler is not invoked so the memory pointed to by cp is lost when the
call returns.
CONSEQUENCES
------------
The memory area pointed to by cp is 144 bytes. Since the variable goes out
of scope, it is permanently lost with no way of finding it again. The
memory losses are cumulative, too. It would take little more than
while true; do telnet localhost chargen < /dev/null; done;
to DOS the services provided by xinetd if you could identify a machine
that uses xinetd to reject connections. Xinetd provides a rich set of
options for rejecting connections, this includes: tcp_wrappers, only_from,
no_access, sensors, access_times, cps, load_avg, etc.
It should also be noted that if you DO NOT have any statements in the
xinetd.conf file that would cause xinetd to reject a connection, then you
are free from this problem.
SOLUTION
--------
Xinetd 2.3.11 fixes the memory leaks as well as other problems discovered
since 2.3.10 was released. All users of xinetd 2.3.10 are strongly urged
to upgrade ASAP to avoid DOS conditions. Anyone running 2.3.9 is also
strongly urged to upgrade since they are leaking file descriptors.
Your xinetd version can be determined by typing "xinetd -version" (that's
version with 1 dash).
The new tarball is: www.xinetd.org/xinetd-2.3.11.tar.gz
This problem has been assigned CAN-2003-0211 to track the bug.
This bug was also reported here:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88537
If you are affected, see if your vendor has an updated xinetd for you.
-Steve Grubb