From 55807240854d0d532a5f70f1f483ed8086eb9d95 Mon Sep 17 00:00:00 2001 From: Mike Rylander Date: Thu, 10 Sep 2015 16:56:13 -0400 Subject: [PATCH] LP#1494486: Limit damage caused by dropped drone XMPP sockets It is apparently possible for drones to get into a state where their XMPP socket is closed but they don't notice. This is bad because the drone can continue to receive requests from its listener but can no longer respond to them. To limit the pain this can cause, we should kill the drone as soon as we notice this condition. To avoid overhead, this commit notices when the socket returns an error (or raises a signal, in Perl) upon write, and exits immediately. One message will be lost, but the drone will no longer be a black hole that does nothing but absorb requests it can never fill. To test ------- [1] Start an OpenSRF stack and look for a drone process. [2] Use lsof to identify which socket that drone is using to talk to XMPP. [3] Use gdb to attach to the process and close the socket, e.g., $ gdb -p $PID (gdb) p close(11) # or whatever the socket number was (gdb) c [4] Use srfsh to make requests of that service. Eventually, one of them will hit the drone. [5] Sans patch, the request will get handled by the drone, but the results will never get sent, and the drone will remain available to handle other requests. [6] With the patch, the drone will exit when it discovers that it can no longer write to the XMPP socket. Signed-off-by: Mike Rylander Signed-off-by: Galen Charlton --- src/libopensrf/osrf_app_session.c | 6 ++++-- src/perl/lib/OpenSRF/Transport/SlimJabber/XMPPReader.pm | 5 +++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/src/libopensrf/osrf_app_session.c b/src/libopensrf/osrf_app_session.c index 8a14d2a..3e00393 100644 --- a/src/libopensrf/osrf_app_session.c +++ b/src/libopensrf/osrf_app_session.c @@ -1011,8 +1011,10 @@ int osrfSendTransportPayload( osrfAppSession* session, const char* payload ) { message_set_osrf_xid( t_msg, osrfLogGetXid() ); int retval = client_send_message( session->transport_handle, t_msg ); - if( retval ) - osrfLogError( OSRF_LOG_MARK, "client_send_message failed" ); + if( retval ) { + osrfLogError( OSRF_LOG_MARK, "client_send_message failed, exit()ing immediately" ); + exit(99); + } osrfLogInfo(OSRF_LOG_MARK, "[%s] sent %d bytes of data to %s", session->remote_service, strlen( payload ), t_msg->recipient ); diff --git a/src/perl/lib/OpenSRF/Transport/SlimJabber/XMPPReader.pm b/src/perl/lib/OpenSRF/Transport/SlimJabber/XMPPReader.pm index 857bee7..d1ebfa1 100644 --- a/src/perl/lib/OpenSRF/Transport/SlimJabber/XMPPReader.pm +++ b/src/perl/lib/OpenSRF/Transport/SlimJabber/XMPPReader.pm @@ -166,6 +166,11 @@ sub tcp_connected { # ----------------------------------------------------------- sub send { my($self, $xml) = @_; + + local $SIG{'PIPE'} = sub { + $logger->error("Disconnected from Jabber server, exiting immediately"); + exit(99); + }; $self->{socket}->print($xml); } -- 2.43.2