dbs [Fri, 13 Aug 2010 16:20:01 +0000 (16:20 +0000)]
Make log length in Perl logger a configurable value to assist debugging
If logging to syslog instead of a file, you might need to adjust the
syslog configuration to accept longer than its own default lengths of
log messages
dbs [Fri, 13 Aug 2010 16:16:05 +0000 (16:16 +0000)]
First cut at adding Debian Squeeze to OpenSRF prerequisite installer
Along the way, make a few other changes:
* drop Debian Etch and Gentoo support
* add dscott@laurentian.ca to the author list
* remove Class::DBI as that's actually a prereq for Evergreen, not OpenSRF
scottmk [Tue, 10 Aug 2010 02:13:39 +0000 (02:13 +0000)]
Provide a way for a service to set the effective buffer size for a
specified method.
Non-atomic methods accumulate RESULT messages into a buffer, from which
they are flushed when the buffer is about to overflow, or when the
closing STATUS message is issued.
The new osrfMethodSetBufferSize() function allows the service to favor
large buffers (for greater throughput) or small ones (for a lower
latency for the first response).
Since the buffersize is not an absolute limit, the effective buffer
size may be set to zero, in which case each RESULT message will be
packaged and sent in a separate XMPP message as soon as it is ready.
Changing the buffer size has no effect on an atomic method, nor on a
method that returns only one RESULT message.
M include/opensrf/osrf_application.h
M src/libopensrf/osrf_application.c
scottmk [Mon, 9 Aug 2010 16:18:05 +0000 (16:18 +0000)]
Cruft removal; there should be no outwardly visible effects.
1. Remove the OSRF_METHOD_ATOMIC and OSRF_METHOD_SYSTEM options
from the interface functions osrfRegisterMethod() and
osrfRegisterExtendedMethod().
An application cannot usefully apply these options when it registers
a method, and if it tries, it will almost certainly not work as
intended anyway.
This change required considerable refactoring of the code
responsible for registering methods.
2. When the attempt to initialize an application fails, remove
the application from the application list and destroy it,
instead of keeping it around in an unusable state.
3. Eliminate some redundant lookups of application by name
when registering system methods.
M include/opensrf/osrf_application.h
M src/libopensrf/osrf_application.c
scottmk [Mon, 9 Aug 2010 12:31:33 +0000 (12:31 +0000)]
Fixed a bug in the chunking.
Short version: The non-atomic system methods weren't working. Now
they are.
Long version: When a method returns a value greater than zero, a
post-processing step sends a STATUS message to signify that the
response is complete. In the old code this post-processing
bypassed the buffer-flushing step, so that anything still in the
buffer didn't get sent.
The non-atomic system methods returned positive return codes and
were therefore subject to this problem. I don't know if any
non-system methods return postive return codes, but if they do,
they would also have been affected.
scottmk [Wed, 4 Aug 2010 03:20:33 +0000 (03:20 +0000)]
Implement the chunking of OSRF messages. I.e. bundle multiple
OSRF messages into an XMPP message, up to about 10k bytes, so
as to reduce networking overhead.
M include/opensrf/osrf_application.h
M src/libopensrf/osrf_application.c
1. Add a buffer to osrfAppSession structure; for future use
2. New function osrfSendTransportPayload(). This a repackaging of
existing functionality pulled out into a separate function so that
it can be reused in other contexts.
These changes are preparation for future changes, and will have no
visible effect by themselves.
M include/opensrf/osrf_app_session.h
M src/libopensrf/osrf_app_session.c
Add response chunking support to the Perl implementation of OpenSRF
Two new optional paramters to register_method are now supported:
* max_chunk_size
* max_chunk_count
OpenSRF has always supported message bundling, but only respond_complete made
use of this fact by sending the final result message and the completion status
message in the same XMPP envelope. Now, on a per method basis, RESULT messages
can be bundled (cached) until one of three conditions occurs:
* The size of the JSON of the RESULT messages matches or exceeds max_chunk_size
* The number of RESULT messages cached matches or exceeds max_chunk_count
* respond_complete is called (which happens implicitly by returning from a method)
Because the overhead of sending multiple XMPP messages far outweighs the caching
and cache management costs of chunking, the default for max_chunk_size is set
at 10240 bytes (10k). The default for max_chunk_count is 0. To turn off chunking
completely, set the max_chunk_size register_method parameter to 0.
1. Make osrfMessageToJSON() available at global scope.
2. New function osrf_message_set_result() -- a more efficient alternative
to osrf_message_set_result_content().
Typically when using the older function, we convert a jsonObject to JSON
text, and then parse the JSON text back into a jsonObject. With the new
function we can avoid the round trip through the text format.
M include/opensrf/osrf_message.h
M src/libopensrf/osrf_message.c
scottmk [Tue, 22 Jun 2010 04:23:09 +0000 (04:23 +0000)]
Rearrange the way that a listener juggles its drones.
Instead of maintaining a single circular linked list including both active
processes and idle processes, keep the active drones in the circular list
and the idle drones in a separate linear list; move them back and forth
as needed. This change simplifies how we search for drones when we need
them, because we don't have to skip over ones that we don't want.
The list of idle drones acts as a stack. When we look for an idle drone
for a new request, we pick the one at the head of the list, which is the
last one to have become idle.
As a result, we pick the drone that is most likely still to be in memory.
In the old arrangment, the list acted as a queue. When picking an idle
drone, we picked the one that had been idle the longest -- which was the
one most likely to have been swapped out.
Also: added a number of doxygen-style comments, and tinkered with the
white space here and there.
Note: an earlier commit tried to make similar changes, but it had to be
backed out because it mangled the linkage pointers in some situations,
leading to segfaults. That problem has now been corrected.
dbs [Tue, 15 Jun 2010 14:03:58 +0000 (14:03 +0000)]
Rename example Apache configuration file to include ".conf" suffix
This is useful on systems like Fedora that require the .conf suffix to act
on files in a conf.d subdirectory, and is generally useful for visually
distinguishing the purpose of the file.
scottmk [Mon, 14 Jun 2010 18:31:32 +0000 (18:31 +0000)]
When traversing a list of osrfMessages: don't free the messages.
The list owns them. Instead, get the list to free them when
you're done, by calling osrfListClear().
scottmk [Fri, 7 May 2010 16:40:35 +0000 (16:40 +0000)]
Write the PID file from the child process (the one that launches the
listeners), not from the parent process; and only if it actually launches
a listener.
Otherwise if there are no C apps to launch, the child would die immediately,
leaving the PID file hanging around to confuse the surrounding shell script.
scottmk [Tue, 4 May 2010 13:41:16 +0000 (13:41 +0000)]
Fix a bug that occasionally caused OSRF not to shut down cleanly.
The osrf_ctl.sh script had been using ps + grep to capture
the process ID (PID) of the opensrf-c daemon so that it could
send a SIGINT signal to it later to shut it down. However the
script was also capturing the PIDs of the daemon's child processes
(i.e. the listener processes), which hadn't yet changed to
application-specific names.
As a result, when shutting down, the listener processes would
receive signals from two different sources: from the opensrf-c
daemon and from the surrounding shell script. If the signal
from opensrf-c got there first, the kill from the script would
fail, and the script would abort, even though the process had
been successfully killed.
The solution is for opensrf.c to write the daemon's PID directly
to a file, instead of relying on ps + grep to capture it. The
file name is specified by an additional command line parameter,
which (for upward compatibility) is currently optional.
Because this change involves a change to the osrf_ctl.sh
script, it will be necessary to run configure before the
usual make and make install. If you are using the usual
configuration, run the following from within the OSRF
trunk directory:
If you don't run configure, the old osrf_ctl.sh script will
continue to work as it has in the past, and you won't get
the benefit of the change.
M include/opensrf/utils.h
M include/opensrf/osrf_system.h
M src/libopensrf/utils.c
M src/libopensrf/opensrf.c
M src/libopensrf/osrf_system.c
M bin/osrf_ctl.sh.in
scottmk [Mon, 3 May 2010 02:23:59 +0000 (02:23 +0000)]
Set the appname to the logging routines at the beginning of
osrfAppRegisterApplication() instead of at the end, so that messages
issued during initialization of the application will be properly
identified. This change particularly affects apps that do a lot
of initialization, such as cstore, rstore, and pcrud.
Specifically: it was misbehaving when a string literal contained
a comma, square bracket, or curly brace. Now it pays attention to
whether those characters are in within quotes or not, and treats
them accordingly.
It also applies more consistent indentation by consuming
extraneous white space that would otherwise lead to ragged margins.
scottmk [Tue, 16 Mar 2010 04:41:04 +0000 (04:41 +0000)]
A small performance tweak.
When we receive a new message, we update current_locale with the
locale of the message.
The tweak: if current_locale is already the same as the locale of
the message (which is presumably most of the time in practice), then
don't update it. That way we avoid a malloc() and a free().
scottmk [Mon, 8 Mar 2010 03:26:03 +0000 (03:26 +0000)]
Miscellaneous minor changes. mostly for clarity:
1. Changed return type of _osrfAppRegisterSysMethods from int to void,
since we never looked at the return code anyway.
2. In osrfAppRegisterApplication(): 0pen the shared object before
allocating an osrfApplication, so that we don't have to free the
osrfApplication if the open fails.
3. In osrfAppRegisterExtendedMethod(): when creating an atomic method,
pass the userData pointer to _osrfAppBuildMethod() instead of
installing it on a separate line.
4. In _osrfAppBuildMethod(): for an atomic method, build the method
name correctly the first time, instead of building it incorrectly
and later replacing it.
5. In osrfAppRunMethod(): rearranged things a bit for clarity.
Simplified the declaration and dereferencing of the meth pointer.
6. In osrfAppIntrospect(): Introduced an early return in order to
reduce the level of indentation of the rest of the function. Moved
some declarations closer to their first uses.
7. In osrfAppIntrospectAll(): moved the declaration of resp into the
loop where it is used.
8. Finished adding doxygen-style comments to document the functions.
Touched up the white space here and there.
scottmk [Fri, 26 Feb 2010 03:57:11 +0000 (03:57 +0000)]
Tidying up various things; nothing very substantial.
1. Added newlines and indentation within long macros to make them look
more like real code.
2. The OSRF_METHOD_VERIFY_DESCRIPTION macro is not used anywhere. I
moved it from the header into the implementation file and commented it
out, preserving it like a fly in amber in case we ever want to revive it.
3. Moved the definition of the struct osrfApplication from the header
into the application file, since no other file references it.
4. Moved the OSRF_SYSMETHOD_* macros from the header into the implementation
file, since they are not referenced elsewhere.
5. Turned _osrfAppFindApplication() and osrfAppFindMethod into inline
functions, since each is a trivial wrapper for another function.
6. Added a formal void parameter to osrfAppRunExitCode, so that its
signature is a prototype rather than a mere declaration.
7. Removed a couple of redundant sanity checks.
8. Further tinkered with comments and white space.
M include/opensrf/osrf_application.h
M src/libopensrf/osrf_application.c
scottmk [Mon, 22 Feb 2010 17:53:35 +0000 (17:53 +0000)]
1. Added doxygen-style comments to document all functions.
2. For osrfConfigHasDefault() and osrfConfigCleanup(): added a formal void parameter
so that the header will contain prototypes rather than mere declarations.
3. In osrfConfigValueObject: added a sanity check for a non-loaded configuration.
4. In osrfConfigGetValueList(): replaced a call to jsonObjectToSimpleString() with
a call to jsonObjectGetString(), in order to eliminate a malloc() and free().
5. Tidied up the white space here and there.
M include/opensrf/osrfConfig.h
M src/libopensrf/osrfConfig.c
scottmk [Sun, 21 Feb 2010 02:31:58 +0000 (02:31 +0000)]
1. Reworked the xmlDocToJSON and _xmlDocToJSON functions for clarity.
In particular: _xmlDocToJSON() now returns void, and requires its
second parameter (the jsonObject*) to be non-NULL. The old version
behaved differently depending on whether the second parameter was NULL.
I found that arrangement confusing.
2. Finished adding doxygen-style comments; removed comments from the
header so that they won't override the more complete comments in the
implementation file.
3. Tinkered with white space here and there.
M include/opensrf/xml_utils.h
M src/libopensrf/xml_utils.c
scottmk [Sat, 6 Feb 2010 20:53:31 +0000 (20:53 +0000)]
1. For the "request" command: removed the calls to osrfAppSessionConnect()
and osrf_app_session_disconnect(). There's no point in opening and closing
a connection for a single request.
2. For the "math_bench" command: coerce the first command argument to a
positive value. In the old code, a zero value led to a division-by-zero,
and a negative value led to a segfault.
3. For the "math_bench" command: accept an optional second parameter
to control whether and when we call osrf_app_session_disconnect(). It
has valid values of 0 (the default), 1, and 2; values out of range are
coerced to valid values.
4. In do_math(): initialize the array of floats by setting each float
to zero, instead of by using memset() to fill the array with
all-bits-zero. The C Standard does not guarantee that all-bits-zero
represents zero for a float.
(This last change shouldn't make any difference anyway, because each float
in the array is overwritten before it is read. Still, anything not worth
doing is not worth doing badly.)
dbs [Thu, 28 Jan 2010 03:14:32 +0000 (03:14 +0000)]
Provide a thread if the translator wasn't passed one in the request headers
The OpenSRF-over-HTTP spec says that the translator will generate a thread
in the response if one wasn't passed in the request; however, we were trying
to pass a null pointer back and that doesn't work so well.
As we're already generating a reasonably unique string for the purposes
of delim in the multipart data response, let's also use that for the
response thread when necessary.
erickson [Mon, 25 Jan 2010 16:29:19 +0000 (16:29 +0000)]
Added some fault tolerance to the inbound-to-unixserver message handoff process. In some conditions, the unix socket will result in sigpipe (anecdotally, seen more oftenvirtualized environments). these changes add a call to select before writing to the socket to check for socket availability and add a sigpipe handler that forces the inbound process to wait a brief period of time before trying the delivery again
erickson [Thu, 21 Jan 2010 03:51:33 +0000 (03:51 +0000)]
similar to recent opensrf-c changes, keep active and idle child procs in separate lists for faster access and simpler code. also some formatting/comment updates
scottmk [Thu, 21 Jan 2010 01:08:37 +0000 (01:08 +0000)]
Bug fix.
When all the servers for a given server go away, the router deletes the
server class from its internal data structures. However that can happen
in the middle of a loop receiving successive messages from that server.
The old code would continue trying to read more messages from the
deleted server class, leading to a segfault.
The new code checks to see whether the server class still exists. If
not, it breaks out of the loop.
erickson [Tue, 19 Jan 2010 18:41:24 +0000 (18:41 +0000)]
Patch from Joe Atzberger:
When running from command-line, logger warns "Logger found no config. Using STDERR" but in fact fails to do so. That blocks atomic testing of dependent modules, including almost all of EG's Application and Utils.
Also, in _write_file when failing to sysopen the target file, the die message reports a different variable than the one actually targeted.
scottmk [Mon, 18 Jan 2010 03:00:13 +0000 (03:00 +0000)]
Instead of keeping all child processes in a single list, keep them in two
separate lists: one list of those which are currently servicing requests,
and another list of idle children. Move the children back and forth as
their status changes.
This change has three advantages:
1. When searching for an active child, there is a smaller list to search.
2. When doing a select() to identify children that have become available,
we can look for messages only from the active children, since the idle
ones won't write anything back to the parent. (Actually we could have
done this with a single list, but we didn't.)
3. Probably most important: since the idle list functions as a stack, we
assign the next request to the child that was most recently active, or was
most recently launched. That's the child that is most likely still to be
physically in memory. Formerly we assigned requests to children approximately
on a round-robin basis. As a result we assigned each new request to the
child that was most likely to be swapped out.
Also, in prefork_clear(): kill each child process individually. Sending a
SIGKILL to the entire process group kills the parent as well as the
children.
scottmk [Fri, 15 Jan 2010 04:52:00 +0000 (04:52 +0000)]
Close a substantial resource leak in drone processes.
A drone inherits the transport_client of its parent process,
including a socket and a substantial amount of memory. The
old code avoided freeing the transport_client in order to
avoid disconnecting the parent from Jabber.
The new code contrives to reclaim the resources without
sending a disconnect to Jabber. Hence the parent remains
connected.
M include/opensrf/transport_client.h
M include/opensrf/transport_session.h
M src/libopensrf/osrf_system.c
M src/libopensrf/transport_session.c
M src/libopensrf/transport_client.c
scottmk [Sat, 9 Jan 2010 20:54:57 +0000 (20:54 +0000)]
1. For keeping track of the child processes: use a doubly-linked
list instead of a singly-linked list. The resulting list manipulations
are both simpler (fewer special cases) and faster (no need to traverse
then entire list just to find the end).
2. Maintain a free list of prefork_child structures that have been
allocated but are not currently in use. Allocate from the free list
when possible, in order to avoid churning through malloc() and free().
3. When initializing prefork_child.appname: assign it the same value
as the corresponding field in the parental prefork_simple, instead of
creating a separate copy. The parental copy will remain valid until
after all the prefork_children are gone.
scottmk [Fri, 8 Jan 2010 21:14:35 +0000 (21:14 +0000)]
Added or modified a lot of comments; made some minor tweaks to white space.
Eliminated the min_children member of the prefork_child structure. We
didn't use it for anything, and it made no sense anyway.
Moved the closing of the child's file descriptors into prefork_child_free()
in order to eliminate duplicated code. Also we were closing two of the
file descriptors twice.
prefork_child_free() now returns void instead of a pointless int.
In reap_children(): reset the child_dead at the beginning of the function
instead of at the end. Otherwise we could miss a SIGCHLD that occurs during
the function.
Pass NULL as the second parameter to waitpid(). We don't do anything with the
children's return codes, so there's no need to capture them.
Eliminated all calls to osrf_clearbuf() as pointless.
In prefork_clear(): instead of sleeping for a second after sending the SIGKILL
to the child processes, go through a waitpid() loop. (In practice this
function is all but unreachable anyway.)
scottmk [Wed, 6 Jan 2010 05:15:24 +0000 (05:15 +0000)]
Moved nested #includes out of osrf_prefork.h and into the implementation files.
Somne of the #includes turned out to be unnecessary, so I eliminated them.
In osrf_prefork.c: instead of dynamically allocating a prefork_simple,
allocate it on the stack. That way we avoid a malloc() and free().
Renamed prefork_free() to prefork_clear(), since it no longer frees the
prefork_simple itself, but only memory owned by the prefork_simple. Also
it now returns void instead of int, since there are no errors to detect
or report.
Added some comments.
M include/opensrf/osrf_prefork.h
M src/libopensrf/osrf_system.c
M src/libopensrf/osrf_prefork.c
scottmk [Sun, 3 Jan 2010 19:28:02 +0000 (19:28 +0000)]
Finished adding doxygen-style comments to document the app session
functions. Removed comments from the header so that they wouldn't
override more complete comments in the implementation file.
Tweaked the white space here and there.
Changed the return code in a few cases, for consistency. A couple of
functions seemed unable to decide whether 0 was good and 1 was bad,
or vice versa.
M include/opensrf/osrf_app_session.h
M src/libopensrf/osrf_app_session.c
scottmk [Fri, 1 Jan 2010 22:16:01 +0000 (22:16 +0000)]
Incorporate _osrf_app_request_resend into osrf_app_session_request_resend(),
which had been its only caller. The code is trivial enough that it will be
simple to break out again if necessary. Meanwhile having it all in a single
function makes it more readable.
Free the hash table of requests correctly; i.e. all of each linked list, not
just the first node.
Various additions and refinements to comments; occasional tweaks to white space.
M include/opensrf/osrf_app_session.h
M src/libopensrf/osrf_app_session.c
scottmk [Fri, 1 Jan 2010 14:49:16 +0000 (14:49 +0000)]
Changed the way pending requests are stored in an osrfAppSession.
Before, pending requests were stored in a so-called request_queue.
However it wasn't a queue at all, except in name. It was an
osrfList, i.e. an expandable pointer array used as a random
access container. Request ids were used as subscripts into the
array.
Since we don't reuse request ids (except in the theoretical case
of a wraparound), the array grew without limit. This unbounded
growth in the memory footprint could create problems for a
long-running busy process. It might have contributed to the
rumored instabililty of chopchop, our homegrown Jabber server.
Now, pending requests are stored in a hash table, where each
of 64 slots holds a doubly linked list. There should be no
effect on performance unless there are hundreds or thousands
of pending requests at once, in which case we would probably
get bogged down anyway.
M include/opensrf/osrf_app_session.h
M src/libopensrf/osrf_app_session.c
scottmk [Mon, 28 Dec 2009 03:50:38 +0000 (03:50 +0000)]
Add routines for calculating SHA1 and MD5 digests, based on the
gnutls package. These routines are designed to replace the
current functions shahash() and md5sum(), but they are not
drop-in replacements because they have different signatures.
Prerequisites: packages gcrypt and gnutls.
This code is based on a contribution by Jason Stephenson.
A include/opensrf/osrf_digest.h
A src/libopensrf/osrf_digest.c
scottmk [Sun, 20 Dec 2009 06:37:02 +0000 (06:37 +0000)]
1. In the parent router process: wait for all of the immediate
child processes to terminate before exiting.
This change eliminates the need for the shell script invoking
the router to sleep before running a ps to identify the effective
router processes (which are grandchildren of the parent). By the
time the parent exits, the children will have launched the
grandchildren and exited.
2. If any of the immediate child processes terminates abnormally
(either a non-zero return code or termination by a signal), issue
a warning message to that effect. This message goes to standard
error, since the parent process never opens a log file.
3. Apply the volatile qualifier to a couple of variables that
are updated asynchronously by a signal handler.
M src/router/osrf_router.c
M src/router/osrf_router_main.c
scottmk [Mon, 14 Dec 2009 20:38:44 +0000 (20:38 +0000)]
1. In osrf_app_session.[ch]: Create a new function
osrfAppSessionSendRequest(), similar to osrfAppSessionMakeRequest()
but without the param_strings parameter.
2. Replace the old function with the new one in srfsh.c.
M include/opensrf/osrf_app_session.h
M src/libopensrf/osrf_app_session.c
M src/srfsh/srfsh.c
projects / OpenSRF.git / log