9 years agoTest for Apache executable in sbin dirs
Dan Scott [Sun, 27 Jan 2013 16:08:12 +0000 (11:08 -0500)]
Test for Apache executable in sbin dirs

Debian apparently doesn't include /usr/sbin in the default user PATH, so
add an APXS2-like lookup with likely locations if we don't find the
apache executable in our standard path.

Signed-off-by: Dan Scott <>
Signed-off-by: Jason Etheridge <>
Signed-off-by: Ben Shum <>
9 years agoSupport Apache 2.4 client IP address lookups
Dan Scott [Thu, 17 Jan 2013 02:59:22 +0000 (21:59 -0500)]
Support Apache 2.4 client IP address lookups

The Apache 2.4 API replaces the conn_rec->remote_ip member with
conn_rec->client_ip or conn_rec->useragent_ip (per client_ip
seems to be the best choice for us, to avoid everything being logged
from the load balancer or proxy instead of the client's address.

This code detects the version of Apache and uses client_ip if it is 2.4
or greater.

Signed-off-by: Dan Scott <>
Signed-off-by: Bill Erickson <>
9 years agoPrevent "uninitialized value" warnings in parameter logging
Bill Erickson [Tue, 26 Jun 2012 13:21:57 +0000 (09:21 -0400)]
Prevent "uninitialized value" warnings in parameter logging

Use of uninitialized value $p[0] in join or string at
/usr/local/share/perl/5.10.1/OpenSRF/ line 130.

This is caused when the array of method params contains null/undef
values.  Prevent the warnings by replacing these values w/ empty

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
Signed-off-by: Galen Charlton <>
9 years agoChange OSRF_JAVA_DEPSDIR so building Java just works.
Jason Stephenson [Wed, 31 Oct 2012 14:21:18 +0000 (10:21 -0400)]
Change OSRF_JAVA_DEPSDIR so building Java just works.

OSRF_JAVA_DEPSDIR was being set to /opt/java by configure without any
documentation to that effect. The src/java/ downloads the deps files
to src/java/deps. This means that you cannot just build java without either
copying the jars from src/java/deps or altering src/java/Makefile.

This edit changes the AC_SUBST call to replace OSRF_JAVA_DEPSDIR with deps,
so that you can at least build java without making any changes.

Signed-off-by: Jason Stephenson <>
Signed-off-by: Bill Erickson <>
9 years agoSafer casting in java config parsing
Bill Erickson [Thu, 1 Nov 2012 13:41:52 +0000 (09:41 -0400)]
Safer casting in java config parsing

Java config parsing, in particular getString and getInt, no longer fail
when encountering a number value in the internl JSON->Map object.

Java does not allow the following cast:

String s = (String) someNumber;

Instead, rely on the object's toString() which is safe and guaranteed
to exist.

Signed-off-by: Bill Erickson <>
Signed-off-by: Jason Stephenson <>
Signed-off-by: Dan Scott <>
10 years agoTypo in 'framweork'
Dan Scott [Mon, 19 Nov 2012 15:51:26 +0000 (10:51 -0500)]
Typo in 'framweork'

Reported by David Busby of edoceo in - thanks David!

Signed-off-by: Dan Scott <>
10 years agoAdd 2.1.2 ChangeLog entries
Dan Scott [Sun, 18 Nov 2012 12:49:28 +0000 (07:49 -0500)]
Add 2.1.2 ChangeLog entries

Signed-off-by: Dan Scott <>
10 years agoBump version numbers for 2.1.2
Dan Scott [Sun, 18 Nov 2012 12:41:46 +0000 (07:41 -0500)]
Bump version numbers for 2.1.2

Signed-off-by: Dan Scott <>
10 years agoWhitespace in OpenSRF/
Dan Scott [Fri, 16 Nov 2012 15:36:28 +0000 (10:36 -0500)]
Whitespace in OpenSRF/

Spaces not tabs.

Signed-off-by: Dan Scott <>
10 years agoMove setting of OpenSRF::Application::shared_conf to load_bootstrap_config.
Jason Stephenson [Thu, 15 Nov 2012 17:20:11 +0000 (12:20 -0500)]
Move setting of OpenSRF::Application::shared_conf to load_bootstrap_config.

shared_conf was not available and broke clients that were not in a
service, such as the Evergreen CStoreEditor.

Signed-off-by: Jason Stephenson <>
Signed-off-by: Dan Scott <>
10 years agoBump release numbers, create 2.1.1 change log
Dan Scott [Wed, 31 Oct 2012 03:39:25 +0000 (23:39 -0400)]
Bump release numbers, create 2.1.1 change log

Signed-off-by: Dan Scott <>

10 years agoChopChop is dead, long live ChopChop.
Bill Erickson [Mon, 21 May 2012 20:42:12 +0000 (16:42 -0400)]
ChopChop is dead, long live ChopChop.

It was never fully baked, nor was it secure, nor does it compile cleanly
with -Wall, nor does it provide any useful examples for working with
opensrf, etc.  Time to clear out.  So long, chopchop.

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoBump libopensrf version-info revision and age
Dan Scott [Tue, 6 Nov 2012 17:32:48 +0000 (12:32 -0500)]
Bump libopensrf version-info revision and age

The log redaction functionality changed the source files (thus 'age'
gets bumped) and is backwards-compatible (thus 'revision' gets bumped)
but maintains the same interface (thus 'current' stays the same).

Signed-off-by: Dan Scott <>
10 years agoPurge the wildly out-of-date Roadmap document
Dan Scott [Wed, 31 Oct 2012 03:27:34 +0000 (23:27 -0400)]
Purge the wildly out-of-date Roadmap document

Signed-off-by: Dan Scott <>
10 years agoProtect against empty/invalid log_protect sections
Dan Wells [Wed, 31 Oct 2012 14:21:14 +0000 (10:21 -0400)]
Protect against empty/invalid log_protect sections

Connectivity would fail if the log_protect section of opensrf_core.xml
was empty, or contained only comments, as it does in the default
example. Add a simple guard against this potential problem.  This fix
will also protect against cases where the 'log_protect' section
contains text rather than separate configuration elements.

Credit to Dan Scott for finding the bug and working out how to fix it.

Signed-off-by: Dan Wells <>
Signed-off-by: Dan Scott <>
10 years agoConsolidate duplicate osrfMethodVerifyContext handler
Bill Erickson [Wed, 17 Oct 2012 19:57:14 +0000 (15:57 -0400)]
Consolidate duplicate osrfMethodVerifyContext handler

Log redaction was not occuring in some cases because the code was using
the OSRF_METHOD_VERIFY_CONTEXT macro instead of the similarly named
function which does the same thing.  This change points the macro at the
function so that all code uses the same underlying code.

Note this change turns on CALL param logging unconditionally, whereas
users of the macro would previously have been able to avoid CALL logging
via the OSRF_LOG_PARAMS variable.  In practice, little code uses the
macro and all code enables OSRF_LOG_PARAMS.  If we need to add this
control back, it can be added directly to osrfMethodVerifyContext().
For now, it's one less env variable we need to define.

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoLog redaction for sensitive input values, Perl side
Dan Wells [Fri, 12 Oct 2012 13:45:35 +0000 (09:45 -0400)]
Log redaction for sensitive input values, Perl side

This commit attempts to do the same as the C log redaction fix,
but now at the Perl level.  The Perl configuration code was a
little more crufty than the C side, so an additional feature was
added to to support the new 'shared' section.  At some
point we should consider a ground-up rewrite of, as the
code seems to suffer some from its INI file roots.

Signed-off-by: Dan Wells <>
Signed-off-by: Dan Scott <>
10 years agoLog redaction for sensitive input values, C side
Dan Wells [Wed, 10 Oct 2012 21:28:07 +0000 (17:28 -0400)]
Log redaction for sensitive input values, C side

Some service/methods deal with sensitive information (passwords,
financial, etc.).  All input values (e.g. gateway method params)
are currently logged in the activity log regardless of your log
level.  This commit will allow you to redact the params of any
method which matches a configurable set of left-anchored string

This commit adds the initial config, and covers redaction of method
parameters sent through the gateway, the translator, and more general
OpenSRF C applications.

Signed-off-by: Dan Wells <>
Signed-off-by: Dan Scott <>
10 years agoFix installation on Ubuntu Precise.
Jason Stephenson [Tue, 6 Nov 2012 14:16:18 +0000 (09:16 -0500)]
Fix installation on Ubuntu Precise.

Remove libreadline5-dev and replace with libreadline-dev for Ubuntu
and Debian.  This installs libreadline6, but everything still works
since there's nothing version specific in our use of libreadline.

Signed-off-by: Jason Stephenson <>
Signed-off-by: Dan Scott <>
10 years agoUse apt-get instead of aptitude on Debian / Ubuntu
Dan Scott [Mon, 22 Oct 2012 15:50:21 +0000 (11:50 -0400)]
Use apt-get instead of aptitude on Debian / Ubuntu

Reports from the field state that aptitude is not installed by default on
Ubuntu any longer; thus go with the sure thing.

Signed-off-by: Dan Scott <>
10 years agoRemove the reference to Evergreen in the README
Dan Scott [Mon, 22 Oct 2012 15:53:24 +0000 (11:53 -0400)]
Remove the reference to Evergreen in the README

Stephen Wills reported problems installing Evergreen because of missing
dependencies, and reading the OpenSRF README shows that it refers to the
Evergreen prerequisite installer in a confusing way that might lead to
users using only the Evergreen Makefile.install, rather than using the
OpenSRF Makefile.install.

In addition, Fedora 16 is going to be out of support soon, and Fedora 17
and 18 are well-tested targets now.

Signed-off-by: Dan Scott <>
10 years agoNo, Ubuntu Hardy is not well-tested for 2.1.0
Dan Scott [Thu, 31 May 2012 03:12:32 +0000 (23:12 -0400)]
No, Ubuntu Hardy is not well-tested for 2.1.0

One more reference to Ubuntu Hardy that we can remove from the

Signed-off-by: Dan Scott <>
10 years agoBump version numbers to help with library compatibility
Dan Scott [Thu, 31 May 2012 02:33:17 +0000 (22:33 -0400)]
Bump version numbers to help with library compatibility

By bumping the OpenSRF version numbers for Perl libs, Python libraries,
and the libtool versioning for the libopensrf library, projects that
build on OpenSRF will be able to flag the minimum required version and
bail out early if an older version of OpenSRF has been installed.

Signed-off-by: Dan Scott <>
10 years agoAdd "tar" to base prereqs to appease autotools
Dan Scott [Tue, 22 May 2012 20:45:33 +0000 (16:45 -0400)]
Add "tar" to base prereqs to appease autotools

On a minimal install of Fedora 17, I was running into the perplexing
error message whilst running 'autoreconf -i':

libtoolize: can not copy `/usr/share/libtool/config/' to `./'

This was confusing because a manual copy would, in fact, copy the file
just fine. finally
lead to enlightenment: autotools uses "tar" to copy files, not "cp".
Thus, to avoid similar head-scratching install problems in the future,
add "tar" to the base prerequisites for building OpenSRF (both to the
DEBs list and the RPMs list).

Signed-off-by: Dan Scott <>
10 years agoUpdate OS support in README / Makefile.install
Dan Scott [Tue, 22 May 2012 06:26:04 +0000 (02:26 -0400)]
Update OS support in README / Makefile.install

Kick Debian Lenny, Ubuntu Hardy, and RHEL 5 / CentOS 5 to the curb.

Also change the name of the README to the more generic & instructive
"Installing OpenSRF" rather than "README for OpenSRF #.#.#".

Signed-off-by: Dan Scott <>
10 years agoNagios Example Plugin for monitoring services
Thomas Berezansky [Wed, 23 May 2012 14:13:41 +0000 (10:13 -0400)]
Nagios Example Plugin for monitoring services

Does not check that all drones in a brick are fully up, just that the
entire brick will respond to each service, even if only one drone happens
to be running the listener properly (and said listener is responding).

Signed-off-by: Thomas Berezansky <>
Signed-off-by: Dan Scott <>
10 years agoDetect and repair multipart/mixed message delivery errors
Bill Erickson [Mon, 2 Apr 2012 18:55:56 +0000 (14:55 -0400)]
Detect and repair multipart/mixed message delivery errors

For unknown reasons, the Content-Type header will occasionally be
included in the XHR.responseText for multipart/mixed messages.  When
this happens, strip the header and newlines from the message body and

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoRemove Evergreen 2.0 from automated builds
Dan Scott [Wed, 23 May 2012 04:50:49 +0000 (00:50 -0400)]
Remove Evergreen 2.0 from automated builds

rel_2_0 is no longer in support, thus it's time to remove it from the
automated testing builds.

Signed-off-by: Dan Scott <>
10 years agofix up index/position type for calls of various osrfList* functions
Galen Charlton [Mon, 21 May 2012 21:29:05 +0000 (17:29 -0400)]
fix up index/position type for calls of various osrfList* functions

osrfListSet, osrfListRemove, osrfListGetIndex, and osrfListExtract
all expect unsigned ints for the index/position parameter.

src/jserver is ignored by this commit as its removal is pending.

Signed-off-by: Galen Charlton <>
10 years agoRemove comparisons that can never evaluate to true
Dan Scott [Sat, 5 May 2012 05:58:22 +0000 (01:58 -0400)]
Remove comparisons that can never evaluate to true

Using clang as the compiler results in 4 warnings like the following:

osrf_list.c:106:23: warning: comparison of unsigned expression < 0 is
always false [-Wtautological-compare]
        if(!list || position < 0) return NULL;
                    ~~~~~~~~ ^ ~

(Explanation: "position" is an unsigned int; thus the comparison to < 0
can never evaluate to true).

Signed-off-by: Dan Scott <>
Signed-off-by: Galen Charlton <>
10 years agoLP954059: Silence uninitialized var warning
Dan Scott [Sat, 5 May 2012 05:32:25 +0000 (01:32 -0400)]
LP954059: Silence uninitialized var warning

Compiling osrf_utf8.c generates the following warning:

osrf_utf8.c:510:29: warning: utf8_char may be used uninitialized in this
function [-Wuninitialized]

So... initialize utf8_char when we declare it, and make the compiler

Signed-off-by: Dan Scott <>
Signed-off-by: Galen Charlton <>
10 years agoLP# 953299 - defend against null and zero-length cache keys
Galen Charlton [Mon, 21 May 2012 21:02:34 +0000 (17:02 -0400)]
LP# 953299 - defend against null and zero-length cache keys

Ignore undefined and zero-length (after key normalization) cache

Signed-off-by: Galen Charlton <>
Signed-off-by: Dan Scott <>
10 years agoLP# 953299 - Prevent get/set of invalid cache keys
Dan Scott [Mon, 21 May 2012 16:18:41 +0000 (12:18 -0400)]
LP# 953299 - Prevent get/set of invalid cache keys

Clients of OpenSRF::Utils::Cache occasionally request cache keys that
contain invalid characters (a particular case is ISBNs that contain
spaces), so strip those out of incoming get/set requests to avoid ugly
memcached errors.

Signed-off-by: Dan Scott <>
Signed-off-by: Galen Charlton <>
10 years ago.gitignore - ignore more built stuff in fewer lines
Dan Scott [Mon, 21 May 2012 16:41:25 +0000 (12:41 -0400)]
.gitignore - ignore more built stuff in fewer lines

Use the power of glob to catch more built files that we want to ignore,
while simplifying the .gitignore file significantly.

Signed-off-by: Dan Scott <>
10 years agoRevert inadvertently introduced debugging code
Dan Scott [Mon, 21 May 2012 15:48:26 +0000 (11:48 -0400)]
Revert inadvertently introduced debugging code

In commit 91b8790c, several debugging lines were accidentally committed
to the OpenSRF caching utility. Revert that change.

Signed-off-by: Dan Scott <>
10 years agoMinor cleanups to Java dependency build process
Dan Scott [Wed, 2 May 2012 02:11:00 +0000 (22:11 -0400)]
Minor cleanups to Java dependency build process

While still hardcodes the expected location of the Java
dependencies (JSON and Memcached classes), we can clean up some of the
lower-level challenges with the Java build.

* StAX and WSTX were rolled into the core Java environment long ago,
  so there's no need to download them and even less need to verify
  that the downloaded version exist.
* The Memcached class is now up to 2.0.1, per, but the
  file was pointing at 1.5.1.
* Speaking of the file, nothing seems to use it; delete it.
* The Makefile was dying because the variables for the Memcached and
  JSON classes weren't defined. Define those at the top of the Makefile.

Signed-off-by: Dan Scott <>
10 years agoClean up root directory: INSTALL, .gitignore, autoreconf
Dan Scott [Wed, 2 May 2012 02:08:23 +0000 (22:08 -0400)]
Clean up root directory: INSTALL, .gitignore, autoreconf

* Update the README to avoid running autoreconf with the "-f" flag.
* Make the INSTALL file a symbolic link to the README to avoid
  creating a generic INSTALL file each time autoreconf is run.
* Add a .gitignore file that ignores all of the built files.

Signed-off-by: Dan Scott <>
10 years agoChange Java to include . in CLASSPATH
Dan Scott [Wed, 2 May 2012 01:31:38 +0000 (21:31 -0400)]
Change Java to include . in CLASSPATH

Following the directions given by running was a little bit
frustrating; we need to work in some documentation on how to use the
Java bits of OpenSRF - and perhaps look at a more standard option like
maven for dependencies.

Signed-off-by: Dan Scott <>
10 years agoJava Gateway interface improved exception handling
Bill Erickson [Fri, 16 Mar 2012 13:56:13 +0000 (09:56 -0400)]
Java Gateway interface improved exception handling

Handle any exceptions that should not reasonably occur in normal
operation under the covers.  Bubble the rest up.  Update test code with

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoJava gateway interface test class
Bill Erickson [Thu, 15 Mar 2012 21:26:16 +0000 (17:26 -0400)]
Java gateway interface test class

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoJava HTTP gateway interface
Bill Erickson [Mon, 27 Feb 2012 23:05:07 +0000 (18:05 -0500)]
Java HTTP gateway interface

Supports sync and async requests.  Async requests support onResponse,
onComplete, and onError handlers.

Supports a max-threads value to limit the number of activately
communicating threads over any connection.  When max-threads is reached,
requests are queued and delivered as soon as there is room.

Note that since this is talking to the OpenSRF gateway and not the
translater, responses are simply collected and passed one at a time to
onResponse.  They are not streamed.  The goal of supporting onResponse
is to provide the same client API for both the gateway and translator.

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoRemove GNU default INSTALL file
Dan Scott [Mon, 23 Apr 2012 18:17:35 +0000 (14:17 -0400)]
Remove GNU default INSTALL file

GNU autotools generate an INSTALL file that is generic and which
conflicts with the instructions given in the README. Installation is
already hard enough without conflicting instructions, so delete the

Signed-off-by: Dan Scott <>
10 years agoautotools - do not explicitly include m4 directory
Dan Scott [Mon, 23 Apr 2012 18:13:37 +0000 (14:13 -0400)]
autotools - do not explicitly include m4 directory

Some versions of autotools complain bitterly if an m4 include directory
is specified that does not exist.

Signed-off-by: Dan Scott <>
10 years agoName the bootstrapping steps in buildbot
Dan Scott [Mon, 23 Apr 2012 17:45:14 +0000 (13:45 -0400)]
Name the bootstrapping steps in buildbot

Signed-off-by: Dan Scott <>
10 years agoFix buildbot configuration
Dan Scott [Mon, 23 Apr 2012 17:16:25 +0000 (13:16 -0400)]
Fix buildbot configuration

Need to define our tests before they're called; also, can't blindly
invoke a step out of the proper scope. Duh.

Signed-off-by: Dan Scott <>
10 years agoSwitch to autoreconf instead of
Dan Scott [Mon, 23 Apr 2012 16:52:57 +0000 (12:52 -0400)]
Switch to autoreconf instead of

Update the buildbot config accordingly to avoid erroneous errors of

Signed-off-by: Dan Scott <>
10 years agoUpdated buildbot.cfg for current build environments
Dan Scott [Sun, 8 Apr 2012 18:53:36 +0000 (14:53 -0400)]
Updated buildbot.cfg for current build environments

* Drop Ubuntu 8.04 from the list of build environments.
* OpenSRF: drop rel_1_6, add rel_2_1
* Evergreen: drop rel_1_6_1, add rel_2_2
* Adjust for buildbot 0.86 API changes to avoid deprecation
* Improve PEP8 compliance

Signed-off-by: Dan Scott <>
10 years agoAdd trailing period, "localhost" to ejabberd.cfg example
Dan Scott [Fri, 23 Mar 2012 17:05:31 +0000 (13:05 -0400)]
Add trailing period, "localhost" to ejabberd.cfg example

Thanks to Yamil Suarez for noticing a discrepancy between the older wiki
instructions and the README.

Signed-off-by: Dan Scott <>
10 years agoPrevent undef warnings on perl socket read nbytes test
Bill Erickson [Wed, 14 Mar 2012 15:38:18 +0000 (11:38 -0400)]
Prevent undef warnings on perl socket read nbytes test

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoDetect remote disconnect in Perl XMPP reader
Bill Erickson [Mon, 20 Feb 2012 19:40:30 +0000 (14:40 -0500)]
Detect remote disconnect in Perl XMPP reader

When the jabber server severs the connection, it leaves the Perl libs
with a socket that perpetually appears ready for reading, but always
returns 0 bytes.  This produces a loop in the client code.  This change
detects this situation, logs an error, and throws an exception.

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoProtect gateway from format-string crashes in data
Dan Wells [Tue, 6 Mar 2012 20:08:33 +0000 (15:08 -0500)]
Protect gateway from format-string crashes in data

As a common security measure, printf-style formatting codes are
not allowed to be directly interpreted from a writable segment.
The gateway code currently has the following function call:

osrfLogActivity( OSRF_LOG_MARK, act->buf );

This is a variadic function which expects the 'act->buf' position
to contain a format string and any trailing arguments to be the
values passed to the formatter.  Since act->buf is the value of
what we passed in, some data inadvertantly contains format strings,
and since it is a writable segment, the program crashes.  Here is
an example of a crash-causing call:


The param is interpreted as "%1n" and abruptly fails.

The simple solution is to include a formatter so that our param gets
demoted to being mere data, i.e.:

osrfLogActivity( OSRF_LOG_MARK, "%s", act->buf );

Signed-off-by: Dan Wells <>
Signed-off-by: Dan Scott <>
10 years agoStop warning about missing parentheses...
Thomas Berezansky [Wed, 7 Mar 2012 21:42:21 +0000 (16:42 -0500)]
Stop warning about missing parentheses... adding them.

Signed-off-by: Thomas Berezansky <>
10 years agoRefresh child file handles on status read
Bill Erickson [Thu, 16 Feb 2012 22:31:29 +0000 (17:31 -0500)]
Refresh child file handles on status read

If a child process dies while the parent is attempting to read status
information from the child, the pipe connecting the two will be cleaned
up.  The parent mustn't attempt to read from the dead pipe or it will
result in a bad file descriptor error, culminating in a high-cpu
infinite loop on the parent (Listener) process.

Signed-off-by: Bill Erickson <>
Signed-off-by: Thomas Berezansky <>
Signed-off-by: Jason Stephenson <>
10 years agoPython libs for OpenSRF ingress tracking
Bill Erickson [Fri, 13 Jan 2012 15:57:59 +0000 (10:57 -0500)]
Python libs for OpenSRF ingress tracking

This also set the "srfsh" ingress value for

Signed-off-by: Bill Erickson <>
Signed-off-by: Mike Rylander <>
10 years agoSet OpenSRF ingress value for srfsh/gateways
Bill Erickson [Fri, 13 Jan 2012 14:35:18 +0000 (09:35 -0500)]
Set OpenSRF ingress value for srfsh/gateways

Sets the "srfsh", "gateway-v1", and "translator-v1" ingress values

For the translater, it's necessary to stamp the unpacked messages with
the updated ingress, then re-serialize before sending the messages along.

Signed-off-by: Bill Erickson <>
Signed-off-by: Mike Rylander <>
10 years agoC libs for OpenSRF ingress tracking
Bill Erickson [Fri, 13 Jan 2012 14:15:05 +0000 (09:15 -0500)]
C libs for OpenSRF ingress tracking


Signed-off-by: Bill Erickson <>
Signed-off-by: Mike Rylander <>
10 years agoPerl libs for OpenSRF ingress tracking
Bill Erickson [Thu, 12 Jan 2012 21:36:16 +0000 (16:36 -0500)]
Perl libs for OpenSRF ingress tracking

Ingress is a free-form text value which represents the entry point for
the client into the opensrf network.  The value is passed within opensrf
messages, similar to "locale".  Clients should specify the ingress
before any opensrf communication occurs.


Stock values include the following:

opensrf (default)

Signed-off-by: Bill Erickson <>
Signed-off-by: Mike Rylander <>
10 years agoCopy opensrf_core.xml.example to a clean opensrf_core.xml file
Dan Scott [Wed, 29 Feb 2012 15:48:38 +0000 (10:48 -0500)]
Copy opensrf_core.xml.example to a clean opensrf_core.xml file

Ben Shum with the assist again.

Also, add ~/.srfsh.xml to the definition list of interesting OpenSRF
configuration files.

Signed-off-by: Dan Scott <>
10 years agoInclude explicit instructions to copy the opensrf config files
Dan Scott [Wed, 29 Feb 2012 15:42:58 +0000 (10:42 -0500)]
Include explicit instructions to copy the opensrf config files

Also noted by the sharp-eyed Ben Shum, we didn't tell people to actually
copy opensrf.xml.example / opensrf_core.xml.example, which could lead to

Signed-off-by: Dan Scott <>
10 years agoUpdate max_user_sessions suggestion to 10000 to match wiki
Dan Scott [Wed, 29 Feb 2012 15:27:36 +0000 (10:27 -0500)]
Update max_user_sessions suggestion to 10000 to match wiki

Ben Shum noticed that the README still specified 1000 for
max_user_sessions in the ejabberd.cfg file, whereas current larger
OpenSRF systems with many services may blow past that limit. Update to
match the current wiki specification of 10000.

Signed-off-by: Dan Scott <>
10 years agoRemove reference to Fedora specific version in prereq installer
Dan Scott [Wed, 4 Jan 2012 20:21:36 +0000 (15:21 -0500)]
Remove reference to Fedora specific version in prereq installer

Fedora changes every 6 months, so it's probably silly to have a make
target of "fedora14" when that is no longer supported by the Fedora
project itself. As Fedora has packaged all of the OpenSRF dependencies
(thanks, Ben Webb!), in theory it should be supported by the latest
releases of Fedora in the future... as long as we don't introduce any
dependencies on deprecated versions of packages.

Signed-off-by: Dan Scott <>
10 years agoUpdate README to match Evergreen's format
Dan Scott [Wed, 4 Jan 2012 20:17:48 +0000 (15:17 -0500)]
Update README to match Evergreen's format

Reference the various accounts in use consistently.

Correct reference to "fedora" target in Makefile.install.

Put the developer preamble up front.

Provide more Fedora examples.

Do not include leading "#" and "$" in bash examples, to make it easier
for people to copy and paste.

Include source highlighting instructions and titles for example

Signed-off-by: Dan Scott <>
10 years agoJava dependencies update
Bill Erickson [Fri, 24 Feb 2012 21:28:27 +0000 (16:28 -0500)]
Java dependencies update

As of Java 6, XML Stax parsing is natively supported.  This change
removes the external Stax dependencies and updates how the JSON libs are
fetched (in, which now come from github.

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoWe're not in Subversion anymore, Dorothy
Dan Scott [Tue, 21 Feb 2012 14:55:57 +0000 (09:55 -0500)]
We're not in Subversion anymore, Dorothy

We're in git now, make the README reflect that accordingly. Thanks to
Warren Layton for the tip!

Signed-off-by: Dan Scott <>
10 years agoPerl parent/child write improvements
Bill Erickson [Tue, 14 Feb 2012 14:10:58 +0000 (09:10 -0500)]
Perl parent/child write improvements

* Updated variable names for clarity
* Added more inline comments
* Added additional error logging
* For severe read errors, allow the child to gracefully skip the request

Signed-off-by: Bill Erickson <>
Signed-off-by: Jason Stephenson <>
Signed-off-by: Mike Rylander <>
10 years agoPerl pipe reading overhaul : data size header
Bill Erickson [Mon, 13 Feb 2012 21:53:59 +0000 (16:53 -0500)]
Perl pipe reading overhaul : data size header

The lockfile mechanism for preventing premature end of reads on child
processes suffers from one serious flaw:  if the data to write exceeds
the pipe buffer size, the parent will block on syswrite and the service
will lock up.  It's also not as effecient (for the normal case) as the
code was without the lockfile, becasue the writes and reads are

This commit replaces the lockfile mechanism with a protocol header in
the data.  The first X (currently 12) bytes of data written to the child
process will contain the full length of the data to be written (minus
the header size).  The child now reads the data in parallel with the parent as
data is available.  If the child reads all available data (in the pipe)
but not all of the expected data, the child will go back into a select()
wait pending more data from the parent.  The process continues until all
data is read.

This same mechanism is already used to commicate status info from child
processes to the parent.

Signed-off-by: Bill Erickson <>
Signed-off-by: Jason Stephenson <>
Signed-off-by: Mike Rylander <>
10 years agoJSON_v0 has been superseded, and it has insidious bugs anyway.
Lebbeous Fogle-Weekley [Fri, 27 Jan 2012 14:05:07 +0000 (09:05 -0500)]
JSON_v0 has been superseded, and it has insidious bugs anyway.

For example, you cannot round-trip this through JSON2js() and js2JSON()
in IE8:

This will make Evergreen's build fail until the change specified in
LP #922609 is applied.

Signed-off-by: Lebbeous Fogle-Weekley <>
Signed-off-by: Bill Erickson <>
10 years agoSync parent/child write/read with lock file
Bill Erickson [Mon, 7 Nov 2011 22:30:44 +0000 (17:30 -0500)]
Sync parent/child write/read with lock file

Wrap parent writes to child socket and initial child reads of the
socket in file lock (via flock()) to prevent rare race condition
where child process reads to the end of the data before the parent
has written all bytes.

This create a new lock file on the system, which resides in the same
directory as the pid files.  The lock file is created and destroyed by
the opensrf perl mods at service start up and shutdown.

See also

Signed-off-by: Bill Erickson <>
Signed-off-by: Dan Scott <>
10 years agoUpdate Perl OpenSRF build and tests for O:A:Validator
Dan Scott [Mon, 12 Dec 2011 18:09:13 +0000 (13:09 -0500)]
Update Perl OpenSRF build and tests for O:A:Validator

Kickstarted by wanting to bring OpenSRF::Application::Validator into the
MANIFEST and testing coverage, I noticed a few other files that needed
to be added to the MANIFEST as well. This gets us closer to acceptable
according to "./Build distcheck"

Signed-off-by: Dan Scott <>
10 years agoOpenSRF Validator Service
Thomas Berezansky [Fri, 9 Dec 2011 20:19:12 +0000 (15:19 -0500)]
OpenSRF Validator Service

Add a new Validator service, and EmailAddress validators.

The service runs a chain of one or more validators, each one being fed the
normalized output of the previous one.

The return from each validator should be a hash of valid (0 or 1), the new
normalized output (the untouched input if invalid or nothing needed to be
changed), and if invalid an error string. Optionally, a validator can also
include an "additionals" hash of extra information to be included in the
final response.

The complete list of validators included is:

    The base validator. Always returns valid.
    Always returns invalid for testing purposes.
    Does a very basic regular expression check on email addresses.
    Uses Net::DNS to look up the domain on an email address

Signed-off-by: Thomas Berezansky <>
Signed-off-by: Dan Scott <>
10 years agoSupport installation for Ubuntu 12.04 Precise Pangolin.
Jason Stephenson [Wed, 7 Dec 2011 20:08:42 +0000 (15:08 -0500)]
Support installation for Ubuntu 12.04 Precise Pangolin.

Update the README to include Precise Pangolin as a supported release.

Update src/extras/Makefile.install to install two new debs required on
Ubuntu 12.04 alpha1.

Signed-off-by: Jason Stephenson <>
Signed-off-by: Chris Sharp <>
11 years agoAdd explicit chown command to README
Dan Scott [Fri, 4 Nov 2011 13:42:50 +0000 (09:42 -0400)]
Add explicit chown command to README

Warren Layton noticed that there was no explicit command to change the
ownership of the files in the /<PREFIX> directory to be owned by
"opensrf", although there was a statement that the files needed to be
owned by "opensrf". To reduce the chance of failure, add the explicit

TODO (for a willing volunteer): teach the installer to change the
ownership at the time the files are installed!

Signed-off-by: Dan Scott <>
11 years agoAdd SIGPIPE retry handling to child data sysread
Bill Erickson [Fri, 28 Oct 2011 15:33:24 +0000 (11:33 -0400)]
Add SIGPIPE retry handling to child data sysread

Similar to the SIGPIPE retry logic wrapped around the parent process'
syswrite call (for sending data to a child process), protect the child's
sysread call (as it reads data from the parent).  In pre-2.0, the
sysread step was handled by Net::Server, but now we need to protect it

Signed-off-by: Bill Erickson <>
Signed-off-by: Galen Charlton <>
11 years agoLP878284: stop_osrf action should stop, not start, Perl
Dan Scott [Wed, 19 Oct 2011 15:39:41 +0000 (11:39 -0400)]
LP878284: stop_osrf action should stop, not start, Perl

As reported by Vicent Mas <> to the Evergreen
developer's mailing list:

It seems I've found a small bug in the
/openils/bin/ script. The line for the osrf_stop action is:

"stop_osrf") stop_python; stop_c; start_perl;;

but should be:

"stop_osrf") stop_python; stop_c; stop_perl;;

Signed-off-by: Dan Scott <>
11 years agoWarn when sending very large messages
Bill Erickson [Tue, 18 Oct 2011 13:17:10 +0000 (09:17 -0400)]
Warn when sending very large messages

Depending on configuration, messages of a certain size sent through a
Jabber server will cause the jabber server to disconnect the client.
This change allows admins to configure a message size warning threshold.
When a message meets or exceeds the size threshold, a warning is issued
to the logs with the message size (in bytes) and the message recipient.
It does not prevent the message from being delivered.  It's purely

Use 1 800 000 as the default threhold.

Signed-off-by: Bill Erickson <>
Signed-off-by: Galen Charlton <>
11 years agoFix README typo: 'mod_offlinex'
Dan Scott [Sat, 10 Sep 2011 16:22:08 +0000 (12:22 -0400)]
Fix README typo: 'mod_offlinex'

mod_offline is not that x-treme.

Signed-off-by: Dan Scott <>
11 years agoDon't define a variable inside a conditional block
Dan Scott [Thu, 25 Aug 2011 15:41:19 +0000 (11:41 -0400)]
Don't define a variable inside a conditional block

./configure --disable-core --enable-javascript was dying with an error
due to CHECK_TESTS not being defined; move it outside the conditional
block so that we can ensure that it is defined as either yes or no.

Signed-off-by: Dan Scott <>
11 years agoMake distro targets in README match Makefile.install
Dan Scott [Mon, 22 Aug 2011 13:49:45 +0000 (09:49 -0400)]
Make distro targets in README match Makefile.install

Swap ubuntu-karmic for ubuntu-lucid and pull fedora14 as it is really
close to end of life.

Signed-off-by: Dan Scott <>
11 years agoAdd a 0 to max_stanza_size in README instructions
Dan Scott [Fri, 19 Aug 2011 13:45:16 +0000 (09:45 -0400)]
Add a 0 to max_stanza_size in README instructions

Yes, that should be 2000000, not 200000 for max_stanza_size. Symptoms
included "Text of error message received from Jabber: XML stanza is too
big" in error logs and crashing OpenSRF processes.

For reference,

Signed-off-by: Dan Scott <>
11 years agoLP 799718: Pass the DESTDIR variable to
Ben Webb [Mon, 25 Jul 2011 20:40:40 +0000 (21:40 +0100)]
LP 799718: Pass the DESTDIR variable to

This allows python modules to be built when creating packages.

Per comment from Dan Scott on the LP bug:
says "Multiple successive slashes are considered to be the same as one
slash." so in theory "--root=$(DESTDIR)/" should be just as safe as
"--root=$(DESTDIR)///", but why not go the distance and use 3 slashes to
avoid any possibility of some oddball POSIX implementation that treats 2
slashes differently for some reason?

Signed-off-by: Ben Webb <>
Signed-off-by: Dan Scott <>
11 years agoChange "Scheduler" to "SingleBranchScheduler" per 0.8.4
Dan Scott [Wed, 6 Jul 2011 14:49:04 +0000 (10:49 -0400)]
Change "Scheduler" to "SingleBranchScheduler" per 0.8.4

In buildbot 0.8.4, the Scheduler becomes a SingleBranchScheduler to make
way for other kinds of Schedulers: see

Signed-off-by: Dan Scott <>
11 years agoRespect DESTDIR during the build process
Ben Webb [Mon, 13 Jun 2011 12:29:36 +0000 (13:29 +0100)]
Respect DESTDIR during the build process

Replace instances of make with the generic $(MAKE)
Add $(DESTDIR) to paths it is currently missing from
Manually specify the apxs2 install location so that DESTDIR can be prepended
Attempt to create the apache directory if it does not exist

Signed-off-by: Ben Webb <>
Signed-off-by: Dan Scott <>
11 years agoMove the buildbot workdirs to a non-volatile directory
Dan Scott [Wed, 25 May 2011 04:02:33 +0000 (00:02 -0400)]
Move the buildbot workdirs to a non-volatile directory

/tmp/ is a good directory for throwing temporary things,
but not so good for things you want to keep around for a relatively
long time. Especially on Ubuntu, which has a default policy of
deleting files in /tmp/ on reboot - meaning that the git repos
that form the basis of the repository have all of their files
wiped - greatly confusing the buildbot, which tries to run
'git reset --head' in an empty directory and fails miserably
in that situation.

So - ensure that WORKDIR is set to something that exists and
which will exist for the duration of your CI server, and you'll
be all right.

Signed-off-by: Dan Scott <>
11 years agoTurn on email and IRC notification about build breaks
Dan Scott [Wed, 18 May 2011 18:38:19 +0000 (14:38 -0400)]
Turn on email and IRC notification about build breaks

Buildbot can notify us when a build breaks via email and IRC.

Now that everything is green on, let's
turn that capability on. As configured, we should only be notified
when a build goes from success to failure, the first time; and
then again, when a build goes from failure to success.

Email notification should go to "interested users" - all of the
email addresses in the collected signed-off-by / author tags in
the commits for a given build.

Signed-off-by: Dan Scott <>
11 years agoTeach Perl Makefile to build Build for every target
Dan Scott [Wed, 18 May 2011 14:23:18 +0000 (10:23 -0400)]
Teach Perl Makefile to build Build for every target

With the previous Perl, running "make check" before
running "make all" would fail on the Perl directory because it
assumed that Build would have been created from Build.PL. Factoring out
the build of Build from Build.PL and making that a prereq of the
pertinent targets avoids that problem on a clean checkout of the
OpenSRF source.

Signed-off-by: Dan Scott <>
11 years agoSpecify explicit branches for each builder, lazy-style
Dan Scott [Mon, 16 May 2011 22:11:59 +0000 (18:11 -0400)]
Specify explicit branches for each builder, lazy-style

SVNPoller was able to rely on change filters to look at every
commit to the repository and hand them off to the scheduler
accordingly; GitPoller, however, looks at one branch per
poller, so we need to instantiate one poller per branch of
interest. This gives us an opportunity to refactor our approach
to be more programmatic.

Signed-off-by: Dan Scott <>
11 years agoComment on why we're pulling in version.m4
Dan Scott [Mon, 16 May 2011 20:01:37 +0000 (16:01 -0400)]
Comment on why we're pulling in version.m4

Commenting code is helpful, I've been told that it is the best
way to reduce bugs in a project. To that end, make the connection

Signed-off-by: Dan Scott <>
11 years agoDefine version number in one place
Dan Scott [Mon, 16 May 2011 19:08:04 +0000 (15:08 -0400)]
Define version number in one place

Adding the version.m4 macro defines VERSION_NUMBER for us in
one place so that we can simplify package releases (and fix
'osrf_config --version' output so that it does not always
tell us 'trunk').

Signed-off-by: Dan Scott <>
11 years agoTypo fix: deprecasted -> deprecated
Dan Scott [Mon, 16 May 2011 18:52:40 +0000 (14:52 -0400)]
Typo fix: deprecasted -> deprecated

It's an annoying typo, and it helps test out the buildbot

Signed-off-by: Dan Scott <>
11 years agoRemove 'branches' reference in Buildbot branches test
Dan Scott [Sun, 15 May 2011 21:38:46 +0000 (17:38 -0400)]
Remove 'branches' reference in Buildbot branches test

We can just check for the branch directly now that we are
in git. Some things really are simpler with git!

Signed-off-by: Dan Scott <>
11 years agoAdd a workdir for each GitPoller source repository
Dan Scott [Sun, 15 May 2011 20:23:32 +0000 (16:23 -0400)]
Add a workdir for each GitPoller source repository

Not 100% sure, but with two GitPollers using the same working
directory, I suspect that there was a conflict between the
underlying repos as to which commit was the newest. Give
each of them an explicit working directory in the attempt
to avoid this.

Signed-off-by: Dan Scott <>
11 years agoTeach Buildbot to poll the new git repos
Dan Scott [Sun, 15 May 2011 19:25:33 +0000 (15:25 -0400)]
Teach Buildbot to poll the new git repos

We're now using GitPoller to check for commits to the branches
of interest in Evergreen and OpenSRF on the default interval
of 10 minutes.

Also, some syntax cleanups following the last refactoring.

Signed-off-by: Dan Scott <>
11 years agoInline TestObject for as well
dbs [Mon, 9 May 2011 17:10:19 +0000 (17:10 +0000)]
Inline TestObject for as well

Signed-off-by: Dan Scott <>
git-svn-id: svn:// 9efc2488-bf62-4759-914b-345cdb29e865

11 years agoFix unit tests for Python in a twistd instance
dbs [Mon, 9 May 2011 13:53:30 +0000 (13:53 +0000)]
Fix unit tests for Python in a twistd instance

twistd could not find testobj; just inline it.

Signed-off-by: Dan Scott <>
git-svn-id: svn:// 9efc2488-bf62-4759-914b-345cdb29e865

11 years agoAdd Python unit testing and coverage report to "make check"
dbs [Mon, 9 May 2011 05:19:51 +0000 (05:19 +0000)]
Add Python unit testing and coverage report to "make check"

If --enable-python is included in the arguments to configure, "make check"
runs all Python unit tests using nosetests and generates a testing coverage
report for all Python code.

The original is factored out to provide a separate file for
testing osrf.net_obj vs. osrf.json vs. osrf.*, when we eventually get there.

Signed-off-by: Dan Scott <>
git-svn-id: svn:// 9efc2488-bf62-4759-914b-345cdb29e865

11 years agoAdd Check dependencies for Fedora
dbs [Sun, 8 May 2011 15:37:10 +0000 (15:37 +0000)]
Add Check dependencies for Fedora

Required to run "make check" for the C unit tests
previously added by Kevin Beswick.

Signed-off-by: Dan Scott <>
git-svn-id: svn:// 9efc2488-bf62-4759-914b-345cdb29e865

11 years agoAdd a C unit testing framework and tests for OpenSRF
dbs [Sun, 8 May 2011 15:35:28 +0000 (15:35 +0000)]
Add a C unit testing framework and tests for OpenSRF

The unit testing framework is "Check" -

These tests can be executed by running 'make check' after you
configure and compile the OpenSRF code.

To run them, you must have the 'check' package installed.

Author: Kevin Beswick <>
Signed-off-by: Dan Scott <>
Signed-off-by: Kevin Beswick <>
git-svn-id: svn:// 9efc2488-bf62-4759-914b-345cdb29e865

11 years agomove jabber_state_machine_struct into header to keep 'make check' happy
erickson [Sat, 30 Apr 2011 22:40:54 +0000 (22:40 +0000)]
move jabber_state_machine_struct into header to keep 'make check' happy

git-svn-id: svn:// 9efc2488-bf62-4759-914b-345cdb29e865

11 years agoPatch from Kevin Beswick to support set operations on certain opensrf message fields...
erickson [Thu, 28 Apr 2011 16:20:47 +0000 (16:20 +0000)]
Patch from Kevin Beswick to support set operations on certain opensrf message fields.  note, this does not currently affect behavior, since values for messages are initially set from parsing JSON without using the access/mutators for setting values.

git-svn-id: svn:// 9efc2488-bf62-4759-914b-345cdb29e865

11 years agoSidestep JSON object ordering in unit tests
dbs [Mon, 25 Apr 2011 04:12:41 +0000 (04:12 +0000)]
Sidestep JSON object ordering in unit tests

Depending on the JS engine and version of Dojo, the order
of attributes in JSON objects may change (as is explicitly
allowed by the JSON spec). To avoid false test failures,
only use one attribute in the object.

git-svn-id: svn:// 9efc2488-bf62-4759-914b-345cdb29e865