Does not check that all drones in a brick are fully up, just that the
entire brick will respond to each service, even if only one drone happens
to be running the listener properly (and said listener is responding).
Signed-off-by: Thomas Berezansky <tsbere@mvlc.org> Signed-off-by: Dan Scott <dan@coffeecode.net>
Bill Erickson [Mon, 2 Apr 2012 18:55:56 +0000 (14:55 -0400)]
Detect and repair multipart/mixed message delivery errors
For unknown reasons, the Content-Type header will occasionally be
included in the XHR.responseText for multipart/mixed messages. When
this happens, strip the header and newlines from the message body and
re-parse.
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Dan Scott <dan@coffeecode.net>
Dan Scott [Mon, 21 May 2012 16:18:41 +0000 (12:18 -0400)]
LP# 953299 - Prevent get/set of invalid cache keys
Clients of OpenSRF::Utils::Cache occasionally request cache keys that
contain invalid characters (a particular case is ISBNs that contain
spaces), so strip those out of incoming get/set requests to avoid ugly
memcached errors.
Signed-off-by: Dan Scott <dan@coffeecode.net> Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Dan Scott [Wed, 2 May 2012 02:11:00 +0000 (22:11 -0400)]
Minor cleanups to Java dependency build process
While configure.ac still hardcodes the expected location of the Java
dependencies (JSON and Memcached classes), we can clean up some of the
lower-level challenges with the Java build.
* StAX and WSTX were rolled into the core Java environment long ago,
so there's no need to download them and even less need to verify
that the downloaded version exist.
* The Memcached class is now up to 2.0.1, per deps.sh, but the deps.inc
file was pointing at 1.5.1.
* Speaking of the deps.inc file, nothing seems to use it; delete it.
* The Makefile was dying because the variables for the Memcached and
JSON classes weren't defined. Define those at the top of the Makefile.
Dan Scott [Wed, 2 May 2012 02:08:23 +0000 (22:08 -0400)]
Clean up root directory: INSTALL, .gitignore, autoreconf
* Update the README to avoid running autoreconf with the "-f" flag.
* Make the INSTALL file a symbolic link to the README to avoid
creating a generic INSTALL file each time autoreconf is run.
* Add a .gitignore file that ignores all of the built files.
Dan Scott [Wed, 2 May 2012 01:31:38 +0000 (21:31 -0400)]
Change Java deps.sh to include . in CLASSPATH
Following the directions given by running deps.sh was a little bit
frustrating; we need to work in some documentation on how to use the
Java bits of OpenSRF - and perhaps look at a more standard option like
maven for dependencies.
Bill Erickson [Mon, 27 Feb 2012 23:05:07 +0000 (18:05 -0500)]
Java HTTP gateway interface
Supports sync and async requests. Async requests support onResponse,
onComplete, and onError handlers.
Supports a max-threads value to limit the number of activately
communicating threads over any connection. When max-threads is reached,
requests are queued and delivered as soon as there is room.
Note that since this is talking to the OpenSRF gateway and not the
translater, responses are simply collected and passed one at a time to
onResponse. They are not streamed. The goal of supporting onResponse
is to provide the same client API for both the gateway and translator.
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Dan Scott <dscott@laurentian.ca>
Dan Scott [Mon, 23 Apr 2012 18:17:35 +0000 (14:17 -0400)]
Remove GNU default INSTALL file
GNU autotools generate an INSTALL file that is generic and which
conflicts with the instructions given in the README. Installation is
already hard enough without conflicting instructions, so delete the
INSTALL file.
Dan Scott [Sun, 8 Apr 2012 18:53:36 +0000 (14:53 -0400)]
Updated buildbot.cfg for current build environments
* Drop Ubuntu 8.04 from the list of build environments.
* OpenSRF: drop rel_1_6, add rel_2_1
* Evergreen: drop rel_1_6_1, add rel_2_2
* Adjust for buildbot 0.86 API changes to avoid deprecation
* Improve PEP8 compliance
Bill Erickson [Mon, 20 Feb 2012 19:40:30 +0000 (14:40 -0500)]
Detect remote disconnect in Perl XMPP reader
When the jabber server severs the connection, it leaves the Perl libs
with a socket that perpetually appears ready for reading, but always
returns 0 bytes. This produces a loop in the client code. This change
detects this situation, logs an error, and throws an exception.
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Dan Scott <dscott@laurentian.ca>
Dan Wells [Tue, 6 Mar 2012 20:08:33 +0000 (15:08 -0500)]
Protect gateway from format-string crashes in data
As a common security measure, printf-style formatting codes are
not allowed to be directly interpreted from a writable segment.
The gateway code currently has the following function call:
osrfLogActivity( OSRF_LOG_MARK, act->buf );
This is a variadic function which expects the 'act->buf' position
to contain a format string and any trailing arguments to be the
values passed to the formatter. Since act->buf is the value of
what we passed in, some data inadvertantly contains format strings,
and since it is a writable segment, the program crashes. Here is
an example of a crash-causing call:
Bill Erickson [Thu, 16 Feb 2012 22:31:29 +0000 (17:31 -0500)]
Refresh child file handles on status read
If a child process dies while the parent is attempting to read status
information from the child, the pipe connecting the two will be cleaned
up. The parent mustn't attempt to read from the dead pipe or it will
result in a bad file descriptor error, culminating in a high-cpu
infinite loop on the parent (Listener) process.
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Thomas Berezansky <tsbere@mvlc.org> Signed-off-by: Jason Stephenson <jstephenson@mvlc.org>
Bill Erickson [Thu, 12 Jan 2012 21:36:16 +0000 (16:36 -0500)]
Perl libs for OpenSRF ingress tracking
Ingress is a free-form text value which represents the entry point for
the client into the opensrf network. The value is passed within opensrf
messages, similar to "locale". Clients should specify the ingress
before any opensrf communication occurs.
OpenSRF::AppSession->ingress($ingress);
Stock values include the following:
opensrf (default)
srfsh
translator-v1
gateway-v1
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Mike Rylander <mrylander@gmail.com>
Dan Scott [Wed, 29 Feb 2012 15:42:58 +0000 (10:42 -0500)]
Include explicit instructions to copy the opensrf config files
Also noted by the sharp-eyed Ben Shum, we didn't tell people to actually
copy opensrf.xml.example / opensrf_core.xml.example, which could lead to
failure.
Dan Scott [Wed, 29 Feb 2012 15:27:36 +0000 (10:27 -0500)]
Update max_user_sessions suggestion to 10000 to match wiki
Ben Shum noticed that the README still specified 1000 for
max_user_sessions in the ejabberd.cfg file, whereas current larger
OpenSRF systems with many services may blow past that limit. Update to
match the current wiki specification of 10000.
Dan Scott [Wed, 4 Jan 2012 20:21:36 +0000 (15:21 -0500)]
Remove reference to Fedora specific version in prereq installer
Fedora changes every 6 months, so it's probably silly to have a make
target of "fedora14" when that is no longer supported by the Fedora
project itself. As Fedora has packaged all of the OpenSRF dependencies
(thanks, Ben Webb!), in theory it should be supported by the latest
releases of Fedora in the future... as long as we don't introduce any
dependencies on deprecated versions of packages.
Bill Erickson [Fri, 24 Feb 2012 21:28:27 +0000 (16:28 -0500)]
Java dependencies update
As of Java 6, XML Stax parsing is natively supported. This change
removes the external Stax dependencies and updates how the JSON libs are
fetched (in deps.sh), which now come from github.
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Dan Scott <dscott@laurentian.ca>
Bill Erickson [Tue, 14 Feb 2012 14:10:58 +0000 (09:10 -0500)]
Perl parent/child write improvements
* Updated variable names for clarity
* Added more inline comments
* Added additional error logging
* For severe read errors, allow the child to gracefully skip the request
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Jason Stephenson <jstephenson@mvlc.org> Signed-off-by: Mike Rylander <mrylander@gmail.com>
Bill Erickson [Mon, 13 Feb 2012 21:53:59 +0000 (16:53 -0500)]
Perl pipe reading overhaul : data size header
The lockfile mechanism for preventing premature end of reads on child
processes suffers from one serious flaw: if the data to write exceeds
the pipe buffer size, the parent will block on syswrite and the service
will lock up. It's also not as effecient (for the normal case) as the
code was without the lockfile, becasue the writes and reads are
serialized.
This commit replaces the lockfile mechanism with a protocol header in
the data. The first X (currently 12) bytes of data written to the child
process will contain the full length of the data to be written (minus
the header size). The child now reads the data in parallel with the parent as
data is available. If the child reads all available data (in the pipe)
but not all of the expected data, the child will go back into a select()
wait pending more data from the parent. The process continues until all
data is read.
This same mechanism is already used to commicate status info from child
processes to the parent.
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Jason Stephenson <jstephenson@mvlc.org> Signed-off-by: Mike Rylander <mrylander@gmail.com>
Bill Erickson [Mon, 7 Nov 2011 22:30:44 +0000 (17:30 -0500)]
Sync parent/child write/read with lock file
Wrap parent writes to child socket and initial child reads of the
socket in file lock (via flock()) to prevent rare race condition
where child process reads to the end of the data before the parent
has written all bytes.
This create a new lock file on the system, which resides in the same
directory as the pid files. The lock file is created and destroyed by
the opensrf perl mods at service start up and shutdown.
See also https://bugs.launchpad.net/opensrf/+bug/883155
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Dan Scott <dscott@laurentian.ca>
Dan Scott [Mon, 12 Dec 2011 18:09:13 +0000 (13:09 -0500)]
Update Perl OpenSRF build and tests for O:A:Validator
Kickstarted by wanting to bring OpenSRF::Application::Validator into the
MANIFEST and testing coverage, I noticed a few other files that needed
to be added to the MANIFEST as well. This gets us closer to acceptable
according to "./Build distcheck"
Add a new Validator service, and EmailAddress validators.
The service runs a chain of one or more validators, each one being fed the
normalized output of the previous one.
The return from each validator should be a hash of valid (0 or 1), the new
normalized output (the untouched input if invalid or nothing needed to be
changed), and if invalid an error string. Optionally, a validator can also
include an "additionals" hash of extra information to be included in the
final response.
The complete list of validators included is:
OpenSRF::Application::Validator::Base
The base validator. Always returns valid.
OpenSRF::Application::Validator::Invalid
Always returns invalid for testing purposes.
OpenSRF::Application::Validator::EmailAddress::Regex
Does a very basic regular expression check on email addresses.
OpenSRF::Application::Validator::EmailAddress::DNS
Uses Net::DNS to look up the domain on an email address
Signed-off-by: Thomas Berezansky <tsbere@mvlc.org> Signed-off-by: Dan Scott <dscott@laurentian.ca>
Dan Scott [Fri, 4 Nov 2011 13:42:50 +0000 (09:42 -0400)]
Add explicit chown command to README
Warren Layton noticed that there was no explicit command to change the
ownership of the files in the /<PREFIX> directory to be owned by
"opensrf", although there was a statement that the files needed to be
owned by "opensrf". To reduce the chance of failure, add the explicit
command.
TODO (for a willing volunteer): teach the installer to change the
ownership at the time the files are installed!
Bill Erickson [Fri, 28 Oct 2011 15:33:24 +0000 (11:33 -0400)]
Add SIGPIPE retry handling to child data sysread
Similar to the SIGPIPE retry logic wrapped around the parent process'
syswrite call (for sending data to a child process), protect the child's
sysread call (as it reads data from the parent). In pre-2.0, the
sysread step was handled by Net::Server, but now we need to protect it
ourselves.
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bill Erickson [Tue, 18 Oct 2011 13:17:10 +0000 (09:17 -0400)]
Warn when sending very large messages
Depending on configuration, messages of a certain size sent through a
Jabber server will cause the jabber server to disconnect the client.
This change allows admins to configure a message size warning threshold.
When a message meets or exceeds the size threshold, a warning is issued
to the logs with the message size (in bytes) and the message recipient.
It does not prevent the message from being delivered. It's purely
informational.
Use 1 800 000 as the default threhold.
Signed-off-by: Bill Erickson <berick@esilibrary.com> Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Dan Scott [Thu, 25 Aug 2011 15:41:19 +0000 (11:41 -0400)]
Don't define a variable inside a conditional block
./configure --disable-core --enable-javascript was dying with an error
due to CHECK_TESTS not being defined; move it outside the conditional
block so that we can ensure that it is defined as either yes or no.
Dan Scott [Fri, 19 Aug 2011 13:45:16 +0000 (09:45 -0400)]
Add a 0 to max_stanza_size in README instructions
Yes, that should be 2000000, not 200000 for max_stanza_size. Symptoms
included "Text of error message received from Jabber: XML stanza is too
big" in error logs and crashing OpenSRF processes.
For reference,
http://list.georgialibraries.org/pipermail/open-ils-dev/2011-August/007494.html
Ben Webb [Mon, 25 Jul 2011 20:40:40 +0000 (21:40 +0100)]
LP 799718: Pass the DESTDIR variable to setup.py
This allows python modules to be built when creating packages.
Per comment from Dan Scott on the LP bug:
http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap03.html#tag_03_266
says "Multiple successive slashes are considered to be the same as one
slash." so in theory "--root=$(DESTDIR)/" should be just as safe as
"--root=$(DESTDIR)///", but why not go the distance and use 3 slashes to
avoid any possibility of some oddball POSIX implementation that treats 2
slashes differently for some reason?
Signed-off-by: Ben Webb <bjwebb67@googlemail.com> Signed-off-by: Dan Scott <dscott@laurentian.ca>
Dan Scott [Wed, 6 Jul 2011 14:49:04 +0000 (10:49 -0400)]
Change "Scheduler" to "SingleBranchScheduler" per 0.8.4
In buildbot 0.8.4, the Scheduler becomes a SingleBranchScheduler to make
way for other kinds of Schedulers: see
http://buildbot.net/buildbot/docs/current/SingleBranchScheduler.html
Ben Webb [Mon, 13 Jun 2011 12:29:36 +0000 (13:29 +0100)]
Respect DESTDIR during the build process
Replace instances of make with the generic $(MAKE)
Add $(DESTDIR) to paths it is currently missing from
Manually specify the apxs2 install location so that DESTDIR can be prepended
Attempt to create the apache directory if it does not exist
Signed-off-by: Ben Webb <bjwebb67@googlemail.com> Signed-off-by: Dan Scott <dan@coffeecode.net>
Dan Scott [Wed, 25 May 2011 04:02:33 +0000 (00:02 -0400)]
Move the buildbot workdirs to a non-volatile directory
/tmp/ is a good directory for throwing temporary things,
but not so good for things you want to keep around for a relatively
long time. Especially on Ubuntu, which has a default policy of
deleting files in /tmp/ on reboot - meaning that the git repos
that form the basis of the repository have all of their files
wiped - greatly confusing the buildbot, which tries to run
'git reset --head' in an empty directory and fails miserably
in that situation.
So - ensure that WORKDIR is set to something that exists and
which will exist for the duration of your CI server, and you'll
be all right.
Dan Scott [Wed, 18 May 2011 18:38:19 +0000 (14:38 -0400)]
Turn on email and IRC notification about build breaks
Buildbot can notify us when a build breaks via email and IRC.
Now that everything is green on testing.evergreen-ils.org, let's
turn that capability on. As configured, we should only be notified
when a build goes from success to failure, the first time; and
then again, when a build goes from failure to success.
Email notification should go to "interested users" - all of the
email addresses in the collected signed-off-by / author tags in
the commits for a given build.
Dan Scott [Wed, 18 May 2011 14:23:18 +0000 (10:23 -0400)]
Teach Perl Makefile to build Build for every target
With the previous Perl Makefile.am, running "make check" before
running "make all" would fail on the Perl directory because it
assumed that Build would have been created from Build.PL. Factoring out
the build of Build from Build.PL and making that a prereq of the
pertinent targets avoids that problem on a clean checkout of the
OpenSRF source.
Dan Scott [Mon, 16 May 2011 22:11:59 +0000 (18:11 -0400)]
Specify explicit branches for each builder, lazy-style
SVNPoller was able to rely on change filters to look at every
commit to the repository and hand them off to the scheduler
accordingly; GitPoller, however, looks at one branch per
poller, so we need to instantiate one poller per branch of
interest. This gives us an opportunity to refactor our approach
to be more programmatic.
Dan Scott [Mon, 16 May 2011 20:01:37 +0000 (16:01 -0400)]
Comment on why we're pulling in version.m4
Commenting code is helpful, I've been told that it is the best
way to reduce bugs in a project. To that end, make the connection
between VERSION_NUMBER and PACKAGE_VERSION in configure.ac
Dan Scott [Mon, 16 May 2011 19:08:04 +0000 (15:08 -0400)]
Define version number in one place
Adding the version.m4 macro defines VERSION_NUMBER for us in
one place so that we can simplify package releases (and fix
'osrf_config --version' output so that it does not always
tell us 'trunk').
Dan Scott [Sun, 15 May 2011 20:23:32 +0000 (16:23 -0400)]
Add a workdir for each GitPoller source repository
Not 100% sure, but with two GitPollers using the same working
directory, I suspect that there was a conflict between the
underlying repos as to which commit was the newest. Give
each of them an explicit working directory in the attempt
to avoid this.
dbs [Mon, 9 May 2011 05:19:51 +0000 (05:19 +0000)]
Add Python unit testing and coverage report to "make check"
If --enable-python is included in the arguments to configure, "make check"
runs all Python unit tests using nosetests and generates a testing coverage
report for all Python code.
The original json_test.py is factored out to provide a separate file for
testing osrf.net_obj vs. osrf.json vs. osrf.*, when we eventually get there.
Signed-off-by: Dan Scott <dan@coffeecode.net>
git-svn-id: svn://svn.open-ils.org/OpenSRF/trunk@2244 9efc2488-bf62-4759-914b-345cdb29e865
dbs [Sun, 8 May 2011 15:35:28 +0000 (15:35 +0000)]
Add a C unit testing framework and tests for OpenSRF
The unit testing framework is "Check" - http://check.sourceforge.net/
These tests can be executed by running 'make check' after you
configure and compile the OpenSRF code.
To run them, you must have the 'check' package installed.
Author: Kevin Beswick <kevinbeswick00@gmail.com> Signed-off-by: Dan Scott <dan@coffeecode.net> Signed-off-by: Kevin Beswick <kevinbeswick00@gmail.com>
git-svn-id: svn://svn.open-ils.org/OpenSRF/trunk@2237 9efc2488-bf62-4759-914b-345cdb29e865
Patch from Kevin Beswick to support set operations on certain opensrf message fields. note, this does not currently affect behavior, since values for messages are initially set from parsing JSON without using the access/mutators for setting values.
Depending on the JS engine and version of Dojo, the order
of attributes in JSON objects may change (as is explicitly
allowed by the JSON spec). To avoid false test failures,
only use one attribute in the object.
dbs [Wed, 30 Mar 2011 04:31:28 +0000 (04:31 +0000)]
Handle the common NXDOMAIN problem with Python a bit more gracefully
Rather than dumping a nasty full stacktrace to the command line, we
print a hopefully helpful pointer to the actual problem in plain
English.
Also, rather than stopping everything if a "stop_all" command was
issued, skip the error status in osrf_ctl.sh for the Python bit
and carry on shutting down the other services. A bit more convenient
if you want to have Python enabled but don't necessarily need it
running.
dbs [Tue, 29 Mar 2011 13:55:09 +0000 (13:55 +0000)]
Make javascript unit tests more granular to ease tracking down errors
Noticed a failure when running with Rhino 1.7RC2 that wasn't happening
with version of Rhino packaged in shrinksafe.jar, and it was a bit
tedious narrowing down which test was actually failing. Breaking up
the tests into smaller sets will make this process easier and might invite
the submission of other tests to target areas that are not yet covered.
dbs [Tue, 29 Mar 2011 02:59:51 +0000 (02:59 +0000)]
Give Python services introspection methods matching C and Perl
Perl and C services offer "opensrf.system.method" and
"opensrf.system.method.all" methods for accessing service
information. Renaming Python's methods to match at least
enables srfsh to access this information, even if it does
still need to be taught to provide that information in a
consistent manner.
dbs [Mon, 28 Mar 2011 15:22:54 +0000 (15:22 +0000)]
Add DOH-based unit test harness for OpenSRF JavaScript
The Dojo Objective Harness enables us to run unit tests from the command
line or within a browser. Running tests from a command line makes it easier
to fold into the continuous integration server. Included in this commit
is a small README for setting up command-line testing and a sample set
of unit tests that exercise parts of JSON_v1.js.
dbs [Sun, 27 Mar 2011 21:36:25 +0000 (21:36 +0000)]
Improved PID file handling in osrf_ctl.sh
Joseph Lewis <joehms22@gmail.com> submitted a patch to address
https://bugs.launchpad.net/evergreen/+bug/741088 ("osrf_ctl.sh Doesn't check if
process is actually running")
As of this patch, when start_perl, start_c, or start_python are started and
existing PID files are found for those processes, instead of assuming that the
processes are actually running, osrf-ctl.sh now has the intelligence to check
the process list to see if there is a matching process. If no running process
is found, then the old PID file is removed and the start command is issued.
Two additional actions have been added, "smart_clear" and "clear_pid".
"smart_clear" checks the PID files against the running processes and removes
PID files where no running process is found. This is effectively invoked
under the covers when start_* is invoked and a PID file is found.
"clear_pid" deletes all PID files without checking to see if there are any
running processes. In most cases, it should only be invoked as part of an
automated boot sequence.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
Signed-off-by: Joseph Lewis <joehms22@gmail.com>
git-svn-id: svn://svn.open-ils.org/OpenSRF/trunk@2215 9efc2488-bf62-4759-914b-345cdb29e865
dbs [Sun, 13 Mar 2011 06:03:46 +0000 (06:03 +0000)]
Run OpenSRF JavaScript through jslint for a light cleanup
Mostly just semicolons, and shifting a bit of code around to reduce
usage of functions/classes before they've been defined. Staying away
from the === / !== operators for null/undefined/0/''/true/false for
now.
dbs [Wed, 2 Mar 2011 03:51:42 +0000 (03:51 +0000)]
Coerce the weird test env for MFHD into working inside buildbot
'use testlib.pm' throws the standard "prove --lib lib -r t" recipe for testing
Perl packages out the window, so we have to override the Perl testing class
and point to the known location of testlib.pm
This may still fail for Evergreen branches prior to rel_2_1, but it's
_something_. Throw me a bone, here, will ya?
Ultimately, the best thing to do is probably to make MFHD tests conform.
projects / OpenSRF.git / log