4 2011-08-19 Dan Scott <dan@coffeecode.net>
6 * README: Add a 0 to max_stanza_size in README instructions
8 Yes, that should be 2000000, not 200000 for max_stanza_size.
9 Symptoms included "Text of error message received from Jabber: XML
10 stanza is too big" in error logs and crashing OpenSRF processes. For reference,
11 http://list.georgialibraries.org/pipermail/open-ils-dev/2011-August/007494.html
13 Signed-off-by: Dan Scott <dscott@laurentian.ca>
15 2011-07-25 Ben Webb <bjwebb67@googlemail.com>
17 * src/python/Makefile.am: LP 799718: Pass the DESTDIR variable to
18 setup.py This allows python modules to be built when creating packages.
20 Per comment from Dan Scott on the LP bug:
21 http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap03.html#tag_03_266
22 says "Multiple successive slashes are considered to be the same as
23 one slash." so in theory "--root=$(DESTDIR)/" should be just as safe
24 as "--root=$(DESTDIR)///", but why not go the distance and use 3
25 slashes to avoid any possibility of some oddball POSIX
26 implementation that treats 2 slashes differently for some reason?
28 Signed-off-by: Ben Webb <bjwebb67@googlemail.com>
29 Signed-off-by: Dan Scott <dscott@laurentian.ca>
31 2011-06-13 Ben Webb <bjwebb67@googlemail.com>
33 * Respect DESTDIR during the build process
35 Replace instances of make with the generic $(MAKE) Add $(DESTDIR) to
36 paths it is currently missing from Manually specify the apxs2
37 install location so that DESTDIR can be prepended Attempt to create
38 the apache directory if it does not exist
40 Signed-off-by: Ben Webb <bjwebb67@googlemail.com>
41 Signed-off-by: Dan Scott <dan@coffeecode.net>
43 Changelog for 2.0.0 final release
44 =================================
48 * src/extras/Makefile.install: Add Check dependencies for Fedora
50 Required to run "make check" for the C unit tests
51 previously added by Kevin Beswick.
53 Signed-off-by: Dan Scott <dan@coffeecode.net>
55 * Makefile.am, configure.ac, src/extras/Makefile.install, tests: Add a C unit
56 testing framework and tests for OpenSRF
58 The unit testing framework is "Check" - http://check.sourceforge.net/
60 These tests can be executed by running 'make check' after you
61 configure and compile the OpenSRF code.
63 To run them, you must have the 'check' package installed.
65 Author: Kevin Beswick <kevinbeswick00@gmail.com>
66 Signed-off-by: Dan Scott <dan@coffeecode.net>
67 Signed-off-by: Kevin Beswick <kevinbeswick00@gmail.com>
69 * include/opensrf/transport_session.h, src/libopensrf/transport_session.c: move
70 jabber_state_machine_struct into header to keep 'make check' happy
74 * src/javascript/opensrf.js: Patch from Kevin Beswick to support set operations on
75 certain opensrf message fields. note, this does not currently affect behavior,
76 since values for messages are initially set from parsing JSON without using the
77 access/mutators for setting values.
81 * src/javascript/tests/testJSON_v1.js: Sidestep JSON object ordering in unit tests
83 Depending on the JS engine and version of Dojo, the order
84 of attributes in JSON objects may change (as is explicitly
85 allowed by the JSON spec). To avoid false test failures,
86 only use one attribute in the object.
90 * src/libopensrf/osrf_prefork.c: change proc managament debug message from Info to
95 * examples/opensrf-rsyslog.conf: additional settings in sample rsyslog config:
96 disable EscapeControlCharactersOnReceive
100 * bin/osrf_ctl.sh.in, src/python/opensrf.py.in: Handle the common NXDOMAIN problem
101 with Python a bit more gracefully
103 Rather than dumping a nasty full stacktrace to the command line, we
104 print a hopefully helpful pointer to the actual problem in plain
107 Also, rather than stopping everything if a "stop_all" command was
108 issued, skip the error status in osrf_ctl.sh for the Python bit
109 and carry on shutting down the other services. A bit more convenient
110 if you want to have Python enabled but don't necessarily need it
115 * src/javascript/tests/testJSON_v1.js: Make javascript unit tests more granular to
116 ease tracking down errors
118 Noticed a failure when running with Rhino 1.7RC2 that wasn't happening
119 with version of Rhino packaged in shrinksafe.jar, and it was a bit
120 tedious narrowing down which test was actually failing. Breaking up
121 the tests into smaller sets will make this process easier and might invite
122 the submission of other tests to target areas that are not yet covered.
124 * src/javascript/tests/testJSON_v1.js: A few more tests for JSON_v1.js
126 We're not really touching the interesting parts of encodeJS/decodeJS
127 yet, but at least we get to open the door a crack.
129 * src/javascript/tests: Add DOH-based unit test harness for OpenSRF JavaScript
131 The Dojo Objective Harness enables us to run unit tests from the command
132 line or within a browser. Running tests from a command line makes it easier
133 to fold into the continuous integration server. Included in this commit
134 is a small README for setting up command-line testing and a sample set
135 of unit tests that exercise parts of JSON_v1.js.
137 * src/python/osrf/app.py: Give Python services introspection methods matching C
140 Perl and C services offer "opensrf.system.method" and
141 "opensrf.system.method.all" methods for accessing service
142 information. Renaming Python's methods to match at least
143 enables srfsh to access this information, even if it does
144 still need to be taught to provide that information in a
149 * bin/osrf_ctl.sh.in: Improved PID file handling in osrf_ctl.sh
151 Joseph Lewis <joehms22@gmail.com> submitted a patch to address
152 https://bugs.launchpad.net/evergreen/+bug/741088 ("osrf_ctl.sh Doesn't check if
153 process is actually running")
155 As of this patch, when start_perl, start_c, or start_python are started and
156 existing PID files are found for those processes, instead of assuming that the
157 processes are actually running, osrf-ctl.sh now has the intelligence to check
158 the process list to see if there is a matching process. If no running process
159 is found, then the old PID file is removed and the start command is issued.
161 Two additional actions have been added, "smart_clear" and "clear_pid".
163 "smart_clear" checks the PID files against the running processes and removes
164 PID files where no running process is found. This is effectively invoked
165 under the covers when start_* is invoked and a PID file is found.
167 "clear_pid" deletes all PID files without checking to see if there are any
168 running processes. In most cases, it should only be invoked as part of an
169 automated boot sequence.
171 Developer's Certificate of Origin 1.1
173 By making a contribution to this project, I certify that:
175 (a) The contribution was created in whole or in part by me and I
176 have the right to submit it under the open source license
177 indicated in the file; or
179 (b) The contribution is based upon previous work that, to the best
180 of my knowledge, is covered under an appropriate open source
181 license and I have the right under that license to submit that
182 work with modifications, whether created in whole or in part
183 by me, under the same open source license (unless I am
184 permitted to submit under a different license), as indicated
187 (c) The contribution was provided directly to me by some other
188 person who certified (a), (b) or (c) and I have not modified
191 (d) I understand and agree that this project and the contribution
192 are public and that a record of the contribution (including all
193 personal information I submit with it, including my sign-off) is
194 maintained indefinitely and may be redistributed consistent with
195 this project or the open source license(s) involved.
197 Signed-off-by: Joseph Lewis <joehms22@gmail.com>
201 * src/javascript/DojoSRF.js, src/javascript/JSON_v0.js, src/javascript/JSON_v1.js,
202 src/javascript/opensrf.js, src/javascript/opensrf_xhr.js,
203 src/javascript/opensrf_xmpp.js: Run OpenSRF JavaScript through jslint for a
206 Mostly just semicolons, and shifting a bit of code around to reduce
207 usage of functions/classes before they've been defined. Staying away
208 from the === / !== operators for null/undefined/0/''/true/false for
213 * README: Forward-port AsciiDoc version of README from 2.0.0-rc2
215 Change log from 2.0.0-rc1 to 2.0.0-rc2
216 ========================================
220 * src/python/srfsh.py: bug fix with srfsh.py introspection; better
221 handling/reporting of service exceptions
223 * src/perl/lib/OpenSRF/Server.pm: consistent w/ sigpipe handling in osrf 1.6,
224 provide a warning and retry mechanism for syswrites that fail as a result of
227 Change log from 2.0.0-beta1 to 2.0.0-rc1
228 ========================================
232 * examples/opensrf.xml.example: Remove max_requests config stanza from C language
233 implementations in example config
235 Only Perl implementations support the max_requests stanza, so let's not confuse
236 matters with an unused element.
238 * examples/opensrf-rsyslog.conf: added sample rsyslog opensrf configuration file
240 * src/python/osrf/app.py: Trivial typo fix (also tests OpenSRF buildbot)
242 * README: Remove extraneous line for dnsmasq / Python
244 * src/libopensrf/osrf_prefork.c: Get that brace back in the right place
246 Unbreak the build. Argh.
248 * src/libopensrf/osrf_prefork.c, src/perl/lib/OpenSRF/Server.pm: Add hints to
249 increase max_children config variable if we're hitting the maximum
251 Already added this hint to the Python implementation, now propagating to
254 * src/python/osrf/server.py: Clean up Python server implementation, guided by
257 Add some docstrings, keep line lengths at 80 or less (MORE
258 WHITESPACE!), remove some dead code, and add a hint to check
259 max_children config value if we're bumping up against the
262 * src/perl/lib/OpenSRF/Application.pm: on second thought, removing debug sleep
263 method, which is (more than most) a DOS waiting to happen.
265 * src/c-apps/Makefile.am: Apply libtools versioning to OpenSRF c-apps as well
267 We apply a -version-info argument of 2:0:2, meaning:
268 * interface = 2: interface '2' of this library
269 * revision = 0: first implementation of interface '2'
270 * age = 2: this library can be linked with executables going back
271 two interface versions
273 See http://sourceware.org/autobook/autobook/autobook_91.html for more
274 information on the -version-info argument.
276 * src/libopensrf/Makefile.am: Prepare for version 2.0.0 with versioning for 2.0.0
278 Use the libtool -version-info option to provide an explicit version
279 for the libopensrf.so library. Introducing versioned libraries will
280 give us the ability to force recompiles of linked applications when
281 binary compatibility has been broken.
283 * src/perl/lib/OpenSRF/Application.pm: added opensrf.system.sleep method; takes 1
284 param (seconds); useful for debugging child process management
286 * src/perl/lib/OpenSRF/Server.pm: Resolve encoding problem found by attempting to
287 register patrons with Unicode names
291 1. a gateway.log entry showing http_translator receiving an XMPP error message
292 2. a corresponding osrfsys.log entry "server: died with error Wide character in
293 syswrite at /usr/local/share/perl5/OpenSRF/Server.pm line 239"
295 Per normal Perl Unicode handling practices, decoding input and encoding output
296 using the Encode module and decode_utf8() / encode_utf8() appears to resolve
301 * Makefile.am: Whitespace consistency (tabs for Makefiles I guess)
303 * src/perl/lib/OpenSRF.pm: Use an Exporter-approved version number (only one
306 * src/perl/Build.PL: Correct copy/paste error in Build.PL metadata
308 OpenSRF uses GPL v2 or later, not the Perl license.
312 * src/perl/t/08-Server.t, src/perl/t/08-UnixServer.t: OpenSRF::UnixServer doesn't
315 This test was still succeeding on hosts that had installed
316 previous versions of OpenSRF, but failed on a shiny new
321 * src/perl/t/07-Transport.t: Set the correct number of expected tests so that we
322 don't fail 100% of the time
324 I guess we haven't run "make test" here since r2024 - heh.
328 * src/libopensrf/osrf_application.c, src/perl/lib/OpenSRF/Application.pm: make the
329 api CALL log line consistent across Perl and C and make it something that can be
330 pasted directly into srfsh
332 * src/python/srfsh.py: repaired call to handle_request for router query; cannot
333 survive without the self.
337 * src/python/srfsh.py: repaired bug in total request time calc, particularly when
338 a request returns no response
342 * src/extras/Makefile.install: Straighten out -f / -d flag confusion in
343 libmemcached prerequisite
345 There are two different styles of checking for previously downloaded
346 tarballs in this Makefile; the libmemcached entry combined them both
347 with unfortunate results.
351 * src/python/srfsh.py: Python srfsh enhancements
353 * Srfsh plugins can now insert new commands and add words to the tab
354 completion word bank.
355 * Addded support reading script files
356 * Added support for service open/close (connect/disconnect) for stateful
358 * Moved to class-based srfsh module for easier state maintenance
363 * src/perl/lib/OpenSRF/Application.pm: on second thought, removing debug sleep
364 method, which is (more than most) a DOS waiting to happen.
366 * src/python/osrf/server.py: improved select/read/write fault tolerance; cleaner
367 and more efficient child process idle/active list management; improved logging
371 * README: Apply patch for dnsmasq documentation
373 Thanks to Michael Giarlo <leftwing@alumni.rutgers.edu> for
374 providing this correction to the dnsmasq configuration line!
378 * src/python/opensrf.py.in: fail gracefully when opensrf.settings is not up and
383 * src/perl/lib/OpenSRF/Server.pm: don't clean up child attributes until we're done
384 with them; added pid map hash for easy/quick access to child via pid
386 * src/perl/lib/OpenSRF/Server.pm: add fault tolerance (e.g. recover from
387 interrupted select/sysread); always perform idle maintenance when there are any
388 active child processes; exit check status early when no active processes exist
390 * src/perl/lib/OpenSRF/Transport/SlimJabber/XMPPReader.pm: avoid calling sysread
391 when select exited from interruption.
393 * src/perl/lib/OpenSRF/Application.pm: added opensrf.system.sleep method; takes 1
394 param (seconds); useful for debugging child process management
398 * src/extras/Makefile.install: Lenny needs zlib1g-dev package to install
399 XML::LibXML and others
401 Per https://rt.cpan.org/Public/Bug/Display.html?id=51439; also,
402 libmemcached failed to configure and build without this.
406 * src/extras/Makefile.install, src/perl/Build.PL: no longer need FreezeThaw
408 Signed-off-by: Galen Charlton <gmc@esilibrary.com>
412 * src/libopensrf/osrf_prefork.c: in the rare case the select() is interrupted
413 while waiting on child statuses, exit early to prevent read()'s on invalid file
414 handles and allow the calling code to loop back around and try again
418 * src/python/srfsh.py: added support for introspect operation
420 * src/Makefile.am: install srfsh.py when --enable-python is set
422 * src/python/srfsh.py: added support for paging through 'less' (like C); added
423 'router' query docs; added separate env vars for pretty printing json and for
424 print network object keys; explicit opensrf disconnect on exit
426 * src/python/osrf/system.py: added network disconnect func
430 * src/python/srfsh.py: added 'router' command for router class info queries;
431 improved redline tab completion; more changes for consistency with C version;
436 * src/python/osrf/net_obj.py: repaired logic in the as-yet-unused
437 import_array_data; use the more efficient enumerate func; capture array length
438 outside of loop for efficiency; avoid calls to set/get_field and access _data
439 directly inside of the NetworkObject class for faster accessor/mutator's
441 * src/python/srfsh.py: removed debug print statement
443 * src/python/osrf/net_obj.py: rely on logic, not failure through try/catch, to
444 determine of an object is a registered network object; if an object is
445 unregistered, do not treat it as an __unknown network object (which obfuscates
446 and provides no value).
448 * src/python/srfsh.py: killed the terminal colors in srfsh.py output. They are
449 cute and all, but are likely the cause of readline oddities/artifacts (and kind
450 of distracting). Default to raw (json) output for consistency
452 Change log for 2.0.0-beta1
453 ==========================
457 * src/perl/lib/OpenSRF.pm: Bump version number for 2.0 branch
461 * .: Create rel_2_0 branch for next major OpenSRF release
465 * Limit list of Python services to those within the domain-specific activeapps
468 Fix up help with correct default PID directory and document list_all action
472 * If an alternate config file is pointed to in osrf_ctl.sh, respect it
474 In a previous edit, I accidentally removed the config file option
475 from the start_python command, which would break if anything other
476 than the configured default config file was specified as an option to
477 osrf_ctl.sh. This makes opensrf.py once again respect the authority
480 * Use the same PID directory as the opensrf-perl.pl and opensrf.py scripts
482 The default PID directories for the opensrf-perl.pl and opensrf.py scripts
483 was /openils/var/run/opensrf (assuming a prefix of '/openils'), while the
484 default PID directory for osrf_ctl.sh was /openils/var/run. This complicated
485 matters when trying to restart individual Python or Perl services, as without
486 passing an explicit PID directory argument the restart command from the
487 language-specific script would check for a PID in a different PID directory,
488 find none, and end up running a second copy of the service instead of
489 restarting the existing service.
491 The change groups all PIDs in the 'run/opensrf' subdirectory to avoid any
492 PID file conflicts (as unlikely as that might be) if OpenSRF is installed
493 in the default location - so rather than /var/local/run/*.pid, one will
494 be able to look at /var/local/run/opensrf/*.pid
498 * Add the sanity-saving step of a munged PERL5LIB to the README
500 Thanks for the nudge from Thomas Berezanksy!
502 * Commit some unit tests for Python osrf.json module
504 Recommendation: add *_test.py scripts that mirror each
505 osrf.* module in src/python/tests. Boo-yah.
511 * Use the right variable name in to_json_raw()
512 * PEP8 compliance for variable naming
514 * Simpler __tabs() implementation
516 * A solution for starting all Python services: make sh do the work
518 * Enhanced do_start() implementation in Python management script
520 Unclean shutdowns and fork() misery can create PID files that have
521 no actual process behind them. Instead of just trusting the PID file, check
522 for the running PID; if the process is not running, then remove the PID file
523 and actually start the service.
525 * Assign, don't test equality (thanks berick)
527 * Make start_all automatically daemonize services
529 * Give opensrf.py reasonable defaults for options
532 opensrf.py -l -d -f /openils/conf/opensrf_core.xml -p /openils/var/run/ -a
536 opensrf.py -l -d -a start_all
538 Isn't that better? Note that we put the PIDs into PID_DIR/run/opensrf/
539 so that if/when OpenSRF is installed outside of the /openils/ prefix,
540 the names of the processes won't conflict with any other application PIDs.
541 Unlikely, but you never know.
543 * Try not to stomp on global variable names and built-ins, add docstrings
545 * Typos: s/wait_for_child/try_wait_child/ and s/min_childen/min_children/
547 * Typo: Pass in the array of servers to the memcached client if we have one
549 * Reference constants properly
551 * Make pylint happier with PEP8-compliant argument name
553 * Fix daemonize problem that surfaced in start_all
555 Thanks to Michael Giarlo for reporting the problem, Bill Erickson
556 for pointing the way to the solution, and http://bugs.python.org/issue5313
557 for providing me with more context for the problem & solution.
559 * More debuggery for the spawning of child processes
561 * Minutes and months were reversed in Python logging timestamps
563 * Add a Python and curl example of invoking the Perl simpletext services
565 * Add a streaming version of the split() method in simpletext example
569 * Make opensrf-perl.pl handle missing configuration for services more gracefully
571 As we might want to ship configuration files with commented out sections,
572 this will alert the user that a service was listed in the hosts section
573 but the configuration for that service could not be found.
577 * Patch from Michael Giarlo to integrate python service control into osrf_ctl.sh.
579 I made the following additional changes:
581 1. copy opensrf.py into the BIN dir during install when --enable-python is used.
583 2. osrf_ctl.sh will not attempt to control python services when python is not
586 * patch from Michael Giarlo for better sanity checking during settings file
587 reading in python (avoid #comment's)
589 * added osrf.apps to installed pachages (thanks for the tip, dbs). bumped version
592 * socket_bundle typo patch from Jason Stephenson
594 Further discussion and DCO:
596 http://libmail.georgialibraries.org/pipermail/open-ils-dev/2010-November/006476.html
600 * consistent w/ the C libs, only log server message processing duration at INFO.
601 log server response processing duration at debug and w/ slighly different
602 wording. let the processing duration log message act as the indication of a
603 successfully handled message to reduce logging
607 * log message processing duration at INFO level, instead of DEBUG, consistent with
612 * Plug a minor memory leak that could occur when we opened a TCP connection
613 to a server. In practice this meant whenever we opened a Jabber session.
615 We call getaddrinfo() to get a dynamically allocated linked list of
616 addresses for a given server name. Then we traverse the list, looking
617 for one that accepts streaming connections over IPV4, in order to get
620 At the end, we call freeaddrinfo() to free the linked list.
622 Previously we would pass to freeaddrinfo() a pointer, not to the
623 head of the list, but to the node that we used for getting an IP
624 address. Prior nodes, if any, would leak.
626 Also: added calls to freeaddrinfo() in the case of early returns, to
627 avoid leaking the list in the event of an error.
629 M src/libopensrf/socket_bundle.c
633 * implemented an optional per-service stderr log for capturing miscellaneous
634 stderr output from services, similar to the old-style _unix.log files, since
635 there are often useful warnings and error messages that never bubble up to
636 syslog. for clarity, the files now use _stderr as a suffix instead of _unix.
637 stderr logs are enabled by default. included opensrf.xml example of how to
638 disable it for a given service
640 * when we add xhr responses onto the end of the queue, stash the original session
641 as well, otherwise we're attempting to ask a session questions about requests
646 * in json encoder, replace 2 sub calls w/ 1 hash lookup in heavily called code
647 path. we lose some abstraction, but it's probably worth it here
651 * Work around a Net::Domain bug that can result in fqdn's like
652 foo.example.com,bar.com
654 The bug manifested when trying to run autogen.sh, with the output:
656 No Response from settings server...going to sleep
658 This was because SettingsParser was generating invalid XPath and consquently
659 errors in the opensrf.settings service; with no response, the attempt to run
660 autogen.sh would die.
662 This workaround splits the fqdn on commas and tries each possible domain in
663 the server setting XPath. Long term we either need to wait for Net::Domain
664 bug #60729 (https://rt.cpan.org/Public/Bug/Display.html?id=60729) to be
665 resolved, or consider alternatives.
669 * return the trace-augmented message as written to the log
673 * removing vestigial references to Inbound.pm, which no longer exists in the
674 repository. this also removes get/set_listener, which were never taken advantage
675 of and are no longer of use since the listener is thoroughly hard-coded into the
680 * Bump trunk version number for OpenSRF up to 1.8.0; better than 1.4.0.
682 perlmodstyle says "A correct CPAN version number is a floating point
683 number with at least 2 digits after the decimal." so we're not in
684 compliance in any case. Oh well.
688 * Update the manifest for Perl modules to avoid build warnings
690 r2016 removed two Perl modules,
691 src/perl/lib/OpenSRF/Transport/SlimJabber/Inbound.pm
692 and src/perl/lib/OpenSRF/UnixServer.pm, and the build gets upset because the
693 manifest suggests they're missing. But no, we know where they are, and we
694 don't expect them to come back.
698 * added support for reading ContinueStatus opensrf messages in the JS opensrf lib
702 * updated mailing address of the FSF
704 * remove executable bit from module files that don't need it
706 Small nit to pick for Debianization.
710 * Replace Net::Server with local pre-forking server
712 Support max/min children and max/min spare children
714 http://libmail.georgialibraries.org/pipermail/open-ils-dev/2010-May/006068.html
718 * remove ntp and ntpdate dependencies
720 Not used by OpenSRF; these are legacies of
721 Evergreen dependencies at the time OpenSRF was
722 split off. Quite likely a bunch of deps in
723 Makefile.install aren't needed by OpenSRF