1 = Easing gently into OpenSRF =
2 Dan Scott <dscott@laurentian.ca>
6 The Evergreen open-source library system serves library consortia composed of
7 hundreds of branches with millions of patrons - for example,
8 http://www.georgialibraries.org/statelibrarian/bythenumbers.pdf[the Georgia
9 Public Library Service PINES system]. One of the claimed advantages of
10 Evergreen over alternative integrated library systems is the underlying Open
11 Service Request Framework (OpenSRF, pronounced "open surf") architecture. This
12 article introduces OpenSRF, demonstrates how to build OpenSRF services through
13 simple code examples, and explains the technical foundations on which OpenSRF
16 == Introducing OpenSRF ==
17 OpenSRF is a message routing network that offers scalability and failover
18 support for individual services and entire servers with minimal development and
19 deployment overhead. You can use OpenSRF to build loosely-coupled applications
20 that can be deployed on a single server or on clusters of geographically
21 distributed servers using the same code and minimal configuration changes.
22 Although copyright statements on some of the OpenSRF code date back to Mike
23 Rylander's original explorations in 2000, Evergreen was the first major
24 application to be developed with, and to take full advantage of, the OpenSRF
25 architecture starting in 2004. The first official release of OpenSRF was 0.1 in
26 February 2005 (http://evergreen-ils.org/blog/?p=21), but OpenSRF's development
27 continues a steady pace of enhancement and refinement, with the release of
28 1.0.0 in October 2008 and the most recent release of 1.2.2 in February 2010.
30 OpenSRF is a distinct break from the architectural approach used by previous
31 library systems and has more in common with modern Web applications. The
32 traditional "scale-up" approach to serve more transactions is to purchase a
33 server with more CPUs and more RAM, possibly splitting the load between a Web
34 server, a database server, and a business logic server. Evergreen, however, is
35 built on the Open Service Request Framework (OpenSRF) architecture, which
36 firmly embraces the "scale-out" approach of spreading transaction load over
37 cheap commodity servers. The http://evergreen-ils.org/blog/?p=56[initial GPLS
38 PINES hardware cluster], while certainly impressive, may have offered the
39 misleading impression that Evergreen is complex and requires a lot of hardware
42 This article hopes to correct any such lingering impression by demonstrating
43 that OpenSRF itself is an extremely simple architecture on which one can easily
44 build applications of many kinds – not just library applications – and that you
45 can use a number of different languages to call and implement OpenSRF methods
46 with a minimal learning curve. With an application built on OpenSRF, when you
47 identify a bottleneck in your application's business logic layer, you can
48 adjust the number of the processes serving that particular bottleneck on each
49 of your servers; or if the problem is that your service is resource-hungry, you
50 could add an inexpensive server to your cluster and dedicate it to running that
51 resource-hungry service.
53 === Programming language support ===
55 If you need to develop an entirely new OpenSRF service, you can choose from a
56 number of different languages in which to implement that service. OpenSRF
57 client language bindings have been written for C, Java, JavaScript, Perl, and
58 Python, and server language bindings have been written for C, Perl, and Python.
59 This article uses Perl examples as a lowest common denominator programming
60 language. Writing an OpenSRF binding for another language is a relatively small
61 task if that language offers libraries that support the core technologies on
62 which OpenSRF depends:
64 * http://tools.ietf.org/html/rfc3920[Extensible Messaging and Presence
65 Protocol] (XMPP, sometimes referred to as Jabber) - provides the base messaging
66 infrastructure between OpenSRF clients and servers
67 * http://json.org[JavaScript Object Notation] (JSON) - serializes the content
68 of each XMPP message in a standardized and concise format
69 * http://memcached.org[memcached] - provides the caching service
70 * http://tools.ietf.org/html/rfc5424[syslog] - the standard UNIX logging
74 http://evergreen-ils.org/dokuwiki/doku.php?id=osrf-devel:primer[OpenSRF
75 reference documentation], although augmented by the
76 http://evergreen-ils.org/dokuwiki/doku.php?id=osrf-devel:primer[OpenSRF
77 glossary], blog posts like http://evergreen-ils.org/blog/?p=36[the description
78 of OpenSRF and Jabber], and even this article, is not a sufficient substitute
79 for a complete specification on which one could implement a language binding.
80 The recommended option for would-be developers of another language binding is
81 to use the Python implementation as the cleanest basis for a port to another
84 === OpenSRF communication flows over XMPP ===
86 The XMPP messaging service underpins OpenSRF, requiring an XMPP server such
87 as http://www.ejabberd.im/[ejabberd]. When you start OpenSRF, the first XMPP
88 clients that connect to the XMPP server are the OpenSRF public and private
89 _routers_. OpenSRF routers maintain a list of available services and connect
90 clients to available services. When an OpenSRF service starts, it establishes a
91 connection to the XMPP server and registers itself with the private router. The
92 OpenSRF configuration contains a list of public OpenSRF services, each of which
93 must also register with the public router. Services and clients connect to the
94 XMPP server using a single set of XMPP client credentials (for example,
95 `opensrf@private.localhost`), but use XMPP resource identifiers to
96 differentiate themselves in the Jabber ID (JID) for each connection. For
97 example, the JID for a copy of the `opensrf.simple-text` service with process
98 ID `6285` that has connected to the `private.localhost` domain using the
99 `opensrf` XMPP client credentials could be
100 `opensrf@private.localhost/opensrf.simple-text_drone_at_localhost_6285`.
102 [id="OpenSRFOverHTTP"]
103 === OpenSRF communication flows over HTTP ===
104 Any OpenSRF service registered with the public router is accessible via the
105 OpenSRF HTTP Translator. The OpenSRF HTTP Translator implements the
106 http://www.open-ils.org/dokuwiki/doku.php?id=opensrf_over_http[OpenSRF-over-HTTP
107 proposed specification] as an Apache module that translates HTTP requests into
108 OpenSRF requests and returns OpenSRF results as HTTP results to the initiating
111 .Issuing an HTTP POST request to an OpenSRF method via the OpenSRF HTTP Translator
113 --------------------------------------------------------------------------------
114 # curl request broken up over multiple lines for legibility
115 curl -H "X-OpenSRF-service: opensrf.simple-text" \ # <1>
116 --data 'osrf-msg=[ \ # <2>
117 {"__c":"osrfMessage","__p":{"threadTrace":0,"locale":"en-CA", \ # <3>
118 "type":"REQUEST","payload": {"__c":"osrfMethod","__p": \
119 {"method":"opensrf.simple-text.reverse","params":["foobar"]} \
122 http://localhost/osrf-http-translator \ # <4>
123 --------------------------------------------------------------------------------
125 <1> The `X-OpenSRF-service` header identifies the OpenSRF service of interest.
127 <2> The POST request consists of a single parameter, the `osrf-msg` value,
128 which contains a JSON array.
130 <3> The first object is an OpenSRF message (`"__c":"osrfMessage"`) with a set of
131 parameters (`"__p":{}`) containing:
133 * the identifier for the request (`"threadTrace":0`); this value is echoed
136 * the message type (`"type":"REQUEST"`)
138 * the locale for the message; if the OpenSRF method is locale-sensitive, it
139 can check the locale for each OpenSRF request and return different information
140 depending on the locale
142 * the payload of the message (`"payload":{}`) containing the OpenSRF method
143 request (`"__c":"osrfMethod"`) and its parameters (`"__p:"{}`), which in turn
146 ** the method name for the request (`"method":"opensrf.simple-text.reverse"`)
148 ** a set of JSON parameters to pass to the method (`"params":["foobar"]`); in
149 this case, a single string `"foobar"`
151 <4> The URL on which the OpenSRF HTTP translator is listening,
152 `/osrf-http-translator` is the default location in the Apache example
153 configuration files shipped with the OpenSRF source, but this is configurable.
156 .Results from an HTTP POST request to an OpenSRF method via the OpenSRF HTTP Translator
158 --------------------------------------------------------------------------------
159 # HTTP response broken up over multiple lines for legibility
160 [{"__c":"osrfMessage","__p": \ # <1>
161 {"threadTrace":0, "payload": \ # <2>
162 {"__c":"osrfResult","__p": \ # <3>
163 {"status":"OK","content":"raboof","statusCode":200} \ # <4>
164 },"type":"RESULT","locale":"en-CA" \ # <5>
167 {"__c":"osrfMessage","__p": \ # <6>
168 {"threadTrace":0,"payload": \ # <7>
169 {"__c":"osrfConnectStatus","__p": \ # <8>
170 {"status":"Request Complete","statusCode":205} \ # <9>
171 },"type":"STATUS","locale":"en-CA" \ # <10>
174 --------------------------------------------------------------------------------
176 <1> The OpenSRF HTTP Translator returns an array of JSON objects in its
177 response. Each object in the response is an OpenSRF message
178 (`"__c":"osrfMessage"`) with a collection of response parameters (`"__p":`).
180 <2> The OpenSRF message identifier (`"threadTrace":0`) confirms that this
181 message is in response to the request matching the same identifier.
183 <3> The message includes a payload JSON object (`"payload":`) with an OpenSRF
184 result for the request (`"__c":"osrfResult"`).
186 <4> The result includes a status indicator string (`"status":"OK"`), the content
187 of the result response - in this case, a single string "raboof"
188 (`"content":"raboof"`) - and an integer status code for the request
189 (`"statusCode":200`).
191 <5> The message also includes the message type (`"type":"RESULT"`) and the
192 message locale (`"locale":"en-CA"`).
194 <6> The second message in the set of results from the response.
196 <7> Again, the message identifier confirms that this message is in response to
197 a particular request.
199 <8> The payload of the message denotes that this message is an
200 OpenSRF connection status message (`"__c":"osrfConnectStatus"`), with some
201 information about the particular OpenSRF connection that was used for this
204 <9> The response parameters for an OpenSRF connection status message include a
205 verbose status (`"status":"Request Complete"`) and an integer status code for
206 the connection status (`"statusCode":205).
208 <10> The message also includes the message type (`"type":"RESULT"`) and the
209 message locale (`"locale":"en-CA"`).
213 Before adding a new public OpenSRF service, ensure that it does
214 not introduce privilege escalation or unchecked access to data. For example,
215 the Evergreen `open-ils.cstore` private service is an object-relational mapper
216 that provides read and write access to the entire Evergreen database, so it
217 would be catastrophic to expose that service publicly. In comparison, the
218 Evergreen `open-ils.pcrud` public service offers the same functionality as
219 `open-ils.cstore` to any connected HTTP client or OpenSRF client, but the
220 additional authentication and authorization layer in `open-ils.pcrud` prevents
221 unchecked access to Evergreen's data.
223 === Stateless and stateful connections ===
225 OpenSRF supports both _stateless_ and _stateful_ connections. When an OpenSRF
226 client issues a `REQUEST` message in a _stateless_ connection, the router
227 forwards the request to the next available service and the service returns the
228 result directly to the client.
230 .REQUEST flow in a stateless connection
231 image:media/REQUEST.png[REQUEST flow in a stateless connection]
233 When an OpenSRF client issues a `CONNECT` message to create a _stateful_ conection, the
234 router returns the Jabber ID of the next available service to the client so
235 that the client can issue one or more `REQUEST` message directly to that
236 particular service and the service will return corresponding `RESULT` messages
237 directly to the client. Until the client issues a `DISCONNECT` message, that
238 particular service is only available to the requesting client. Stateful connections
239 are useful for clients that need to make many requests from a particular service,
240 as it avoids the intermediary step of contacting the router for each request, as
241 well as for operations that require a controlled sequence of commands, such as a
242 set of database INSERT, UPDATE, and DELETE statements within a transaction.
244 .CONNECT, REQUEST, and DISCONNECT flow in a stateful connection
245 image:media/CONNECT.png[CONNECT, REQUEST, and DISCONNECT flow in a stateful connection]
247 == Enough jibber-jabber: writing an OpenSRF service ==
248 Imagine an application architecture in which 10 lines of Perl or Python, using
249 the data types native to each language, are enough to implement a method that
250 can then be deployed and invoked seamlessly across hundreds of servers. You
251 have just imagined developing with OpenSRF – it is truly that simple. Under the
252 covers, of course, the OpenSRF language bindings do an incredible amount of
253 work on behalf of the developer. An OpenSRF application consists of one or more
254 OpenSRF services that expose methods: for example, the `opensrf.simple-text`
255 http://svn.open-ils.org/trac/OpenSRF/browser/trunk/src/perl/lib/OpenSRF/Application/Demo/SimpleText.pm[demonstration
256 service] exposes the `opensrf.simple-text.split()` and
257 `opensrf.simple-text.reverse()` methods. Each method accepts zero or more
258 arguments and returns zero or one results. The data types supported by OpenSRF
259 arguments and results are typical core language data types: strings, numbers,
260 booleans, arrays, and hashes.
262 To implement a new OpenSRF service, perform the following steps:
264 1. Include the base OpenSRF support libraries
265 2. Write the code for each of your OpenSRF methods as separate procedures
266 3. Register each method
267 4. Add the service definition to the OpenSRF configuration files
269 For example, the following code implements an OpenSRF service. The service
270 includes one method named `opensrf.simple-text.reverse()` that accepts one
271 string as input and returns the reversed version of that string:
274 --------------------------------------------------------------------------------
277 package OpenSRF::Application::Demo::SimpleText;
281 use OpenSRF::Application;
282 use parent qw/OpenSRF::Application/;
285 my ($self , $conn, $text) = @_;
286 my $reversed_text = scalar reverse($text);
287 return $reversed_text;
290 __PACKAGE__->register_method(
291 method => 'text_reverse',
292 api_name => 'opensrf.simple-text.reverse'
294 --------------------------------------------------------------------------------
296 Ten lines of code, and we have a complete OpenSRF service that exposes a single
297 method and could be deployed quickly on a cluster of servers to meet your
298 application's ravenous demand for reversed strings! If you're unfamiliar with
299 Perl, the `use OpenSRF::Application; use parent qw/OpenSRF::Application/;`
300 lines tell this package to inherit methods and properties from the
301 `OpenSRF::Application` module. For example, the call to
302 `__PACKAGE__->register_method()` is defined in `OpenSRF::Application` but due to
303 inheritance is available in this package (named by the special Perl symbol
304 `__PACKAGE__` that contains the current package name). The `register_method()`
305 procedure is how we introduce a method to the rest of the OpenSRF world.
307 [id="serviceRegistration"]
308 === Registering a service with the OpenSRF configuration files ===
310 Two files control most of the configuration for OpenSRF:
312 * `opensrf.xml` contains the configuration for the service itself as well as
313 a list of which application servers in your OpenSRF cluster should start
315 * `opensrf_core.xml` (often referred to as the "bootstrap configuration"
316 file) contains the OpenSRF networking information, including the XMPP server
317 connection credentials for the public and private routers; you only need to touch
318 this for a new service if the new service needs to be accessible via the
321 Begin by defining the service itself in `opensrf.xml`. To register the
322 `opensrf.simple-text` service, add the following section to the `<apps>`
323 element (corresponding to the XPath `/opensrf/default/apps/`):
326 --------------------------------------------------------------------------------
328 <opensrf.simple-text> <!-- <1> -->
329 <keepalive>3</keepalive> <!-- <2> -->
330 <stateless>1</stateless> <!-- <3> -->
331 <language>perl</language> <!-- <4> -->
332 <implementation>OpenSRF::Application::Demo::SimpleText</implementation> <!-- <5> -->
333 <max_requests>100</max_requests> <!-- <6> -->
335 <max_requests>1000</max_requests> <!-- <7> -->
336 <unix_log>opensrf.simple-text_unix.log</unix_log> <!-- <8> -->
337 <unix_sock>opensrf.simple-text_unix.sock</unix_sock> <!-- <9> -->
338 <unix_pid>opensrf.simple-text_unix.pid</unix_pid> <!-- <10> -->
339 <min_children>5</min_children> <!-- <11> -->
340 <max_children>15</max_children> <!-- <12> -->
341 <min_spare_children>2</min_spare_children> <!-- <13> -->
342 <max_spare_children>5</max_spare_children> <!-- <14> -->
344 </opensrf.simple-text>
346 <!-- other OpenSRF services registered here... -->
348 --------------------------------------------------------------------------------
350 <1> The element name is the name that the OpenSRF control scripts use to refer
353 <2> Specifies the interval (in seconds) between checks to determine if the
354 service is still running.
356 <3> Specifies whether OpenSRF clients can call methods from this service
357 without first having to create a connection to a specific service backend
358 process for that service. If the value is `1`, then the client can simply
359 issue a request and the router will forward the request to an available
360 service and the result will be returned directly to the client.
362 <4> Specifies the programming language in which the service is implemented
364 <5> Specifies the name of the library or module in which the service is implemented
366 <6> (C implementations): Specifies the maximum number of requests a process
367 serves before it is killed and replaced by a new process.
369 <7> (Perl implementations): Specifies the maximum number of requests a process
370 serves before it is killed and replaced by a new process.
372 <8> The name of the log file for language-specific log messages such as syntax
375 <9> The name of the UNIX socket used for inter-process communications.
377 <10> The name of the PID file for the master process for the service.
379 <11> The minimum number of child processes that should be running at any given
382 <12> The maximum number of child processes that should be running at any given
385 <13> The minimum number of child processes that should be available to handle
386 incoming requests. If there are fewer than this number of spare child
387 processes, new processes will be spawned.
389 <14> The minimum number of child processes that should be available to handle
390 incoming requests. If there are more than this number of spare child processes,
391 the extra processes will be killed.
393 To make the service accessible via the public router, you must also
394 edit the `opensrf_core.xml` configuration file to add the service to the list
395 of publicly accessible services:
397 .Making a service publicly accessible in `opensrf_core.xml`
399 --------------------------------------------------------------------------------
400 <router> <!-- <1> -->
401 <!-- This is the public router. On this router, we only register applications
402 which should be accessible to everyone on the opensrf network -->
404 <domain>public.localhost</domain> <!-- <2> -->
406 <service>opensrf.math</service>
407 <service>opensrf.simple-text</service> <!-- <3> -->
410 --------------------------------------------------------------------------------
412 <1> This section of the `opensrf_core.xml` file is located at XPath
413 `/config/opensrf/routers/`.
415 <2> `public.localhost` is the canonical public router domain in the OpenSRF
416 installation instructions.
418 <3> Each `<service>` element contained in the `<services>` element
419 offers their services via the public router as well as the private router.
421 Once you have defined the new service, you must restart the OpenSRF Router
422 to retrieve the new configuration and start or restart the service itself.
424 === Calling an OpenSRF method ===
425 OpenSRF clients in any supported language can invoke OpenSRF services in any
426 supported language. So let's see a few examples of how we can call our fancy
427 new `opensrf.simple-text.reverse()` method:
429 ==== Calling OpenSRF methods from the srfsh client ====
430 `srfsh` is a command-line tool installed with OpenSRF that you can use to call
431 OpenSRF methods. To call an OpenSRF method, issue the `request` command and pass
432 the OpenSRF service and method name as the first two arguments; then pass a list
433 of JSON objects as the arguments to the method being invoked.
435 The following example calls the `opensrf.simple-text.reverse` method of the
436 `opensrf.simple-text` OpenSRF service, passing the string `"foobar"` as the
437 only method argument:
440 --------------------------------------------------------------------------------
442 srfsh # request opensrf.simple-text opensrf.simple-text.reverse "foobar"
444 Received Data: "raboof"
446 =------------------------------------
447 Request Completed Successfully
448 Request Time in seconds: 0.016718
449 =------------------------------------
450 --------------------------------------------------------------------------------
452 [id="opensrfIntrospection"]
453 ==== Getting documentation for OpenSRF methods from the srfsh client ====
455 The `srfsh` client also gives you command-line access to retrieving metadata
456 about OpenSRF services and methods. For a given OpenSRF method, for example,
457 you can retrieve information such as the minimum number of required arguments,
458 the data type and a description of each argument, the package or library in
459 which the method is implemented, and a description of the method. To retrieve
460 the documentation for an opensrf method from `srfsh`, issue the `introspect`
461 command, followed by the name of the OpenSRF service and (optionally) the
462 name of the OpenSRF method. If you do not pass a method name to the `introspect`
463 command, `srfsh` lists all of the methods offered by the service. If you pass
464 a partial method name, `srfsh` lists all of the methods that match that portion
468 The quality and availability of the descriptive information for each
469 method depends on the developer to register the method with complete and
470 accurate information. The quality varies across the set of OpenSRF and
471 Evergreen APIs, although some effort is being put towards improving the
472 state of the internal documentation.
475 --------------------------------------------------------------------------------
476 srfsh# introspect opensrf.simple-text "opensrf.simple-text.reverse"
477 --> opensrf.simple-text
480 "__c":"opensrf.simple-text",
484 "object_hint":"OpenSRF_Application_Demo_SimpleText",
486 "package":"OpenSRF::Application::Demo::SimpleText", \ # <2>
487 "api_name":"opensrf.simple-text.reverse", \ # <3>
488 "server_class":"opensrf.simple-text",
489 "signature":{ \ # <4>
492 "desc":"The string to reverse",
497 "desc":"Returns the input string in reverse order\n", \ # <6>
499 "desc":"Returns the input string in reverse order",
503 "method":"text_reverse", \ # <8>
507 --------------------------------------------------------------------------------
509 <1> `stream` denotes whether the method supports streaming responses or not.
511 <2> `package` identifies which package or library implements the method.
513 <3> `api_name` identifies the name of the OpenSRF method.
515 <4> `signature` is a hash that describes the parameters for the method.
517 <5> `params` is an array of hashes describing each parameter in the method;
518 each parameter has a description (`desc`), name (`name`), and type (`type`).
520 <6> `desc` is a string that describes the method itself.
522 <7> `return` is a hash that describes the return value for the method; it
523 contains a description of the return value (`desc`) and the type of the
524 returned value (`type`).
526 <8> `method` identifies the name of the function or method in the source
529 <9> `argc` is an integer describing the minimum number of arguments that
530 must be passed to this method.
532 ==== Calling OpenSRF methods from Perl applications ====
534 To call an OpenSRF method from Perl, you must connect to the OpenSRF service,
535 issue the request to the method, and then retrieve the results.
538 --------------------------------------------------------------------------------
541 use OpenSRF::AppSession;
544 OpenSRF::System->bootstrap_client(config_file => '/openils/conf/opensrf_core.xml'); # <1>
546 my $session = OpenSRF::AppSession->create("opensrf.simple-text"); # <2>
548 print "substring: Accepts a string and a number as input, returns a string\n";
549 my $result = $session->request("opensrf.simple-text.substring", "foobar", 3); # <3>
550 my $request = $result->gather(); # <4>
551 print "Substring: $request\n\n";
553 print "split: Accepts two strings as input, returns an array of strings\n";
554 $request = $session->request("opensrf.simple-text.split", "This is a test", " "); # <5>
555 my $output = "Split: [";
557 while ($element = $request->recv()) { # <6>
558 $output .= $element->content . ", "; # <7>
561 print $output . "\n\n";
563 print "statistics: Accepts an array of strings as input, returns a hash\n";
565 "First I think I'll have breakfast",
566 "Then I think that lunch would be nice",
567 "And then seventy desserts to finish off the day"
570 $result = $session->request("opensrf.simple-text.statistics", @many_strings); # <8>
571 $request = $result->gather(); # <9>
572 print "Length: " . $result->{'length'} . "\n";
573 print "Word count: " . $result->{'word_count'} . "\n";
575 $session->disconnect(); # <10>
576 --------------------------------------------------------------------------------
578 <1> The `OpenSRF::System->bootstrap_client()` method reads the OpenSRF
579 configuration information from the indicated file and creates an XMPP client
580 connection based on that information.
582 <2> The `OpenSRF::AppSession->create()` method accepts one argument - the name
583 of the OpenSRF service to which you want to want to make one or more requests -
584 and returns an object prepared to use the client connection to make those
587 <3> The `OpenSRF::AppSession->request()` method accepts a minimum of one
588 argument - the name of the OpenSRF method to which you want to make a request -
589 followed by zero or more arguments to pass to the OpenSRF method as input
590 values. This example passes a string and an integer to the
591 `opensrf.simple-text.substring` method defined by the `opensrf.simple-text`
594 <4> The `gather()` method, called on the result object returned by the
595 `request()` method, iterates over all of the possible results from the result
596 object and returns a single variable.
598 <5> This `request()` call passes two strings to the `opensrf.simple-text.split`
599 method defined by the `opensrf.simple-text` OpenSRF service and returns (via
600 `gather()`) a reference to an array of results.
602 <6> The `opensrf.simple-text.split()` method is a streaming method that
603 returns an array of results with one element per `recv()` call on the
604 result object. We could use the `gather()` method to retrieve all of the
605 results in a single array reference, but instead we simply iterate over
606 the result variable until there are no more results to retrieve.
608 <7> While the `gather()` convenience method returns only the content of the
609 complete set of results for a given request, the `recv()` method returns an
610 OpenSRF result object with `status`, `statusCode`, and `content` fields as
611 we saw in <<httpResults,the HTTP results example>>.
613 <8> This `request()` call passes an array to the
614 `opensrf.simple-text.statistics` method defined by the `opensrf.simple-text`
617 <9> The result object returns a hash reference via `gather()`. The hash
618 contains the `length` and `word_count` keys we defined in the method.
620 <10> The `OpenSRF::AppSession->disconnect()` method closes the XMPP client
621 connection and cleans up resources associated with the session.
623 === Accepting and returning more interesting data types ===
625 Of course, the example of accepting a single string and returning a single
626 string is not very interesting. In real life, our applications tend to pass
627 around multiple arguments, including arrays and hashes. Fortunately, OpenSRF
628 makes that easy to deal with; in Perl, for example, returning a reference to
629 the data type does the right thing. In the following example of a method that
630 returns a list, we accept two arguments of type string: the string to be split,
631 and the delimiter that should be used to split the string.
633 .Text splitting method - streaming mode
635 --------------------------------------------------------------------------------
640 my $delimiter = shift || ' ';
642 my @split_text = split $delimiter, $text;
646 __PACKAGE__->register_method(
647 method => 'text_split',
648 api_name => 'opensrf.simple-text.split'
650 --------------------------------------------------------------------------------
652 We simply return a reference to the list, and OpenSRF does the rest of the work
653 for us to convert the data into the language-independent format that is then
654 returned to the caller. As a caller of a given method, you must rely on the
655 documentation used to register to determine the data structures - if the developer has
656 added the appropriate documentation.
658 === Accepting and returning Evergreen objects ===
660 OpenSRF is agnostic about objects; its role is to pass JSON back and forth
661 between OpenSRF clients and services, and it allows the specific clients and
662 services to define their own semantics for the JSON structures. On top of that
663 infrastructure, Evergreen offers the fieldmapper: an object-relational mapper
664 that provides a complete definition of all objects, their properties, their
665 relationships to other objects, the permissions required to create, read,
666 update, or delete objects of that type, and the database table or view on which
669 The Evergreen fieldmapper offers a great deal of convenience for working with
670 complex system objects beyond the basic mapping of classes to database
671 schemas. Although the result is passed over the wire as a JSON object
672 containing the indicated fields, fieldmapper-aware clients then turn those
673 JSON objects into native objects with setter / getter methods for each field.
675 All of this metadata about Evergreen objects is defined in the
676 fieldmapper configuration file (`/openils/conf/fm_IDL.xml`), and access to
677 these classes is provided by the `open-ils.cstore`, `open-ils.pcrud`, and
678 `open-ils.reporter-store` OpenSRF services which parse the fieldmapper
679 configuration file and dynamically register OpenSRF methods for creating,
680 reading, updating, and deleting all of the defined classes.
682 .Example fieldmapper class definition for "Open User Summary"
684 --------------------------------------------------------------------------------
685 <class id="mous" controller="open-ils.cstore open-ils.pcrud"
686 oils_obj:fieldmapper="money::open_user_summary"
687 oils_persist:tablename="money.open_usr_summary"
688 reporter:label="Open User Summary"> <!-- <1> -->
689 <fields oils_persist:primary="usr" oils_persist:sequence=""> <!-- <2> -->
690 <field name="balance_owed" reporter:datatype="money" /> <!-- <3> -->
691 <field name="total_owed" reporter:datatype="money" />
692 <field name="total_paid" reporter:datatype="money" />
693 <field name="usr" reporter:datatype="link"/>
696 <link field="usr" reltype="has_a" key="id" map="" class="au"/> <!-- <4> -->
698 <permacrud xmlns="http://open-ils.org/spec/opensrf/IDL/permacrud/v1"> <!-- <5> -->
700 <retrieve permission="VIEW_USER"> <!-- <6> -->
701 <context link="usr" field="home_ou"/> <!-- <7> -->
706 --------------------------------------------------------------------------------
708 <1> The `<class>` element defines the class:
710 * The `id` attribute defines the _class hint_ that identifies the class both
711 elsewhere in the fieldmapper configuration file, such as in the value of the
712 `field` attribute of the `<link>` element, and in the JSON object itself when
713 it is instantiated. For example, an "Open User Summary" JSON object would have
714 the top level property of `"__c":"mous"`.
716 * The `controller` attribute identifies the services that have direct access
717 to this class. If `open-ils.pcrud` is not listed, for example, then there is
718 no means to directly access members of this class through a public service.
720 * The `oils_obj:fieldmapper` attribute defines the name of the Perl
721 fieldmapper class that will be dynamically generated to provide setter and
722 getter methods for instances of the class.
724 * The `oils_persist:tablename` attribute identifies the schema name and table
725 name of the database table that stores the data that represents the instances
726 of this class. In this case, the schema is `money` and the table is
729 * The `reporter:label` attribute defines a human-readable name for the class
730 used in the reporting interface to identify the class. These names are defined
731 in English in the fieldmapper configuration file; however, they are extracted
732 so that they can be translated and served in the user's language of choice.
734 <2> The `<fields>` element lists all of the fields that belong to the object.
736 * The `oils_persist:primary` attribute identifies the field that acts as the
737 primary key for the object; in this case, the field with the name `usr`.
739 * The `oils_persist:sequence` attribute identifies the sequence object
740 (if any) in this database provides values for new instances of this class. In
741 this case, the primary key is defined by a field that is linked to a different
742 table, so no sequence is used to populate these instances.
744 <3> Each `<field>` element defines a single field with the following attributes:
746 * The `name` attribute identifies the column name of the field in the
747 underlying database table as well as providing a name for the setter / getter
748 method that can be invoked in the JSON or native version of the object.
750 * The `reporter:datatype` attribute defines how the reporter should treat
751 the contents of the field for the purposes of querying and display.
753 * The `reporter:label` attribute can be used to provide a human-readable name
754 for each field; without it, the reporter falls back to the value of the `name`
757 <4> The `<links>` element contains a set of zero or more `<link>` elements,
758 each of which defines a relationship between the class being described and
761 * The `field` attribute identifies the field named in this class that links
762 to the external class.
764 * The `reltype` attribute identifies the kind of relationship between the
765 classes; in the case of `has_a`, each value in the `usr` field is guaranteed
766 to have a corresponding value in the external class.
768 * The `key` attribute identifies the name of the field in the external
769 class to which this field links.
771 * The rarely-used `map` attribute identifies a second class to which
772 the external class links; it enables this field to define a direct
773 relationship to an external class with one degree of separation, to
774 avoid having to retrieve all of the linked members of an intermediate
775 class just to retrieve the instances from the actual desired target class.
777 * The `class` attribute identifies the external class to which this field
780 <5> The `<permacrud>` element defines the permissions that must have been
781 granted to a user to operate on instances of this class.
783 <6> The `<retrieve>` element is one of four possible children of the
784 `<actions>` element that define the permissions required for each action:
785 create, retrieve, update, and delete.
787 * The `permission` attribute identifies the name of the permission that must
788 have been granted to the user to perform the action.
790 * The `contextfield` attribute, if it exists, defines the field in this class
791 that identifies the library within the system for which the user must have
792 prvileges to work. If a user has been granted a given permission, but has not been
793 granted privileges to work at a given library, they can not perform the action
796 <7> The rarely-used `<context>` element identifies a linked field (`link`
797 attribute) in this class which links to an external class that holds the field
798 (`field` attribute) that identifies the library within the system for which the
799 user must have privileges to work.
801 When you retrieve an instance of a class, you can ask for the result to
802 _flesh_ some or all of the linked fields of that class, so that the linked
803 instances are returned embedded directly in your requested instance. In that
804 same request you can ask for the fleshed instances to in turn have their linked
805 fields fleshed. By bundling all of this into a single request and result
806 sequence, you can avoid the network overhead of requiring the client to request
807 the base object, then request each linked object in turn.
809 You can also iterate over a collection of instances and set the automatically
810 generated `isdeleted`, `isupdated`, or `isnew` properties to indicate that
811 the given instance has been deleted, updated, or created respectively.
812 Evergreen can then act in batch mode over the collection to perform the
813 requested actions on any of the instances that have been flagged for action.
815 === Returning streaming results ===
817 In the previous implementation of the `opensrf.simple-text.split` method, we
818 returned a reference to the complete array of results. For small values being
819 delivered over the network, this is perfectly acceptable, but for large sets of
820 values this can pose a number of problems for the requesting client. Consider a
821 service that returns a set of bibliographic records in response to a query like
822 "all records edited in the past month"; if the underlying database is
823 relatively active, that could result in thousands of records being returned as
824 a single network request. The client would be forced to block until all of the
825 results are returned, likely resulting in a significant delay, and depending on
826 the implementation, correspondingly large amounts of memory might be consumed
827 as all of the results are read from the network in a single block.
829 OpenSRF offers a solution to this problem. If the method returns results that
830 can be divided into separate meaningful units, you can register the OpenSRF
831 method as a streaming method and enable the client to loop over the results one
832 unit at a time until the method returns no further results. In addition to
833 registering the method with the provided name, OpenSRF also registers an additional
834 method with `.atomic` appended to the method name. The `.atomic` variant gathers
835 all of the results into a single block to return to the client, giving the caller
836 the ability to choose either streaming or atomic results from a single method
839 In the following example, the text splitting method has been reimplemented to
840 support streaming; very few changes are required:
842 .Text splitting method - streaming mode
844 --------------------------------------------------------------------------------
849 my $delimiter = shift || ' ';
851 my @split_text = split $delimiter, $text;
852 foreach my $string (@split_text) { # <1>
853 $conn->respond($string);
858 __PACKAGE__->register_method(
859 method => 'text_split',
860 api_name => 'opensrf.simple-text.split',
863 --------------------------------------------------------------------------------
865 <1> Rather than returning a reference to the array, a streaming method loops
866 over the contents of the array and invokes the `respond()` method of the
867 connection object on each element of the array.
869 <2> Registering the method as a streaming method instructs OpenSRF to also
870 register an atomic variant (`opensrf.simple-text.split.atomic`).
872 === Error! Warning! Info! Debug! ===
873 As hard as it may be to believe, it is true: applications sometimes do not
874 behave in the expected manner, particularly when they are still under
875 development. The server language bindings for OpenSRF include integrated
876 support for logging messages at the levels of ERROR, WARNING, INFO, DEBUG, and
877 the extremely verbose INTERNAL to either a local file or to a syslogger
878 service. The destination of the log files, and the level of verbosity to be
879 logged, is set in the `opensrf_core.xml` configuration file. To add logging to
880 our Perl example, we just have to add the `OpenSRF::Utils::Logger` package to our
881 list of used Perl modules, then invoke the logger at the desired logging level.
883 You can include many calls to the OpenSRF logger; only those that are higher
884 than your configured logging level will actually hit the log. The following
885 example exercises all of the available logging levels in OpenSRF:
888 --------------------------------------------------------------------------------
889 use OpenSRF::Utils::Logger;
890 my $logger = OpenSRF::Utils::Logger;
891 # some code in some function
893 $logger->error("Hmm, something bad DEFINITELY happened!");
894 $logger->warn("Hmm, something bad might have happened.");
895 $logger->info("Something happened.");
896 $logger->debug("Something happened; here are some more details.");
897 $logger->internal("Something happened; here are all the gory details.")
899 --------------------------------------------------------------------------------
901 If you call the mythical OpenSRF method containing the preceding OpenSRF logger
902 statements on a system running at the default logging level of INFO, you will
903 only see the INFO, WARN, and ERR messages, as follows:
905 .Results of logging calls at the default level of INFO
906 --------------------------------------------------------------------------------
907 [2010-03-17 22:27:30] opensrf.simple-text [ERR :5681:SimpleText.pm:277:] Hmm, something bad DEFINITELY happened!
908 [2010-03-17 22:27:30] opensrf.simple-text [WARN:5681:SimpleText.pm:278:] Hmm, something bad might have happened.
909 [2010-03-17 22:27:30] opensrf.simple-text [INFO:5681:SimpleText.pm:279:] Something happened.
910 --------------------------------------------------------------------------------
912 If you then increase the the logging level to INTERNAL (5), the logs will
913 contain much more information, as follows:
915 .Results of logging calls at the default level of INTERNAL
916 --------------------------------------------------------------------------------
917 [2010-03-17 22:48:11] opensrf.simple-text [ERR :5934:SimpleText.pm:277:] Hmm, something bad DEFINITELY happened!
918 [2010-03-17 22:48:11] opensrf.simple-text [WARN:5934:SimpleText.pm:278:] Hmm, something bad might have happened.
919 [2010-03-17 22:48:11] opensrf.simple-text [INFO:5934:SimpleText.pm:279:] Something happened.
920 [2010-03-17 22:48:11] opensrf.simple-text [DEBG:5934:SimpleText.pm:280:] Something happened; here are some more details.
921 [2010-03-17 22:48:11] opensrf.simple-text [INTL:5934:SimpleText.pm:281:] Something happened; here are all the gory details.
922 [2010-03-17 22:48:11] opensrf.simple-text [ERR :5934:SimpleText.pm:283:] Resolver did not find a cache hit
923 [2010-03-17 22:48:21] opensrf.simple-text [INTL:5934:Cache.pm:125:] Stored opensrf.simple-text.test_cache.masaa => "here" in memcached server
924 [2010-03-17 22:48:21] opensrf.simple-text [DEBG:5934:Application.pm:579:] Coderef for [OpenSRF::Application::Demo::SimpleText::test_cache] has been run
925 [2010-03-17 22:48:21] opensrf.simple-text [DEBG:5934:Application.pm:586:] A top level Request object is responding de nada
926 [2010-03-17 22:48:21] opensrf.simple-text [DEBG:5934:Application.pm:190:] Method duration for [opensrf.simple-text.test_cache]: 10.005
927 [2010-03-17 22:48:21] opensrf.simple-text [INTL:5934:AppSession.pm:780:] Calling queue_wait(0)
928 [2010-03-17 22:48:21] opensrf.simple-text [INTL:5934:AppSession.pm:769:] Resending...0
929 [2010-03-17 22:48:21] opensrf.simple-text [INTL:5934:AppSession.pm:450:] In send
930 [2010-03-17 22:48:21] opensrf.simple-text [DEBG:5934:AppSession.pm:506:] AppSession sending RESULT to opensrf@private.localhost/_dan-karmic-liblap_1268880489.752154_5943 with threadTrace [1]
931 [2010-03-17 22:48:21] opensrf.simple-text [DEBG:5934:AppSession.pm:506:] AppSession sending STATUS to opensrf@private.localhost/_dan-karmic-liblap_1268880489.752154_5943 with threadTrace [1]
933 --------------------------------------------------------------------------------
935 To see everything that is happening in OpenSRF, try leaving your logging level
936 set to INTERNAL for a few minutes - just ensure that you have a lot of free disk
937 space available if you have a moderately busy system!
939 === Caching results: one secret of scalability ===
940 If you have ever used an application that depends on a remote Web service
941 outside of your control-say, if you need to retrieve results from a
942 microblogging service-you know the pain of latency and dependability (or the
943 lack thereof). To improve response time in OpenSRF applications, you can take
944 advantage of the support offered by the `OpenSRF::Utils::Cache` module for
945 communicating with a local instance or cluster of memcache daemons to store
946 and retrieve persistent values.
949 --------------------------------------------------------------------------------
950 use OpenSRF::Utils::Cache; # <1>
954 my $test_key = shift;
955 my $cache = OpenSRF::Utils::Cache->new('global'); # <2>
956 my $cache_key = "opensrf.simple-text.test_cache.$test_key"; # <3>
957 my $result = $cache->get_cache($cache_key) || undef; # <4>
959 $logger->info("Resolver found a cache hit");
963 my $cache_timeout = 300; # <6>
964 $cache->put_cache($cache_key, "here", $cache_timeout); # <7>
965 return "There was no cache hit.";
967 --------------------------------------------------------------------------------
971 <1> Imports the OpenSRF::Utils::Cache module
973 <2> Creates a cache object
975 <3> Creates a unique cache key based on the OpenSRF method name and
978 <4> Checks to see if the cache key already exists; if so, it immediately
981 <5> If the cache key does not exist, the code sleeps for 10 seconds to
982 simulate a call to a slow remote Web service, or an intensive process
984 <6> Sets a value for the lifetime of the cache key in seconds
986 <7> When the code has retrieved its value, then it can create the cache
987 entry, with the cache key, value to be stored ("here"), and the timeout
988 value in seconds to ensure that we do not return stale data on subsequent
991 === Initializing the service and its children: child labour ===
992 When an OpenSRF service is started, it looks for a procedure called
993 `initialize()` to set up any global variables shared by all of the children of
994 the service. The `initialize()` procedure is typically used to retrieve
995 configuration settings from the `opensrf.xml` file.
997 An OpenSRF service spawns one or more children to actually do the work
998 requested by callers of the service. For every child process an OpenSRF service
999 spawns, the child process clones the parent environment and then each child
1000 process runs the `child_init()` process (if any) defined in the OpenSRF service
1001 to initialize any child-specific settings.
1003 When the OpenSRF service kills a child process, it invokes the `child_exit()`
1004 procedure (if any) to clean up any resources associated with the child process.
1005 Similarly, when the OpenSRF service is stopped, it calls the `DESTROY()`
1006 procedure to clean up any remaining resources.
1008 === Retrieving configuration settings ===
1009 The settings for OpenSRF services are maintained in the `opensrf.xml` XML
1010 configuration file. The structure of the XML document consists of a root
1011 element `<opensrf>` containing two child elements:
1013 * `<default>` contains an `<apps>` element describing all
1014 OpenSRF services running on this system -- see <<serviceRegistration>> --, as
1015 well as any other arbitrary XML descriptions required for global configuration
1016 purposes. For example, Evergreen uses this section for email notification and
1017 inter-library patron privacy settings.
1018 * `<hosts>` contains one element per host that participates in
1019 this OpenSRF system. Each host element must include an `<activeapps>` element
1020 that lists all of the services to start on this host when the system starts
1021 up. Each host element can optionally override any of the default settings.
1023 OpenSRF includes a service named `opensrf.settings` to provide distributed
1024 cached access to the configuration settings with a simple API:
1026 * `opensrf.settings.default_config.get`: accepts zero arguments and returns
1027 the complete set of default settings as a JSON document
1028 * `opensrf.settings.host_config.get`: accepts one argument (hostname) and
1029 returns the complete set of settings, as customized for that hostname, as a
1031 * `opensrf.settings.xpath.get`: accepts one argument (an
1032 http://www.w3.org/TR/xpath/[XPath] expression) and returns the portion of
1033 the configuration file that matches the expression as a JSON document
1035 For example, to determine whether an Evergreen system uses the opt-in
1036 support for sharing patron information between libraries, you could either
1037 invoke the `opensrf.settings.default_config.get` method and parse the
1038 JSON document to determine the value, or invoke the `opensrf.settings.xpath.get`
1039 method with the XPath `/opensrf/default/share/user/opt_in` argument to
1040 retrieve the value directly.
1042 In practice, OpenSRF includes convenience libraries in all of its client
1043 language bindings to simplify access to configuration values. C offers
1044 osrfConfig.c, Perl offers `OpenSRF::Utils::SettingsClient`, Java offers
1045 `org.opensrf.util.SettingsClient`, and Python offers `osrf.set`. These
1046 libraries locally cache the configuration file to avoid network roundtrips for
1047 every request and enable the developer to request specific values without
1048 having to manually construct XPath expressions.
1050 == Getting under the covers with OpenSRF ==
1051 Now that you have seen that it truly is easy to create an OpenSRF service, we
1052 can take a look at what is going on under the covers to make all of this work
1055 === Get on the messaging bus - safely ===
1056 One of the core innovations of OpenSRF was to use the Extensible Messaging and
1057 Presence Protocol (XMPP, more colloquially known as Jabber) as the messaging
1058 bus that ties OpenSRF services together across servers. XMPP is an "XML
1059 protocol for near-real-time messaging, presence, and request-response services"
1060 (http://www.ietf.org/rfc/rfc3920.txt) that OpenSRF relies on to handle most of
1061 the complexity of networked communications. OpenSRF achieves a measure of
1062 security for its services through the use of public and private XMPP domains;
1063 all OpenSRF services automatically register themselves with the private XMPP
1064 domain, but only those services that register themselves with the public XMPP
1065 domain can be invoked from public OpenSRF clients.
1067 In a minimal OpenSRF deployment, two XMPP users named "router" connect to the
1068 XMPP server, with one connected to the private XMPP domain and one connected to
1069 the public XMPP domain. Similarly, two XMPP users named "opensrf" connect to
1070 the XMPP server via the private and public XMPP domains. When an OpenSRF
1071 service is started, it uses the "opensrf" XMPP user to advertise its
1072 availability with the corresponding router on that XMPP domain; the XMPP server
1073 automatically assigns a Jabber ID (JID) based on the client hostname to each
1074 service's listener process and each connected drone process waiting to carry
1075 out requests. When an OpenSRF router receives a request to invoke a method on a
1076 given service, it connects the requester to the next available listener in the
1077 list of registered listeners for that service.
1079 The opensrf and router user names, passwords, and domain names, along with the
1080 list of services that should be public, are contained in the `opensrf_core.xml`
1083 === Message body format ===
1084 OpenSRF was an early adopter of JavaScript Object Notation (JSON). While XMPP
1085 is an XML protocol, the Evergreen developers recognized that the compactness of
1086 the JSON format offered a significant reduction in bandwidth for the volume of
1087 messages that would be generated in an application of that size. In addition,
1088 the ability of languages such as JavaScript, Perl, and Python to generate
1089 native objects with minimal parsing offered an attractive advantage over
1090 invoking an XML parser for every message. Instead, the body of the XMPP message
1091 is a simple JSON structure. For a simple request, like the following example
1092 that simply reverses a string, it looks like a significant overhead: but we get
1093 the advantages of locale support and tracing the request from the requester
1094 through the listener and responder (drone).
1096 .A request for opensrf.simple-text.reverse("foobar"):
1098 --------------------------------------------------------------------------------
1099 <message from='router@private.localhost/opensrf.simple-text'
1100 to='opensrf@private.localhost/opensrf.simple-text_listener_at_localhost_6275'
1101 router_from='opensrf@private.localhost/_karmic_126678.3719_6288'
1102 router_to='' router_class='' router_command='' osrf_xid=''
1104 <thread>1266781414.366573.12667814146288</thread>
1107 {"__c":"osrfMessage","__p":
1108 {"threadTrace":"1","locale":"en-US","type":"REQUEST","payload":
1109 {"__c":"osrfMethod","__p":
1110 {"method":"opensrf.simple-text.reverse","params":["foobar"]}
1117 --------------------------------------------------------------------------------
1119 .A response from opensrf.simple-text.reverse("foobar")
1121 --------------------------------------------------------------------------------
1122 <message from='opensrf@private.localhost/opensrf.simple-text_drone_at_localhost_6285'
1123 to='opensrf@private.localhost/_karmic_126678.3719_6288'
1124 router_command='' router_class='' osrf_xid=''
1126 <thread>1266781414.366573.12667814146288</thread>
1129 {"__c":"osrfMessage","__p":
1130 {"threadTrace":"1","payload":
1131 {"__c":"osrfResult","__p":
1132 {"status":"OK","content":"raboof","statusCode":200}
1133 } ,"type":"RESULT","locale":"en-US"}
1135 {"__c":"osrfMessage","__p":
1136 {"threadTrace":"1","payload":
1137 {"__c":"osrfConnectStatus","__p":
1138 {"status":"Request Complete","statusCode":205}
1139 },"type":"STATUS","locale":"en-US"}
1144 --------------------------------------------------------------------------------
1146 The content of the `<body>` element of the OpenSRF request and result should
1147 look familiar; they match the structure of the <<OpenSRFOverHTTP,OpenSRF over
1148 HTTP examples>> that we previously dissected.
1150 === Registering OpenSRF methods in depth ===
1151 Let's explore the call to `__PACKAGE__->register_method()`; most of the elements
1152 of the hash are optional, and for the sake of brevity we omitted them in the
1153 previous example. As we have seen in the results of the <<opensrfIntrospection,introspection call>>, a
1154 verbose regisration method call is recommended to better enable the internal
1155 documentation. So, for the sake of completeness here, is the set of elements
1156 that you should pass to `__PACKAGE__->register_method()`:
1158 * `method`: the name of the procedure in this module that is being registered as an OpenSRF method
1159 * `api_name`: the invocable name of the OpenSRF method; by convention, the OpenSRF service name is used as the prefix
1160 * `api_level`: (optional) can be used for versioning the methods to allow the use of a deprecated API, but in practical use is always 1
1161 * `argc`: (optional) the minimal number of arguments that the method expects
1162 * `stream`: (optional) if this argument is set to any value, then the method supports returning multiple values from a single call to subsequent requests, and OpenSRF automatically creates a corresponding method with ".atomic" appended to its name that returns the complete set of results in a single request; streaming methods are useful if you are returning hundreds of records and want to act on the results as they return
1163 * `signature`: (optional) a hash describing the method's purpose, arguments, and return value
1164 ** `desc`: a description of the method's purpose
1165 ** `params`: an array of hashes, each of which describes one of the method arguments
1166 *** `name`: the name of the argument
1167 *** `desc`: a description of the argument's purpose
1168 *** `type`: the data type of the argument: for example, string, integer, boolean, number, array, or hash
1169 ** `return`: a hash describing the return value of the method
1170 *** `desc`: a description of the return value
1171 *** `type`: the data type of the return value: for example, string, integer, boolean, number, array, or hash
1173 == Evergreen-specific OpenSRF services ==
1175 Evergreen is currently the primary showcase for the use of OpenSRF as an
1176 application architecture. Evergreen 1.6.0 includes the following
1177 set of OpenSRF services:
1179 * `open-ils.actor`: Supports common tasks for working with user accounts
1181 * `open-ils.auth`: Supports authentication of Evergreen users.
1182 * `open-ils.cat`: Supports common cataloging tasks, such as creating,
1183 modifying, and merging bibliographic and authority records.
1184 * `open-ils.circ`: Supports circulation tasks such as checking out items and
1185 calculating due dates.
1186 * `open-ils.collections`: Supports tasks to assist collections services for
1187 contacting users with outstanding fines above a certain threshold.
1188 * `open-ils.cstore`: Supports unrestricted access to Evergreen fieldmapper
1189 objects. This is a private service.
1190 * `open-ils.ingest`: Supports tasks for importing bibliographic and authority
1192 * `open-ils.pcrud`: Supports permission-based access to Evergreen fieldmapper
1194 * `open-ils.penalty`: Supports the calculation of penalties for users, such as
1195 being blocked from further borrowing, for conditions such as having too many
1196 items checked out or too many unpaid fines.
1197 * `open-ils.reporter`: Supports the creation and scheduling of reports.
1198 * `open-ils.reporter-store`: Supports access to Evergreen fieldmapper objects
1199 for the reporting service. This is a private service.
1200 * `open-ils.search`: Supports searching across bibliographic records,
1201 authority records, serial records, Z39.50 sources, and ZIP codes.
1202 * `open-ils.storage`: A deprecated method of providing access to Evergreen
1203 fieldmapper objects. Implemented in Perl, this service has largely been
1204 replaced by the much faster C-based `open-ils.cstore` service.
1205 * `open-ils.supercat`: Supports transforms of MARC records into other formats,
1206 such as MODS, as well as providing Atom and RSS feeds and SRU access.
1207 * `open-ils.trigger`: Supports event-based triggers for actions such as overdue
1208 and holds available notification emails.
1209 * `open-ils.vandelay`: Supports the import and export of batches of
1210 bibliographic and authority records.
1212 Of some interest is that the `open-ils.reporter-store` and `open-ils.cstore`
1213 services have identical implementations. Surfacing them as separate services
1214 enables a deployer of Evergreen to ensure that the reporting service does not
1215 interfere with the performance-critical `open-ils.cstore` service. One can also
1216 direct the reporting service to a read-only database replica to, again, avoid
1217 interference with `open-ils.cstore` which must write to the master database.
1219 There are only a few significant services that are not built on OpenSRF in
1220 Evergreen 1.6.0, such as the SIP and Z39.50 servers. These services implement
1221 different protocols and build on existing daemon architectures (Simple2ZOOM
1222 for Z39.50), but still rely on the other OpenSRF services to provide access
1223 to the Evergreen data. The non-OpenSRF services are reasonably self-contained
1224 and can be deployed on different servers to deliver the same sort of deployment
1225 flexibility as OpenSRF services, but have the disadvantage of not being
1226 integrated into the same configuration and control infrastructure as the
1229 == Evergreen after one year: reflections on OpenSRF ==
1231 http://projectconifer.ca[Project Conifer] has been live on Evergreen for just
1232 over a year now, and as one of the primary technologists I have had to work
1233 closely with the OpenSRF infrastructure during that time. As such, I am in
1234 a position to identify some of the strengths and weaknesses of OpenSRF based
1237 === Strengths of OpenSRF ===
1239 As a service infrastructure, OpenSRF has been remarkably reliable. We initially
1240 deployed Evergreen on an unreleased version of both OpenSRF and Evergreen due
1241 to our requirements for some functionality that had not been delivered in a
1242 stable release at that point in time, and despite this risky move we suffered
1243 very little unplanned downtime in the opening months. On July 27, 2009 we
1244 moved to a newer (but still unreleased) version of the OpenSRF and Evergreen
1245 code, and began formally tracking our downtime. Since then, we have achieved
1246 more than 99.9% availability - including scheduled downtime for maintenance.
1247 This compares quite favourably to the maximum of 75% availability that we were
1248 capable of achieving on our previous library system due to the nightly downtime
1249 that was required for our backup process. The OpenSRF "maximum request"
1250 configuration parameter for each service that kills off drone processes after
1251 they have served a given number of requests provides a nice failsafe for
1252 processes that might otherwise suffer from a memory leak or hung process. It
1253 also helps that when we need to apply an update to a Perl service that is
1254 running on multiple servers, we can apply the updated code, then restart the
1255 service on one server at a time to avoid any downtime.
1257 As promised by the OpenSRF infrastructure, we have also been able to tune our
1258 cluster of servers to provide better performance. For example, we were able to
1259 change the number of maximum concurrent processes for our database services
1260 when we noticed that we seeing a performance bottleneck with database access.
1261 Making a configuration change go live simply requires you to restart the
1262 `opensrf.setting` service to pick up the configuration change, then restart the
1263 affected service on each of your servers. We were also able to turn off some of
1264 the less-used OpenSRF services, such as `open-ils.collections`, on one of our
1265 servers to devote more resources on that server to the more frequently used
1266 services and other performance-critical processes such as Apache.
1268 The support for logging and caching that is built into OpenSRF has been
1269 particularly helpful with the development of a custom service for SFX holdings
1270 integration into our catalogue. Once I understood how OpenSRF works, most of
1271 the effort required to build that SFX integration service was spent on figuring
1272 out how to properly invoke the SFX API to display human-readable holdings.
1273 Adding a new OpenSRF service and registering several new methods for the
1274 service was relatively easy. The support for directing log messages to syslog
1275 in OpenSRF has also been a boon for both development and debugging when
1276 problems arise in a cluster of five servers; we direct all of our log messages
1277 to a single server where we can inspect the complete set of messages for the
1278 entire cluster in context, rather than trying to piece them together across
1283 The primary weakness of OpenSRF is the lack of either formal or informal
1284 documentation for OpenSRF. There are many frequently asked questions on the
1285 Evergreen mailing lists and IRC channel that indicate that some of the people
1286 running Evergreen or trying to run Evergreen have not been able to find
1287 documentation to help them understand, even at a high level, how the OpenSRF
1288 Router and services work with XMPP and the Apache Web server to provide a
1289 working Evergreen system. Also, over the past few years several developers
1290 have indicated an interest in developing Ruby and PHP bindings for OpenSRF, but
1291 the efforts so far have resulted in no working code. Without a formal
1292 specification, clearly annotated examples, and unit tests for the major OpenSRF
1293 communication use cases that could be ported to the new language as a base set
1294 of expectations for a working binding, the hurdles for a developer new to
1295 OpenSRF are significant. As a result, Evergreen integration efforts with
1296 popular frameworks like Drupal, Blacklight, and VuFind result in the best
1297 practical option for a developer with limited time -- database-level
1298 integration -- which has the unfortunate side effect of being much more likely
1299 to break after an upgrade.
1301 In conjunction with the lack of documentation that makes it hard to get started
1302 with the framework, a disincentive for new developers to contribute to OpenSRF
1303 itself is the lack of integrated unit tests. For a developer to contribute a
1304 significant, non-obvious patch to OpenSRF, they need to manually run through
1305 various (undocumented, again) use cases to try and ensure that the patch
1306 introduced no unanticipated side effects. The same problems hold for Evergreen
1307 itself, although the
1308 http://svn.open-ils.org/ILS-Contrib/constrictor[Constrictor] stress-testing
1309 framework offers a way of performing some automated system testing and
1310 performance testing.
1312 These weaknesses could be relatively easily overcome with the effort through
1313 contributions from people with the right skill sets. This article arguably
1314 offers a small set of clear examples at both the networking and application
1315 layer of OpenSRF. A technical writer who understands OpenSRF could contribute a
1316 formal specification to the project. With a formal specification at their
1317 disposal, a quality assurance expert could create an automated test harness and
1318 a basic set of unit tests that could be incrementally extended to provide more
1319 coverage over time. If one or more continual integration environments are set
1320 up to track the various OpenSRF branches of interest, then the OpenSRF
1321 community would have immediate feedback on build quality. Once a unit testing
1322 framework is in place, more developers might be willing to develop and
1323 contribute patches as they could sanity check their own code without an intense
1324 effort before exposing it to their peers.
1327 In this article, I attempted to provide both a high-level and detailed overview
1328 of how OpenSRF works, how to build and deploy new OpenSRF services, how to make
1329 requests to OpenSRF method from OpenSRF clients or over HTTP, and why you
1330 should consider it a possible infrastructure for building your next
1331 high-performance system that requires the capability to scale out. In addition,
1332 I surveyed the Evergreen services built on OpenSRF and reflected on the
1333 strengths and weaknesses of the platform based on the experiences of Project
1334 Conifer after a year in production, with some thoughts about areas where the
1335 right application of skills could make a significant difference to the Evergreen
1336 and OpenSRF projects.
1338 == Appendix: Python client ==
1340 Following is a Python client that makes the same OpenSRF calls as the Perl
1344 --------------------------------------------------------------------------------
1345 include::python_client.py[]
1346 --------------------------------------------------------------------------------
1348 NOTE: Python's `dnspython` module refuses to read `/etc/resolv.conf`, so to
1349 access hostnames that are not served up via DNS, such as the extremely common
1350 case of `localhost`, you may need to install a package like `dnsmasq` to act
1351 as a local DNS server for those hostnames.
1353 // vim: set syntax=asciidoc: