4 == About the SIP Protocol ==
6 indexterm:[Automated Circulation System]
8 indexterm:[Automated Material Handling]
10 +SIP+, standing for +Standard Interchange Protocol+, was developed by the +3M corporation+ to be a common
11 protocol for data transfer between ILS' (referred to in +SIP+ as an _ACS_, or _Automated Circulation System_) and a
12 third party device. Originally, the protocol was developed for use with _3M SelfCheck_ (often abbreviated SC, not to
13 be confused with Staff Client) systems, but has since expanded to other companies and devices. It is now common
14 to find +SIP+ in use in several other vendors' SelfCheck systems, as well as other non-SelfCheck devices. Some
17 * Patron Authentication (computer access, subscription databases)
18 * Automated Material Handling (AMH)
19 ** The automated sorting of items, often to bins or book carts, based on shelving location or other programmable
22 == Installing the SIP Server ==
26 This is a rough intro to installing the +SIP+ server for Evergreen.
28 === Getting the code ===
30 Current +SIP+ server code lives at in the Evergreen git repository:
33 git clone git://git.evergreen-ils.org/SIPServer.git SIPServer
36 === Configuring the Server ===
38 indexterm:[configuration files, oils_sip.xml]
40 . Type the following commands from the command prompt:
44 $ cp oils_sip.xml.example oils_sip.xml
46 . Edit oils_sip.xml. Change the commented out <server-params> section to this:
55 . max_servers will directly correspond to the number of allowed +SIP+ clients. Set the number accordingly, but
56 bear in mind that too many connections can exhaust memory. On a 4G RAM/4 CPU server (that is also running
57 evergreen), it is not recommended to exceed 100 +SIP+ client connections.
59 ==== Setting the encoding ====
61 SIPServer looks for the encoding in the following
64 1. An +encoding+ attribute on the +account+ element for the currently active SIP account.
65 2. The +encoding+ element that is a child of the +institution+ element of the currently active SIP account.
66 3. The +encoding+ element that is a child of the +implementation_config+ element that is itself a child of the +institution+ element of the currently active SIP account.
67 4. If none of the above exist, then the default encoding (ASCII) is used.
69 Option 3 is a legacy option. It is recommended that you alter your configuration to
70 move this element out of the +implementation_config+ element and into
71 its parent +institution+ element. Ideally, SIPServer should *not* look into
72 the implementation config, and this check may be removed at some time
77 The `msg64_hold_datatype` setting is similar to `msg64_summary_datatype`, but affects holds instead of circulations.
78 When set to `barcode`, holds information will be delivered as a set of copy barcodes instead of title strings for
79 patron info requests. With barcodes, SIP clients can both find the title strings for display (via item info requests)
80 and make subsequent hold-related action requests, like holds cancellation.
83 === Adding SIP Users ===
85 indexterm:[configuration files, oils_sip.xml]
87 . Type the following commands from the command prompt:
92 . In the +<accounts>+ section, add +SIP+ client login information. Make sure that all +<logins>+ use the same
93 institution attribute, and make sure the institution is listed in +<institutions>+. All attributes in the
94 +<login>+ section will be used by the +SIP+ client.
96 . In Evergreen, create a new profile group called +SIP+. This group should be a sub-group of +Users+ (not +Staff+
97 or +Patrons+). Set _Editing Permission_ as *group_application.user.sip_client* and give the group the following
105 VIEW_COPY_CHECKOUT_HISTORY
108 VIEW_USER_FINES_SUMMARY
109 VIEW_USER_TRANSACTIONS
114 INSERT INTO permission.grp_tree (name,parent,description,application_perm)
115 VALUES ('SIP', 1, 'SIP2 Client Systems', 'group_application.user.sip_client');
118 permission.grp_perm_map (grp, perm, depth, grantable)
122 permission.grp_tree g,
123 permission.perm_list p
131 'VIEW_COPY_CHECKOUT_HISTORY',
132 'VIEW_PERMIT_CHECKOUT',
134 'VIEW_USER_FINES_SUMMARY',
135 'VIEW_USER_TRANSACTIONS'
142 FROM permission.grp_perm_map pgpm
143 INNER JOIN permission.perm_list ppl ON pgpm.perm = ppl.id
144 INNER JOIN permission.grp_tree pgt ON pgt.id = pgpm.grp
145 WHERE pgt.name = 'SIP';
149 . For each account created in the +<login>+ section of oils_sip.xml, create a user (via the staff client user
150 editor) that has the same username and password and put that user into the +SIP+ group.
154 The expiration date will affect the +SIP+ users' connection so you might want to make a note of this
158 === Running the server ===
160 To start the +SIP+ server type the following commands from the command prompt:
165 $ oils_ctl.sh -a [start|stop|restart]_sip
177 It is useful to log +SIP+ requests to a separate file especially during initial setup by modifying your syslog config file.
181 $ sudo vi /etc/syslog.conf # maybe /etc/rsyslog.conf
186 local6.* -/var/log/SIP_evergreen.log
188 . Syslog expects the logfile to exist so create the file.
190 $ sudo touch /var/log/SIP_evergreen.log
194 $ sudo /etc/init.d/sysklogd restart
199 indexterm:[syslog-NG]
201 . Edit logging config.
203 sudo vi /etc/syslog-ng/syslog-ng.conf
207 # +SIP2+ for Evergreen
208 filter f_eg_sip { level(warn, err, crit) and facility(local6); };
209 destination eg_sip { file("var/log/SIP_evergreen.log"); };
210 log { source(s_all); filter(f_eg_sip); destination(eg_sip); };
212 . Syslog-ng expects the logfile to exist so create the file.
214 $ sudo touch /var/log/SIP_evergreen.log
218 $ sudo /etc/init.d/syslog-ng restart
224 === Testing Your SIP Connection ===
226 * In the root directory of the SIPServer code:
230 * Edit SIPtest.pm, change the $instid, $server, $username, and $password variables. This will be
231 enough to test connectivity. To run all tests, you'll need to change all the variables in the _Configuration_ section.
233 $ PERL5LIB=../ perl 00sc_status.t
235 This should produce something like:
239 ok 1 - Invalid username
240 ok 2 - Invalid username
244 * Don't be dismayed at *Invalid Username*. That's just one of the many tests that are run.
248 Once you have opened up either the +SIP+ OR +SIP2+ ports to be accessible from outside you can do some testing
249 via +telnet+. In the following tests:
251 * Replace +$server+ with your server hostname (or +localhost+ if you want to
252 skip testing external access for now);
253 * Replace +$username+, +$password+, and +$instid+ with the corresponding values
254 in the +<accounts>+ section of your SIP configuration file;
255 * Replace the +$user_barcode+ and +$user_password+ variables with the values
257 * Replace the +$item_barcode+ variable with the values for a valid item.
260 Comments because we don't want to indent these numbered bullets!
263 . Start by testing your ability to log into the SIP server:
266 ======================
267 We are using 6001 here which is associated with +SIP2+ as per our configuration.
268 ======================
270 $ telnet $server 6001
271 Connected to $server.
272 Escape character is '^]'.
273 9300CN$username|CO$password|CP$instid
275 If successful, the SIP server returns a +941+ result. A result of +940+,
276 however, indicates an unsuccessful login attempt. Check the +<accounts>+
277 section of your SIP configuration and try again.
279 . Once you have logged in successfully, replace the variables in the following
280 line and paste it into the telnet session:
282 2300120080623 172148AO$instid|AA$user_barcode|AC$password|AD$user_password
284 If successful, the SIP server returns the patron information for $user_barcode,
285 similar to the following:
287 24 Y 00120100113 170738AEFirstName MiddleName LastName|AA$user_barcode|BLY|CQY
288 |BHUSD|BV0.00|AFOK|AO$instid|
290 The response declares it is a valid patron BLY with a valid password CQY and shows the user's +$name+.
292 . To test the SIP server's item information response, issue the following request:
294 1700120080623 172148AO$instid|AB$item_barcode|AC$password
296 If successful, the SIP server returns the item information for $item_barcode,
297 similar to the following:
299 1803020120160923 190132AB30007003601852|AJRégion de Kamouraska|CK001|AQOSUL|APOSUL|BHCAD
300 |BV0.00|BGOSUL|CSCA2 PQ NR46 73R
302 The response declares it is a valid item, with the title, owning library,
303 permanent and current locations, and call number.
307 == SIP Communication ==
309 indexterm:[SIP Server, SIP Communication]
311 +SIP+ generally communicates over a +TCP+ connection (either raw sockets or over +telnet+), but can also
312 communicate via serial connections and other methods. In Evergreen, the most common deployment is a +RAW+ socket
313 connection on port 6001.
315 +SIP+ communication consists of strings of messages, each message request and response begin with a 2-digit
316 ``command'' - Requests usually being an odd number and responses usually increased by 1 to be an even number. The
317 combination numbers for the request command and response is often referred to as a _Message Pair_ (for example,
318 a 23 command is a request for patron status, a 24 response is a patron status, and the message pair 23/24 is patron
319 status message pair). The table in the next section shows the message pairs and a description of them.
321 For clarification, the ``Request'' is from the device (selfcheck or otherwise) to the ILS/ACS. The response is… the
322 response to the request ;).
324 Within each request and response, a number of fields (either a fixed width or separated with a | [pipe symbol] and
325 preceded with a 2-character field identifier) are used. The fields vary between message pairs.
327 |===========================================================================
328 | *Pair* | *Name* | *Supported?* |*Details*
329 | 01 | Block Patron | Yes |<<sip_01_block_patron, 01/Block_Patron>> - ACS responds with 24 Patron Status Response
330 | 09-10 | Checkin | Yes (with extensions) |<<sip_09-10_checkin, 09/10_Checkin>>
331 | 11-12 | Checkout | Yes (no renewals) |<<sip_11-12_checkout, 11/12_Checkout>>
332 | 15-16 | Hold | Partially supported |<<sip_15-16_hold, 15/16_Hold>>
333 | 17-18 | Item Information | Yes (no extensions) |<<sip_17-18_item_information, 17/18_Item_Information>>
334 | 19-20 | Item Status Update | No |<<sip_19-20_item_status_update, 19/20_Item_Status_Update>> - Returns Patron Enable response, but doesn't make any changes in EG
335 | 23-24 | Patron Status | Yes |<<sip_23-24_patron_status, 23/24_Patron_Status>> - 63/64 ``Patron Information'' preferred
336 | 25-26 | Patron Enable | No |<<sip_25-26_patron_enable, 25/26_Patron_Enable>> - Used during system testing and validation
337 | 29-30 | Renew | Yes |<<sip_29-30_renew, 29/30_Renew>>
338 | 35-36 | End Session | Yes |<<sip_35-36_end_session, 35/36_End_Session>>
339 | 37-38 | Fee Paid | Yes |<<sip_37-38_fee_paid, 37/38_Fee_Paid>>
340 | 63-64 | Patron Information | Yes (no extensions) |<<sip_63-64_patron_information, 63/64_Patron_Information>>
341 | 65-66 | Renew All | Yes |<<sip_65-66_renew_all, 65/66_Renew_All>>
342 | 93-94 | Login | Yes |<<sip_93-94_login, 93/94_Login>> - Must be first command to Evergreen ACS (via socket) or +SIP+ will terminate
343 | 97-96 | Resend last message | Yes |<<sip_97-96_resend, 97/96_Resend>>
344 | 99-98 | SC-ACS Status | Yes |<<sip_99-98_sc_and_acs_status, 99/98_SC_and_ACS_Status>>
345 |===========================================================================
347 [#sip_01_block_patron]
349 === 01 Block Patron ===
351 indexterm:[SelfCheck]
353 A selfcheck will issue a *Block Patron* command if a patron leaves their card in a selfcheck machine or if the
354 selfcheck detects tampering (such as attempts to disable multiple items during a single item checkout, multiple failed
357 In Evergreen, this command does the following:
359 * User alert message: _CARD BLOCKED BY SELF-CHECK MACHINE_ (this is independent of the AL _Blocked
360 Card Message_ field).
362 * Card is marked inactive.
364 The request looks like:
366 01<card retained><date>[fields AO, AL, AA, AC]
368 _Card Retained_: A single character field of Y or N - tells the ACS whether the SC has retained the card (ex: left in
371 _Date_: An 18 character field for the date/time when the block occurred.
373 _Format_: YYYYMMDDZZZZHHMMSS (ZZZZ being zone - 4 blanks when local time, ``Z'' (3 blanks and a Z)
374 represents UTC(GMT/Zulu)
376 _Fields_: See <<fields, Fields>> for more details.
378 The response is a 24 ``Patron Status Response'' with the following:
380 * Charge privileges denied
381 * Renewal privileges denied
382 * Recall privileges denied (hard-coded in every 24 or 64 response)
383 * hold privileges denied
384 * Screen Message 1 (AF): _blocked_
389 === 09/10 Checkin ===
391 ~The request looks like:
393 09<No block (Offline)><xact date><return date>[Fields AP,AO,AB,AC,CH,BI]
395 _No Block (Offline)_: A single character field of _Y_ or _N_ - Offline transactions are not currently supported so send _N_.
397 _xact date_: an 18 character field for the date/time when the checkin occurred. Format:
398 YYYYMMDDZZZZHHMMSS (ZZZZ being zone - 4 blanks when local time, ``Z'' (3 blanks and a Z) represents
401 _Fields_: See <<fields, Fields>> for more details.
403 The response is a 10 ``Checkin Response'' with the following:
405 10<resensitize><magnetic media><alert><xact date>[Fields AO,AB,AQ,AJ,CL,AA,CK,CH,CR,CS,CT,CV,CY,DA,AF,AG]
407 Example (with a remote hold):
409 09N20100507 16593720100507 165937APCheckin Bin 5|AOBR1|AB1565921879|ACsip_01|
411 101YNY20100623 165731AOBR1|AB1565921879|AQBR1|AJPerl 5 desktop reference|CK001|CSQA76.73.P33V76 1996
412 |CTBR3|CY373827|DANicholas Richard Woodard|CV02|
414 Here you can see a hold alert for patron CY _373827_, named DA _Nicholas Richard Woodard_, to be picked up at CT
415 ``BR3''. Since the transaction is happening at AO ``BR1'', the alert type CV is 02 for _hold at remote library_. The
416 possible values for CV are:
424 * 03: ILL transfer (not used by EG)
430 indexterm:[magnetic media]
434 The logic for Evergreen to determine whether the content is magnetic_media comes from
435 or search_config_circ_modifier. The default is non-magnetic. The same is true for media_type (default
436 001). Evergreen does not populate the collection_code because it does not really have any, but it will provide
437 the call_number where available.
439 Unlike the +item_id+ (barcode), the +title_id+ is actually a title string, unless the configuration forces the
440 return of the bib ID.
442 Don't be confused by the different branches that can show up in the same response line.
444 * AO is where the transaction took place,
446 * AQ is the ``permanent location'', and
448 * CT is the _destination location_ (i.e., pickup lib for a hold or target lib for a transfer).
451 [#sip_11-12_checkout]
453 === 11/12 Checkout ===
460 Evergreen supports the Hold message for the purpose of canceling
461 holds. It does not currently support creating hold requests via SIP2.
464 [#sip_17-18_item_information]
466 === 17/18 Item Information ===
468 The request looks like:
470 17<xact_date>[fields: AO,AB,AC]
472 The request is very terse. AC is optional.
474 The following response structure is for +SIP2+. (Version 1 of the protocol had only 6 total fields.)
476 18<circulation_status><security_marker><fee_type><xact_date>
477 [fields: CF,AH,CJ,CM,AB,AJ,BG,BH,BV,CK,AQ,AP,CH,AF,AG,+CT,+CS]
481 1720060110 215612AOBR1|ABno_such_barcode|
483 1801010120100609 162510ABno_such_barcode|AJ|
485 1720060110 215612AOBR1|AB1565921879|
487 1810020120100623 171415AB1565921879|AJPerl 5 desktop reference|CK001|AQBR1|APBR1|BGBR1
488 |CTBR3|CSQA76.73.P33V76 1996|
490 The first case is with a bogus barcode. The latter shows an item with a circulation_status of _10_ for _in transit between
491 libraries_. The known values of +circulation_status+ are enumerated in the spec.
493 indexterm:[Automated Material Handling (AMH)]
495 EXTENSIONS: The CT field for _destination location_ and CS _call number_ are used by Automated Material Handling
499 [#sip_19-20_item_status_update]
501 === 19/20 Item Status Update ===
504 [#sip_23-24_patron_status]
506 === 23/24 Patron Status ===
510 2300120060101 084235AOUWOLS|AAbad_barcode|ACsip_01|ADbad_password|
512 24YYYY 00120100507 013934AE|AAbad_barcode|BLN|AOUWOLS|
514 2300120060101 084235AOCONS|AA999999|ACsip_01|ADbad_password|
516 24 Y 00120100507 022318AEDoug Fiander|AA999999|BLY|CQN|BHUSD|BV0.00|AFOK|AOCONS|
518 2300120060101 084235AOCONS|AA999999|ACsip_01|ADuserpassword|LY|CQN|BHUSD|BV0.00|AFOK|AOCONS|
520 24 Y 00120100507 022803AEDoug Fiander|AA999999|BLY|CQY|BHUSD|BV0.00|AFOK|AOCONS|
522 . The BL field (+SIP2+, optional) is _valid patron_, so the _N_ value means _bad_barcode_ doesn't match a patron, the
523 _Y_ value means 999999 does.
525 . The CQ field (+SIP2+, optional) is _valid password_, so the _N_ value means _bad_password_ doesn't match 999999's
526 password, the _Y_ means _userpassword_ does.
528 So if you were building the most basic +SIP2+ authentication client, you would check for _|CQY|_ in the response to
529 know the user's barcode and password are correct (|CQY| implies |BLY|, since you cannot check the password
530 unless the barcode exists). However, in practice, depending on the application, there are other factors to consider in
531 authentication, like whether the user is blocked from checkout, owes excessive fines, reported their card lost, etc.
532 These limitations are reflected in the 14-character _patron status_ string immediately following the _24_ code. See the
533 field definitions in your copy of the spec.
536 [#sip_25-26_patron_enable]
538 === 25/26 Patron Enable ===
547 Evergreen supports the Renew message. Evergreen checks whether a penalty is specifically configured to block
548 renewals before blocking any SIP renewal.
551 [#sip_35-36_end_session]
553 === 35/36 End Session ===
555 3520100505 115901AOBR1|AA999999|
557 36Y20100507 161213AOCONS|AA999999|AFThank you!|
559 The _Y/N_ code immediately after the 36 indicates _success/failure_. Failure is not particularly meaningful or important
560 in this context, and for evergreen it is hardcoded _Y_.
564 [#sip_37-38_fee_paid]
566 === 37/38 Fee Paid ===
568 Evergreen supports the Fee Paid message.
571 [#sip_63-64_patron_information]
573 === 63/64 Patron Information ===
575 Attempting to retrieve patron info with a bad barcode:
577 6300020060329 201700 AOBR1|AAbad_barcode|
579 64YYYY 00020100623 141130000000000000000000000000AE|AAbad_barcode|BLN|AOBR1|
581 Attempting to retrieve patron info with a good barcode (but bad patron password):
583 6300020060329 201700 AOBR1|AA999999|ADbadpwd|
585 64 Y 00020100623 141130000000000000000000000000AA999999|AEDavid J. Fiander|BHUSD|BV0.00
586 |BD2 Meadowvale Dr. St Thomas, ON Canada
588 90210|BEdjfiander@somemail.com|BF(519) 555 1234|AQBR1|BLY|CQN|PB19640925|PCPatrons
589 |PIUnfiltered|AFOK|AOBR1|
591 See <<sip_23-24_patron_status, 23/24 Patron Status>> for info on +BL+ and +CQ+ fields.
595 [#sip_65-66_renew_all]
597 === 65/66 Renew All ===
599 Evergreen supports the Renew All message.
608 9300CNsip_01|CObad_value|CPBR1|
610 [Connection closed by foreign host.]
613 9300CNsip_01|COsip_01|CPBR1|
617 _941_ means successful terminal login. _940_ or getting dropped means failure.
619 When using a version of SIPServer that supports the feature, the Location (CP) field of the Login (93) message will be used as the workstation name if supplied. Blank or missing location fields will be ignored. This allows users or reports to determine which selfcheck performed a circulation.
627 [#sip_99-98_sc_and_acs_status]
629 === 99/98 SC and ACS Status ===
631 99<status code><max print width><protocol version>
633 All 3 fields are required:
637 * 1: SC is out of paper
639 * 2: SC shutting down
641 * status code - 1 character
643 * max print width - 3 characters - the integer number of characters the client can print
645 * protocol version - 4 characters - x.xx
647 98<on-line status><checkin ok><checkout ok><ACS renewal policy>
648 <status update ok><offline ok><timeout period>
650 <retries allowed><date/time sync><protocol version><institution id>
651 <library name><supported messages><terminal
653 location><screen message><print line>
659 98YYYYNN60000320100510 1717202.00AOCONS|BXYYYYYYYYYNYNNNYN|
661 The Supported Messages field +BX+ appears only in +SIP2+, and specifies whether 16 different +SIP+ commands are
662 supported by the +ACS+ or not.
669 All fixed-length fields in a communication will appear before the first variable-length field. This allows for simple
670 parsing. Variable-length fields are by definition delimited, though there will not necessarily be an initial delimiter
671 between the last fixed-length field and the first variable-length one. It would be unnecessary, since you should know
672 the exact position where that field begins already.
675 == Patron privacy and the SIP protocol ==
677 SIP traffic includes a lot of patron information, and is not
678 encrypted by default. It is strongly recommended that you
679 encrypt any SIP traffic.
681 === SIP server configuration ===
683 On the SIP server, use `iptables` or `etc/hosts` to allow SSH connections on port 22 from the SIP client machine. You will probably want to have very restrictive rules
684 on which IP addresses can connect to this server.
687 === SSH tunnels on SIP clients ===
689 SSH tunnels are a good fit for use cases like self-check machines, because it is relatively easy to automatically open the connection. Using a VPN is another option,
690 but many VPN clients require manual steps to open the VPN connection.
692 . If the SIP client will be on a Windows machine, install cygwin on the SIP client.
693 . On the SIP client, use `ssh-keygen` to generate an SSH key.
694 . Add the public key to /home/my_sip_user/.ssh/authorized_keys on your SIP server to enable logins without using the UNIX password.
695 . Configure an SSH tunnel to open before every connection. You can do this in several ways:
696 .. If the SIP client software allows you to run an arbitrary command before
697 each SIP connection, use something like this:
701 ssh -f -L 6001:localhost:6001 my_sip_user@my_sip_server.com sleep 10
704 .. If you feel confident that the connection won't get interrupted, you can have something like this run at startup:
708 ssh -f -N -L 6001:localhost:6001 my_sip_user@my_sip_server.com
711 .. If you want to constantly poll to make sure that the connection is still running, you can do something like this as a cron job or scheduled task on the SIP client machine:
715 instances=`/bin/ps -ef | /bin/grep ssh | /bin/grep -v grep | /bin/wc -l`
716 if [ $instances -eq 0 ]; then
717 echo "Restarting ssh tunnel"
718 /usr/bin/ssh -L 6001:localhost:6001 my_sip_user@my_sip_server.com -f -N