SIP manual testing formatting cleanup

[Evergreen.git] / docs / admin / sip_server.txt

SIP Server
----------

indexterm:[Automated Circulation System]
indexterm:[SelfCheck]
indexterm:[Automated Material Handling]

+SIP+, standing for +Standard Interchange Protocol+, was developed by the +3M corporation+ to be a common 
protocol for data transfer between ILS' (referred to in +SIP+ as an _ACS_, or _Automated Circulation System_)  and a 
third party device. Originally, the protocol was developed for use with _3M SelfCheck_ (often abbreviated SC, not to 
be confused with Staff Client) systems, but has since expanded to other companies and devices. It is now common 
to find +SIP+ in use in several other vendors' SelfCheck systems, as well as other non-SelfCheck devices. Some 
examples include:

* Patron Authentication (computer access, subscription databases) 
* Automated Material Handling (AMH) 
** The automated sorting of items, often to bins or book carts, based on shelving location or other programmable 
criteria

Installing the SIP Server
~~~~~~~~~~~~~~~~~~~~~~~~~



This is a rough intro to installing the +SIP+ server for Evergreen. 

Getting the code 
^^^^^^^^^^^^^^^^

Current +SIP+ server code lives at in the Evergreen git repository:

   cd /opt 
   git clone git://git.evergreen-ils.org/SIPServer.git SIPServer 


Configuring the Server 
^^^^^^^^^^^^^^^^^^^^^^

indexterm:[configuration files, oils_sip.xml]

. Type the following commands from the command prompt: 

   $ sudo su opensrf 
   $ cd /openils/conf 
   $ cp oils_sip.xml.example oils_sip.xml 

. Edit oils_sip.xml. Change the commented out <server-params> section to this: 

   <server-params
      min_spare_servers='1' 
      max_spare_servers='2' 
      min_servers='3'
      max_servers='25' 
                />

. max_servers will directly correspond to the number of allowed +SIP+ clients. Set the number accordingly, but 
bear in mind that too many connections can exhaust memory. On a 4G RAM/4 CPU server (that is also running 
evergreen), it is not recommended to exceed 100 +SIP+ client connections. 

Adding SIP Users
^^^^^^^^^^^^^^^^

indexterm:[configuration files, oils_sip.xml]

. Type the following commands from the command prompt: 

  $ sudo su opensrf 
  $ cd /openils/conf

. In the +<accounts>+ section, add +SIP+ client login information. Make sure that all +<logins>+ use the same 
institution attribute, and make sure the institution is listed in +<institutions>+. All attributes in the 
+<login>+ section will be used by the +SIP+ client. 

. In Evergreen, create a new profile group called +SIP+. This group should be a sub-group of +Users+ (not +Staff+ 
or +Patrons+). Set _Editing Permission_ as *group_application.user.sip_client* and give the group the following 
permissions: 
+
     COPY_CHECKIN 
     COPY_CHECKOUT 
     CREATE_PAYMENT
     RENEW_CIRC 
     VIEW_CIRCULATIONS 
     VIEW_COPY_CHECKOUT_HISTORY 
     VIEW_PERMIT_CHECKOUT 
     VIEW_USER 
     VIEW_USER_FINES_SUMMARY 
     VIEW_USER_TRANSACTIONS 
+
OR use SQL like: 
+
   
   INSERT INTO permission.grp_tree (name,parent,description,application_perm) 
   VALUES ('SIP', 1, 'SIP2 Client Systems', 'group_application.user.sip_client'); 
  
   INSERT INTO 
     permission.grp_perm_map (grp, perm, depth, grantable) 
   SELECT 
     g.id, p.id, 0, FALSE 
   FROM 
     permission.grp_tree g, 
     permission.perm_list p 
   WHERE 
     g.name = 'SIP' AND 
     p.code IN ( 
       'COPY_CHECKIN', 
       'COPY_CHECKOUT', 
       'RENEW_CIRC', 
       'VIEW_CIRCULATIONS', 
       'VIEW_COPY_CHECKOUT_HISTORY', 
       'VIEW_PERMIT_CHECKOUT', 
       'VIEW_USER', 
       'VIEW_USER_FINES_SUMMARY', 
       'VIEW_USER_TRANSACTIONS' 
    ); 
+
Verify:
+
    
    SELECT * 
    FROM permission.grp_perm_map pgpm 
        INNER JOIN permission.perm_list ppl ON pgpm.perm = ppl.id 
        INNER JOIN permission.grp_tree pgt ON pgt.id = pgpm.grp 
    WHERE pgt.name = 'SIP';
    
 
    
. For each account created in the +<login>+ section of oils_sip.xml, create a user (via the staff client user 
editor) that has the same username and password and put that user into the +SIP+ group. 

[NOTE]
===================
The expiration date will affect the +SIP+ users' connection so you might want to make a note of this 
somewhere. 
===================

Running the server 
^^^^^^^^^^^^^^^^^^

To start the +SIP+ server type the following commands from the command prompt: 


   $ sudo su opensrf 

   $ oils_ctl.sh -a [start|stop|restart]_sip 

indexterm:[SIP]


Logging-SIP 
^^^^^^^^^^^

Syslog
++++++

indexterm:[syslog]


It is useful to log +SIP+ requests to a separate file especially during initial setup by modifying your syslog config file. 

. Edit syslog.conf. 

   $ sudo vi /etc/syslog.conf  # maybe /etc/rsyslog.conf


. Add this: 

   local6.*                -/var/log/SIP_evergreen.log 
 
. Syslog expects the logfile to exist so create the file. 

   $ sudo touch /var/log/SIP_evergreen.log 

. Restart sysklogd. 

   $ sudo /etc/init.d/sysklogd restart 


Syslog-NG
+++++++++

indexterm:[syslog-NG]

. Edit logging config. 

   sudo vi /etc/syslog-ng/syslog-ng.conf 

. Add: 

   # +SIP2+ for Evergreen 
   filter    f_eg_sip { level(warn, err, crit) and facility(local6); }; 
   destination eg_sip { file("var/log/SIP_evergreen.log"); }; 
   log { source(s_all); filter(f_eg_sip); destination(eg_sip); }; 

. Syslog-ng expects the logfile to exist so create the file. 

   $ sudo touch /var/log/SIP_evergreen.log

. Restart syslog-ng 

   $ sudo /etc/init.d/syslog-ng restart 


indexterm:[SIP]


Testing Your SIP Connection 
^^^^^^^^^^^^^^^^^^^^^^^^^^^

* In the root directory of the SIPServer code: 

   $ cd SIPServer/t 

* Edit SIPtest.pm, change the $instid, $server, $username, and $password variables. This will be 
enough to test connectivity. To run all tests, you'll need to change all the variables in the _Configuration_ section. 

   $ PERL5LIB=../ perl 00sc_status.t 
+
This should produce something like:
+

   1..4 
   ok 1 - Invalid username 
   ok 2 - Invalid username 
   ok 3 - login 
   ok 4 - SC status 

* Don't be dismayed at *Invalid Username*. That's just one of the many tests that are run. 

More Testing
^^^^^^^^^^^^

Once you have opened up either the +SIP+ OR +SIP2+ ports to be accessible from outside you can do some testing 
via +telnet+. In the following tests:

* Replace +$server+ with your server hostname (or +localhost+ if you want to
  skip testing external access for now);
* Replace +$username+, +$password+, and +$instid+ with the corresponding values
  in the +<accounts>+ section of your SIP configuration file;
* Replace the +$user_barcode+ and +$user_password+ variables with the values
  for a valid user.

///////////////
Comments because we don't want to indent these numbered bullets!
///////////////

. Start by testing your ability to log into the SIP server:
+
[NOTE]
======================
We are using 6001 here which is associated with +SIP2+ as per our configuration. 
======================
+
   $ telnet $server 6001 
   Connected to $server. 
   Escape character is '^]'. 
   9300CN$username|CO$password|CP$instid
+
If successful, the SIP server returns a +941+ result. A result of +940+,
however, indicates an unsuccessful login attempt. Check the +<accounts>+
section of your SIP configuration and try agin.

. Once you have logged in successfully, replace the variables in the following
line and paste it into the telnet session:
+
    2300120080623    172148AO$instid|AA$user_barcode|AC$password|AD$user_password
+
If successful, the SIP server returns the patron information for $barcode,
similar to the following:
+
   24  Y           00120100113    170738AEFirstName MiddleName LastName|AA$barcode|BLY|CQY 
   |BHUSD|BV0.00|AFOK|AO$instid| 
+
The response declares it is a valid patron BLY with a valid password CQY and shows the user's +$name+.


indexterm:[SIP]

SIP Communication
~~~~~~~~~~~~~~~~~

indexterm:[SIP Server, SIP Communication]

+SIP+ generally communicates over a +TCP+ connection (either raw sockets or over +telnet+), but can also 
communicate via serial connections and other methods. In Evergreen, the most common deployment is a +RAW+ socket 
connection on port 6001.

+SIP+ communication consists of strings of messages, each message request and response begin with a 2-digit 
``command'' - Requests usually being an odd number and responses usually increased by 1 to be an even number. The 
combination numbers for the request command and response is often referred to as a _Message Pair_ (for example, 
a 23 command is a request for patron status, a 24 response is a patron status, and the message pair 23/24 is patron 
status message pair). The table in the next section shows the message pairs and a description of them. 

For clarification, the ``Request'' is from the device (selfcheck or otherwise) to the ILS/ACS. The response is… the 
response to the request ;). 

Within each request and response, a number of fields (either a fixed width or separated with a | [pipe symbol] and 
preceded with a 2-character field identifier) are used. The fields vary between message pairs. 

|===========================================================================
| *Pair* | *Name*              | *Supported?*          |*Details* 
| 01     | Block Patron        | Yes                   |<<sip_01_block_patron, 01/Block_Patron>> - ACS responds with 24 Patron Status Response 
| 09-10  | Checkin             | Yes (with extensions) |<<sip_09-10_checkin, 09/10_Checkin>> 
| 11-12  | Checkout            | Yes (no renewals)     |<<sip_11-12_checkout, 11/12_Checkout>> 
| 15-16  | Hold                | Partially supported   |<<sip_15-16_hold, 15/16_Hold>>  
| 17-18  | Item Information    | Yes (no extensions)   |<<sip_17-18_item_information, 17/18_Item_Information>> 
| 19-20  | Item Status Update  | No                    |<<sip_19-20_item_status_update, 19/20_Item_Status_Update>> - Returns Patron Enable response, but doesn't make any changes in EG  
| 23-24  | Patron Status       | Yes                   |<<sip_23-24_patron_status, 23/24_Patron_Status>> - 63/64 ``Patron Information'' preferred 
| 25-26  | Patron Enable       | No                    |<<sip_25-26_patron_enable, 25/26_Patron_Enable>> - Used during system testing and validation 
| 29-30  | Renew               | Yes                   |<<sip_29-30_renew, 29/30_Renew>> 
| 35-36  | End Session         | Yes                   |<<sip_35-36_end_session, 35/36_End_Session>>
| 37-38  | Fee Paid            | Yes                   |<<sip_37-38_fee_paid, 37/38_Fee_Paid>> 
| 63-64  | Patron Information  | Yes (no extensions)   |<<sip_63-64_patron_information, 63/64_Patron_Information>> 
| 65-66  | Renew All           | Yes                   |<<sip_65-66_renew_all, 65/66_Renew_All>> 
| 93-94  | Login               | Yes                   |<<sip_93-94_login, 93/94_Login>> - Must be first command to Evergreen ACS (via socket) or +SIP+ will terminate 
| 97-96  | Resend last message | Yes                   |<<sip_97-96_resend, 97/96_Resend>> 
| 99-98  | SC-ACS Status       | Yes                   |<<sip_99-98_sc_and_acs_status, 99/98_SC_and_ACS_Status>> 
|===========================================================================

anchor:sip_01_block_patron[]

01 Block Patron
^^^^^^^^^^^^^^^

indexterm:[SelfCheck]

A selfcheck will issue a *Block Patron* command if a patron leaves their card in a selfcheck machine or if the 
selfcheck detects tampering (such as attempts to disable multiple items during a single item checkout, multiple failed 
pin entries, etc). 

In Evergreen, this command does the following:

* User alert message: _CARD BLOCKED BY SELF-CHECK MACHINE_ (this is independent of the AL _Blocked 
Card Message_ field). 

* Card is marked inactive. 

The request looks like: 

   01<card retained><date>[fields AO, AL, AA, AC] 

_Card Retained_: A single character field of Y or N - tells the ACS whether the SC has retained the card (ex: left in 
the machine) or not. 

_Date_: An 18 character field for the date/time when the block occurred. 

_Format_: YYYYMMDDZZZZHHMMSS (ZZZZ being zone - 4 blanks when local time, ``Z'' (3 blanks and a Z) 
represents UTC(GMT/Zulu) 

_Fields_: See <<fields, Fields>> for more details.

The response is a 24 ``Patron Status Response'' with the following: 

* Charge privileges denied 
* Renewal privileges denied 
* Recall privileges denied (hard-coded in every 24 or 64 response) 
* hold privileges denied 
* Screen Message 1 (AF): _blocked_ 
* Patron 

anchor:sip_09-10_checkin[]

09/10 Checkin
^^^^^^^^^^^^^

~The request looks like: 

   09<No block (Offline)><xact date><return date>[Fields AP,AO,AB,AC,CH,BI] 

_No Block (Offline)_: A single character field of _Y_ or _N_ - Offline transactions are not currently supported so send _N_. 

_xact date_: an 18 character field for the date/time when the checkin occurred. Format: 
YYYYMMDDZZZZHHMMSS (ZZZZ being zone - 4 blanks when local time, ``Z'' (3 blanks and a Z) represents 
UTC(GMT/Zulu) 

_Fields_: See <<fields, Fields>> for more details. 

The response is a 10 ``Checkin Response'' with the following: 

   10<resensitize><magnetic media><alert><xact date>[Fields AO,AB,AQ,AJ,CL,AA,CK,CH,CR,CS,CT,CV,CY,DA,AF,AG] 

Example (with a remote hold): 

  09N20100507    16593720100507    165937APCheckin Bin 5|AOBR1|AB1565921879|ACsip_01| 

  101YNY20100623    165731AOBR1|AB1565921879|AQBR1|AJPerl 5 desktop reference|CK001|CSQA76.73.P33V76 1996 
  |CTBR3|CY373827|DANicholas Richard Woodard|CV02|

Here you can see a hold alert for patron CY _373827_, named DA _Nicholas Richard Woodard_, to be picked up at CT 
``BR3''. Since the transaction is happening at AO ``BR1'', the alert type CV is 02 for _hold at remote library_. The 
possible values for CV are: 

* 00: unknown 

* 01: local hold 

* 02: remote hold 

* 03: ILL transfer (not used by EG) 

* 04: transfer 

* 99: other 

indexterm:[magnetic media]

[NOTE]
===============
The logic for Evergreen to determine whether the content is magnetic_media comes from either legacy circ 
scripts or search_config_circ_modifier. The default is non-magnetic. The same is true for media_type (default 
001). Evergreen does not populate the collection_code because it does not really have any, but it will provide 
the call_number where available.

Unlike the +item_id+ (barcode), the +title_id+ is actually a title string, unless the configuration forces the 
return of the bib ID. 

Don't be confused by the different branches that can show up in the same response line. 

* AO is where the transaction took place, 

* AQ is the ``permanent location'', and 

* CT is the _destination location_ (i.e., pickup lib for a hold or target lib for a transfer). 
================

anchor:sip_11-12_checkout[]

11/12 Checkout
^^^^^^^^^^^^^^


anchor:sip_15-16_hold[]

15/16 Hold
^^^^^^^^^^

Evergreen supports the Hold message for the purpose of canceling
holds.  It does not currently support creating hold requests via SIP2.


anchor:sip_17-18_item_information[]

17/18 Item Information 
^^^^^^^^^^^^^^^^^^^^^^

The request looks like: 

    17<xact_date>[fields: AO,AB,AC] 

The request is very terse. AC is optional. 

The following response structure is for +SIP2+. (Version 1 of the protocol had only 6 total fields.) 

    18<circulation_status><security_marker><fee_type><xact_date> 
    [fields: CF,AH,CJ,CM,AB,AJ,BG,BH,BV,CK,AQ,AP,CH,AF,AG,+CT,+CS] 

Example:

   1720060110    215612AOBR1|ABno_such_barcode| 

   1801010120100609    162510ABno_such_barcode|AJ| 

   1720060110    215612AOBR1|AB1565921879| 

   1810020120100623    171415AB1565921879|AJPerl 5 desktop reference|CK001|AQBR1|APBR1|BGBR1 
   |CTBR3|CSQA76.73.P33V76 1996| 

The first case is with a bogus barcode. The latter shows an item with a circulation_status of _10_ for _in transit between 
libraries_. The known values of +circulation_status+ are enumerated in the spec. 

indexterm:[Automated Material Handling (AMH)]

EXTENSIONS: The CT field for _destination location_ and CS _call number_ are used by Automated Material Handling 
systems. 


anchor:sip_19-20_item_status_update[]

19/20 Item Status Update
^^^^^^^^^^^^^^^^^^^^^^^^


anchor:sip_23-24_patron_status[]

23/24 Patron Status 
^^^^^^^^^^^^^^^^^^^

Example: 

   2300120060101    084235AOUWOLS|AAbad_barcode|ACsip_01|ADbad_password| 

   24YYYY          00120100507    013934AE|AAbad_barcode|BLN|AOUWOLS|

   2300120060101    084235AOCONS|AA999999|ACsip_01|ADbad_password| 

   24  Y           00120100507    022318AEDoug Fiander|AA999999|BLY|CQN|BHUSD|BV0.00|AFOK|AOCONS| 

   2300120060101    084235AOCONS|AA999999|ACsip_01|ADuserpassword|LY|CQN|BHUSD|BV0.00|AFOK|AOCONS| 

   24  Y           00120100507    022803AEDoug Fiander|AA999999|BLY|CQY|BHUSD|BV0.00|AFOK|AOCONS| 

. The BL field (+SIP2+, optional) is _valid patron_, so the _N_ value means _bad_barcode_ doesn't match a patron, the 
_Y_ value means 999999 does. 

. The CQ field (+SIP2+, optional) is _valid password_, so the _N_ value means _bad_password_ doesn't match 999999's 
password, the _Y_ means _userpassword_ does. 

So if you were building the most basic +SIP2+ authentication client, you would check for _|CQY|_ in the response to 
know the user's barcode and password are correct (|CQY| implies |BLY|, since you cannot check the password 
unless the barcode exists). However, in practice, depending on the application, there are other factors to consider in 
authentication, like whether the user is blocked from checkout, owes excessive fines, reported their card lost, etc. 
These limitations are reflected in the 14-character _patron status_ string immediately following the _24_ code. See the 
field definitions in your copy of the spec. 


anchor:sip_25-26_patron_enable[]

25/26 Patron Enable 
^^^^^^^^^^^^^^^^^^^

Not yet supported. 


anchor:sip_29-30_renew[]

29/30 Renew
^^^^^^^^^^^

Evergreen supports the Renew message.


anchor:sip_35-36_end_session[]

35/36 End Session
^^^^^^^^^^^^^^^^^

   3520100505    115901AOBR1|AA999999|

   36Y20100507    161213AOCONS|AA999999|AFThank you!| 
   
The _Y/N_ code immediately after the 36 indicates _success/failure_. Failure is not particularly meaningful or important 
in this context, and for evergreen it is hardcoded _Y_. 



anchor:sip_37-38_fee_paid[]

37/38 Fee Paid 
^^^^^^^^^^^^^^

Evergreen supports the Fee Paid message.


anchor:sip_63-64_patron_information[]

63/64 Patron Information 
^^^^^^^^^^^^^^^^^^^^^^^^

Attempting to retrieve patron info with a bad barcode:

   6300020060329    201700          AOBR1|AAbad_barcode| 
   
   64YYYY          00020100623    141130000000000000000000000000AE|AAbad_barcode|BLN|AOBR1| 

Attempting to retrieve patron info with a good barcode (but bad patron password): 

   6300020060329    201700          AOBR1|AA999999|ADbadpwd| 

   64  Y           00020100623    141130000000000000000000000000AA999999|AEDavid J. Fiander|BHUSD|BV0.00 
   |BD2 Meadowvale Dr. St Thomas, ON Canada 

   90210|BEdjfiander@somemail.com|BF(519) 555 1234|AQBR1|BLY|CQN|PB19640925|PCPatrons 
   |PIUnfiltered|AFOK|AOBR1| 

See <<sip_23-24_patron_status, 23/24 Patron Status>> for info on +BL+ and +CQ+ fields. 



anchor:sip_65-66_renew_all[]

65/66 Renew All
^^^^^^^^^^^^^^^

Evergreen supports the Renew All message.


anchor:sip_93-94_login[]

93/94 Login 
^^^^^^^^^^^

Example: 

   9300CNsip_01|CObad_value|CPBR1| 
  
   [Connection closed by foreign host.] 
   ... 

   9300CNsip_01|COsip_01|CPBR1|

   941 

_941_ means successful terminal login. _940_ or getting dropped means failure.


anchor:sip_97-96_resend[]

97/96 Resend
^^^^^^^^^^^^


anchor:sip_99-98_sc_and_acs_status[]

99/98 SC and ACS Status
^^^^^^^^^^^^^^^^^^^^^^^

   99<status code><max print width><protocol version> 
 
All 3 fields are required:

* 0: SC is OK

* 1: SC is out of paper 

* 2: SC shutting down

* status code - 1 character 

* max print width - 3 characters - the integer number of characters the client can print 

* protocol version - 4 characters - x.xx 

  98<on-line status><checkin ok><checkout ok><ACS renewal policy>
  <status update ok><offline ok><timeout period> 

  <retries allowed><date/time sync><protocol version><institution id> 
  <library name><supported messages><terminal 

  location><screen message><print line> 

Example: 

  9910302.00 

  98YYYYNN60000320100510    1717202.00AOCONS|BXYYYYYYYYYNYNNNYN|

The Supported Messages field +BX+ appears only in +SIP2+, and specifies whether 16 different +SIP+ commands are 
supported by the +ACS+ or not. 


anchor:fields[]

Fields
^^^^^^

All fixed-length fields in a communication will appear before the first variable-length field. This allows for simple 
parsing. Variable-length fields are by definition delimited, though there will not necessarily be an initial delimiter 
between the last fixed-length field and the first variable-length one. It would be unnecessary, since you should know 
the exact position where that field begins already.