1.6/admin/troubleshooting.xml

   1 <?xml version="1.0" encoding="utf-8"?>
   2 <chapter xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"
   3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:id="troubleshooting">
   4                 <title>Troubleshooting System Errors</title>
   5                 <para>If you have Evergreen installed and are encountering systematic errors, here is the steps to find the
   6                 cause and solution to most problems. These instructions assume standard locations and file names for Evergreen
   7                 installations, and may also include commands for specific Linux distributions.</para>
   8                 <procedure>
   9                         <title>Systematic Evergreen Restart to isolate Errors</title>
  10                         <step>
  11                                 <para>Stop Apache:</para>
  12                                 <screen>/etc/init.d/apache2 stop</screen>
  13                                 <para>or</para>
  14                                 <screen>apache2ctl stop</screen>
  15                         </step>
  16                         <step>
  17                                 <para>Stop OpenSRF:</para>
  18                                 <screen>osrf_ctl.sh -l -a stop_all</screen>
  19                                 <para>You should get either output resembling this:</para>
  20                                 <screen>Stopping OpenSRF C process 12515...</screen>
  21                                 <screen>Stopping OpenSRF C process 12520...</screen>
  22                                 <screen>Stopping OpenSRF C process 12526...</screen>
  23                                 <screen>Stopping OpenSRF Perl process 12471...</screen>
  24                                 <screen>Stopping OpenSRF Router process 12466...</screen>
  25                                 <para>Or, if services have already been stopped, output maay look like this:</para>
  26                                 <screen>OpenSRF C not running</screen>
  27                                 <screen>OpenSRF Perl not running</screen>
  28                                 <screen>OpenSRF Router not running</screen>
  29                                 <para>Occasionally osrf_ctl.sh fails to kill OpenSRF processes, so we should check to make
  30                                 sure that none are still running with the command:</para>
  31                                 <screen>ps -aef | grep OpenSRF</screen>
  32                                 <para>You should manually kill any OpenSRF processes.</para>
  33                                 <para>If you were unable to stop OpenSRF with the above methods, you could also try this
  34                                 command:</para>
  35                                 <screen>rm –R /openils/var/run/*.pid</screen>
  36                                 <para>This will remove the temporary OpenSRF process files from the run directory which may
  37                                 have been left over from a previous system boot cycle.</para>
  38                         </step>
  39                         <step>
  40                                 <para>Restart Ejabberd and Memcached with the following commands:</para>
  41                                 <screen>sudo /etc/init.d/ejabberd restart</screen>
  42                                 <screen>sudo /etc/init.d/memcached restart</screen>
  43                         </step>
  44                         <step>
  45                                 <para>Start the OpenSRF router and check for errors</para>
  46                                 <screen>/openils/bin/osrf_ctl.sh -l -a start_router</screen>
  47                                 <para>If the router started correctly, output will be:</para>
  48                                 <screen>Starting OpenSRF Router</screen>
  49                                 <para>If router does not start correcltly, you should check the router error log files
  50                                 for error information.</para>
  51                                 <para>Evergreen 1.6 uses two routers, a public one and a private one, with two different
  52                                 logfiles:</para>
  53                                 <para><filename >/openils/var/log/private.router.log</filename></para>
  54                                 <para><filename >/openils/var/log/public.router.log</filename></para>
  55                                 <para>A quick way to find error information in the logs is with the grep command.</para>
  56                                         <screen>grep ERR /openils/var/log/*router.log</screen>
  57                                 <para>As a final sanity check, look for router processes using the process status
  58                                 command:</para>
  59                                 <screen>ps -aef | grep Router</screen>
  60                         </step>
  61                         <step>
  62                                 <para>Start the OpenSRF perl services and check for errors</para>
  63                                 <screen>/openils/bin/osrf_ctl.sh -l -a start_perl</screen>
  64                                 <para>You should see the following output:</para>
  65                                 <screen>Starting OpenSRF Perl</screen>
  66                                 <screen>* starting all services for norcrossfx.norcross.esi</screen>
  67                                 <screen>* starting servivce pid=7484 opensrf.settings</screen>
  68                                 <screen>* starting servivce pid=7493 open-ils.cat</screen>
  69                                 <screen>* starting servivce pid=7495 open-ils.supercat</screen>
  70                                 <screen>* starting servivce pid=7497 open-ils.search</screen>
  71                                 <screen>* starting servivce pid=7499 open-ils.circ</screen>
  72                                 <screen>* starting servivce pid=7501 open-ils.actor</screen>
  73                                 <screen>* starting servivce pid=7502 open-ils.storage</screen>
  74                                 <screen>* starting servivce pid=7509 open-ils.penalty</screen>
  75                                 <screen>* starting servivce pid=7512 open-ils.collections</screen>
  76                                 <screen>* starting servivce pid=7514 open-ils.ingest</screen>
  77                                 <screen>* starting servivce pid=7517 open-ils.permacrud</screen>
  78                                 <screen>* starting servivce pid=7522 open-ils.fielder</screen>
  79                                 <screen>* starting servivce pid=7527 open-ils.vandelay</screen>
  80                                 <screen>* starting servivce pid=7516 open-ils.reporter</screen>
  81                                 <para>If the perl services do not start correclty or you receive errors, search for errors
  82                                 in the following log files:</para>
  83                                 <screen>/openils/var/log/router.log</screen>
  84                                 <screen>/openils/var/log/osrfsys.log</screen>
  85                                 <para>At this point you can use the <command>grep</command> command to find errors in
  86                                 any of the Evrgreen log files:</para>
  87                                 <screen>grep ERR /openils/var/log/*.log</screen>
  88                                 <para>As a final sanity check, look for OpenSRF processes:</para>
  89                                 <screen>ps -aef | grep -i opensrf</screen>
  90                         </step>
  91                         <step>
  92                                 <para>Start the OpenSRF C services and check for errors:</para>
  93                                 <screen>/openils/bin/osrf_ctl.sh -l -a start_c</screen>
  94                                 <para>And output should be:</para>
  95                                 <para>Starting OpenSRF C (host=localhost)</para>
  96                                 <para>If the <emphasis>c</emphasis> service does not start, check for errors by grepping
  97                                 the log files for errors:</para>
  98                                 <screen>grep ERR /openils/var/log/*.log</screen>
  99                                 <para>Check for OpenSRF processes:</para>
 100                                 <screen>ps -aef | grep -i opensrf</screen>
 101                         </step>
 102                         <step>
 103                                 <para>Smoke test with autogen.sh</para>
 104                                 <para>The Autogen tool will take some dynamic information from the database and generate
 105                                 static Javascript files for use by the OPAC and staff client. It is also able to refresh
 106                                 the proximity map between libraries for the purpose of efficiently routing hold
 107                                 requests.</para>
 108                                 <para>As user opensrf, you invoke Autogen with the command:</para>
 109                                 <screen>/openils/bin/autogen.sh -c /openils/conf/opensrf_core.xml -u</screen>
 110                                 <para>If Autogen completes successfully, the output will be:</para>
 111                                 <screen>Updating fieldmapper</screen>
 112                                 <screen>Updating web_fieldmapper</screen>
 113                                 <screen>Updating OrgTree</screen>
 114                                 <screen>removing OrgTree from the cache...</screen>
 115                                 <screen>Updating OrgTree HTML</screen>
 116                                 <screen>Updating locales selection HTML</screen>
 117                                 <screen>Updating Search Groups</screen>
 118                                 <screen>Refreshing proximity of org units</screen>
 119                                 <screen>Successfully updated the organization proximity</screen>
 120                                 <screen>Done</screen>
 121                                 <para>If Autogen does not complete its task and you recieve errors, use
 122                                 <command>grep</command> to find errors in the log files:</para>
 123                                 <screen>grep ERR /openils/var/log/*.log</screen>
 124                         </step>
 125                         <step>
 126                                 <para>Connect to Evergreen using the srfsh command-line OpenSRF client</para>
 127                                 <screen>/openils/bin/srfsh</screen>
 128                                 <note>
 129                                         <para>In order for you to connect using <command>srfsh</command>, you will need to
 130                                         have set up the .srfsh.xml configuration file in your home directory as as
 131                                         described in the installation chapter.</para>
 132                                 </note>
 133                                 <para>You will then see the <emphasis>srfsh</emphasis> prompt:</para>
 134                                 <screen>srfsh#</screen>
 135                                 <para>At the srfsh prompt, enter this command:</para>
 136                                 <screen>login admin open-ils</screen>
 137                                 <para>You should the request verification:</para>
 138                                 <screen>Received Data: "6f63ff5542da1fead4431c6c280efc75"</screen>
 139                                 <screen>------------------------------------</screen>
 140                                 <screen>Request Completed Successfully</screen>
 141                                 <screen>Request Time in seconds: 0.018414</screen>
 142                                 <screen>------------------------------------</screen>
 143                                 <screen></screen>
 144                                 <screen>Received Data: {</screen>
 145                                 <screen>"ilsevent":0,</screen>
 146                                 <screen>"textcode":"SUCCESS",</screen>
 147                                 <screen>"desc":" ",</screen>
 148                                 <screen>"pid":7793,</screen>
 149                                 <screen>"stacktrace":"oils_auth.c:312",</screen>
 150                                 <screen>"payload":{</screen>
 151                                 <screen>"authtoken":"28804ebf99508496e2a4d2593aaa930e",</screen>
 152                                 <screen> "authtime":420.000000</screen>
 153                                 <screen>}</screen>
 154                                 <screen>}</screen>
 155                                 <screen></screen>
 156                                 <screen>------------------------------------</screen>
 157                                 <screen>Request Completed Successfully</screen>
 158                                 <screen> Request Time in seconds: 0.552430</screen>
 159                                 <screen>------------------------------------</screen>
 160                                 <screen> Login Session: 28804.  Session timeout: 420.000</screen>
 161                                 <screen>srfsh#</screen>
 162                                 <para>If you encounter errors or if you are unable to connect, you should consult the
 163                                 <filename>srfsh.log</filename> file. The location of this file is configured in your
 164                                 <filename>.srfsh.xml</filename> configuration file and is
 165                                 <filename>/openils/var/log/srfsh.log</filename> by default.</para>
 166                                 <para>Pressing
 167                                         <keycombo>
 168                                                 <keycap>Ctrl</keycap>
 169                                                 <keycap>D</keycap>
 170                                         </keycombo>
 171                                 or entering <quote>exit</quote> will terminate srfsh.</para>
 172                         </step>
 173                         <step>
 174                                 <para>Start <application>Apache</application> and check for errors:</para>
 175                                 <screen>/etc/init.d/apache2 start</screen>
 176                                 <para>or</para>
 177                                 <screen>apache2ctl start</screen>
 178                                 <para>You should see output:</para>
 179                                 <screen>* Starting web server apache2</screen>
 180                                 <screen>...done.</screen>
 181                                 <para>the Apache OpenSRF modules write to the
 182                                 <filename>/openils/var/log/gateway.log</filename></para>
 183                                 <para>However, you should check all of the log files for errors:</para>
 184                                 <screen>grep ERR /openils/var/log/*.log</screen>
 185                                 <para>Another place to check for erros is the <application>Apache</application> error logs
 186                                 generally located in in the <filename class="directory">/var/log/Apapche2</filename>
 187                                 directory</para>
 188                                 <para>If you encounter errors with Apache, a common source of potential problems are the
 189                                 Evergreen site configuration files <filename>/etc/apache2/eg_vhost.conf</filename> and
 190                                 <filename>/etc/apache2/sites-available/eg.conf</filename></para>
 191                         </step>
 192                         <step>
 193                                 <para>Try to authenticate with the staff client</para>
 194                         </step>
 195                         <step>
 196                                 <para>Try to authenticate with the OPAC</para>
 197                         </step>
 198                 </procedure>
 199 </chapter>