1.6/admin/troubleshooting.xml

   1 <?xml version="1.0" encoding="utf-8"?>\r
   2 <chapter xmlns="http://docbook.org/ns/docbook" xmlns:xi="http://www.w3.org/2001/XInclude"\r
   3 xmlns:xl="http://www.w3.org/1999/xlink" version="5.0" xml:id="troubleshooting">\r
   4         <title>Troubleshooting System Errors</title>\r
   5         <para>If you have Evergreen installed and are encountering systematic errors, here is the steps to find the \r
   6         cause and solution to most problems. These instructions assume standard locations and file names for Evergreen\r
   7         installations, and may also include commands for specific Linux distributions.</para>\r
   8         <procedure>\r
   9                 <title>Systematic Evergreen Restart to Isolate Errors</title>\r
  10                 <step>\r
  11                         <para>Stop Apache:</para><indexterm><primary>web server</primary><secondary>Apache</secondary><tertiery>stopping</tertiery></indexterm>\r
  12                         <screen><userinput>/etc/init.d/apache2 stop</userinput></screen>\r
  13                         <para>or</para>\r
  14                         <screen><userinput>apache2ctl stop</userinput></screen>\r
  15                 </step>\r
  16                 <step>\r
  17                         <para>Stop OpenSRF:</para><indexterm><primary>OpenSRF</primary></indexterm>\r
  18                         <screen><userinput>osrf_ctl.sh -l -a stop_all</userinput></screen>\r
  19                         <para>You should get either output simlar to this:</para>\r
  20 <screen>\r
  21 Stopping OpenSRF C process 12515...\r
  22 Stopping OpenSRF C process 12520...\r
  23 Stopping OpenSRF C process 12526...\r
  24 Stopping OpenSRF Perl process 12471...\r
  25 Stopping OpenSRF Router process 12466...\r
  26 </screen>\r
  27                         <para>Or, if services have already been stopped, output may look like this:</para>\r
  28                         <screen>OpenSRF C not running</screen>\r
  29                         <screen>OpenSRF Perl not running</screen>\r
  30                         <screen>OpenSRF Router not running</screen>\r
  31                         <para>Occasionally osrf_ctl.sh fails to kill OpenSRF processes, so we should check to make \r
  32                         sure that none are still running with the command:</para>\r
  33                         <screen><userinput>ps -aef | grep OpenSRF</userinput></screen>\r
  34                         <para>You should manually kill any OpenSRF processes.</para>\r
  35                         <para>If you were unable to stop OpenSRF with the above methods, you could also try this \r
  36                         command:</para>\r
  37                         <screen><userinput>rm –R /openils/var/run/*.pid</userinput></screen>\r
  38                         <para>This will remove the temporary OpenSRF process files from the run directory which may \r
  39                         have been left over from a previous system boot cycle.</para>\r
  40                 </step>\r
  41                 <step>\r
  42                         <para>Restart <systemitem class="service">Ejabberd</systemitem> and \r
  43                         <systemitem class="service">Memcached</systemitem> with the following commands:</para><indexterm><primary>Ejabberd</primary></indexterm>\r
  44                         <screen><userinput>sudo /etc/init.d/ejabberd restart</userinput></screen>\r
  45                         <screen><userinput>sudo /etc/init.d/memcached restart</userinput></screen><indexterm><primary>memcached</primary></indexterm>\r
  46                 </step>\r
  47                 <step>  \r
  48                         <para>Start the OpenSRF <systemitem class="service">router</systemitem> and check for errors</para>\r
  49                         <indexterm><primary>OpenSRF</primary><secondary>services</secondary><tertiery>router</tertiery></indexterm>\r
  50                         <screen><userinput>/openils/bin/osrf_ctl.sh -l -a start_router</userinput></screen>\r
  51                         <para>If the <systemitem class="service">router</systemitem> started correctly, output will be:</para>\r
  52                         <screen>Starting OpenSRF Router</screen>\r
  53                         <para>If <systemitem class="service">router</systemitem> does not start correctly, you should check the <systemitem class="service">router</systemitem> error log files \r
  54                         for error information.</para>\r
  55                         <para>Evergreen 1.6 uses two routers, a public one and a private one, with two different \r
  56                         logfiles:</para>\r
  57                         <para><filename >/openils/var/log/private.router.log</filename></para><indexterm><primary>logs</primary><secondary>router</secondary></indexterm>\r
  58                         <para><filename >/openils/var/log/public.router.log</filename></para>\r
  59                         <para>A quick way to find error information in the logs is with the grep command.</para>\r
  60                                 <screen>grep ERR /openils/var/log/*router.log</screen>\r
  61                         <para>As a final sanity check, look for router processes using the process status \r
  62                         command:</para>\r
  63                         <screen><userinput>ps -aef | grep Router</userinput></screen>\r
  64                 </step>\r
  65                 <step>\r
  66                         <para>Start the OpenSRF <systemitem class="service">perl</systemitem> services and check for errors</para>\r
  67                 <indexterm><primary>OpenSRF</primary><secondary>services</secondary><tertiery>perl</tertiery></indexterm>\r
  68                         <screen><userinput>/openils/bin/osrf_ctl.sh -l -a start_perl</userinput></screen>\r
  69                         <para>You should see the output similar to the following:</para>\r
  70 <screen>\r
  71 Starting OpenSRF Perl\r
  72 * starting all services for ...\r
  73 * starting service pid=7484 opensrf.settings\r
  74 * starting service pid=7493 open-ils.cat\r
  75 * starting service pid=7495 open-ils.supercat\r
  76 * starting service pid=7497 open-ils.search\r
  77 * starting service pid=7499 open-ils.circ\r
  78 * starting service pid=7501 open-ils.actor\r
  79 * starting service pid=7502 open-ils.storage\r
  80 ...\r
  81 </screen>\r
  82                         <para>If the <systemitem class="service">perl</systemitem> services do not start correctly or you receive errors, search for errors \r
  83                         in the following log files:</para>\r
  84                         <itemizedlist>  \r
  85                                 <listitem>/openils/var/log/router.log</listitem>\r
  86                                 <listitem>/openils/var/log/osrfsys.log</listitem><indexterm><primary>logs</primary><secondary>osrfsys.log</secondary></indexterm>\r
  87                         </itemizedlist>\r
  88                         <para>At this point you can use the <command>grep</command> command to find errors in \r
  89                         any of the Evergreen log files:</para>\r
  90                         <screen><userinput>grep ERR /openils/var/log/*.log</userinput></screen>\r
  91                         <para>As a final sanity check, look for OpenSRF processes:</para>\r
  92                         <screen><userinput>ps -aef | grep -i opensrf</userinput></screen>\r
  93                 </step>\r
  94                 <step>\r
  95                         <para>Start the OpenSRF <systemitem class="service">c</systemitem> services and check for errors:</para>]\r
  96                         <indexterm><primary>OpenSRF</primary><secondary>services</secondary><tertiery>C</tertiery></indexterm>\r
  97                         <screen>/openils/bin/osrf_ctl.sh -l -a start_c</screen>\r
  98                         <para>And output should be:</para>\r
  99                         <screen>Starting OpenSRF C (host=localhost)</screen>\r
 100                         <para>If the <systemitem class="service">c</systemitem> service does not start, check for errors by grepping \r
 101                         the log files for errors:</para>\r
 102                         <screen><userinput>grep ERR /openils/var/log/*.log</userinput></screen>\r
 103                         <para>Check for <application>OpenSRF</application> processes:</para>\r
 104                         <screen><userinput>ps -aef | grep -i opensrf</userinput></screen>\r
 105                 </step>\r
 106                 <step>\r
 107                         <para>Smoke test with <filename>autogen.sh</filename><indexterm><primary>autogen</primary></indexterm></para>\r
 108                         <para>The <command>autogen</command> tool will take some dynamic information from the database and generate \r
 109                         static JavaScript <indexterm><primary>JavaScript</primary></indexterm> files for use by the OPAC and staff client. It is also able to refresh \r
 110                         the proximity map<indexterm><primary>proximity map</primary></indexterm> between libraries for the purpose of efficiently routing hold \r
 111                         requests.</para>\r
 112                         <para>As user <systemitem class="username">opensrf</systemitem>, you invoke <command>autogen</command> with the command:</para>\r
 113                         <screen><userinput>/openils/bin/autogen.sh -c /openils/conf/opensrf_core.xml -u</userinput></screen>\r
 114                         <para>If Autogen completes successfully, the output will be:</para>\r
 115 <screen>\r
 116 Updating fieldmapper\r
 117 Updating web_fieldmapper\r
 118 Updating OrgTree\r
 119 removing OrgTree from the cache...\r
 120 Updating OrgTree HTML\r
 121 Updating locales selection HTML\r
 122 Updating Search Groups\r
 123 Refreshing proximity of org units\r
 124 Successfully updated the organization proximity\r
 125 Done\r
 126 </screen>\r
 127                         <para>If Autogen does not complete its task and you receive errors, use \r
 128                         <command>grep</command> to find errors in the log files:</para>  \r
 129                         <screen><userinput>grep ERR /openils/var/log/*.log</userinput></screen>\r
 130                 </step>\r
 131                 <step>\r
 132                         <para>Connect to Evergreen using the <command>srfsh</command> command-line OpenSRF client<indexterm><primary>srfsh (command-line OpenSRF client)</primary></indexterm></para>\r
 133                         <screen><userinput>/openils/bin/srfsh</userinput></screen>\r
 134                         <note>\r
 135                                 <para>In order for you to connect using <command>srfsh</command>, you will need to \r
 136                                 have set up the .srfsh.xml configuration file in your home directory as as \r
 137                                 described in the installation chapter.</para>\r
 138                         </note> \r
 139                         <para>You will then see the <emphasis>srfsh</emphasis> prompt:</para>\r
 140                         <screen><prompt>srfsh#</prompt></screen>\r
 141                         <para>At the srfsh prompt, enter this command:</para>\r
 142                         <screen><userinput>login admin open-ils</userinput></screen>\r
 143                         <para>You should the request verification:</para>\r
 144 <screen>\r
 145 Received Data: "6f63ff5542da1fead4431c6c280efc75"\r
 146 ------------------------------------\r
 147 Request Completed Successfully\r
 148 Request Time in seconds: 0.018414\r
 149 ------------------------------------\r
 150 \r
 151 Received Data: {\r
 152 "ilsevent":0,\r
 153 "textcode":"SUCCESS",\r
 154 "desc":" ",\r
 155 "pid":7793,\r
 156 "stacktrace":"oils_auth.c:312",\r
 157 "payload":{\r
 158 "authtoken":"28804ebf99508496e2a4d2593aaa930e",\r
 159  "authtime":420.000000\r
 160 }\r
 161 }\r
 162 \r
 163 ------------------------------------\r
 164 Request Completed Successfully\r
 165 Request Time in seconds: 0.552430\r
 166 ------------------------------------\r
 167 Login Session: 28804.  Session timeout: 420.000\r
 168 srfsh#</screen>        \r
 169                         <para>If you encounter errors or if you are unable to connect, you should consult the \r
 170                         <filename>srfsh.log</filename> file. The location of this file is configured in your <indexterm><primary>logs</primary><secondary>srfsh.log</secondary></indexterm>\r
 171                         <filename>.srfsh.xml</filename> configuration file and is \r
 172                         <filename>/openils/var/log/srfsh.log</filename> by default.</para>\r
 173                         <para>Pressing \r
 174                                 <keycombo>\r
 175                                         <keycap>Ctrl</keycap>\r
 176                                         <keycap>D</keycap>\r
 177                                 </keycombo>\r
 178                         or entering <quote>exit</quote> will terminate srfsh.</para>\r
 179                 </step>\r
 180                 <step>\r
 181                         <para>Start <application>Apache</application> and check for errors:</para>\r
 182                         <indexterm><primary>web server</primary><secondary>Apache</secondary><tertiery>starting</tertiery></indexterm>\r
 183                         <screen>/etc/init.d/apache2 start</screen>\r
 184                         <para>or</para>\r
 185                         <screen>apache2ctl start</screen>\r
 186                         <para>You should see output:</para>\r
 187 <screen>\r
 188 * Starting web server apache2\r
 189 ...done.\r
 190 </screen>\r
 191                         <para>the Apache OpenSRF modules write to the \r
 192                         <filename>/openils/var/log/gateway.log</filename></para><indexterm><primary>logs</primary><secondary>gateway.log</secondary></indexterm>\r
 193                         <para>However, you should check all of the log files for errors:</para>\r
 194                         <screen>grep ERR /openils/var/log/*.log</screen>\r
 195                         <para>Another place to check for errors is the <application>Apache</application> error logs\r
 196                         generally located in in the <filename class="directory">/var/log/Apache2</filename> <indexterm><primary>logs</primary><secondary>Apache</secondary></indexterm>\r
 197                         <indexterm><primary>Apache</primary><secondary>logs</secondary></indexterm>\r
 198                         directory</para>\r
 199                         <para>If you encounter errors with Apache, a common source of potential problems are the \r
 200                         Evergreen site configuration files <filename>/etc/apache2/eg_vhost.conf</filename> and \r
 201                         <filename>/etc/apache2/sites-available/eg.conf</filename></para>\r
 202                         <indexterm><primary>configuration files</primary><secondary>Apache</secondary><tertiery>eg.conf</tertiery></indexterm>\r
 203                         <indexterm><primary>configuration files</primary><secondary>Apache</secondary><tertiery>eg_vhost.conf</tertiery></indexterm>\r
 204                 </step>\r
 205 \r
 206                 <step>\r
 207                                         <title>Testing with <command>settings-tester.pl</command></title>\r
 208                                         <para>As the <systemitem class="username">opensrf</systemitem> user, run the script <command>settings-tester.pl</command> to see if it finds any \r
 209                                         system configuration problems.</para>\r
 210 <screen>                                        \r
 211 <userinput>cd /home/opensrf/Evergreen-ILS-1.6.0.0</userinput>\r
 212 <userinput>perl Open-ILS/src/support-scripts/settings-tester.pl</userinput>\r
 213 </screen>\r
 214                                         <para>Here is example output from running <command>settings-tester.pl</command>:</para>\r
 215 <screen>\r
 216 LWP::UserAgent version 5.810\r
 217 XML::LibXML version 1.70\r
 218 XML::LibXML::XPathContext version 1.70\r
 219 XML::LibXSLT version 1.70\r
 220 Net::Server::PreFork version 0.97\r
 221 Cache::Memcached version 1.24\r
 222 Class::DBI version 0.96\r
 223 Class::DBI::AbstractSearch version 0.07\r
 224 Template version 2.19\r
 225 DBD::Pg version 2.8.2\r
 226 Net::Z3950::ZOOM version 1.24\r
 227 MARC::Record version 2.0.0\r
 228 MARC::Charset version 1.1\r
 229 MARC::File::XML version 0.92\r
 230 Text::Aspell version 0.04\r
 231 CGI version 3.29\r
 232 DateTime::TimeZone version 0.7701\r
 233 DateTime version 0.42\r
 234 DateTime::Format::ISO8601 version 0.06\r
 235 DateTime::Format::Mail version 0.3001\r
 236 Unix::Syslog version 1.1\r
 237 GD::Graph3d version 0.63\r
 238 JavaScript::SpiderMonkey version 0.19\r
 239 Log::Log4perl version 1.16\r
 240 Email::Send version 2.192\r
 241 Text::CSV version 1.06\r
 242 Text::CSV_XS version 0.52\r
 243 Spreadsheet::WriteExcel::Big version 2.20\r
 244 Tie::IxHash version 1.21\r
 245 Parse::RecDescent version 1.95.1\r
 246 SRU version 0.99\r
 247 JSON::XS version 2.27\r
 248 \r
 249 Checking Jabber connection for user opensrf, domain private.localhost\r
 250 * Jabber successfully connected\r
 251 \r
 252 Checking Jabber connection for user opensrf, domain public.localhost\r
 253 * Jabber successfully connected\r
 254 \r
 255 Checking Jabber connection for user router, domain public.localhost\r
 256 * Jabber successfully connected\r
 257 \r
 258 Checking Jabber connection for user router, domain private.localhost\r
 259 * Jabber successfully connected\r
 260 \r
 261 Checking database connections\r
 262 * /opensrf/default/reporter/setup :: Successfully connected to database dbi:Pg:dbname=evergreen;host=localhost;port=5432\r
 263   * Database has the expected server encoding UTF8.\r
 264 * /opensrf/default/apps/open-ils.storage/app_settings/databases :: Successfully connected to database dbi:...\r
 265   * Database has the expected server encoding UTF8.\r
 266 * /opensrf/default/apps/open-ils.cstore/app_settings :: Successfully connected to database dbi:Pg:dbname...\r
 267   * Database has the expected server encoding UTF8.\r
 268 * /opensrf/default/apps/open-ils.pcrud/app_settings :: Successfully connected to database dbi:Pg:dbname...\r
 269   * Database has the expected server encoding UTF8.\r
 270 * /opensrf/default/apps/open-ils.reporter-store/app_settings :: Successfully connected to database dbi:Pg:dbname...\r
 271   * Database has the expected server encoding UTF8.\r
 272 \r
 273 Checking database drivers to ensure &lt;driver&gt; matches &lt;language&gt;\r
 274 * OK: Pg language is undefined for reporter base configuration\r
 275 * OK: Pg language is undefined for reporter base configuration\r
 276 * OK: Pg language is perl in /opensrf/default/apps/open-ils.storage/language\r
 277 * OK: pgsql language is C in /opensrf/default/apps/open-ils.cstore/language\r
 278 * OK: pgsql language is C in /opensrf/default/apps/open-ils.pcrud/language\r
 279 * OK: pgsql language is C in /opensrf/default/apps/open-ils.reporter-store/language\r
 280 \r
 281 Checking libdbi and libdbi-drivers\r
 282   * OK - found locally installed libdbi.so and libdbdpgsql.so in shared library path\r
 283 \r
 284 Checking hostname\r
 285  * OK: found hostname 'localhost' in &lt;hosts&gt; section of opensrf.xml\r
 286 $\r
 287 </screen>\r
 288                                         <para>If the output from the script does not help you find the problem, please do not make any further significant changes to your configuration. \r
 289                                         Follow the steps in the troubleshooting guide in <xref linkend="troubleshooting"/>.</para>\r
 290                 </step>\r
 291                 <step>\r
 292                         <para>Try to login from the <link linkend="staffclient-login">staff client</link></para>\r
 293                 </step>\r
 294                 <step xml:id="testing-opac">\r
 295                                         <title>Testing the Catalog</title>\r
 296                                         <para>By default, the OPAC will live at the URL <uri>http://my.domain.com/opac/</uri>.</para>\r
 297                                         <para>Navigate to this URL and the front page of the OPAC should load. There is a basic text entry field with some extra search options. If you have any \r
 298                                         problems loading this page, check the Apache error logs. If the page loads but does not function correctly, then check for possible javascript errors. We \r
 299                                         highly recommend testing with the <application>Firefox</application> browser because of the helpful javascript debugging tools.</para>\r
 300                                         <para>Assuming that the OPAC is functioning and there is data in your database, you can now perform other simple functional tests \r
 301                                         (e.g., searching the catalog).</para>\r
 302                 </step> \r
 303         </procedure>    \r
 304 </chapter>\r