LP#1704116: fix intermittant failure of parallel building

[OpenSRF.git] / README

Installing OpenSRF
==================

Preamble: referenced user accounts
----------------------------------

In subsequent sections, we will refer to a number of different accounts, as
follows:

  * Linux user accounts:
    ** The *user* Linux account is the account that you use to log onto the
       Linux system as a regular user.
    ** The *root* Linux account is an account that has system administrator
       privileges. On Debian and Fedora you can switch to this account from
       your *user* account by issuing the `su -` command and entering the
       password for the *root* account when prompted. On Ubuntu you can switch
       to this account from your *user* account using the `sudo su -` command
       and entering the password for your *user* account when prompted.
    ** The *opensrf* Linux account is an account that you will create as part
       of installing OpenSRF. You can switch to this account from the *root*
       account by issuing the `su - opensrf` command.

Installing prerequisites
------------------------

OpenSRF has a number of prerequisite packages that must be installed
before you can successfully configure, compile, and install OpenSRF.
On Debian and Ubuntu, the easiest way to install these prerequisites
is to use the Makefile.install prerequisite installer.

Issue the following commands as the *root* Linux account to install 
prerequisites using the Makefile.install prerequisite installer, substituting 
your operating system identifier for <osname> below:

[source, bash]
---------------------------------------------------------------------------
apt-get install make
make -f src/extras/Makefile.install <osname>
---------------------------------------------------------------------------

Well-tested values for <osname> include:

  * `debian-jessie` for Debian 8.0
  * `debian-wheezy` for Debian 7.0
  * `ubuntu-precise` for Ubuntu 12.04
  * `ubuntu-trusty` for Ubuntu 14.04
  * `fedora` for Fedora 17 and later

Patches and suggestions for improvement from users of these distributions,
or others, are welcome!

When the prerequisite installer reaches the Perl module stage, you may 
be prompted for configuration of Comprehensive Perl Archive Network (CPAN)
on your server. You can generally accept the defaults by pressing <return>
for all of the prompts, except for the country configuration.

Preamble: Developer instructions
--------------------------------

[NOTE]
Skip this section if you are using an official release tarball downloaded
from http://evergreen-ils.org/downloads

Developers working directly with the source code from the Git repository,
rather than an official release tarball, must install some extra packages
and perform one step before they can proceed with the `./configure` step.

As the *root* Linux account, install the following packages:

  * autoconf
  * automake
  * libtool

As the *user* Linux account, issue the following command in the OpenSRF
source directory to generate the configure script and Makefiles:

[source, bash]
------------------------------------------------------------------------------
autoreconf -i
------------------------------------------------------------------------------

Configuration and compilation instructions
------------------------------------------

Use the `configure` command to configure OpenSRF, and the `make` command to
build OpenSRF. The default installation prefix (PREFIX) for OpenSRF is
`/opensrf/`.

If you are building OpenSRF for Evergreen, issue the following commands as
the *user* Linux account to configure and build OpenSRF:

[source, bash]
---------------------------------------------------------------------------
./configure --prefix=/openils --sysconfdir=/openils/conf
make
---------------------------------------------------------------------------

By default, OpenSRF includes C, Perl, and JavaScript support.
You can add the `--enable-python` option to the configure command
to build Python support and `--enable-java` for Java support.

Installation instructions
-------------------------

1. Once you have configured and compiled OpenSRF, issue the following
   command as the *root* Linux account to install OpenSRF:
+
[source, bash]
---------------------------------------------------------------------------
make install
---------------------------------------------------------------------------

Create and set up the opensrf Unix user environment
---------------------------------------------------

This user is used to start and stop all OpenSRF processes, and must own all
files contained in the PREFIX directory hierarchy. Issue the following
commands as the *root* Linux account to create the `opensrf` user and set up
its environment, substituting <PREFIX> with the value you passed to `--prefix`
in your configure command:

.Creating the `opensrf` user
[source, bash]
---------------------------------------------------------------------------
useradd -m -s /bin/bash opensrf
echo "export PATH=\$PATH:/<PREFIX>/bin" >> /home/opensrf/.bashrc
passwd opensrf
chown -R opensrf:opensrf /<PREFIX>
---------------------------------------------------------------------------

Define your public and private OpenSRF domains
----------------------------------------------

For security purposes, OpenSRF uses Jabber domains to separate services
into public and private realms. Throughout these instructions, we will use
the example domains `public.localhost` and `private.localhost`. 

On a single-server system, the easiest way to define public and private
domains is to define separate hostnames by adding entries to the `/etc/hosts`
file. Here are entries that you could add to a stock `/etc/hosts` file for our
example domains:

.Example added entries for `/etc/hosts`
[source, bash]
---------------------------------------------------------------------------
127.0.1.2       public.localhost        public
127.0.1.3       private.localhost       private
---------------------------------------------------------------------------

Adjust the system dynamic library path
--------------------------------------

Add `<PREFIX>/lib/` to the system's dynamic library path, and then run
`ldconfig` as the *root* Linux account.

On Debian, Ubuntu, and Fedora systems, run the following commands as the *root*
Linux account:

.Adjusting the system dynamic library path
[source, bash]
---------------------------------------------------------------------------
echo <PREFIX>/lib > /etc/ld.so.conf.d/opensrf.conf
ldconfig
---------------------------------------------------------------------------

On most other systems, you can add these entries to `/etc/ld.so.conf`, or
create a file within the `/etc/ld.so.conf.d/` directory, and then run
`ldconfig` as the *root* Linux account.

Configure the ejabberd server
-----------------------------

OpenSRF requires an XMPP (Jabber) server. For performance reasons, ejabberd is
the Jabber server of choice for the OpenSRF project. In most cases, you only
have to make a few changes to the default configuration file to make ejabberd
work for OpenSRF. 

1. Stop ejabberd before making any changes to its configuration by issuing the
   following command as the *root* Linux account:
+
.(Debian / Ubuntu) Stopping ejabberd
[source, bash]
---------------------------------------------------------------------------
/etc/init.d/ejabberd stop
---------------------------------------------------------------------------
+
.(Fedora) Stopping ejabberd
[source, bash]
---------------------------------------------------------------------------
systemctl stop ejabberd.service
---------------------------------------------------------------------------
+
2. Edit the ejabberd config file.
+
(Debian Wheezy / Ubuntu) Ejabberd 2.x.x::
Open `/etc/ejabberd/ejabberd.cfg` and make the following
changes:
  a. Define your public and private domains in the `hosts` directive. For
   example:
+
[source, bash]
---------------------------------------------------------------------------
{hosts, ["localhost", "private.localhost", "public.localhost"]}.
---------------------------------------------------------------------------
+
  b. Change all `max_stanza_size` values to 2000000
  c. Change all `maxrate` values to 500000
  d. Increase the `max_user_sessions` value to 10000
  e. Comment out the `mod_offline` directive
+
(Debian Jessie) Ejabberd 13.x and 14.x::
Open `/etc/ejabberd/ejabberd.yml` and make the following
changes:
  a. Define your public and private domains in the `hosts` directive. For
   example:
+
[source, bash]
---------------------------------------------------------------------------
hosts:
  - "localhost"
  - "private.localhost"
  - "public.localhost"
---------------------------------------------------------------------------
+
  b. Change all `max_stanza_size` values to 2000000
  c. Change `shaper:` `normal` and `fast` values to 500000
  d. Increase the `max_user_sessions:` `all:` value to 10000
  e. Comment out the `mod_offline` directive
+
-----------------------
##mod_offline:
    ##access_max_user_messages: max_user_offline_messages
-----------------------
+
3. Restart the ejabberd server to make the changes take effect:
+
.(Debian / Ubuntu) Starting ejabberd
[source, bash]
---------------------------------------------------------------------------
/etc/init.d/ejabberd start
---------------------------------------------------------------------------
+
.(Fedora) Starting ejabberd
[source, bash]
---------------------------------------------------------------------------
systemctl start ejabberd.service
---------------------------------------------------------------------------

Create the OpenSRF Jabber users
-------------------------------

On each domain, you need two Jabber users to manage the OpenSRF communications:

  * a `router` user, to whom all requests to connect to an OpenSRF service
    will be routed; this Jabber user must be named `router`
  * an `opensrf` user, which clients use to connect to OpenSRF services; this
    user can be named anything you like

Create the Jabber users by issuing the following commands as the *root* Linux
account. Substitute `<password>` for your chosen passwords for each user
respectively:

.Creating the OpenSRF Jabber users
[source, bash]
---------------------------------------------------------------------------
ejabberdctl register router private.localhost <password>
ejabberdctl register opensrf private.localhost <password>
ejabberdctl register router public.localhost <password>
ejabberdctl register opensrf public.localhost <password>
---------------------------------------------------------------------------

Update the OpenSRF configuration files
--------------------------------------

About the OpenSRF configuration files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are several configuration files that you must update to make OpenSRF
work. SYSCONFDIR is `/opensrf/etc` by default, or the value that you passed to
`--sysconfdir` during the configuration phase.

  * `SYSCONFDIR/opensrf.xml` - this file lists the services that this
    OpenSRF installation supports; if you create a new OpenSRF service,
    you need to add it to this file.
      ** The `<hosts>` element at the bottom of the file lists the services
         that should be started for each hostname. You can force the system
         to use `localhost`, so in most cases you will leave this section
         as-is.
    
  * `SYSCONFDIR/opensrf_core.xml` - this file lists the Jabber connection
    information that will be used for the system, as well as determining
    logging verbosity and defining which services will be exposed on the
    HTTP gateway.

  * `~/.srfsh.xml` - this file gives a Linux account the ability to use
    the `srfsh` interpreter to communicate with OpenSRF services.

Updating the OpenSRF configuration files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  1. As the *opensrf* Linux account, copy the example configuration files
     to create your locally customizable OpenSRF configuration files:
+
.Copying the example OpenSRF configuration files
[source, bash]
---------------------------------------------------------------------------
cd SYSCONFDIR
cp opensrf_core.xml.example opensrf_core.xml
cp opensrf.xml.example opensrf.xml
---------------------------------------------------------------------------
+
  2. Edit the `SYSCONFDIR/opensrf_core.xml` file to update the four username
     / password pairs to match the Jabber user accounts you just created:

    a. `<config><opensrf>` = use the private Jabber `opensrf` user
    b. `<config><gateway>` = use the public Jabber `opensrf` user
    c. `<config><routers><router>` = use the public Jabber `router` user
    d. `<config><routers><router>` = use the private Jabber `router` user
  3. Create a `.srfsh.xml` file in the home directory of each user
     that you want to use `srfsh` to communicate with OpenSRF services. For
     example, to enable the *opensrf* Linux account to use `srfsh`:
    a. `cp SYSCONFDIR/srfsh.xml.example ~/.srfsh.xml`
    b. Open `~/.srfsh.xml` in your text editor of choice and update the
       password to match the password you set for the Jabber `opensrf` user
       at the `private.localhost` domain.

Starting and stopping OpenSRF services
--------------------------------------

To start all OpenSRF services with a hostname of `localhost`, issue the
following command as the *opensrf* Linux account:

[source, bash]
---------------------------------------------------------------------------
osrf_control --localhost --start-all
---------------------------------------------------------------------------

To stop all OpenSRF services with a hostname of `localhost`, issue the
following command as the *opensrf* Linux account:

[source, bash]
---------------------------------------------------------------------------
osrf_control --localhost --stop-all
---------------------------------------------------------------------------

Testing the default OpenSRF services
------------------------------------

By default, OpenSRF ships with an `opensrf.math` service that performs basic
calculations involving two integers. Once you have started the OpenSRF
services, test the services as follows:

1. Start the `srfsh` interactive OpenSRF shell by issuing the following
   command as the *opensrf* Linux account:
+
.Starting the `srfsh` interactive OpenSRF shell
[source, bash]
---------------------------------------------------------------------------
srfsh
---------------------------------------------------------------------------
+
2. Issue the following request to test the `opensrf.math` service:
+
[source, bash]
---------------------------------------------------------------------------
srfsh# request opensrf.math add 2,2
---------------------------------------------------------------------------
+
You should receive the value `4`.

Optional: Websockets installation instructions
----------------------------------------------
Websockets are new to OpenSRF 2.4+ and are required for operating the new web-based
staff client for Evergreen.  Complete the following steps as the *root* Linux 
account:

1. Install git if not already present:
+
[source, bash]
---------------------------------------------------------------------------
apt-get install git-core
---------------------------------------------------------------------------
+
2. Install the apache-websocket module:
+
[source, bash]
---------------------------------------------------------------------------
# Use a temporary directory
cd /tmp
git clone https://github.com/disconnect/apache-websocket
cd apache-websocket
apxs2 -i -a -c mod_websocket.c
---------------------------------------------------------------------------
+
3. Create the websocket Apache instance (more information about this in 
   `/usr/share/doc/apache2/README.multiple-instances`)
+
.(Debian / Ubuntu Precise)
[source, bash]
---------------------------------------------------------------------------
sh /usr/share/doc/apache2.2-common/examples/setup-instance websockets
---------------------------------------------------------------------------
+
.(Ubuntu Trusty)
[source, bash]
---------------------------------------------------------------------------
sh /usr/share/doc/apache2/examples/setup-instance websockets
---------------------------------------------------------------------------
+
4. Remove from the main apache instance
+
[source, bash]
---------------------------------------------------------------------------
a2dismod websocket
---------------------------------------------------------------------------
+
5. Copy into place the config files
+
.(Debian / Ubuntu Precise)
[source, bash]
---------------------------------------------------------------------------
cp examples/apache2/websockets/apache2.conf /etc/apache2-websockets/
---------------------------------------------------------------------------
+
.(Ubuntu Trusty)
[source, bash]
---------------------------------------------------------------------------
cp examples/apache_24/websockets/apache2.conf /etc/apache2-websockets/
---------------------------------------------------------------------------
+
6. OPTIONAL: add these configuration variables to `/etc/apache2-websockets/envvars`
   and adjust as needed.
+
[source, bash]
---------------------------------------------------------------------------
export OSRF_WEBSOCKET_IDLE_TIMEOUT=120
export OSRF_WEBSOCKET_IDLE_CHECK_INTERVAL=5
export OSRF_WEBSOCKET_CONFIG_FILE=/openils/conf/opensrf_core.xml
export OSRF_WEBSOCKET_CONFIG_CTXT=gateway
export OSRF_WEBSOCKET_MAX_REQUEST_WAIT_TIME=600
---------------------------------------------------------------------------
+
  * `IDLE_TIMEOUT` specifies how long we will allow a client to stay connected
    while idle.  A longer timeout means less network traffic (from fewer
    websocket CONNECT calls), but it also means more Apache processes are
    tied up doing nothing.
  * `IDLE_CHECK_INTERVAL` specifies how often we wake to check the idle status
    of the connected client.
  * `MAX_REQUEST_WAIT_TIME` is the maximum amount of time the gateway will
    wait before declaring a client as idle when there is a long-running
    outstanding request, yet no other activity is occurring.  This is
    primarily a fail-safe to allow idle timeouts when one or more requests
    died on the server, and thus no response was ever delivered to the gateway.
  * `CONFIG_FILE / CTXT` are the standard opensrf core config options.

7. Before you can start websockets, you must install a valid SSL certificate 
   in `/etc/apache2/ssl/`.  It is possible, but not recommended, to generate a 
   self-signed SSL certificate.  For example, if you need to test with a self-signed 
   certicate on Chrome or Chromimum browsers, one workaround is to start the browser 
   with `--ignore-certificate-errors`.

8. After OpenSRF is up and running (or after any re-install),
   fire up the secondary Apache instance. Errors will appear in
   `/var/log/apache2-websockets/error.log`. Start apache2-websockets with:
+
[source, bash]
---------------------------------------------------------------------------
/etc/init.d/apache2-websockets start
---------------------------------------------------------------------------

Troubleshooting note for Python users
-------------------------------------

If you are running a Python client and trying to connect to OpenSRF running on
localhost rather than a hostname that can be resolved via DNS, you will
probably receive exceptions about `dns.resolver.NXDOMAIN`. If this happens,
you need to install the `dnsmasq` package, configure it to serve up a DNS
entry for localhost, and point your local DNS resolver to `dnsmasq`. For example,
on Ubuntu you can issue the following commands as the *root* Linux account:

.(Debian / Ubuntu) Installing and starting `dnsmasq`
[source, bash]
---------------------------------------------------------------------------
aptitude install dnsmasq
/etc/init.d/dnsmasq restart
---------------------------------------------------------------------------

Then edit `/etc/resolv.conf` and ensure that `nameserver 127.0.0.1` is the
first entry in the file.

Getting help
------------

Need help installing or using OpenSRF? Join the mailing lists at
http://evergreen-ils.org/communicate/mailing-lists/ or contact us 
on the Freenode IRC network on the #evergreen channel.