docs: some edits to the 3.7 release notes

[Evergreen.git] / docs / RELEASE_NOTES_3_7.adoc

Evergreen 3.7-beta Release Notes
================================
:toc:
:numbered:

Upgrade notes
-------------

New Features
------------



Administration
~~~~~~~~~~~~~~



Single Sign On (Shibboleth) OPAC integration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Evergreen OPAC can now be used as a Service Provider (SP) in a
Single Sign On infrastructure.  This allows system administrators to
connect the Evergreen OPAC to an identity provider (IdP).  Such a scenario
offers significant usability improvements to patrons:

* They can use the same, IdP-provided login screen and credentials that they
use for other applications (SPs).
* If they have already logged into another participating application, when
they arrive at the Evergreen OPAC, they can be logged in without needing to
enter any credentials at all.
* Evergreen can be configured to offer a Single Sign-out service, where
logging out of the Evergreen OPAC will also log the user out of all other SPs.

It can also offer security benefits, if it enables a Shibboleth-enabled
Evergreen installation to move away from insecure autogenerated user passwords
(e.g. year of birth or last four digits of a phone number).

Different Org Units can use different IdPs.  This development also supports a
mix of Shibboleth and non-Shibboleth libraries.

Note that only the OPAC can be integrated with Shibboleth at this time; no such
support exists for the staff client, self-check, etc.

Also note that this development does not include automatic provisioning of
accounts.  At this time, matching accounts must already exist in Evergreen
for a patron to successfully authenticate into the OPAC via Single Sign On.

Installation
++++++++++++

Installing and configuring Shibboleth support is a complex project.  In
broad strokes, the process includes:

. Installing Shibboleth and the Shibboleth Apache module (`apt install libapache2-mod-shib2` on Debian and Ubuntu)
. Configuring Shibboleth, including:
  * Setting up a certificate
  * assigning an Entity ID
  * getting metadata about the IdP from the IdP (perhaps "locally maintained
  metadata", where an XML file from the IdP is copied into place on your
  Evergreen server)
  * Understanding what attributes the IdP will provide about your users,
  describing those in the `attribute-map.xml` file.
. Providing your Entity ID, information about possible bindings, and any
other requested information to the IdP administrator.  Much of this information
will be available at http://YOUR_EVERGREEN_DOMAIN/Shibboleth.sso/Metadata
. Configuring Apache, including:
  * Enabling Shibboleth authentication in the `eg_vhost.conf` file
  * (Optional) Using the new _sso_loc_ Apache variable to identify
  which org unit should be used as the context location when fetching
  Shibboleth-related library settings.
. As a user with the new `SSO_ADMIN` permission, configure Evergreen using
the Library Settings Editor, including:
  * Enable Shibboleth SSO for the OPAC
  * (Optional) Configure whether you will use SSO exclusively, or offer
  patrons a choice between SSO and standard Evergreen authentication
  * (Optional) Configure whether or not you will use Single Log Out
  * (Optional) In scenarios where a single Evergreen installation is
  connected to multiple IdPs, assign org units to the relevant IdPs,
  referenced by the IdP's Entity Id.
  * Of the attributes defined in `attribute-map.xml`, configure which one
  should be used to match users in the Evergreen database.  This defaults
  to uid.
  * For the attribute you chose in the previous step, configure which
  Evergreen field it should match against.  Options are usrname (default),
  barcode, and email.

This https://www.youtube.com/watch?v=SvppXbpv-5k[video on the SAML protocol] can
be very helpful for introducing the basic concepts used in the installation and
configuration processes.




Architecture
~~~~~~~~~~~~



Block Login of Expired Staff Accounts
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Evergreen now has the ability to prevent staff users whose
accounts have expired from logging in. This is controlled
by the new global flag "auth.block_expired_staff_login", which
is not enabled by default. If that flag is turned on, accounts
that have the `STAFF_LOGIN` permission and whose expiration date
is in the past are prevented from logging into any Evergreen
interface, including the staff client, the public catalog, and SIP2.

It should be noted that ordinary patrons are allowed to log into
the public catalog if their circulation privileges have expired. This
feature prevents expired staff users from logging into the public catalog
(and all other Evergreen interfaces and APIs) outright in order to
prevent them from getting into the staff interface anyway by
creative use of Evergreen's authentication APIs.

Evergreen admins are advised to check the expiration status of staff
accounts before turning on the global flag, as otherwise it is
possible to lock staff users out unexpectedly. The following SQL
query will identify expired but otherwise un-deleted users that
would be blocked by turning on the flag:

[source,sql]
----
SELECT DISTINCT usrname, expire_date
FROM actor.usr au, permission.usr_has_perm_at_all(id, 'STAFF_LOGIN')
WHERE active
AND NOT deleted
AND NOT barred
AND expire_date < NOW()
----

Note that this query can take a long time to run in large databases
given the general way that it checks for users that have the
`STAFF_LOGIN` permission. Replacing the use of
`permission.usr_has_perm_at_all()` with a query on expired users
with profiles known to have the `STAFF_LOGIN` permission will
be much faster.




Migration From GIST to GIN Indexes for Full Text Search
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Evergreen now uses GIN indexes for full text search in PostgreSQL.
GIN indexes offer better performance than GIST.  For more information
on the differences in the two index types, please refer to the
https://www.postgresql.org/docs/current/textsearch-indexes.html[PostgreSQL
documentation].

An upgrade script is provided as part of this migration.  If you
upgrade normally from a previous release of Evergreen, this upgrade
script should run as part of the upgrade process.  The migration
script recommends that you run a `VACUUM ANALYZE` in PostgreSQL on the
tables that had the indexes changed.  The migration process does not
do this for you, so you should do it as soon as is convenient after
the upgrade.

Updating Your Own Indexes
+++++++++++++++++++++++++

If you have added your own full text indexes of type GIST, and you
wish to migrate them to GIN, you may do so.  The following query, when
run in your Evergreen databsase after the migration from GIST to GIN,
will identify the remaining GIST indexes in your database:

[source,sql]
----------------------------------------
SELECT schemaname, indexname
FROM pg_indexes
WHERE indexdef ~* 'gist';
----------------------------------------

If the above query produces output, you can run the next query to
output a SQL script to migrate the remaining indexes from GIST to GIN:

[source,sql]
----------------------------------------
SELECT 'DROP INDEX ' || schemaname || '.' || indexname || E';\n' ||
       REGEXP_REPLACE(indexdef, 'gist', 'gin', 'i') || E';\n' ||
       'VACUUM ANAlYZE ' || schemaname || '.' || tablename || ';'
FROM pg_indexes
WHERE indexdef ~* 'gist';
----------------------------------------




Removal of Custom Dojo Build
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Evergreen had a
https://wiki.evergreen-ils.org/doku.php?id=scratchpad:random_magic_spells#custom_dojo_build[method
of making a custom build of the Dojo JavaScript library].  Followiing
this procedure could improve the load times for the OPAC and other
interfaces that use Dojo.  However, very few sites took advantage of
this process or even knew of its existence.

As a part of the process, an `openils_dojo.js` file was built and
installed along with the other Dojo files.  Evergreen had many
references to load this optional file.  For the majority of sites that
did not use this custom Dojo process, this file did not exist.
Browsers would spend time and resources requesting this nonexistent
file.  This situation also contributed noise to the Apache logs with
the 404 errors from these requests.

In keeping with the goal of eliminating Dojo from Evergreen, all
references to `openils_dojo.js` have been removed from the OPAC and
other files.  The profile script required to make the custom Dojo
build has also been removed.





Cataloging
~~~~~~~~~~



Czech language records in sample data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This release adds 7 Czech-language MARC records to the
sample data set (also known as Concerto data set).





Publisher Catalog Display Includes 264 Tag
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Publisher values are now extracted for display from tags 260 OR 264.

Upgrade Notes
+++++++++++++

A partial reingest is required to extract the new publisher data for display.
This query may be long-running.

[source,sql]
--------------------------------------------------------------------------
WITH affected_bibs AS (
    SELECT DISTINCT(bre.id) AS id
    FROM biblio.record_entry bre
    JOIN metabib.real_full_rec mrfr
    ON (mrfr.record = bre.id AND mrfr.tag = '264')
    WHERE NOT bre.deleted
)
SELECT metabib.reingest_metabib_field_entries(id, TRUE, FALSE, TRUE, TRUE)
FROM affected_bibs;
--------------------------------------------------------------------------





Circulation
~~~~~~~~~~~



Hold Groups
^^^^^^^^^^^

This feature allows staff to add multiple users to a named hold group
bucket and place title-level holds for a record for that entire set of users.
Users can be added to such a hold group bucket from either the patron
search result interface, via the Add to Bucket dropdown, or through a dedicated
Hold Group interface available from the Circulation menu.  Adding new
patrons to a hold group bucket will require staff have the PLACE_HOLD
permission.

Holds can be placed for the users in a hold group bucket either directly from
the normal staff-place hold interface in the embedded OPAC, or by supplying the
record ID within the hold group bucket interface.  In the latter case, the
list of users for which a hold was attempted but failed to be placed can be
downloaded by staff in order to address any placement issues.  Placing a
hold group bucket hold will requires staff have the MANAGE_HOLD_GROUPS
permission, which is new with this development.

In the event of a mistaken hold group hold, staff with the MANAGE_HOLD_GROUPS
permission will have the ability to cancel all unfulfilled holds created as
part of a hold group event.

A link to the title's hold interface is available from the list of hold group
events in the dedicated hold group interface.





Scan Item as Missing Pieces Angular Port
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The 'Scan Item As Missing Pieces' interface is now an Angular interface.
The functionality is the same, but the interface displays more details
on the item in question (title/author/callnum) before proceeding with the 
missing pieces process.




Opt-In Setting for Overdue and Predue Emails
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The "Receive Overdue and Courtesy Emails" user setting permits users to
control whether they receive email notifications about overdue items.

To use the setting, modify any action trigger event definitions which
send emails about overdue items, setting the "Opt In Setting" to
"circ.default_overdue_notices_enabled" and the "User Field" to "usr".
You can accomplish this by running the following query in your database:

----
UPDATE action_trigger.event_definition
SET opt_in_setting = 'circ.default_overdue_notices_enabled',
    usr_field = 'usr'
WHERE opt_in_setting IS NULL
    AND hook = 'checkout.due'
    AND reactor = 'SendEmail';
----

Once this is done, the patron registration screen in the staff client
will show a "Receive Overdue and Courtesy Emails" checkbox, which will
be checked by default.  To ensure that existing patrons continue to
recieve email notifications, you will need to add the user setting to
their accounts, which you can do by running the following query in your
database:

----
INSERT INTO actor.usr_setting (usr, name, value)
SELECT
    id,
    'circ.default_overdue_notices_enabled',
    'true'
FROM actor.usr;
----






Allow Circulation Renewal for Expired Patrons
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The "Allow renewal request if renewal recipient privileges have
expired" organizational unit setting can be set to true to permit
expired patrons to renew circulations.  Allowing renewals for expired
patrons reduces the number of auto-renewal failures and assumes that a
patron with items out eligible for renewals has not been expired for
very long and that such patrons are likely to renew their privileges
in a timely manner.

The setting is referenced based on the current circulation library for
the renewal.  It takes into account the global flags for "Circ: Use
original circulation library on desk renewal instead of the
workstation library" and "Circ: Use original circulation library on
opac renewal instead of user home library."




OPAC
~~~~



Consistent Ordering for Carousels
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Carousel ordering is now stable and predictable:

 * Newly Cataloged Item and Newest Items by Shelving Location carousels are ordered from most recently cataloged to least recently cataloged.
 * Recently Returned Item carousels is ordered is from most recently returned to least recently returned.
 * Top Circulated Items carousels is ordered is from most circulated to least circulated.
 * Manual carousels (as of now, without the ability to adjust the position of items) are in the order they are added to the backing bucket.
  ** Emptying and refilling the bucket allows reordering.





Default Public Catalog to the Bootstrap Skin
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The public catalog now defaults to the Bootstrap skin rather than the
legacy TPAC skin.

Bootstrap is now the default in order to encourage more testing, but
users should be aware of the following 
https://bugs.launchpad.net/evergreen/+bugs?field.tag=bootstrap-blocker[issues];
certain specific functionality is available only in the TPAC skin.

The TPAC skin remains available for use, but current Evergreen users
should start actively considering migrating to the Bootstrap skin.

In order to continue to use the TPAC skin, comment out the following
line in `eg_vhost.conf`

[source,conf]
-------------------
PerlAddVar OILSWebTemplatePath "@localstatedir@/templates-bootstrap" # Comment this line out to use the legacy TPAC
-------------------




Did You Mean? Single word search suggestions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This feature is the first in the series to add native search suggestions
to the Evergreen search logic.  A significant portion of the code is
dedicated to infrastructure that will be used in later enhancements to
the functionality.

Overview
++++++++
When searching the public or staff catalog in a single search class (title,
author, subject, series, identifier, or keyword) with a single search term
users can be presented with alternate search terms.  Depending on how the
instance has been configured, suggestions may be provided for only
misspelled words (as defined by existence in the bibliographic corpus),
terms that are spelled properly but occur very few times, or on every
single-term search.

Settings
++++++++

The following new library settings control the behavior of the suggestions:

* Maximum search result count at which spelling suggestions may be offered
* Minimum required uses of a spelling suggestions that may be offered
* Maximum number of spelling suggestions that may be offered
* Pg_trgm score weighting in OPAC spelling suggestions
* Soundex score weighting in OPAC spelling suggestions
* QWERTY Keyboard similarity score weighting in OPAC spelling suggestions 

There are also two new internal flags:

* symspell.prefix_length
* symspell.max_edit_distance

Upgrading
+++++++++
This feature requires the addition of new Perl module dependencies.  Please
run the app server and database server dependency Makefiles before applying
the database and code updates.

At the end of the database upgrade script, the administrator is presented
with a set of instructions necessary to precompute the suggestion
dictionary based on the current bibliographic database.  The first half
of this procedure can be started even before the upgrade begins, as soon
as the Evergreen database is no longer accessible to users that might
cause changes to bibliographic records.  For very large instances, this
dictionary generation can take several hours and needs to be run on a
server with significant RAM and CPU resources. Please look at the upgrade
script before beginning an upgrade and plan this dictionary creation as
part of the overall upgrade procedure.

Given a server, such as a database server with 64G of RAM, you should
be able to run all six of the shell commands in parallel in screen
sessions or with a tool such as GNU parallel.

These commands invoke a script that will generate a class-specific sub-
set of the dictionary, and can be used to recreate the dictionary if
necessary in the future.





Sort Holdings by Geographical Proximity
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This functionality integrates 3rd party geographic lookup services to allow patrons
to enter an address on the record details page in the OPAC and sort the holdings
for that record based on proximity of their circulating libraries to the entered
address. To support this, latitude and longitude coordinates may be associated with
each org unit. Care is given to not log or leak patron provided addresses or the
context in which they are used.

Requires the following Perl modules: `Geo::Coder::Free`, `Geo::Coder::Google`, and `Geo::Coder::OSM`




Library Groups
^^^^^^^^^^^^^^
The Library Groups search feature revives a longstanding internal
concept in Evergreen called “Lassos,” which allows an administrator
to define a group of organizational units for searching outside of
the standard organizational unit hierarchy.

Use case examples include creating a group of law or science
libraries within a university consortium, or grouping all school
libraries together within a mixed school/public library consortium.

Searches can be restricted to a particular Library Group from the
library selector in the public catalog basic search page and from
the new "Where" selector on the advanced search page.

Restricting catalog searches by Library Group is available only
in the public catalog and "traditional" staff catalog; it is not
available in the Angular staff catalog.

This feature adds a new permission, `ADMIN_LIBRARY_GROUPS`, that
allows updating Library Groups and Library Group Maps. This permission
is not associated with any profiles by default, and replaces
the `CREATE_LASSO`, `UPDATE_LASSO`, and `DELETE_LASSO` permissions. 

To define new library groups, use the Server Administration Library
Groups and Library Group Maps pages. An autogen and a reload of
Apache should be performed after making changes to Library Groups.


Easier Styling of Public Catalog Logo and Cart Images
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Evergreen now has IDs associated with logos and cart images in the TPAC and Bootstrap OPACs to aid in customization.  Images are as follows:

* small Evergreen logo in navigation bar is 'topnav_logo_image'
* the large Evergreen logo in the center of the splash page of the TPAC is 'homesearch_main_logo_image' 
* the cart icon is 'cart_icon_image' 
* the small logo in the footer is 'footer_logo_image'

The Bootstrap OPAC does not have a homesearch logo icon as it is added in the background by CSS and can be directly styled through the CSS.





Easier TPAC Customization via colors.tt2
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Twelve new colors for TPAC have been added to the colors.tt2 file as well as 
having coresponding changes to the style.css.tt2 file. These use 
descriptive rather than abstract names. These changes help avoid 
situations were unreadable values are placed on top of each other 
and where different values are wanted for elements that only refernece 
a single color previously. Guidelines are below for setting values that 
correspond to the previous values used in the colors.tt2 file.  
For more diverse customizations the OPAC should be reviewed before 
a production load.

* 'footer' is used for the background color of the footer. It replaces the 
'primary'.
* 'footer_text' sets the text color in the footer and replaces 'text_invert' 
* 'header' sets the background of the header and replaces 'primary_fade'
* 'header_text' sets the color of text in the header and replaces 'text_invert'
* 'header_links_bar' sets the background of the links bar that separates the 
header on the front page of the opac and replaces 'background_invert'
* 'header_links_text' sets the text on the links bar and replaces 'text_invert'
* 'header_links_text_hover' set the hover text color on the links bar and 
replaces 'primary'
* 'opac_button' sets the background color of the My Opac button and replaces 
'control'
* 'opac_button_text' explicitly sets the text color on the My Opac button  
* 'opac_button_hover' sets the background color of the My Opac button when the 
mouse is hovering over it and replaces 'primary'
* 'opac_button_hover_text' sets the text color of the My Opac button when the 
mouse is hovering over it and replaces 'text invert'

Note that is patch is primarily meant for users who wish to continue
using TPAC rather than the Bootstrap skin for a while; new Evergreen
users are advised to use the now-default Bootstrap skin.




Configurable Read More Accordion for OPAC Search and Record View (TPAC)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Read More Button
++++++++++++++++
Public catalog record fields (in the TPAC skin only) now truncate
themselves based on a configurable amount of characters.  The full
field may be displayed upon hitting a (Read More) link, which will
then toggle into a (Read Less) link to re-truncate the field.

Configuration
+++++++++++++
`Open-ILS/src/templates/opac/parts/config.tt2` contains two new
configuration variables:


* `truncate_contents` (default: 1)
* `contents_truncate_length` (default: 50).

Setting `truncate_contents` to 0 will disable the read more
functionality.  The variable `contents_truncate_length` corresponds
to the amount of characters to display before truncating the text.
If `contents_truncate_length` is removed, it will default to 100.

Additional configuration for note fields can be made in
`Open-ILS/src/templates/opac/parts/record/contents.tt2`, allowing a
`trunc_length` variable for each individual type of note, which will
override `contents_truncate_length` for that specific
type of note.


Adding Read More Functionality to further fields
++++++++++++++++++++++++++++++++++++++++++++++++
To add Read More functionality to any additional fields, you may use
the macro `accordion()`, defined in `misc_util.tt2`. It can take three
variables: `str`, `trunc_length`, and `element`. `str` corresponds to
the string you want to apply it to, `trunc_length` (optional) will
override `contents_truncate_length` if supplied, and `element`
(optional) provides an alternative HTML element to look at for the
truncation process (useful in situations such as the Authors and Cast
fields, where each field is processed individually, but needs to be
treated as a single field).




Reports
~~~~~~~



Reports Scheduler Improvements
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Previously, the reports scheduler allowed duplicated reports
under certain circumstances.  A uniqueness constraint now
disallows this without adversely affecting the reports process.



Miscellaneous
-------------


* The 'Create Reservation' form in the Booking module now includes
  an option to search for the patron by attributes other than just
  their barcode.
* The form to add a user to a Course now includes an option to search
  for the patron by attributes other than just their barcode.
* For consistency with the menu action Cataloging => Retrieve Record by
  TCN Value, the staff catalog Numeric Search => TCN search now includes
  deleted bib records.
* Add a new command-line script, `overdrive-api-checker.pl`, for testing
  the OverDrive API.
* The Shelving Location Groups editor is ported to Angular.
* The staff catalog now has the ability to add all search results (up to
  1,000 titles) to the basket in one fell swoop.
* Add 'All Videos' as a search format.
* Server-side print templates can now have print contexts set.
* Add ability to set the print context for a print template to "No-Print"
  to specify, well, that a given receipt should never be printed.
* Add Check Number as an available column to the Bill History grids.
* Adds a new control to the item table in the TPAC public catalog only to
  specify that only items that are available should be displayed.

Acknowledgments
---------------
The Evergreen project would like to acknowledge the following
organizations that commissioned developments in this release of
Evergreen:

* BC Libraries Cooperative
* Community Library (Sunbury)
* Consortium of Ohio Libraries (COOL)
* Evergreen Community Development Initiative
* Evergreen Indiana
* Georgia PINES
* Linn-Benton Community College
* Pennsylvania Integrated Library System (PaILS)


We would also like to thank the following individuals who contributed
code, translations, documentation, patches, and tests to this release of
Evergreen:

* Felicia Beaudry
* Galen Charlton
* Jason Etheridge
* Andrea Buntz Neiman
* Mike Rylander
* Carmen Oleskevich

We also thank the following organizations whose employees contributed
patches:

* Equinox Open Library Initiative

We regret any omissions.  If a contributor has been inadvertently
missed, please open a bug at http://bugs.launchpad.net/evergreen/
with a correction.