LP1895699: Add an on_reserve search filter This filter allows users to limit their search to records that are attached to courses at particular libraries, like so: biology on_reserve(5, 10) The filter can also be negated to search for records that are not attached to a course: biology -on_reserve(5) Also includes some additional bib records and courses to facilitate testing. A test using the concerto data set: 1) In the OPAC, run the following search at BR3: type 2) Note that you get 5 results 3) Run this search at BR3: type on_reserve(6) 4) Note that it's down to 2 results 5) Run this search at BR3: type -on_reserve(6) 6) Note that you get 3 results -- the 3 that were included in step 2 but not step 5. Another test using concerto data set: 1) Ensure that your org is opted in to the Course Materials module, and that you don't have an old value of that library setting cached. 2) Use the advanced search screen to search for `graphic` at BR3 3) Check that the "Limit to course materials" checkbox works as expected. Signed-off-by: Jane Sandberg <sandbergja@gmail.com> Signed-off-by: Terran McCanna <tmccanna@georgialibraries.org>
LP#2004055: Simpler Display Field highlighting This commit uses database functions to precompute the normalized and tokenized tsquery required for highlighting before it is returned to the user, and disallows highlight-time compilation of the highlight map. The primary purpose of this is to avoid the chance for user input to find its way directly into SQL statements, but an additional benefit is that it becomes much simpler for high level application code to make use of Display Field highlighting in non-search contexts. Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Galen Charlton <gmc@equinoxOLI.org> Signed-off-by: Jason Boyer <JBoyer@equinoxOLI.org>
Allow QP tester to run without a full stack in simple situations Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Jason Stephenson <jason@sigio.com> Signed-off-by: Galen Charlton <gmc@equinoxOLI.org> Signed-off-by: Jason Boyer <JBoyer@equinoxOLI.org>
LP#1775958: Rework pullup mechanism to flatten more nested queries The bulk of this commit reworks the query tree pullup logic, which is responsible for simplifying the query tree that is used to generate the SQL query for search. In particular, we now do a better job of finding opportunities to merge adjacent parts of the query that have the same requested_class (pre-dealiasing) in the face of boolean OR operators, explicit grouping, and alternating requested_class values. The result is fewer joins in the SQL, which should speed up all but the most trivial searches, and generally help protect the database from mis- or mal-constructed queries. We also now use CTEs to separate branches of the logical search tree into descrete subqueries, which helps reduce the total core query JOINs, and provides the planner with more options for join order. This also does away with the conversion of a negated atom into an "un-phrase". Instead, we just detect and handle those directly as atoms with a prefix, as appropriate. This allows single negated words to be used directly in the core tsquery construct, rather than having them require a separate join and special where clause. Additionally, this commit handles phrases differently at both the QP and SQL level, making use of Postgres's phrase support in modern versions and simplifying how they're handled within the base parse tree structure. Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Jason Stephenson <jason@sigio.com> Signed-off-by: Galen Charlton <gmc@equinoxOLI.org> Signed-off-by: Jason Boyer <JBoyer@equinoxOLI.org>
LP#1997485: Multi-term Did You Mean Expanding on the previous single-class, single-term search suggestion development, this feature provides suggestions for single-class searches with multiple terms. * The Library Settings that were previously used to control the global behavior of search suggestions have been moved to search class configuration fields. This was done because the data in each search class benefits from different setting values. * If a patron's search brings back a suggestion that matches an authority variant heading, the system will provide the main heading as a suggestion as well, along with spelling-corrected suggestions. * Quoated phrases in user input require strict term order and adjacency for the phrase portion of the suggestion generated for the phrase(s), whereas unquoted input (or the portion that is not quoted) does not. MARC Search/Facet Class (config.metabib_class) field additions: * variant_authority_suggestion Whether this class should attempt variant authority suggestions based on search-class/browse-axis mapping * symspell_transfer_case Whether suggestions should retain user-supplied letter case * symspell_skip_correct Only supply suggestions to misspelled words * symspell_suggestion_verbosity Setting that controls the amount of effort, and therefore time, spent on suggestion generation * max_phrase_edit_distance Maximum average per-word edit distance when evaluating suggestions * suggestion_word_option_count Maximum alternate suggestions per word * max_suggestions Maximum suggstions to present * low_result_threshold Maximum hit count beyond which suggestions are not provided * min_suggestion_use_threshold Minimum number of times a suggestion must exist in the corpus * pg_trgm_weight Weight of the trigram similarity metric; 0 avoids calculation costs * soundex_weight Weight of the soundex similarity metric; 0 avoids calculation costs * keyboard_distance_weight Weight of the keyboard distance similarity metric; 0 avoids calculation costs Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Ruth Frasur <rfrasur@library.in.gov> Signed-off-by: Galen Charlton <gmc@equinoxOLI.org>
LP#1999274: Improve Search Performance on Pg 12+ Use OFFSET as an optimization fence to keep newer PGs from trying to fold the c_attr and b_attr CTEs into the main search query. Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Jason Stephenson <jason@sigio.com> Signed-off-by: Jane Sandberg <js7389@princeton.edu>
LP#1746800 Exclude empty records on filtered staff searches Excludes empty bibs on staff searches when filtering on location or location group or when using the availability modifier. Signed-off-by: Dan Briem <dbriem@wlsmail.org> Signed-off-by: Jennifer Weston <jennifer.weston@equinoxoli.org> Signed-off-by: Mike Rylander <mrylander@gmail.com>
lp1846354 additional tweaks and fixes * fixes for SIP * patron merge & purge * also tweak some storage code, which probably isn't being used for this anywhere * remove some legacy note code * don't filter out penalties with deleted messages for Notes count in patron staff display Signed-off-by: Jason Etheridge <jason@EquinoxInitiative.org> Signed-off-by: Chris Sharp <csharp@georgialibraries.org>
LP1923076: Add cast to int in more places Add cast to int of scalar used on arrays in more places where it seems necessary. Signed-off-by: Jason Stephenson <jason@sigio.com> Signed-off-by: Michele Morgan <mmorgan@noblenet.org> Signed-off-by: Mike Rylander <mrylander@gmail.com>
LP#1815815: Library Groups This branch implements Library Groups (what used to be called "lassos") for Evergreen. Evergreen has, internally, a concept called "lassos" that allows an administrator to define a group of org units to search that has no relation to the hierarchical org tree. For instance, one might create a group of law or science libraries within a university consortium, or group all school libraries together. In addition to the previous always-visible type of Library Group (lasso), one can now make them context-aware so that that only show up if the current search location is included as one of the org units in the Library Group. This is implemented without regard to the org unit hierarchy, and so requires that the relevant ancestor and descendent org units be included in the group along with those that actually hold copies, but allows for complete flexibility in context-aware Library Group configuration. Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Ruth Frasur <rfrasur@library.in.gov> Signed-off-by: Terran McCanna <tmccanna@georgialibraries.org> Signed-off-by: Galen Charlton <gmc@equinoxinitiative.org>
LP#1882937: Comment and dead-code cleanup Adjust comments in Storage::Driver::Pg to remove references to Class::DBI::Replication, which we did not end up using. Also remove a commented-out sub that purported to make use of a function that did not exist in the home-grown multi-db implementation. NOTE: While unused, the multi-db implementation in Storage::Driver::Pg intelligently decides when to use a read-only secondary based on the existence of a transaction or the "write-iness" state controlled by the caller, and does not require separate main (or primary) and secondary db lists. Signed-off-by: Mike Rylander <mrylander@gmail.com>
LP1882937 Replace master/slave terminology In Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Driver/Pg.pm changed master/slave terminology to more neutral primary/standby. Signed-off-by: Mike Rylander <mrylander@gmail.com>
Lp 1835620: Require some Storage submodules instead of use them Switch from "use" to "require" for submodules in OpenILS::Application ::Storage::CDBI and OpenILS::Application::Storage::Driver::Pg. Circular dependencies and other issues cause the following errors at compile time for those modules: Compilation failed in require at /usr/local/share/perl/5.28.1/OpenILS/Application/Storage/CDBI.pm line 9. Compilation failed in require at /usr/local/share/perl/5.26.1/OpenILS/Application/Storage/Driver/Pg.pm line 15. Require loads the submodules at run time and does not attempt to import any methods. This is the behavior we want, eliminates the above errors, and everything still works. Add comments above the required modules explaining why require is needed and not use. Enable and reorder all of the tests in Open-ILS/src/perlmods/t/ 09-OpenILS-Application-Storage-Driver.t. Repairing the above issue also repairs these tests. Signed-off-by: Jason Stephenson <jason@sigio.com> Signed-off-by: Galen Charlton <gmc@equinoxinitiative.org> Signed-off-by: Jason Boyer <jboyer@equinoxinitiative.org>
LP#1836963: reduce the cost of utility functions, speeding up search For large org trees, some several seconds are spent testing org visibility. The immediate cause is that AppUtils::get_org_tree() does not populate the process-local cache with a memcache-cached org tree. That only happens when memcache does not have a copy of the org tree. This is obviously a simple oversight, which is addressed by making sure any memcache return value is pushed into the the process local cache. Additionally, the visibility check is making heavy use of lots of indirection and delegation to utility code, when some slightly smarter code could avoid many repeated function calls. We now supply some local utility code to flesh and unflesh the parent_ou field of objects in the org tree, allowing us to avoid using find_org() and instead just calling parent_ou() when walking "up" the tree. Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Galen Charlton <gmc@equinoxinitiative.org>
LP#1552778: copy some date/time utils from OpenSRF As preparation for subsequent bugfixes, this patch copies several date/time routines from OpenSRF::Utils to a new module, OpenILS::Utils::DateTime. Specifically, the routines copied over are: * clean_ISO8601() (renaming of the OpenSRF cleanse_ISO8601) * gmtime_ISO8601() * interval_to_seconds() * seconds_to_interval() This move will allow us to fix bugs in this core routines without requiring a mandatory OpenSRF upgrade. Furthermore, with the exception of interval_to_seconds() (and in only one place), none of those routines are used by OpenSRF itself. To test ------- [1] Apply the patch. [2] Verify that unit tests pass. [3] Verify that all Perl services start correctly. [4] Verify that date/time and interval calculations continue to work as expected, particularly in circulation. Signed-off-by: Galen Charlton <gmc@equinoxinitiative.org> Signed-off-by: Kathy Lussier <klussier@masslnc.org>
LP#1745207: tie together copy_tag() and site() search filters The copy_tag() filter will now restrict itself to records that have copies that both have matching copy tags and a circ_lib within the site() scope. To test ------- [1] Define two digital bookplate copy tags at two difference org units, e.g., "BR1 Smith" at BR1 and "BR4 Smith" at "BR4". [2] Arrange that a record has items in both BR1 and BR4, and assign the "BR4 Smith" copy tag to one of the BR4 items. [3] On a separate record, set up a similar situation, but use BR1 instead. [4] Enable the digital bookplate search option and do a bookplate search for "Smith" with the scope limited to BR1. [5] Note that both of the records set up in #2 and #3 are included in the results set. [6] Apply the patch and repeat step 5. This time, only the record set up in #3 should be included. Signed-off-by: Galen Charlton <gmc@equinoxinitiative.org> Signed-off-by: Garry Collum <gcollum@gmail.com> Signed-off-by: Chris Sharp <csharp@georgialibraries.org>
LP#1773832: Empty deleted records can be surfaced in search results There are cases where we avoid checking the deleted flag of bibs as an optimization that avoids a JOIN in the search query. However, for some shapes of real-world data, particularly in upgraded instances, deleted records can slip through. This commit adds an explicit check for the flag to all search queries. Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Daniel Pearl <dpearl@cwmars.org> Signed-off-by: Jason Stephenson <jason@sigio.com>
LP#1745233: Don't test for LURIs during copy location searches When a user is searching with a copy location or location group filter, Located URIs should not be used to include records in the result set because they are not "within" a copy location. Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Cesar Velez <cesar.velez@equinoxinitiative.org> Signed-off-by: Kathy Lussier <klussier@masslnc.org>
LP#1744385: Treat phrase terms the same as non-phrase terms WRT dictionaries Signed-off-by: Mike Rylander <mrylander@gmail.com> Signed-off-by: Kathy Lussier <klussier@masslnc.org> Signed-off-by: Dan Wells <dbw2@calvin.edu>