From dafb624be8bc31b83e8b503d248f7131fed4c580 Mon Sep 17 00:00:00 2001 From: Mike Rylander Date: Mon, 11 Sep 2017 15:24:55 -0400 Subject: [PATCH] LP#1744385: Search and Result Display improvements == Virtual Index Definitions The practical purpose of Virtual Index Definitions is to supply an Evergreen administrator with the ability to control the weighting and field inclusion of values in the general keyword index, commonly referred to as "the blob," without requiring tricky configuration that has subtle semantics, an over-abundance of index definitions which can slow search generally, or the need to reingest all records on a regular basis as experiments are performed and the configuration refined. Significant results of recasting keyword indexes as a set of one or more Virtual Index Definitions will be simpler search configuration management, faster search speed overall, and more practical reconfiguration and adjustment as needed. Previous to this commit, in order to provide field-specific weighting to keyword matches against titles or authors, an administrator must duplicate many other index definitions and supply overriding weights to those duplicates. This not only complicates configuration, but slows down record ingest as well as search. It is also fairly ineffective at achieving the goal of weighted keyword fields. Virtual Index Definitions will substantially alleviate the need for these workarounds and their consequences. * A Virtual Index Definition is not required supply any configuration for extracting bibliographic data from records, but instead can become a sink for data collected by other index definitions which is then colocated together to supply a search target made up of the separately extracted data. Virtual Index Definitions are effectively treated as aggregate definitions, matching across all values extracted from constituent non-virtual index definitions. They can further make use of the Combined class functionality to colocate all values in a class together for matching even across virtual fields. * Configuration allows for weighting of constituent index definitions that participate in a Virtual Index Definition. This weighting is separate from the weighting supplied when the index definition itself is a search target. * The Evergreen QueryParser driver returns the list of fields actually searched using every user-supplied term set, including constituent expansion when a Virtual Index Definition is searched. In particular, this will facilitate Search Term Highlighting described below. * Stock configuration changes make use of pre-existing, non-virtual index definitions mapped to new a Virtual Index Definition that implements the functionality provided by the keyword|keyword index definition. The keyword|keyword definition is left in place for the time being, until more data can be gathered about the real-world effect of removing it entirely and replacing it with Virtual Index Definition mappings. * New system administration functions will be created to facilitate modification of Virtual Index Definition mapping, avoiding the need for a full reingest when existing index definitions are added or removed from a virtual field. == Increased use of Metabib Display Fields In extention of changes proposed in other available branches, we here use Metabib Display Fields to render catalog search results, intermediate metarecord results, and record detail pages. This will requires the addition of several new Metabib Display Field definitions, as well as Perl services to gather and render the data. == Search Term Highlighting This commit enables Search Term Highlighting in the OPAC on the main search results page, the record detail page, and intermediate pages such as metarecord grouped results page. Highlighting search terms will help the user determine why a particular record (or set of records) was retrieved. Highlighting of matched terms uses the same stemming used to accomplish the search, as configured per field and class. This feature will help the user more quickly determine the relevance of a particular record by calling their attention to search terms in context. Lastly, it will help familiarize the user with how records are searched, including which fields are searched as well as exposing concepts like stemming. == Interfaces A new AngularJS "MARC Search/Facet Fields" interface has been created to replace the Dojo version, and both have been extended to support Virtual Index Definition data supplier mapping and weighting. == Settings & Permissions The new Virtual Index Definition data supplier mapping table, config.metabib_field_virtual_map, requires the same permissions as the MARC Search/Facet Fields interface: CREATE_METABIB_FIELD, UPDATE_METABIB_FIELD, DELETE_METABIB_FIELD, or ADMIN_METABIB_FIELD for all actions There is a new template-level global configuration variable in config.tt2 called search.no_highlight which disables highlighting for users of that config.tt2 instance. == Public Catalog The public and staff catalog will make use of new APIs to identify and display highlight-augmented values for those Display Fields used to render the search result pages, intermediate metarecord constituent pages, and record detail pages. Highlighting of terms will be performed using the application of Template::Toolkit-driven CSS. A generic CSS class identifying a highlighted term, along with CSS classes identifying the search class and each search field will be available for use for customization of the highlighting. A stock CSS template is provided as a baseline upon which sites may expand. When highlighting is generally enabled, it may be turned on or off on a per-page basis through the use of a UI component which will request the page again without highlighting. == Backend There now exist several new database tables and functions primarily in support of search highlighting. Additionally, the QueryParser driver for Evergreen has been augmented to be able to return a data structure describing how the search was performed, in a way that allows a separate support API to gather a highlighted version of the Display Field data for a given record. == Re-ingest or Indexing Dependencies With the addition and modification of many Index Definitions, a full reingest is recommended. However, search will continue to work as it did before the changes in this commit for those records that have not yet been reingested during that process. Therefore a slow, rolling reingest is recommended. == Performance Implications or Concerns Because the Metabib Display Fields infrastructure will eventually replace functionality that is significantly more CPU-intensive in the various forms of XML parsing, XSLT transformation, XPath calculation, and Metabib Virtual Record construction, it is expected that the overall CPU load will be reduced by this development, and ideally the overall time required to perform and render a search will likewise drop. It is unlikely that the speed increase will be visible to users on a per-search basis, but that search in aggregate will become a smaller consumer of resources. Signed-off-by: Mike Rylander Signed-off-by: Kathy Lussier Signed-off-by: Dan Wells --- Open-ILS/examples/fm_IDL.xml | 44 +- .../lib/OpenILS/Application/Search/Biblio.pm | 24 + .../Application/Storage/CDBI/config.pm | 7 + .../Storage/Driver/Pg/QueryParser.pm | 225 +- .../Application/Storage/Publisher/metabib.pm | 48 + .../Application/Storage/QueryParser.pm | 21 +- .../perlmods/lib/OpenILS/WWW/EGCatLoader.pm | 10 + Open-ILS/src/sql/Pg/002.schema.config.sql | 25 +- Open-ILS/src/sql/Pg/030.schema.metabib.sql | 136 +- .../src/sql/Pg/300.schema.staged_search.sql | 133 + Open-ILS/src/sql/Pg/950.data.seed-values.sql | 136 +- .../XXXX.data.display-field-seed-data.sql | 51 +- .../upgrade/XXXX.schema.highlight_search.sql | 559 ++ ...YYY.data.mods-title-punctuation-change.sql | 6817 +++++++++++++++++ .../upgrade/YYYY.data.virtual_index_defs.sql | 42 + .../test-scripts/query_parser.pl | 12 +- .../conify/global/config/metabib_field.tt2 | 24 +- .../config/metabib_field_virtual_map.tt2 | 77 + Open-ILS/src/templates/opac/css/style.css.tt2 | 10 + .../src/templates/opac/parts/misc_util.tt2 | 48 +- .../templates/opac/parts/record/authors.tt2 | 32 +- .../templates/opac/parts/record/contents.tt2 | 16 +- .../templates/opac/parts/record/subjects.tt2 | 37 +- .../templates/opac/parts/record/summary.tt2 | 2 +- .../src/templates/opac/parts/result/table.tt2 | 4 +- .../admin/server/config/metabib_field.tt2 | 64 + .../config/metabib_field_virtual_map.tt2 | 41 + .../admin/server/config/metabib_field.js | 93 + .../config/metabib_field_virtual_map.js | 98 + .../staff/services/fm_record_editor.js | 19 +- .../web/js/ui/default/staff/services/grid.js | 4 +- 31 files changed, 8714 insertions(+), 145 deletions(-) create mode 100644 Open-ILS/src/sql/Pg/upgrade/XXXX.schema.highlight_search.sql create mode 100644 Open-ILS/src/sql/Pg/upgrade/YYYY.data.mods-title-punctuation-change.sql create mode 100644 Open-ILS/src/sql/Pg/upgrade/YYYY.data.virtual_index_defs.sql create mode 100644 Open-ILS/src/templates/conify/global/config/metabib_field_virtual_map.tt2 create mode 100644 Open-ILS/src/templates/staff/admin/server/config/metabib_field.tt2 create mode 100644 Open-ILS/src/templates/staff/admin/server/config/metabib_field_virtual_map.tt2 create mode 100644 Open-ILS/web/js/ui/default/staff/admin/server/config/metabib_field.js create mode 100644 Open-ILS/web/js/ui/default/staff/admin/server/config/metabib_field_virtual_map.js diff --git a/Open-ILS/examples/fm_IDL.xml b/Open-ILS/examples/fm_IDL.xml index a84b556385..ae534967bd 100644 --- a/Open-ILS/examples/fm_IDL.xml +++ b/Open-ILS/examples/fm_IDL.xml @@ -2761,7 +2761,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA - + @@ -2801,7 +2801,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA - + @@ -2827,26 +2827,32 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA - - + - + + + + + + - + + + @@ -2857,6 +2863,28 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + + + + + + + + + + + + + + + + + + + + + + @@ -3736,6 +3764,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA + @@ -3839,8 +3868,7 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA oils_persist:tablename="config.display_field_map" oils_obj:fieldmapper="config::display_field_map" oils_persist:field_safe="true" - reporter:label="Display Field Map" - oils_persist:readonly="true"> + reporter:label="Display Field Map"> diff --git a/Open-ILS/src/perlmods/lib/OpenILS/Application/Search/Biblio.pm b/Open-ILS/src/perlmods/lib/OpenILS/Application/Search/Biblio.pm index 06ec97d92d..96ce5da3ee 100644 --- a/Open-ILS/src/perlmods/lib/OpenILS/Application/Search/Biblio.pm +++ b/Open-ILS/src/perlmods/lib/OpenILS/Application/Search/Biblio.pm @@ -1306,6 +1306,30 @@ sub staged_search { return cache_facets($facet_key, $new_ids, $IAmMetabib, $ignore_facet_classes) if $docache; } +sub passthrough_fetch_display_fields { + my $self = shift; + my $conn = shift; + my $highlight_map = shift; + my @records = @_; + + return $U->storagereq( + 'open-ils.storage.fetch.metabib.display_field.highlight', + $highlight_map, + @records + ) if (@records == 1); + + return $U->storagereq( + 'open-ils.storage.fetch.metabib.display_field.highlight.atomic', + $highlight_map, + \@records + ); +} +__PACKAGE__->register_method( + method => 'passthrough_fetch_display_fields', + api_name => 'open-ils.search.fetch.metabib.display_field.highlight' +); + + sub tag_circulated_records { my ($auth, $results, $metabib) = @_; my $e = new_editor(authtoken => $auth); diff --git a/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/CDBI/config.pm b/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/CDBI/config.pm index 12f4f45223..e91ff207ea 100644 --- a/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/CDBI/config.pm +++ b/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/CDBI/config.pm @@ -34,6 +34,13 @@ __PACKAGE__->columns(Primary => 'id'); __PACKAGE__->columns(Essential => qw/field_class name xpath weight format search_field facet_field display_xpath display_field/); #------------------------------------------------------------------------------- +package config::metabib_field_virtual_map; +use base qw/config/; +__PACKAGE__->table('config_metabib_field_virtual_map'); +__PACKAGE__->columns(Primary => 'id'); +__PACKAGE__->columns(Essential => qw/real virtual/); +#------------------------------------------------------------------------------- + package config::identification_type; use base qw/config/; __PACKAGE__->table('config_identification_type'); diff --git a/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Driver/Pg/QueryParser.pm b/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Driver/Pg/QueryParser.pm index 4b4548397e..a763f4dd1c 100644 --- a/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Driver/Pg/QueryParser.pm +++ b/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Driver/Pg/QueryParser.pm @@ -8,6 +8,8 @@ use OpenSRF::Utils qw/:datetime/; use OpenSRF::Utils::JSON; use OpenILS::Application::AppUtils; use OpenILS::Utils::CStoreEditor; +use OpenSRF::Utils::Logger qw($logger); +use Data::Dumper; my $U = 'OpenILS::Application::AppUtils'; my ${spc} = ' ' x 2; @@ -228,6 +230,15 @@ sub search_field_id_map { return $self->custom_data->{search_field_id_map}; } +sub search_field_virtual_map { + my $self = shift; + my $map = shift; + + $self->custom_data->{search_field_virtual_map} ||= {}; + $self->custom_data->{search_field_virtual_map} = $map if ($map); + return $self->custom_data->{search_field_virtual_map}; +} + sub add_search_field_id_map { my $self = shift; my $class = shift; @@ -246,6 +257,18 @@ sub add_search_field_id_map { }; } +sub add_search_field_virtual_map { + my $self = shift; + my $realid = shift; + my $virtid = shift; + + $self->search_field_virtual_map->{by_virt}{$virtid} ||= []; + push @{$self->search_field_virtual_map->{by_virt}{$virtid}}, $realid; + + $self->search_field_virtual_map->{by_real}{$realid} ||= []; + push @{$self->search_field_virtual_map->{by_real}{$realid}}, $virtid; +} + sub search_field_class_by_id { my $self = shift; my $id = shift; @@ -400,6 +423,17 @@ sub initialize_search_field_id_map { return $self->search_field_id_map; } +sub initialize_search_field_virtual_map { + my $self = shift; + my $cmfvm_list = shift; + + __PACKAGE__->add_search_field_virtual_map( $_->real, $_->virtual ) + for (@$cmfvm_list); + + $logger->debug('Virtual field map: ' . Dumper($self->search_field_virtual_map)); + return $self->search_field_virtual_map; +} + sub initialize_aliases { my $self = shift; my $cmsa_list = shift; @@ -513,6 +547,9 @@ sub initialize { $self->initialize_search_field_id_map( $args{config_metabib_field} ) if ($args{config_metabib_field}); + $self->initialize_search_field_virtual_map( $args{config_metabib_field_virtual_map} ) + if ($args{config_metabib_field_virtual_map}); + $self->initialize_aliases( $args{config_metabib_search_alias} ) if ($args{config_metabib_search_alias}); @@ -598,6 +635,8 @@ sub TEST_SETUP { __PACKAGE__->add_relevance_bump( keyword => keyword => first_word => 1 ); __PACKAGE__->add_relevance_bump( keyword => keyword => full_match => 1 ); + __PACKAGE__->add_search_field_virtual_map( 6 => 15 ); + __PACKAGE__->class_ts_config( 'series', undef, 1, 'english_nostop' ); __PACKAGE__->class_ts_config( 'title', undef, 1, 'english_nostop' ); __PACKAGE__->class_ts_config( 'author', undef, 1, 'english_nostop' ); @@ -635,6 +674,8 @@ sub TEST_SETUP { __PACKAGE__->add_search_field_alias( subject => name => 'bib.subjectName' ); + #__PACKAGE__->search_class_combined( keyword => 1 ); + __PACKAGE__->search_class_combined( author => 1 ); } __PACKAGE__->default_search_class( 'keyword' ); @@ -1169,17 +1210,6 @@ sub flatten { next; } - my $table = $node->table; - my $ctable = $node->combined_table; - my $talias = $node->table_alias; - - my $node_rank = 'COALESCE(' . $node->rank . " * ${talias}.weight, 0.0)"; - - $from .= "\n" . ${spc} x 4 ."LEFT JOIN (\n" - . ${spc} x 5 . "SELECT fe.*, fe_weight.weight, ${talias}_xq.tsq, ${talias}_xq.tsq_rank /* search */\n" - . ${spc} x 6 . "FROM $table AS fe"; - $from .= "\n" . ${spc} x 7 . "JOIN config.metabib_field AS fe_weight ON (fe_weight.id = fe.field)"; - my @bump_fields; my @field_ids; if (@{$node->fields} > 0) { @@ -1196,22 +1226,37 @@ sub flatten { @bump_fields = @{$self->QueryParser->search_fields->{$node->classname}}; } + # use search_field_list to handle virtual index defs + my $search_field_list = $self->QueryParser->search_field_ids_by_class($node->classname); + $search_field_list = [@field_ids] if (@field_ids); + + my $table = $node->table; + my $ctable = $node->combined_table; + my $talias = $node->table_alias; + + my $node_rank = 'COALESCE(' . $node->rank . " * ${talias}.weight * 1000, 0.0)"; + + $from .= "\n" . ${spc} x 4 ."LEFT JOIN (\n" + . ${spc} x 5 . "SELECT fe.*, fe_weight.weight, ${talias}_xq.tsq, ${talias}_xq.tsq_rank /* search */\n" + . ${spc} x 6 . "FROM $table AS fe\n" + . ${spc} x 7 . "JOIN config.metabib_field AS fe_weight ON (fe_weight.id = fe.field)"; + if ($node->dummy_count < @{$node->only_atoms} ) { $with .= ",\n " if $with; $with .= "${talias}_xq AS (SELECT ". $node->tsquery ." AS tsq,". $node->tsquery_rank ." AS tsq_rank )"; if ($node->combined_search) { - $from .= "\n" . ${spc} x 6 . "JOIN $ctable AS com ON (com.record = fe.source"; + $from .= "\n" . ${spc} x 7 . "JOIN $ctable AS com ON (com.record = fe.source"; if (@field_ids) { $from .= " AND com.metabib_field IN (" . join(',',@field_ids) . "))"; } else { $from .= " AND com.metabib_field IS NULL)"; } - $from .= "\n" . ${spc} x 6 . "JOIN ${talias}_xq ON (com.index_vector @@ ${talias}_xq.tsq)"; + $from .= "\n" . ${spc} x 7 . "JOIN ${talias}_xq ON (com.index_vector @@ ${talias}_xq.tsq)"; } else { - $from .= "\n" . ${spc} x 6 . "JOIN ${talias}_xq ON (fe.index_vector @@ ${talias}_xq.tsq)"; + $from .= "\n" . ${spc} x 7 . "JOIN ${talias}_xq ON (fe.index_vector @@ ${talias}_xq.tsq)"; } } else { - $from .= "\n" . ${spc} x 6 . ", (SELECT NULL::tsquery AS tsq, NULL:tsquery AS tsq_rank ) AS ${talias}_xq"; + $from .= "\n" . ${spc} x 7 . ", (SELECT NULL::tsquery AS tsq, NULL::tsquery AS tsq_rank ) AS ${talias}_xq"; } if (@field_ids) { @@ -1219,6 +1264,47 @@ sub flatten { join(',', @field_ids) . ")"; } + # Even though virtual fields have all the real field data in + # their combined version, and thus a search against the real + # fields is not necessary to match records, we still want to + # UNION them in so we can get their virtual weight if they + # would match a search directly against them. + if ($node->dummy_count < @{$node->only_atoms} ) { # no point in searching real fields with no search terms + for my $possible_vfield (@$search_field_list) { + my $real_fields = $self->QueryParser->search_field_virtual_map->{by_virt}->{$possible_vfield}; + if ($real_fields and @$real_fields) { # this is a virt field + # UNION in the others ... group by class? + for my $real_field (@$real_fields) { + $node->add_vfield($real_field); + $logger->debug("Looking up virtual field for real field $real_field"); + my $vclass = $self->QueryParser->search_field_class_by_id($real_field)->{classname}; + my $vtable = $node->table($vclass); + my $vfield = 'field'; + my $vrecord = 'source'; + $from .= "\n" . ${spc} x 8 . "UNION ALL\n"; + + if ($node->combined_search) { # real fields inherit combine'dness from the virtual field + $vtable = $node->combined_table($vclass); + $vfield = 'metabib_field'; + $vrecord = 'record'; + $from .= ${spc} x 5 . "SELECT 0::BIGINT AS id, fe.record AS source, fe.metabib_field AS field, " + . "'' AS value, fe.index_vector, fe_weight.weight, ${talias}_xq.tsq, ${talias}_xq.tsq_rank /* virtual field addition */\n"; + } else { + $from .= ${spc} x 5 . "SELECT fe.id, fe.source, fe.field, fe.value, fe.index_vector, " + . "fe_weight.weight, ${talias}_xq.tsq, ${talias}_xq.tsq_rank /* virtual field addition */\n"; + } + + $from .= ${spc} x 6 . "FROM $vtable AS fe\n" + . ${spc} x 7 . "JOIN config.metabib_field_virtual_map AS fe_weight ON (" + ."fe_weight.virtual = $possible_vfield AND " + ."fe_weight.real = $real_field AND " + ."fe_weight.real = fe.$vfield)\n" + . ${spc} x 7 . "JOIN ${talias}_xq ON (fe.index_vector @@ ${talias}_xq.tsq)"; + } + } + } + } + $from .= "\n" . ${spc} x 4 . ") AS $talias ON (m.source = ${talias}.source)"; my %used_bumps; @@ -1239,9 +1325,9 @@ sub flatten { if(scalar @bumps > 0 && scalar @{$node->only_positive_atoms} > 0) { # Note: Previous rank function used search_normalize outright. Duplicating that here. - $node_rank .= "\n" . ${spc} x 5 . "* evergreen.rel_bump(('{' || quote_literal(search_normalize("; + $node_rank .= "\n" . ${spc} x 5 . "* COALESCE(evergreen.rel_bump(('{' || quote_literal(search_normalize("; $node_rank .= join(")) || ',' || quote_literal(search_normalize(",map { $self->QueryParser->quote_phrase_value($_->content) } @{$node->only_positive_atoms}); - $node_rank .= ")) || '}')::TEXT[], " . $node->table_alias . ".value, '{" . join(",",@bumps) . "}'::TEXT[], '{" . join(",",@bumpmults) . "}'::NUMERIC[])"; + $node_rank .= ")) || '}')::TEXT[], " . $node->table_alias . ".value, '{" . join(",",@bumps) . "}'::TEXT[], '{" . join(",",@bumpmults) . "}'::NUMERIC[]),1.0)"; } my $NOT = ''; @@ -1665,11 +1751,13 @@ sub fields { sub table_alias { my $self = shift; + my $suffix = shift; my $table_alias = "$self"; $table_alias =~ s/^.*\(0(x[0-9a-fA-F]+)\)$/$1/go; $table_alias .= '_' . $self->name; $table_alias =~ s/\|/_/go; + $table_alias .= "_$suffix" if ($suffix); return $table_alias; } @@ -1777,6 +1865,86 @@ sub buildSQL { #------------------------------- package OpenILS::Application::Storage::Driver::Pg::QueryParser::query_plan::node; use base 'QueryParser::query_plan::node'; +use List::MoreUtils qw/uniq/; +use Data::Dumper; + +sub abstract_node_additions { + my $self = shift; + my $aq = shift; + + my $hm = $self->plan + ->QueryParser + ->parse_tree + ->get_abstract_data('highlight_map') || {}; + + my $field_set = $self->fields; + $field_set = $self->plan->QueryParser->search_fields->{$self->classname} + if (!@$field_set); + + my @field_ids = grep defined, ( + map { + $self->plan->QueryParser->search_field_ids_by_class( + $self->classname, $_ + )->[0] + } @$field_set + ); + + push @field_ids, @{$self->{vfields}} if $self->{vfields}; + + my $ts_query = $self->tsquery_rank; + + # We need to rework the hash so fields are only ever pointed at once. + # That means if a field is already being looked at elsewhere then we'll + # need to separate it out and combine its preexisting tsqueries. This + # will be fairly brute-force, and could be improved later, likely, with + # a clever algorithm. + + my %inverted_hm; + for my $t (keys %$hm) { + for my $f (@{$$hm{$t}}) { + $inverted_hm{$f} = $t; + } + } + + # Then, loop over new fields and put them in the inverted hash. + my @existing_fields = keys %inverted_hm; + + for my $f (@field_ids) { + if (grep { $f == $_ } @existing_fields) { # We've seen the field, should we combine? + my $t = $inverted_hm{$f}; + if ($t ne $ts_query) { # Different tsquery, do it! + $t .= ' || '. $ts_query; + $inverted_hm{$f} = $t; + } + } else { # New field + $inverted_hm{$f} = $ts_query; + } + } + + # Now, flip it back over. + $hm = {}; + for my $f (keys %inverted_hm) { + my $t = $inverted_hm{$f}; + if ($$hm{$t}) { + push @{$$hm{$t}}, $f; + } else { + $$hm{$t} = [$f]; + } + } + + $self->plan + ->QueryParser + ->parse_tree + ->set_abstract_data('highlight_map', $hm); +} + +sub add_vfield { + my $self = shift; + my $vfield = shift; + + $self->{vfields} ||= []; + push @{$self->{vfields}}, $vfield; +} sub only_atoms { my $self = shift; @@ -1824,18 +1992,14 @@ sub dummy_count { sub table { my $self = shift; - my $table = shift; - $self->{table} = $table if ($table); - return $self->{table} if $self->{table}; - return $self->table( 'metabib.' . $self->classname . '_field_entry' ); + my $classname = shift || $self->classname; + return 'metabib.' . $classname . '_field_entry'; } sub combined_table { my $self = shift; - my $ctable = shift; - $self->{ctable} = $ctable if ($ctable); - return $self->{ctable} if $self->{ctable}; - return $self->combined_table( 'metabib.combined_' . $self->classname . '_field_entry' ); + my $classname = shift || $self->classname; + return 'metabib.combined_' . $classname . '_field_entry'; } sub combined_search { @@ -1845,16 +2009,15 @@ sub combined_search { sub table_alias { my $self = shift; - my $table_alias = shift; - $self->{table_alias} = $table_alias if ($table_alias); - return $self->{table_alias} if ($self->{table_alias}); + my $suffix = shift; - $table_alias = "$self"; + my $table_alias = "$self"; $table_alias =~ s/^.*\(0(x[0-9a-fA-F]+)\)$/$1/go; $table_alias .= '_' . $self->requested_class; $table_alias =~ s/\|/_/go; + $table_alias .= "_$suffix" if ($suffix); - return $self->table_alias( $table_alias ); + return $table_alias; } sub tsquery { @@ -1881,7 +2044,7 @@ sub tsquery_rank { push @atomlines, "\n" . ${spc} x 3 . $atom->sql; } $self->{tsquery_rank} = join(' ||', @atomlines); - $self->{tsquery_rank} = 'NULL::tsquery' unless $self->{tsquery_rank}; + $self->{tsquery_rank} = "''::tsquery" unless $self->{tsquery_rank}; return $self->{tsquery_rank}; } diff --git a/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Publisher/metabib.pm b/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Publisher/metabib.pm index c8c3ea76e7..eb3f423279 100644 --- a/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Publisher/metabib.pm +++ b/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Publisher/metabib.pm @@ -40,6 +40,11 @@ sub _initialize_parser { 'open-ils.cstore.direct.config.metabib_field.search.atomic', { id => { "!=" => undef } } )->gather(1), + config_metabib_field_virtual_map => + $cstore->request( + 'open-ils.cstore.direct.config.metabib_field_virtual_map.search.atomic', + { id => { "!=" => undef } } + )->gather(1), config_metabib_search_alias => $cstore->request( 'open-ils.cstore.direct.config.metabib_search_alias.search.atomic', @@ -87,6 +92,49 @@ sub _initialize_parser { die("Cannot initialize $parser!") unless ($parser->initialization_complete); } +sub fetch_highlighted_display_fields { + my $self = shift; + my $client = shift; + my $records = shift; + my $highlight_map = shift; + + unless ($records) { + $client->respond_complete; + return; + } + + my $hl_map_string = "''::HSTORE"; + if (ref($highlight_map)) { + $hl_map_string = ""; + for my $tsq (keys %$highlight_map) { + my $field_list = join(',', @{$$highlight_map{$tsq}}); + $hl_map_string .= ' || ' if $hl_map_string; + $hl_map_string .= "hstore(($tsq)\:\:TEXT,'$field_list')"; + } + } + + my $sth = metabib::metarecord_source_map->db_Main->prepare( + "SELECT * FROM search.highlight_display_fields(?, $hl_map_string)" + ); + + $records = [$records] unless ref($records); + for my $record ( @$records ) { + next unless $record; + $sth->execute($record); + my $rows = $sth->fetchall_arrayref({}); + $client->respond($rows); + } + + return undef; +} +__PACKAGE__->register_method( + api_name => 'open-ils.storage.fetch.metabib.display_field.highlight', + method => 'fetch_highlighted_display_fields', + api_level => 1, + stream => 1 +); + + sub ordered_records_from_metarecord { # XXX Replace with QP-based search-within-MR my $self = shift; my $client = shift; diff --git a/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/QueryParser.pm b/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/QueryParser.pm index 090b944f7c..199a4152c0 100644 --- a/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/QueryParser.pm +++ b/Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/QueryParser.pm @@ -1520,6 +1520,19 @@ package QueryParser::query_plan; use Data::Dumper; $Data::Dumper::Indent = 0; +sub get_abstract_data { + my $self = shift; + my $key = shift; + return $self->{abstract_data}{$key}; +} + +sub set_abstract_data { + my $self = shift; + my $key = shift; + my $value = shift; + $self->{abstract_data}{$key} = $value; +} + sub atoms_only { my $self = shift; return @{$self->filters} == 0 && @@ -1741,7 +1754,7 @@ sub QueryParser { sub new { my $pkg = shift; $pkg = ref($pkg) || $pkg; - my %args = (query => [], joiner => '&', @_); + my %args = (abstract_data => {}, query => [], joiner => '&', @_); return bless \%args => $pkg; } @@ -2076,6 +2089,9 @@ sub to_abstract_query { } $abstract_query->{children} ||= { QueryParser::_util::default_joiner() => $kids }; + $$abstract_query{additional_data} = $self->{abstract_data} + if (keys(%{$self->{abstract_data}})); + return $abstract_query; } @@ -2301,6 +2317,9 @@ sub to_abstract_query { "fields" => $self->fields }; + $self->abstract_node_additions($abstract_query) + if ($self->can('abstract_node_additions')); + my $kids = []; for my $qatom (@{$self->query_atoms}) { diff --git a/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader.pm b/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader.pm index 6aa1f5811d..e386d4375a 100644 --- a/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader.pm +++ b/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader.pm @@ -341,6 +341,16 @@ sub load_common { $self->load_org_util_funcs; $self->load_perm_funcs; + $ctx->{fetch_display_fields} = sub { + my $id = shift; + return $U->simplereq( + 'open-ils.search', + 'open-ils.search.fetch.metabib.display_field.highlight', + $id, + $ctx->{query_struct}{additional_data}{highlight_map} + ); + }; + return Apache2::Const::OK; } diff --git a/Open-ILS/src/sql/Pg/002.schema.config.sql b/Open-ILS/src/sql/Pg/002.schema.config.sql index 491f3bb7f7..09b59cf9ba 100644 --- a/Open-ILS/src/sql/Pg/002.schema.config.sql +++ b/Open-ILS/src/sql/Pg/002.schema.config.sql @@ -202,7 +202,7 @@ CREATE TABLE config.metabib_field ( field_class TEXT NOT NULL REFERENCES config.metabib_class (name), name TEXT NOT NULL, label TEXT NOT NULL, - xpath TEXT NOT NULL, + xpath TEXT, weight INT NOT NULL DEFAULT 1, format TEXT NOT NULL REFERENCES config.xml_transform (name) DEFAULT 'mods33', search_field BOOL NOT NULL DEFAULT TRUE, @@ -215,7 +215,7 @@ CREATE TABLE config.metabib_field ( authority_xpath TEXT, joiner TEXT, restrict BOOL DEFAULT FALSE NOT NULL, - display_field BOOL NOT NULL DEFAULT FALSE + display_field BOOL NOT NULL DEFAULT TRUE ); COMMENT ON TABLE config.metabib_field IS $$ XPath used for record indexing ingest @@ -226,6 +226,27 @@ a "class" of either title, subject, author, keyword, series or identifier. $$; +CREATE TABLE config.metabib_field_virtual_map ( + id SERIAL PRIMARY KEY, + real INT NOT NULL REFERENCES config.metabib_field (id), + virtual INT NOT NULL REFERENCES config.metabib_field (id), + weight INT NOT NULL DEFAULT 1 +); +COMMENT ON TABLE config.metabib_field_virtual_map IS $$ +Maps between real (physically extracted) index definitions +and virtual (target sync, no required extraction of its own) +index definitions. + +The virtual side may not extract any data of its own, but +will collect data from all of the real fields. This reduces +extraction (ingest) overhead by eliminating duplcated extraction, +and allows for searching across novel combinations of fields, such +as names used as either subjects or authors. By preserving this +mapping rather than defining duplicate extractions, information +about the originating, "real" index definitions can be used +in interesting ways, such as highlighting in search results. +$$; + CREATE UNIQUE INDEX config_metabib_field_class_name_idx ON config.metabib_field (field_class, name); CREATE TABLE config.display_field_map ( diff --git a/Open-ILS/src/sql/Pg/030.schema.metabib.sql b/Open-ILS/src/sql/Pg/030.schema.metabib.sql index 499073b943..2f37b555d4 100644 --- a/Open-ILS/src/sql/Pg/030.schema.metabib.sql +++ b/Open-ILS/src/sql/Pg/030.schema.metabib.sql @@ -173,6 +173,19 @@ CREATE UNIQUE INDEX metabib_combined_series_field_entry_fakepk_idx ON metabib.co CREATE INDEX metabib_combined_series_field_entry_index_vector_idx ON metabib.combined_series_field_entry USING GIST (index_vector); CREATE INDEX metabib_combined_series_field_source_idx ON metabib.combined_series_field_entry (metabib_field); +CREATE VIEW metabib.combined_all_field_entry AS + SELECT * FROM metabib.combined_title_field_entry + UNION ALL + SELECT * FROM metabib.combined_author_field_entry + UNION ALL + SELECT * FROM metabib.combined_subject_field_entry + UNION ALL + SELECT * FROM metabib.combined_keyword_field_entry + UNION ALL + SELECT * FROM metabib.combined_identifier_field_entry + UNION ALL + SELECT * FROM metabib.combined_series_field_entry; + CREATE TABLE metabib.facet_entry ( id BIGSERIAL PRIMARY KEY, source BIGINT NOT NULL, @@ -230,28 +243,76 @@ CREATE VIEW metabib.compressed_display_entry AS CREATE VIEW metabib.wide_display_entry AS /* Table-like view of well-known display fields. This VIEW expands as well-known display fields are added. */ - SELECT + SELECT bre.id AS source, COALESCE(mcde_title.value, 'null') AS title, COALESCE(mcde_author.value, 'null') AS author, - COALESCE(mcde_subject.value, 'null') AS subject, + COALESCE(mcde_subject_geographic.value, 'null') AS subject_geographic, + COALESCE(mcde_subject_name.value, 'null') AS subject_name, + COALESCE(mcde_subject_temporal.value, 'null') AS subject_temporal, + COALESCE(mcde_subject_topic.value, 'null') AS subject_topic, COALESCE(mcde_creators.value, 'null') AS creators, - COALESCE(mcde_isbn.value, 'null') AS isbn - -- ensure one row per bre regardless of the presence of display entries - FROM biblio.record_entry bre - LEFT JOIN metabib.compressed_display_entry mcde_title + COALESCE(mcde_isbn.value, 'null') AS isbn, + COALESCE(mcde_issn.value, 'null') AS issn, + COALESCE(mcde_upc.value, 'null') AS upc, + COALESCE(mcde_tcn.value, 'null') AS tcn, + COALESCE(mcde_edition.value, 'null') AS edition, + COALESCE(mcde_physical_description.value, 'null') AS physical_description, + COALESCE(mcde_publisher.value, 'null') AS publisher, + COALESCE(mcde_series_title.value, 'null') AS series_title, + COALESCE(mcde_abstract.value, 'null') AS abstract, + COALESCE(mcde_toc.value, 'null') AS toc, + COALESCE(mcde_pubdate.value, 'null') AS pubdate, + COALESCE(mcde_type_of_resource.value, 'null') AS type_of_resource + FROM biblio.record_entry bre + LEFT JOIN metabib.compressed_display_entry mcde_title ON (bre.id = mcde_title.source AND mcde_title.name = 'title') - LEFT JOIN metabib.compressed_display_entry mcde_author + LEFT JOIN metabib.compressed_display_entry mcde_author ON (bre.id = mcde_author.source AND mcde_author.name = 'author') - LEFT JOIN metabib.compressed_display_entry mcde_subject + LEFT JOIN metabib.compressed_display_entry mcde_subject ON (bre.id = mcde_subject.source AND mcde_subject.name = 'subject') - LEFT JOIN metabib.compressed_display_entry mcde_creators + LEFT JOIN metabib.compressed_display_entry mcde_subject_geographic + ON (bre.id = mcde_subject_geographic.source + AND mcde_subject_geographic.name = 'subject_geographic') + LEFT JOIN metabib.compressed_display_entry mcde_subject_name + ON (bre.id = mcde_subject_name.source + AND mcde_subject_name.name = 'subject_name') + LEFT JOIN metabib.compressed_display_entry mcde_subject_temporal + ON (bre.id = mcde_subject_temporal.source + AND mcde_subject_temporal.name = 'subject_temporal') + LEFT JOIN metabib.compressed_display_entry mcde_subject_topic + ON (bre.id = mcde_subject_topic.source + AND mcde_subject_topic.name = 'subject_topic') + LEFT JOIN metabib.compressed_display_entry mcde_creators ON (bre.id = mcde_creators.source AND mcde_creators.name = 'creators') - LEFT JOIN metabib.compressed_display_entry mcde_isbn + LEFT JOIN metabib.compressed_display_entry mcde_isbn ON (bre.id = mcde_isbn.source AND mcde_isbn.name = 'isbn') + LEFT JOIN metabib.compressed_display_entry mcde_issn + ON (bre.id = mcde_issn.source AND mcde_issn.name = 'issn') + LEFT JOIN metabib.compressed_display_entry mcde_upc + ON (bre.id = mcde_upc.source AND mcde_upc.name = 'upc') + LEFT JOIN metabib.compressed_display_entry mcde_tcn + ON (bre.id = mcde_tcn.source AND mcde_tcn.name = 'tcn') + LEFT JOIN metabib.compressed_display_entry mcde_edition + ON (bre.id = mcde_edition.source AND mcde_edition.name = 'edition') + LEFT JOIN metabib.compressed_display_entry mcde_physical_description + ON (bre.id = mcde_physical_description.source + AND mcde_physical_description.name = 'physical_description') + LEFT JOIN metabib.compressed_display_entry mcde_publisher + ON (bre.id = mcde_publisher.source AND mcde_publisher.name = 'publisher') + LEFT JOIN metabib.compressed_display_entry mcde_series_title + ON (bre.id = mcde_series_title.source AND mcde_series_title.name = 'series_title') + LEFT JOIN metabib.compressed_display_entry mcde_abstract + ON (bre.id = mcde_abstract.source AND mcde_abstract.name = 'abstract') + LEFT JOIN metabib.compressed_display_entry mcde_toc + ON (bre.id = mcde_toc.source AND mcde_toc.name = 'toc') + LEFT JOIN metabib.compressed_display_entry mcde_pubdate + ON (bre.id = mcde_pubdate.source AND mcde_pubdate.name = 'pubdate') + LEFT JOIN metabib.compressed_display_entry mcde_type_of_resource + ON (bre.id = mcde_type_of_resource.source + AND mcde_type_of_resource.name = 'type_of_resource') ; - CREATE TABLE metabib.browse_entry ( id BIGSERIAL PRIMARY KEY, value TEXT, @@ -719,13 +780,14 @@ BEGIN -- Loop over the indexing entries FOR idx IN SELECT * FROM config.metabib_field WHERE id = ANY (only_fields) ORDER BY format LOOP + CONTINUE WHEN idx.xpath IS NULL OR idx.xpath = ''; -- pure virtual field process_idx := FALSE; IF idx.display_field AND 'display' = ANY (field_types) THEN process_idx = TRUE; END IF; IF idx.browse_field AND 'browse' = ANY (field_types) THEN process_idx = TRUE; END IF; IF idx.search_field AND 'search' = ANY (field_types) THEN process_idx = TRUE; END IF; IF idx.facet_field AND 'facet' = ANY (field_types) THEN process_idx = TRUE; END IF; - CONTINUE WHEN process_idx = FALSE; + CONTINUE WHEN process_idx = FALSE; -- disabled for all types joiner := COALESCE(idx.joiner, default_joiner); @@ -884,10 +946,14 @@ BEGIN END LOOP; END; - $func$ LANGUAGE PLPGSQL; CREATE OR REPLACE FUNCTION metabib.update_combined_index_vectors(bib_id BIGINT) RETURNS VOID AS $func$ +DECLARE + rdata TSVECTOR; + vclass TEXT; + vfield INT; + rfields INT[]; BEGIN DELETE FROM metabib.combined_keyword_field_entry WHERE record = bib_id; INSERT INTO metabib.combined_keyword_field_entry(record, metabib_field, index_vector) @@ -937,6 +1003,32 @@ BEGIN SELECT bib_id, NULL, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) FROM metabib.identifier_field_entry WHERE source = bib_id; + -- For each virtual def, gather the data from the combined real field + -- entries and append it to the virtual combined entry. + FOR vfield, rfields IN SELECT virtual, ARRAY_AGG(real) FROM config.metabib_field_virtual_map GROUP BY virtual LOOP + SELECT field_class INTO vclass + FROM config.metabib_field + WHERE id = vfield; + + SELECT string_agg(index_vector::TEXT,' ')::tsvector INTO rdata + FROM metabib.combined_all_field_entry + WHERE record = bib_id + AND metabib_field = ANY (rfields); + + BEGIN -- I cannot wait for INSERT ON CONFLICT ... 9.5, though + EXECUTE $$ + INSERT INTO metabib.combined_$$ || vclass || $$_field_entry + (record, metabib_field, index_vector) VALUES ($1, $2, $3) + $$ USING bib_id, vfield, rdata; + EXCEPTION WHEN unique_violation THEN + EXECUTE $$ + UPDATE metabib.combined_$$ || vclass || $$_field_entry + SET index_vector = index_vector || $3 + WHERE record = $1 + AND metabib_field = $2 + $$ USING bib_id, vfield, rdata; + END; + END LOOP; END; $func$ LANGUAGE PLPGSQL; @@ -1404,7 +1496,12 @@ BEGIN END; $func$ LANGUAGE PLPGSQL; -CREATE OR REPLACE FUNCTION metabib.remap_metarecord_for_bib( bib_id BIGINT, fp TEXT, bib_is_deleted BOOL DEFAULT FALSE, retain_deleted BOOL DEFAULT FALSE ) RETURNS BIGINT AS $func$ +CREATE OR REPLACE FUNCTION metabib.remap_metarecord_for_bib( + bib_id bigint, + fp text, + bib_is_deleted boolean DEFAULT false, + retain_deleted boolean DEFAULT false +) RETURNS bigint AS $function$ DECLARE new_mapping BOOL := TRUE; source_count INT; @@ -1415,11 +1512,11 @@ BEGIN -- We need to make sure we're not a deleted master record of an MR IF bib_is_deleted THEN - FOR old_mr IN SELECT id FROM metabib.metarecord WHERE master_record = bib_id LOOP + IF NOT retain_deleted THEN -- Go away for any MR that we're master of, unless retained + DELETE FROM metabib.metarecord_source_map WHERE source = bib_id; + END IF; - IF NOT retain_deleted THEN -- Go away for any MR that we're master of, unless retained - DELETE FROM metabib.metarecord_source_map WHERE source = bib_id; - END IF; + FOR old_mr IN SELECT id FROM metabib.metarecord WHERE master_record = bib_id LOOP -- Now, are there any more sources on this MR? SELECT COUNT(*) INTO source_count FROM metabib.metarecord_source_map WHERE metarecord = old_mr; @@ -1491,8 +1588,7 @@ BEGIN RETURN old_mr; END; -$func$ LANGUAGE PLPGSQL; - +$function$ LANGUAGE plpgsql; CREATE OR REPLACE FUNCTION biblio.map_authority_linking (bibid BIGINT, marc TEXT) RETURNS BIGINT AS $func$ DELETE FROM authority.bib_linking WHERE bib = $1; diff --git a/Open-ILS/src/sql/Pg/300.schema.staged_search.sql b/Open-ILS/src/sql/Pg/300.schema.staged_search.sql index 820313f5f6..71a2e7d76f 100644 --- a/Open-ILS/src/sql/Pg/300.schema.staged_search.sql +++ b/Open-ILS/src/sql/Pg/300.schema.staged_search.sql @@ -1315,6 +1315,139 @@ BEGIN END; $p$ LANGUAGE PLPGSQL ROWS 10; +CREATE OR REPLACE VIEW search.best_tsconfig AS + SELECT m.id AS id, + COALESCE(f.ts_config, c.ts_config, 'simple') AS ts_config + FROM config.metabib_field m + LEFT JOIN config.metabib_class_ts_map c ON (c.field_class = m.field_class AND c.index_weight = 'C') + LEFT JOIN config.metabib_field_ts_map f ON (f.metabib_field = m.id AND c.index_weight = 'C'); + +CREATE TYPE search.highlight_result AS ( id BIGINT, source BIGINT, field INT, value TEXT, highlight TEXT ); + +CREATE OR REPLACE FUNCTION search.highlight_display_fields( + rid BIGINT, + tsq TEXT, + field_list INT[] DEFAULT '{}'::INT[], + css_class TEXT DEFAULT 'oils_SH', + hl_all BOOL DEFAULT TRUE, + minwords INT DEFAULT 5, + maxwords INT DEFAULT 25, + shortwords INT DEFAULT 0, + maxfrags INT DEFAULT 0, + delimiter TEXT DEFAULT ' ... ' +) RETURNS SETOF search.highlight_result AS $f$ +DECLARE + opts TEXT := ''; + v_css_class TEXT := css_class; + v_delimiter TEXT := delimiter; + v_field_list INT[] := field_list; + hl_query TEXT; +BEGIN + IF v_delimiter LIKE $$%'%$$ OR v_delimiter LIKE '%"%' THEN --" + v_delimiter := ' ... '; + END IF; + + IF NOT hl_all THEN + opts := opts || 'MinWords=' || minwords; + opts := opts || ', MaxWords=' || maxwords; + opts := opts || ', ShortWords=' || shortwords; + opts := opts || ', MaxFragments=' || maxfrags; + opts := opts || ', FragmentDelimiter="' || delimiter || '"'; + ELSE + opts := opts || 'HighlightAll=TRUE'; + END IF; + + IF v_css_class LIKE $$%'%$$ OR v_css_class LIKE '%"%' THEN -- " + v_css_class := 'oils_SH'; + END IF; + + opts := opts || $$, StopSel=, StartSel=""$xx$ -- "' + ) AS highlight + FROM metabib.display_entry de + JOIN config.metabib_field mf ON (mf.id = de.field) + JOIN search.best_tsconfig t ON (t.id = de.field) + WHERE de.source = $2 + AND field = ANY ($3) + ORDER BY de.id;$$; + + RETURN QUERY EXECUTE hl_query USING opts, rid, v_field_list; +END; +$f$ LANGUAGE PLPGSQL; + +CREATE OR REPLACE FUNCTION evergreen.escape_for_html (TEXT) RETURNS TEXT AS $$ + SELECT regexp_replace( + regexp_replace( + regexp_replace( + $1, + '&', + '&', + 'g' + ), + '<', + '<', + 'g' + ), + '>', + '>', + 'g' + ); +$$ LANGUAGE SQL IMMUTABLE LEAKPROOF STRICT COST 10; + +CREATE OR REPLACE FUNCTION search.highlight_display_fields( + rid BIGINT, + tsq_map HSTORE, -- { '(a | b) & c' => '1,2,3,4', ...} + css_class TEXT DEFAULT 'oils_SH', + hl_all BOOL DEFAULT TRUE, + minwords INT DEFAULT 5, + maxwords INT DEFAULT 25, + shortwords INT DEFAULT 0, + maxfrags INT DEFAULT 0, + delimiter TEXT DEFAULT ' ... ' +) RETURNS SETOF search.highlight_result AS $f$ +DECLARE + tsq TEXT; + fields TEXT; + afields INT[]; + seen INT[]; +BEGIN + FOR tsq, fields IN SELECT key, value FROM each(tsq_map) LOOP + SELECT ARRAY_AGG(unnest::INT) INTO afields + FROM unnest(regexp_split_to_array(fields,',')); + seen := seen || afields; + + RETURN QUERY + SELECT * FROM search.highlight_display_fields( + rid, tsq, afields, css_class, hl_all,minwords, + maxwords, shortwords, maxfrags, delimiter + ); + END LOOP; + + RETURN QUERY + SELECT id, + source, + field, + value, + value AS highlight + FROM metabib.display_entry + WHERE source = rid + AND NOT (field = ANY (seen)); +END; +$f$ LANGUAGE PLPGSQL ROWS 10; COMMIT; diff --git a/Open-ILS/src/sql/Pg/950.data.seed-values.sql b/Open-ILS/src/sql/Pg/950.data.seed-values.sql index 5a2b0ffb54..94ea0307be 100644 --- a/Open-ILS/src/sql/Pg/950.data.seed-values.sql +++ b/Open-ILS/src/sql/Pg/950.data.seed-values.sql @@ -150,7 +150,7 @@ INSERT INTO config.metabib_field ( id, field_class, name, label, format, xpath, INSERT INTO config.metabib_field ( id, field_class, name, label, format, xpath, browse_field, display_field ) VALUES (15, 'keyword', 'keyword', oils_i18n_gettext(15, 'General Keywords', 'cmf', 'label'), 'mods32', $$//mods32:mods/*[not(local-name()='originInfo')]$$, FALSE, FALSE ); -- /* to fool vim */; INSERT INTO config.metabib_field ( id, field_class, name, label, format, xpath, browse_field, display_field ) VALUES - (16, 'subject', 'complete', oils_i18n_gettext(16, 'All Subjects', 'cmf', 'label'), 'mods32', $$//mods32:mods/mods32:subject$$, FALSE, TRUE ); + (16, 'subject', 'complete', oils_i18n_gettext(16, 'All Subjects', 'cmf', 'label'), 'mods32', $$//mods32:mods/mods32:subject[not(descendant::mods32:geographicCode)]$$, FALSE, TRUE ); INSERT INTO config.metabib_field ( id, field_class, name, label, format, xpath, browse_field ) VALUES (17, 'identifier', 'accession', oils_i18n_gettext(17, 'Accession Number', 'cmf', 'label'), 'marcxml', $$//marc:controlfield[@tag='001']$$, FALSE ); @@ -187,6 +187,7 @@ INSERT INTO config.metabib_field ( id, field_class, name, label, format, xpath, INSERT INTO config.metabib_field ( id, field_class, name, label, format, xpath, browse_field, facet_field, facet_xpath, joiner ) VALUES (33, 'identifier', 'genre', oils_i18n_gettext(33, 'Genre', 'cmf', 'label'), 'marcxml', $$//marc:datafield[@tag='655']$$, FALSE, TRUE, $$//*[local-name()='subfield' and contains('abvxyz',@code)]$$, ' -- ' ); -- /* to fool vim */; +UPDATE config.metabib_field SET display_xpath = facet_xpath, display_field = TRUE WHERE id = 33; UPDATE config.metabib_field SET joiner = ' -- ' WHERE field_class = 'subject' AND name NOT IN ('name'); @@ -210,6 +211,122 @@ INSERT INTO config.metabib_field ( id, field_class, name, label, (37, 'author', 'creator', oils_i18n_gettext(37, 'All Creators', 'cmf', 'label'), 'mods32', $$//mods32:mods/mods32:name[mods32:role/mods32:roleTerm[text()='creator']]$$, TRUE, $$//*[local-name()='namePart']$$ ); -- /* to fool vim */; +INSERT INTO config.metabib_field (id, field_class, name, + label, xpath, display_field, search_field, browse_field) +VALUES ( + 38, 'keyword', 'edition', + oils_i18n_gettext(38, 'Edition', 'cmf', 'label'), + $$//mods33:mods/mods33:originInfo//mods33:edition[1]$$, + TRUE, TRUE, FALSE +); + +INSERT INTO config.metabib_field (id, field_class, name, + label, xpath, display_field, search_field, browse_field) +VALUES ( + 39, 'keyword', 'physical_description', + oils_i18n_gettext(39, 'Physical Descrption', 'cmf', 'label'), + $$(//mods33:mods/mods33:physicalDescription/mods33:form|//mods33:mods/mods33:physicalDescription/mods33:extent|//mods33:mods/mods33:physicalDescription/mods33:reformattingQuality|//mods33:mods/mods33:physicalDescription/mods33:internetMediaType|//mods33:mods/mods33:physicalDescription/mods33:digitalOrigin)$$, + TRUE, TRUE, FALSE +); + +INSERT INTO config.metabib_field (id, field_class, name, + label, xpath, display_field, search_field, browse_field) +VALUES ( + 40, 'keyword', 'publisher', + oils_i18n_gettext(40, 'Publisher', 'cmf', 'label'), + $$//mods33:mods/mods33:originInfo//mods33:publisher[1]$$, + TRUE, TRUE, FALSE +); + +INSERT INTO config.metabib_field (id, field_class, name, + label, xpath, display_field, search_field, browse_field) +VALUES ( + 41, 'keyword', 'abstract', + oils_i18n_gettext(41, 'Abstract', 'cmf', 'label'), + $$//mods33:mods/mods33:abstract$$, + TRUE, TRUE, FALSE +); + +INSERT INTO config.metabib_field (id, field_class, name, + label, xpath, display_field, search_field, browse_field) +VALUES ( + 42, 'keyword', 'toc', + oils_i18n_gettext(42, 'Table of Contents', 'cmf', 'label'), + $$//mods33:tableOfContents$$, + TRUE, TRUE, FALSE +); + +INSERT INTO config.metabib_field (id, field_class, name, + label, xpath, display_field, search_field, browse_field) +VALUES ( + 43, 'identifier', 'type_of_resource', + oils_i18n_gettext(43, 'Type of Resource', 'cmf', 'label'), + $$//mods33:mods/mods33:typeOfResource$$, + TRUE, FALSE, FALSE +); + +INSERT INTO config.metabib_field (id, field_class, name, + label, xpath, display_field, search_field, browse_field) +VALUES ( + 44, 'identifier', 'pubdate', + oils_i18n_gettext(44, 'Publication Date', 'cmf', 'label'), + $$//mods33:mods/mods33:originInfo//mods33:dateIssued[@encoding="marc"]|//mods33:mods/mods33:originInfo//mods33:dateIssued[1]$$, + TRUE, FALSE, FALSE +); + +INSERT INTO config.metabib_field (id, field_class, name, label, browse_field) + VALUES (45, 'keyword', 'blob', oils_i18n_gettext(45, 'All searchable fields', 'cmf', 'label'), FALSE); + +INSERT INTO config.metabib_field (id, field_class, name, + label, xpath, display_field, search_field, browse_field) +VALUES ( + 46, 'keyword', 'bibliography', + oils_i18n_gettext(46, 'Bibliography', 'cmf', 'label'), + $$//mods33:note[@type='bibliography']$$, + TRUE, TRUE, FALSE +),( + 47, 'keyword', 'thesis', + oils_i18n_gettext(47, 'Thesis', 'cmf', 'label'), + $$//mods33:note[@type='thesis']$$, + TRUE, TRUE, FALSE +),( + 48, 'keyword', 'production_credits', + oils_i18n_gettext(48, 'Creation/Production Credits', 'cmf', 'label'), + $$//mods33:note[@type='creation/production credits']$$, + TRUE, TRUE, FALSE +),( + 49, 'keyword', 'performers', + oils_i18n_gettext(49, 'Performers', 'cmf', 'label'), + $$//mods33:note[@type='performers']$$, + TRUE, TRUE, FALSE +),( + 50, 'keyword', 'general_note', + oils_i18n_gettext(50, 'General Note', 'cmf', 'label'), + $$//mods33:note[not(@type)]$$, + TRUE, TRUE, FALSE +) +; + +INSERT INTO config.metabib_field_virtual_map (real, virtual) + SELECT id, + 45 + FROM config.metabib_field + WHERE search_field + AND id NOT IN (15, 45) + AND id NOT IN (SELECT real FROM config.metabib_field_virtual_map); + +-- Modify existing config.metabib_field entries + +UPDATE config.metabib_field SET display_field = TRUE WHERE id IN ( + 1, -- seriestitle + 11, -- subject_geographic + 12, -- subject_name + 13, -- subject_temporal + 14, -- subject_topic + 19, -- ISSN + 20, -- UPC + 26 -- TCN +); INSERT INTO config.metabib_field_index_norm_map (field,norm) SELECT m.id, @@ -226,7 +343,22 @@ INSERT INTO config.display_field_map (name, field, multi) VALUES ('author', 8, FALSE), ('creators', 37, TRUE), ('subject', 16, TRUE), - ('isbn', 18, TRUE) + ('isbn', 18, TRUE), + ('series_title', 1, FALSE), + ('subject_geographic', 11, TRUE), + ('subject_name', 12, TRUE), + ('subject_temporal', 13, TRUE), + ('subject_topic', 14, TRUE), + ('issn', 19, TRUE), + ('upc', 20, TRUE), + ('tcn', 26, FALSE), + ('edition', 38, FALSE), + ('physical_description',39, TRUE), + ('publisher', 40, FALSE), + ('abstract', 41, FALSE), + ('toc', 42, FALSE), + ('type_of_resource', 43, FALSE), + ('pubdate', 44, FALSE) ; INSERT INTO config.metabib_search_alias (alias,field_class) VALUES ('kw','keyword'); diff --git a/Open-ILS/src/sql/Pg/upgrade/XXXX.data.display-field-seed-data.sql b/Open-ILS/src/sql/Pg/upgrade/XXXX.data.display-field-seed-data.sql index ea55482eb8..6b1788b0d1 100644 --- a/Open-ILS/src/sql/Pg/upgrade/XXXX.data.display-field-seed-data.sql +++ b/Open-ILS/src/sql/Pg/upgrade/XXXX.data.display-field-seed-data.sql @@ -7,46 +7,46 @@ BEGIN; INSERT INTO config.metabib_field (id, field_class, name, label, xpath, display_field, search_field, browse_field) VALUES ( - 38, 'identifier', 'edition', + 38, 'keyword', 'edition', oils_i18n_gettext(38, 'Edition', 'cmf', 'label'), $$//mods33:mods/mods33:originInfo//mods33:edition[1]$$, - TRUE, FALSE, FALSE + TRUE, TRUE, FALSE ); INSERT INTO config.metabib_field (id, field_class, name, label, xpath, display_field, search_field, browse_field) VALUES ( - 39, 'identifier', 'physical_description', + 39, 'keyword', 'physical_description', oils_i18n_gettext(39, 'Physical Descrption', 'cmf', 'label'), $$(//mods33:mods/mods33:physicalDescription/mods33:form|//mods33:mods/mods33:physicalDescription/mods33:extent|//mods33:mods/mods33:physicalDescription/mods33:reformattingQuality|//mods33:mods/mods33:physicalDescription/mods33:internetMediaType|//mods33:mods/mods33:physicalDescription/mods33:digitalOrigin)$$, - TRUE, FALSE, FALSE + TRUE, TRUE, FALSE ); INSERT INTO config.metabib_field (id, field_class, name, label, xpath, display_field, search_field, browse_field) VALUES ( - 40, 'identifier', 'publisher', + 40, 'keyword', 'publisher', oils_i18n_gettext(40, 'Publisher', 'cmf', 'label'), $$//mods33:mods/mods33:originInfo//mods33:publisher[1]$$, - TRUE, FALSE, FALSE + TRUE, TRUE, FALSE ); INSERT INTO config.metabib_field (id, field_class, name, label, xpath, display_field, search_field, browse_field) VALUES ( - 41, 'identifier', 'abstract', + 41, 'keyword', 'abstract', oils_i18n_gettext(41, 'Abstract', 'cmf', 'label'), $$//mods33:mods/mods33:abstract$$, - TRUE, FALSE, FALSE + TRUE, TRUE, FALSE ); INSERT INTO config.metabib_field (id, field_class, name, label, xpath, display_field, search_field, browse_field) VALUES ( - 42, 'identifier', 'toc', + 42, 'keyword', 'toc', oils_i18n_gettext(42, 'Table of Contents', 'cmf', 'label'), $$//mods33:tableOfContents$$, - TRUE, FALSE, FALSE + TRUE, TRUE, FALSE ); INSERT INTO config.metabib_field (id, field_class, name, @@ -177,34 +177,3 @@ CREATE VIEW metabib.wide_display_entry AS COMMIT; -/** ROLLBACK - -BEGIN; -DELETE FROM metabib.display_entry WHERE field IN (1,11,12,13,14,19,20,26,38,39,40,41,42,43,44); -DELETE FROM config.display_field_map WHERE field IN (1,11,12,13,14,19,20,26,38,39,40,41,42,43,44); -DELETE FROM config.metabib_field WHERE id IN (38,39,40,41,42,43,44); -COMMIT; - -*/ - --- Perform a full display field reingest, since we didn't do one during --- the 3.0 upgrade when display fields were introduced. - -\qecho -\qecho Reingesting display field entries. This may take a while. -\qecho This command can be stopped (control-c) and rerun later if needed: -\qecho -\qecho SELECT metabib.reingest_metabib_field_entries(id, TRUE, FALSE, TRUE, TRUE, -\qecho (SELECT ARRAY_AGG(id)::INT[] FROM config.metabib_field WHERE display_field)) -\qecho FROM biblio.record_entry WHERE id > 0; -\qecho - --- avoid displaying a row per entry by selecting the total count. --- NOTE: extracting display data for deleted bibs because we occasionally --- display deleted bib records. -SELECT COUNT(*) AS bib_count FROM ( - SELECT metabib.reingest_metabib_field_entries(id, TRUE, FALSE, TRUE, TRUE, - (SELECT ARRAY_AGG(id)::INT[] FROM config.metabib_field WHERE display_field)) - FROM biblio.record_entry WHERE id > 0 -) x; - diff --git a/Open-ILS/src/sql/Pg/upgrade/XXXX.schema.highlight_search.sql b/Open-ILS/src/sql/Pg/upgrade/XXXX.schema.highlight_search.sql new file mode 100644 index 0000000000..2fc357b7bc --- /dev/null +++ b/Open-ILS/src/sql/Pg/upgrade/XXXX.schema.highlight_search.sql @@ -0,0 +1,559 @@ +BEGIN; + +ALTER TABLE config.metabib_field ALTER COLUMN xpath DROP NOT NULL; + +CREATE TABLE config.metabib_field_virtual_map ( + id SERIAL PRIMARY KEY, + real INT NOT NULL REFERENCES config.metabib_field (id), + virtual INT NOT NULL REFERENCES config.metabib_field (id), + weight INT NOT NULL DEFAULT 1 +); +COMMENT ON TABLE config.metabib_field_virtual_map IS $$ +Maps between real (physically extracted) index definitions +and virtual (target sync, no required extraction of its own) +index definitions. + +The virtual side may not extract any data of its own, but +will collect data from all of the real fields. This reduces +extraction (ingest) overhead by eliminating duplcated extraction, +and allows for searching across novel combinations of fields, such +as names used as either subjects or authors. By preserving this +mapping rather than defining duplicate extractions, information +about the originating, "real" index definitions can be used +in interesting ways, such as highlighting in search results. +$$; + +CREATE OR REPLACE VIEW metabib.combined_all_field_entry AS + SELECT * FROM metabib.combined_title_field_entry + UNION ALL + SELECT * FROM metabib.combined_author_field_entry + UNION ALL + SELECT * FROM metabib.combined_subject_field_entry + UNION ALL + SELECT * FROM metabib.combined_keyword_field_entry + UNION ALL + SELECT * FROM metabib.combined_identifier_field_entry + UNION ALL + SELECT * FROM metabib.combined_series_field_entry; + + +CREATE OR REPLACE FUNCTION biblio.extract_metabib_field_entry ( + rid BIGINT, + default_joiner TEXT, + field_types TEXT[], + only_fields INT[] +) RETURNS SETOF metabib.field_entry_template AS $func$ +DECLARE + bib biblio.record_entry%ROWTYPE; + idx config.metabib_field%ROWTYPE; + xfrm config.xml_transform%ROWTYPE; + prev_xfrm TEXT; + transformed_xml TEXT; + xml_node TEXT; + xml_node_list TEXT[]; + facet_text TEXT; + display_text TEXT; + browse_text TEXT; + sort_value TEXT; + raw_text TEXT; + curr_text TEXT; + joiner TEXT := default_joiner; -- XXX will index defs supply a joiner? + authority_text TEXT; + authority_link BIGINT; + output_row metabib.field_entry_template%ROWTYPE; + process_idx BOOL; +BEGIN + + -- Start out with no field-use bools set + output_row.browse_field = FALSE; + output_row.facet_field = FALSE; + output_row.display_field = FALSE; + output_row.search_field = FALSE; + + -- Get the record + SELECT INTO bib * FROM biblio.record_entry WHERE id = rid; + + -- Loop over the indexing entries + FOR idx IN SELECT * FROM config.metabib_field WHERE id = ANY (only_fields) ORDER BY format LOOP + CONTINUE WHEN idx.xpath IS NULL OR idx.xpath = ''; -- pure virtual field + + process_idx := FALSE; + IF idx.display_field AND 'display' = ANY (field_types) THEN process_idx = TRUE; END IF; + IF idx.browse_field AND 'browse' = ANY (field_types) THEN process_idx = TRUE; END IF; + IF idx.search_field AND 'search' = ANY (field_types) THEN process_idx = TRUE; END IF; + IF idx.facet_field AND 'facet' = ANY (field_types) THEN process_idx = TRUE; END IF; + CONTINUE WHEN process_idx = FALSE; -- disabled for all types + + joiner := COALESCE(idx.joiner, default_joiner); + + SELECT INTO xfrm * from config.xml_transform WHERE name = idx.format; + + -- See if we can skip the XSLT ... it's expensive + IF prev_xfrm IS NULL OR prev_xfrm <> xfrm.name THEN + -- Can't skip the transform + IF xfrm.xslt <> '---' THEN + transformed_xml := oils_xslt_process(bib.marc,xfrm.xslt); + ELSE + transformed_xml := bib.marc; + END IF; + + prev_xfrm := xfrm.name; + END IF; + + xml_node_list := oils_xpath( idx.xpath, transformed_xml, ARRAY[ARRAY[xfrm.prefix, xfrm.namespace_uri]] ); + + raw_text := NULL; + FOR xml_node IN SELECT x FROM unnest(xml_node_list) AS x LOOP + CONTINUE WHEN xml_node !~ E'^\\s*<'; + + -- XXX much of this should be moved into oils_xpath_string... + curr_text := ARRAY_TO_STRING(evergreen.array_remove_item_by_value(evergreen.array_remove_item_by_value( + oils_xpath( '//text()', -- get the content of all the nodes within the main selected node + REGEXP_REPLACE( xml_node, E'\\s+', ' ', 'g' ) -- Translate adjacent whitespace to a single space + ), ' '), ''), -- throw away morally empty (bankrupt?) strings + joiner + ); + + CONTINUE WHEN curr_text IS NULL OR curr_text = ''; + + IF raw_text IS NOT NULL THEN + raw_text := raw_text || joiner; + END IF; + + raw_text := COALESCE(raw_text,'') || curr_text; + + -- autosuggest/metabib.browse_entry + IF idx.browse_field THEN + + IF idx.browse_xpath IS NOT NULL AND idx.browse_xpath <> '' THEN + browse_text := oils_xpath_string( idx.browse_xpath, xml_node, joiner, ARRAY[ARRAY[xfrm.prefix, xfrm.namespace_uri]] ); + ELSE + browse_text := curr_text; + END IF; + + IF idx.browse_sort_xpath IS NOT NULL AND + idx.browse_sort_xpath <> '' THEN + + sort_value := oils_xpath_string( + idx.browse_sort_xpath, xml_node, joiner, + ARRAY[ARRAY[xfrm.prefix, xfrm.namespace_uri]] + ); + ELSE + sort_value := browse_text; + END IF; + + output_row.field_class = idx.field_class; + output_row.field = idx.id; + output_row.source = rid; + output_row.value = BTRIM(REGEXP_REPLACE(browse_text, E'\\s+', ' ', 'g')); + output_row.sort_value := + public.naco_normalize(sort_value); + + output_row.authority := NULL; + + IF idx.authority_xpath IS NOT NULL AND idx.authority_xpath <> '' THEN + authority_text := oils_xpath_string( + idx.authority_xpath, xml_node, joiner, + ARRAY[ + ARRAY[xfrm.prefix, xfrm.namespace_uri], + ARRAY['xlink','http://www.w3.org/1999/xlink'] + ] + ); + + IF authority_text ~ '^\d+$' THEN + authority_link := authority_text::BIGINT; + PERFORM * FROM authority.record_entry WHERE id = authority_link; + IF FOUND THEN + output_row.authority := authority_link; + END IF; + END IF; + + END IF; + + output_row.browse_field = TRUE; + -- Returning browse rows with search_field = true for search+browse + -- configs allows us to retain granularity of being able to search + -- browse fields with "starts with" type operators (for example, for + -- titles of songs in music albums) + IF idx.search_field THEN + output_row.search_field = TRUE; + END IF; + RETURN NEXT output_row; + output_row.browse_field = FALSE; + output_row.search_field = FALSE; + output_row.sort_value := NULL; + END IF; + + -- insert raw node text for faceting + IF idx.facet_field THEN + + IF idx.facet_xpath IS NOT NULL AND idx.facet_xpath <> '' THEN + facet_text := oils_xpath_string( idx.facet_xpath, xml_node, joiner, ARRAY[ARRAY[xfrm.prefix, xfrm.namespace_uri]] ); + ELSE + facet_text := curr_text; + END IF; + + output_row.field_class = idx.field_class; + output_row.field = -1 * idx.id; + output_row.source = rid; + output_row.value = BTRIM(REGEXP_REPLACE(facet_text, E'\\s+', ' ', 'g')); + + output_row.facet_field = TRUE; + RETURN NEXT output_row; + output_row.facet_field = FALSE; + END IF; + + -- insert raw node text for display + IF idx.display_field THEN + + IF idx.display_xpath IS NOT NULL AND idx.display_xpath <> '' THEN + display_text := oils_xpath_string( idx.display_xpath, xml_node, joiner, ARRAY[ARRAY[xfrm.prefix, xfrm.namespace_uri]] ); + ELSE + display_text := curr_text; + END IF; + + output_row.field_class = idx.field_class; + output_row.field = -1 * idx.id; + output_row.source = rid; + output_row.value = BTRIM(REGEXP_REPLACE(display_text, E'\\s+', ' ', 'g')); + + output_row.display_field = TRUE; + RETURN NEXT output_row; + output_row.display_field = FALSE; + END IF; + + END LOOP; + + CONTINUE WHEN raw_text IS NULL OR raw_text = ''; + + -- insert combined node text for searching + IF idx.search_field THEN + output_row.field_class = idx.field_class; + output_row.field = idx.id; + output_row.source = rid; + output_row.value = BTRIM(REGEXP_REPLACE(raw_text, E'\\s+', ' ', 'g')); + + output_row.search_field = TRUE; + RETURN NEXT output_row; + output_row.search_field = FALSE; + END IF; + + END LOOP; + +END; +$func$ LANGUAGE PLPGSQL; + +CREATE OR REPLACE FUNCTION metabib.update_combined_index_vectors(bib_id BIGINT) RETURNS VOID AS $func$ +DECLARE + rdata TSVECTOR; + vclass TEXT; + vfield INT; + rfields INT[]; +BEGIN + DELETE FROM metabib.combined_keyword_field_entry WHERE record = bib_id; + INSERT INTO metabib.combined_keyword_field_entry(record, metabib_field, index_vector) + SELECT bib_id, field, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.keyword_field_entry WHERE source = bib_id GROUP BY field; + INSERT INTO metabib.combined_keyword_field_entry(record, metabib_field, index_vector) + SELECT bib_id, NULL, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.keyword_field_entry WHERE source = bib_id; + + DELETE FROM metabib.combined_title_field_entry WHERE record = bib_id; + INSERT INTO metabib.combined_title_field_entry(record, metabib_field, index_vector) + SELECT bib_id, field, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.title_field_entry WHERE source = bib_id GROUP BY field; + INSERT INTO metabib.combined_title_field_entry(record, metabib_field, index_vector) + SELECT bib_id, NULL, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.title_field_entry WHERE source = bib_id; + + DELETE FROM metabib.combined_author_field_entry WHERE record = bib_id; + INSERT INTO metabib.combined_author_field_entry(record, metabib_field, index_vector) + SELECT bib_id, field, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.author_field_entry WHERE source = bib_id GROUP BY field; + INSERT INTO metabib.combined_author_field_entry(record, metabib_field, index_vector) + SELECT bib_id, NULL, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.author_field_entry WHERE source = bib_id; + + DELETE FROM metabib.combined_subject_field_entry WHERE record = bib_id; + INSERT INTO metabib.combined_subject_field_entry(record, metabib_field, index_vector) + SELECT bib_id, field, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.subject_field_entry WHERE source = bib_id GROUP BY field; + INSERT INTO metabib.combined_subject_field_entry(record, metabib_field, index_vector) + SELECT bib_id, NULL, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.subject_field_entry WHERE source = bib_id; + + DELETE FROM metabib.combined_series_field_entry WHERE record = bib_id; + INSERT INTO metabib.combined_series_field_entry(record, metabib_field, index_vector) + SELECT bib_id, field, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.series_field_entry WHERE source = bib_id GROUP BY field; + INSERT INTO metabib.combined_series_field_entry(record, metabib_field, index_vector) + SELECT bib_id, NULL, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.series_field_entry WHERE source = bib_id; + + DELETE FROM metabib.combined_identifier_field_entry WHERE record = bib_id; + INSERT INTO metabib.combined_identifier_field_entry(record, metabib_field, index_vector) + SELECT bib_id, field, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.identifier_field_entry WHERE source = bib_id GROUP BY field; + INSERT INTO metabib.combined_identifier_field_entry(record, metabib_field, index_vector) + SELECT bib_id, NULL, strip(COALESCE(string_agg(index_vector::TEXT,' '),'')::tsvector) + FROM metabib.identifier_field_entry WHERE source = bib_id; + + -- For each virtual def, gather the data from the combined real field + -- entries and append it to the virtual combined entry. + FOR vfield, rfields IN SELECT virtual, ARRAY_AGG(real) FROM config.metabib_field_virtual_map GROUP BY virtual LOOP + SELECT field_class INTO vclass + FROM config.metabib_field + WHERE id = vfield; + + SELECT string_agg(index_vector::TEXT,' ')::tsvector INTO rdata + FROM metabib.combined_all_field_entry + WHERE record = bib_id + AND metabib_field = ANY (rfields); + + BEGIN -- I cannot wait for INSERT ON CONFLICT ... 9.5, though + EXECUTE $$ + INSERT INTO metabib.combined_$$ || vclass || $$_field_entry + (record, metabib_field, index_vector) VALUES ($1, $2, $3) + $$ USING bib_id, vfield, rdata; + EXCEPTION WHEN unique_violation THEN + EXECUTE $$ + UPDATE metabib.combined_$$ || vclass || $$_field_entry + SET index_vector = index_vector || $3 + WHERE record = $1 + AND metabib_field = $2 + $$ USING bib_id, vfield, rdata; + END; + END LOOP; +END; +$func$ LANGUAGE PLPGSQL; + +CREATE OR REPLACE VIEW search.best_tsconfig AS + SELECT m.id AS id, + COALESCE(f.ts_config, c.ts_config, 'simple') AS ts_config + FROM config.metabib_field m + LEFT JOIN config.metabib_class_ts_map c ON (c.field_class = m.field_class AND c.index_weight = 'C') + LEFT JOIN config.metabib_field_ts_map f ON (f.metabib_field = m.id AND c.index_weight = 'C'); + +CREATE TYPE search.highlight_result AS ( id BIGINT, source BIGINT, field INT, value TEXT, highlight TEXT ); + +CREATE OR REPLACE FUNCTION search.highlight_display_fields( + rid BIGINT, + tsq TEXT, + field_list INT[] DEFAULT '{}'::INT[], + css_class TEXT DEFAULT 'oils_SH', + hl_all BOOL DEFAULT TRUE, + minwords INT DEFAULT 5, + maxwords INT DEFAULT 25, + shortwords INT DEFAULT 0, + maxfrags INT DEFAULT 0, + delimiter TEXT DEFAULT ' ... ' +) RETURNS SETOF search.highlight_result AS $f$ +DECLARE + opts TEXT := ''; + v_css_class TEXT := css_class; + v_delimiter TEXT := delimiter; + v_field_list INT[] := field_list; + hl_query TEXT; +BEGIN + IF v_delimiter LIKE $$%'%$$ OR v_delimiter LIKE '%"%' THEN --" + v_delimiter := ' ... '; + END IF; + + IF NOT hl_all THEN + opts := opts || 'MinWords=' || minwords; + opts := opts || ', MaxWords=' || maxwords; + opts := opts || ', ShortWords=' || shortwords; + opts := opts || ', MaxFragments=' || maxfrags; + opts := opts || ', FragmentDelimiter="' || delimiter || '"'; + ELSE + opts := opts || 'HighlightAll=TRUE'; + END IF; + + IF v_css_class LIKE $$%'%$$ OR v_css_class LIKE '%"%' THEN -- " + v_css_class := 'oils_SH'; + END IF; + + opts := opts || $$, StopSel=, StartSel=""$xx$ -- "' + ) AS highlight + FROM metabib.display_entry de + JOIN config.metabib_field mf ON (mf.id = de.field) + JOIN search.best_tsconfig t ON (t.id = de.field) + WHERE de.source = $2 + AND field = ANY ($3) + ORDER BY de.id;$$; + + RETURN QUERY EXECUTE hl_query USING opts, rid, v_field_list; +END; +$f$ LANGUAGE PLPGSQL; + +CREATE OR REPLACE FUNCTION evergreen.escape_for_html (TEXT) RETURNS TEXT AS $$ + SELECT regexp_replace( + regexp_replace( + regexp_replace( + $1, + '&', + '&', + 'g' + ), + '<', + '<', + 'g' + ), + '>', + '>', + 'g' + ); +$$ LANGUAGE SQL IMMUTABLE LEAKPROOF STRICT COST 10; + +CREATE OR REPLACE FUNCTION search.highlight_display_fields( + rid BIGINT, + tsq_map HSTORE, -- { '(a | b) & c' => '1,2,3,4', ...} + css_class TEXT DEFAULT 'oils_SH', + hl_all BOOL DEFAULT TRUE, + minwords INT DEFAULT 5, + maxwords INT DEFAULT 25, + shortwords INT DEFAULT 0, + maxfrags INT DEFAULT 0, + delimiter TEXT DEFAULT ' ... ' +) RETURNS SETOF search.highlight_result AS $f$ +DECLARE + tsq TEXT; + fields TEXT; + afields INT[]; + seen INT[]; +BEGIN + FOR tsq, fields IN SELECT key, value FROM each(tsq_map) LOOP + SELECT ARRAY_AGG(unnest::INT) INTO afields + FROM unnest(regexp_split_to_array(fields,',')); + seen := seen || afields; + + RETURN QUERY + SELECT * FROM search.highlight_display_fields( + rid, tsq, afields, css_class, hl_all,minwords, + maxwords, shortwords, maxfrags, delimiter + ); + END LOOP; + + RETURN QUERY + SELECT id, + source, + field, + value, + value AS highlight + FROM metabib.display_entry + WHERE source = rid + AND NOT (field = ANY (seen)); +END; +$f$ LANGUAGE PLPGSQL ROWS 10; + +CREATE OR REPLACE FUNCTION metabib.remap_metarecord_for_bib( + bib_id bigint, + fp text, + bib_is_deleted boolean DEFAULT false, + retain_deleted boolean DEFAULT false +) RETURNS bigint AS $function$ +DECLARE + new_mapping BOOL := TRUE; + source_count INT; + old_mr BIGINT; + tmp_mr metabib.metarecord%ROWTYPE; + deleted_mrs BIGINT[]; +BEGIN + + -- We need to make sure we're not a deleted master record of an MR + IF bib_is_deleted THEN + IF NOT retain_deleted THEN -- Go away for any MR that we're master of, unless retained + DELETE FROM metabib.metarecord_source_map WHERE source = bib_id; + END IF; + + FOR old_mr IN SELECT id FROM metabib.metarecord WHERE master_record = bib_id LOOP + + -- Now, are there any more sources on this MR? + SELECT COUNT(*) INTO source_count FROM metabib.metarecord_source_map WHERE metarecord = old_mr; + + IF source_count = 0 AND NOT retain_deleted THEN -- No other records + deleted_mrs := ARRAY_APPEND(deleted_mrs, old_mr); -- Just in case... + DELETE FROM metabib.metarecord WHERE id = old_mr; + + ELSE -- indeed there are. Update it with a null cache and recalcualated master record + UPDATE metabib.metarecord + SET mods = NULL, + master_record = ( SELECT id FROM biblio.record_entry WHERE fingerprint = fp AND NOT deleted ORDER BY quality DESC LIMIT 1) + WHERE id = old_mr; + END IF; + END LOOP; + + ELSE -- insert or update + + FOR tmp_mr IN SELECT m.* FROM metabib.metarecord m JOIN metabib.metarecord_source_map s ON (s.metarecord = m.id) WHERE s.source = bib_id LOOP + + -- Find the first fingerprint-matching + IF old_mr IS NULL AND fp = tmp_mr.fingerprint THEN + old_mr := tmp_mr.id; + new_mapping := FALSE; + + ELSE -- Our fingerprint changed ... maybe remove the old MR + DELETE FROM metabib.metarecord_source_map WHERE metarecord = tmp_mr.id AND source = bib_id; -- remove the old source mapping + SELECT COUNT(*) INTO source_count FROM metabib.metarecord_source_map WHERE metarecord = tmp_mr.id; + IF source_count = 0 THEN -- No other records + deleted_mrs := ARRAY_APPEND(deleted_mrs, tmp_mr.id); + DELETE FROM metabib.metarecord WHERE id = tmp_mr.id; + END IF; + END IF; + + END LOOP; + + -- we found no suitable, preexisting MR based on old source maps + IF old_mr IS NULL THEN + SELECT id INTO old_mr FROM metabib.metarecord WHERE fingerprint = fp; -- is there one for our current fingerprint? + + IF old_mr IS NULL THEN -- nope, create one and grab its id + INSERT INTO metabib.metarecord ( fingerprint, master_record ) VALUES ( fp, bib_id ); + SELECT id INTO old_mr FROM metabib.metarecord WHERE fingerprint = fp; + + ELSE -- indeed there is. update it with a null cache and recalcualated master record + UPDATE metabib.metarecord + SET mods = NULL, + master_record = ( SELECT id FROM biblio.record_entry WHERE fingerprint = fp AND NOT deleted ORDER BY quality DESC LIMIT 1) + WHERE id = old_mr; + END IF; + + ELSE -- there was one we already attached to, update its mods cache and master_record + UPDATE metabib.metarecord + SET mods = NULL, + master_record = ( SELECT id FROM biblio.record_entry WHERE fingerprint = fp AND NOT deleted ORDER BY quality DESC LIMIT 1) + WHERE id = old_mr; + END IF; + + IF new_mapping THEN + INSERT INTO metabib.metarecord_source_map (metarecord, source) VALUES (old_mr, bib_id); -- new source mapping + END IF; + + END IF; + + IF ARRAY_UPPER(deleted_mrs,1) > 0 THEN + UPDATE action.hold_request SET target = old_mr WHERE target IN ( SELECT unnest(deleted_mrs) ) AND hold_type = 'M'; -- if we had to delete any MRs above, make sure their holds are moved + END IF; + + RETURN old_mr; + +END; +$function$ LANGUAGE plpgsql; + +COMMIT; + diff --git a/Open-ILS/src/sql/Pg/upgrade/YYYY.data.mods-title-punctuation-change.sql b/Open-ILS/src/sql/Pg/upgrade/YYYY.data.mods-title-punctuation-change.sql new file mode 100644 index 0000000000..5efb1ff967 --- /dev/null +++ b/Open-ILS/src/sql/Pg/upgrade/YYYY.data.mods-title-punctuation-change.sql @@ -0,0 +1,6817 @@ +BEGIN; + +update config.xml_transform set xslt = $XXXX$ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BK + SE + + + BK + MM + CF + MP + VM + MU + + + + + + + + + b + afgk + + + + + abfgk + + + + + + + + + + + ,;/ + + + + + + + + + + <xsl:value-of select="substring($titleChop,@ind2+1)"/> + + + + + <xsl:value-of select="$titleChop"/> + + + + + + + + + b + b + afgk + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">abfgk</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + abfgk + + + + + + + + + + + <xsl:value-of select="substring($titleBrowseChop,@ind2+1)"/> + + + + + <xsl:value-of select="$titleBrowseChop"/> + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">a</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + a + + + + + + + + + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + + + + + + + + <xsl:value-of select="substring($titleChop,@ind2+1)"/> + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <!-- 1/04 removed $h, $b --> + <xsl:with-param name="codes">af</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <xsl:value-of select="$titleChop"/> + + + + + + + + + + + <xsl:value-of select="substring($titleChop,$nfi+1)"/> + + + + + <xsl:value-of select="$titleChop"/> + + + + + + + + + + + + ah + + + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + <xsl:value-of select="substring($titleChop,@ind1+1)"/> + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + + + creator + + + + + + + + + + creator + + + + + + + + + + creator + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + personal + + + + + + + + + + + yes + + + yes + + + text + cartographic + notated music + sound recording-nonmusical + sound recording-musical + still image + moving image + three dimensional object + software, multimedia + mixed material + + + + globe + + + remote sensing image + + + + + + map + + + atlas + + + + + + + + database + + + loose-leaf + + + series + + + newspaper + + + periodical + + + web site + + + + + + + + abstract or summary + + + bibliography + + + catalog + + + dictionary + + + encyclopedia + + + handbook + + + legal article + + + index + + + discography + + + legislation + + + theses + + + survey of literature + + + review + + + programmed text + + + filmography + + + directory + + + statistics + + + technical report + + + legal case and case notes + + + law report or digest + + + treaty + + + + + + conference publication + + + + + + + + numeric data + + + database + + + font + + + game + + + + + + patent + + + festschrift + + + + biography + + + + + essay + + + drama + + + comic strip + + + fiction + + + humor, satire + + + letter + + + novel + + + short story + + + speech + + + + + + + biography + + + conference publication + + + drama + + + essay + + + fiction + + + folktale + + + history + + + humor, satire + + + memoir + + + poetry + + + rehearsal + + + reporting + + + sound + + + speech + + + + + + + art original + + + kit + + + art reproduction + + + diorama + + + filmstrip + + + legal article + + + picture + + + graphic + + + technical drawing + + + motion picture + + + chart + + + flash card + + + microscope slide + + + model + + + realia + + + slide + + + transparency + + + videorecording + + + toy + + + + + + + + + + abvxyz + - + + + + + + + + + code + marccountry + + + + + + + + code + iso3166 + + + + + + + + text + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + :,;/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + monographic + continuing + + + + + + + ab + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + reformatted digital + + + digitized microfilm + + + digitized other analog + + + + + + + + + + + + + + + +
braille
+ + +
print
+ + +
electronic
+ + +
microfiche
+ + +
microfilm
+ + + + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ +
+ + + + + access + + + preservation + + + replacement + + + + + +
chip cartridge
+ + +
computer optical disc cartridge
+ + +
magnetic disc
+ + +
magneto-optical disc
+ + +
optical disc
+ + +
remote
+ + +
tape cartridge
+ + +
tape cassette
+ + +
tape reel
+ + + +
celestial globe
+ + +
earth moon globe
+ + +
planetary or lunar globe
+ + +
terrestrial globe
+ + + +
kit
+ + + +
atlas
+ + +
diagram
+ + +
map
+ + +
model
+ + +
profile
+ + +
remote-sensing image
+ + +
section
+ + +
view
+ + + +
aperture card
+ + +
microfiche
+ + +
microfiche cassette
+ + +
microfilm cartridge
+ + +
microfilm cassette
+ + +
microfilm reel
+ + +
microopaque
+ + + +
film cartridge
+ + +
film cassette
+ + +
film reel
+ + + +
chart
+ + +
collage
+ + +
drawing
+ + +
flash card
+ + +
painting
+ + +
photomechanical print
+ + +
photonegative
+ + +
photoprint
+ + +
picture
+ + +
print
+ + +
technical drawing
+ + + +
notated music
+ + + +
filmslip
+ + +
filmstrip cartridge
+ + +
filmstrip roll
+ + +
other filmstrip type
+ + +
slide
+ + +
transparency
+ + +
remote-sensing image
+ + +
cylinder
+ + +
roll
+ + +
sound cartridge
+ + +
sound cassette
+ + +
sound disc
+ + +
sound-tape reel
+ + +
sound-track film
+ + +
wire recording
+ + + +
braille
+ + +
combination
+ + +
moon
+ + +
tactile, with no writing system
+ + + +
braille
+ + +
large print
+ + +
regular print
+ + +
text in looseleaf binder
+ + + +
videocartridge
+ + +
videocassette
+ + +
videodisc
+ + +
videoreel
+ + + + + + + + + + + abce + + + + + + + + + + + + + + ab + + + + + + + + agrt + + + + + + + ab + + + + + + + + + adolescent + + + adult + + + general + + + juvenile + + + preschool + + + specialized + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + defg + + + + + + + + + + + + marcgac + + + + + + iso3166 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ab + + + + + + + abx + + + + + + + ab + + + + + + + + + + + + + + + + + + + + + + + + + + + + ab + + + + + + + + + + av + + + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + <xsl:value-of select="substring($titleChop,@ind2+1)"/> + + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">av</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + abcx3 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklmorsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">g</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + aq + t + g + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklmorsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">dg</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + c + t + dgn + + + + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">g</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + aqdc + t + gn + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">adfgklmorsv</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + <xsl:value-of select="substring($titleChop,@ind1+1)"/> + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklmorsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">g</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + aq + t + g + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklmorsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">dg</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + c + t + dgn + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">g</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + aqdc + t + gn + + + + + + + + + + + + + + adfgklmorsv + + + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + <xsl:value-of select="substring($titleChop,@ind2+1)"/> + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + + + + + + + + isbn + + + + + + + + + + isrc + + + + + + + + + + ismn + + + + + + + + + + sici + + + + ab + + + + + + issn + + + + + + + + lccn + + + + + + + + + + issue number + matrix number + music plate + music publisher + videorecording identifier + + + + + + + ba + ab + + + + + + + + + + ab + + + + + + + + doi + hdl + uri + + + + + + + + + + + + + + + + + y3z + + + + + + + + + + + + + + + + + + + + + y3 + + + + + + + z + + + + + + + + + + + + + + + + + + abje + + + + + + + + abcd35 + + + + + + + abcde35 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + n + n + fgkdlmor + + + + + p + p + fgkdlmor + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + g + g + pst + + + + + p + p + fgkdlmor + + + + + + + + + + + + + + + + + + + + + + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + cdn + + + + + + + + + + aq + + + + :,;/ + + + + + + + + + + acdeq + + + + + + constituent + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="."></xsl:value-of> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="."></xsl:value-of> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="."></xsl:value-of> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="."></xsl:value-of> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + code + marcgac + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + lcsh + lcshac + mesh + + nal + csh + rvm + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + aq + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + cdnp + + + + + + + + + + + + + + + + abcdeqnp + + + + + + + + + + + + + + + + + + + + + adfhklor + + + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + <xsl:value-of select="substring($titleChop,@ind1+1)"/> + + + + + + <xsl:value-of select="$titleChop" /> + + + + + + + + + + + + + + + + + abcd + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + bc + + + + + + + + + + + + + + + + + + + + + + + + + + + yes + + + + + + + + + + + + + + + + + + + + + + + + + + + Arabic + Latin + Chinese, Japanese, Korean + Cyrillic + Hebrew + Greek + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + summary or subtitle + sung or spoken text + libretto + table of contents + accompanying material + translation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + summary or subtitle + sung or spoken text + libretto + table of contents + accompanying material + translation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + .:,;/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +$XXXX$ where name = $$mods32$$; + +update config.xml_transform set xslt = $XXXX$ + + + + !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ + + + + ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ + + + + !'()*-.0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz~ + + + 0123456789ABCDEF + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + BK + SE + + + BK + MM + CF + MP + VM + MU + + + + + + + + + b + afgk + + + + + abfgk + + + + + + + + + + + ,;/ + + + + + + + + + + <xsl:value-of select="substring($titleChop,@ind2+1)"/> + + + + + <xsl:value-of select="$titleChop"/> + + + + + + + + + b + b + afgk + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">a</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <!-- 1/04 removed $h, b --> + <xsl:with-param name="codes">a</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <!-- 1/04 removed $h, $b --> + <xsl:with-param name="codes">af</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + <xsl:call-template name="uri"/> + + <xsl:variable name="str"> + <xsl:for-each select="marc:subfield"> + <xsl:if + test="(contains('adfklmors',@code) and (not(../marc:subfield[@code='n' or @code='p']) or (following-sibling::marc:subfield[@code='n' or @code='p'])))"> + <xsl:value-of select="text()"/> + <xsl:text> </xsl:text> + </xsl:if> + </xsl:for-each> + </xsl:variable> + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="substring($str,1,string-length($str)-1)"/> + </xsl:with-param> + </xsl:call-template> + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">ah</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + creator + + + + + + + + + + + + creator + + + + + + + + + + + + creator + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + personal + + + + + + + + + + + yes + + + yes + + + text + cartographic + notated music + sound recording-nonmusical + sound recording-musical + still image + moving image + three dimensional object + software, multimedia + mixed material + + + + globe + + + remote-sensing image + + + + + + map + + + atlas + + + + + + + + database + + + loose-leaf + + + series + + + newspaper + + + periodical + + + web site + + + + + + + + abstract or summary + + + bibliography + + + catalog + + + dictionary + + + encyclopedia + + + handbook + + + legal article + + + index + + + discography + + + legislation + + + theses + + + survey of literature + + + review + + + programmed text + + + filmography + + + directory + + + statistics + + + technical report + + + legal case and case notes + + + law report or digest + + + treaty + + + + + + conference publication + + + + + + + + numeric data + + + database + + + font + + + game + + + + + + patent + + + offprint + + + festschrift + + + + biography + + + + + essay + + + drama + + + comic strip + + + fiction + + + humor, satire + + + letter + + + novel + + + short story + + + speech + + + + + + + biography + + + conference publication + + + drama + + + essay + + + fiction + + + folktale + + + history + + + humor, satire + + + memoir + + + poetry + + + rehearsal + + + reporting + + + sound + + + speech + + + + + + + art original + + + kit + + + art reproduction + + + diorama + + + filmstrip + + + legal article + + + picture + + + graphic + + + technical drawing + + + motion picture + + + chart + + + flash card + + + microscope slide + + + model + + + realia + + + slide + + + transparency + + + videorecording + + + toy + + + + + + + + + + + + + abcdef + - + + + + + + + + + + abvxyz + - + + + + + + + + + code + marccountry + + + + + + + + code + iso3166 + + + + + + + + text + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + :,;/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + monographic + continuing + + + + + + + ab + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + reformatted digital + + + digitized microfilm + + + digitized other analog + + + + + + + + + + + + + + + +
braille
+ + +
print
+ + +
electronic
+ + +
microfiche
+ + +
microfilm
+ + + + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + + + + +
+ + +
+ + +
+ + + + access + + + preservation + + + replacement + + + + + +
chip cartridge
+ + +
computer optical disc cartridge
+ + +
magnetic disc
+ + +
magneto-optical disc
+ + +
optical disc
+ + +
remote
+ + +
tape cartridge
+ + +
tape cassette
+ + +
tape reel
+ + + +
celestial globe
+ + +
earth moon globe
+ + +
planetary or lunar globe
+ + +
terrestrial globe
+ + + +
kit
+ + + +
atlas
+ + +
diagram
+ + +
map
+ + +
model
+ + +
profile
+ + +
remote-sensing image
+ + +
section
+ + +
view
+ + + +
aperture card
+ + +
microfiche
+ + +
microfiche cassette
+ + +
microfilm cartridge
+ + +
microfilm cassette
+ + +
microfilm reel
+ + +
microopaque
+ + + +
film cartridge
+ + +
film cassette
+ + +
film reel
+ + + +
chart
+ + +
collage
+ + +
drawing
+ + +
flash card
+ + +
painting
+ + +
photomechanical print
+ + +
photonegative
+ + +
photoprint
+ + +
picture
+ + +
print
+ + +
technical drawing
+ + + +
notated music
+ + + +
filmslip
+ + +
filmstrip cartridge
+ + +
filmstrip roll
+ + +
other filmstrip type
+ + +
slide
+ + +
transparency
+ + +
remote-sensing image
+ + +
cylinder
+ + +
roll
+ + +
sound cartridge
+ + +
sound cassette
+ + +
sound disc
+ + +
sound-tape reel
+ + +
sound-track film
+ + +
wire recording
+ + + +
braille
+ + +
combination
+ + +
moon
+ + +
tactile, with no writing system
+ + + +
braille
+ + +
large print
+ + +
regular print
+ + +
text in looseleaf binder
+ + + +
videocartridge
+ + +
videocassette
+ + +
videodisc
+ + +
videoreel
+ + + + + + + + + + + abce + + + + + + + + + + + + + + ab + + + + + + + + agrt + + + + + + + ab + + + + + + + + + adolescent + + + adult + + + general + + + juvenile + + + preschool + + + specialized + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + defg + + + + + + + + + + + + marcgac + + + + + + iso3166 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ab + + + + + + + abx + + + + + + + ab + + + + + + + + + + + + + + + + + + + + + + + + + + + + ab + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">av</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">av</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + abcx3 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklmorsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">g</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + aq + t + g + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklmorsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">dg</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + c + t + dgn + + + + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">g</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + aqdc + t + gn + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">adfgklmorsv</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="marc:subfield[@code='a']"/> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklmorsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">g</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + aq + t + g + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklmorsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">dg</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + c + t + dgn + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="specialSubfieldSelect"> + <xsl:with-param name="anyCodes">tfklsv</xsl:with-param> + <xsl:with-param name="axis">t</xsl:with-param> + <xsl:with-param name="afterCodes">g</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + aqdc + t + gn + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">adfgklmorsv</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + isbn + + + + + + + + + + isrc + + + + + + + + + + ismn + + + + + + + + + + sici + + + + ab + + + + + + + issn + + + + + + + + issn-l + + + + + + + + + + + + lccn + + + + + + + + + + issue number + matrix number + music plate + music publisher + videorecording identifier + + + + + + + + ba + ab + + + + + + + + + + + ab + + + + + + + + doi + hdl + uri + + + + + + + + + + + + + + + + + y3z + + + + + + + + + + + + + + + + + + + + + + + + + y3 + + + + + + + z + + + + + + + + + + + + + + + + y3 + + + + + + + z + + + + + + + + + + + + + + + + + + abe + + + + + + + + + u + + + + + + + + hijklmt + + + + + + + + + + abcd35 + + + + + + + abcde35 + + + + + + + + + + aacr2 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + n + n + fgkdlmor + + + + + p + p + fgkdlmor + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + g + g + pst + + + + + p + p + fgkdlmor + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + cdn + + + + + + + + + + aq + + + + :,;/ + + + + + + + + + + acdeq + + + + + + constituent + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="."/> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="."/> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="."/> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:value-of select="."/> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + code + marcgac + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + lcsh + lcshac + mesh + + nal + csh + rvm + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + aq + + + + + + + + + + + + + + + + + + + + + + + + + + + + + cdnp + + + + + + + + + + + + + + + abcdeqnp + + + + + + + + + + + + + + + + + + + <xsl:call-template name="chopPunctuation"> + <xsl:with-param name="chopString"> + <xsl:call-template name="subfieldSelect"> + <xsl:with-param name="codes">adfhklor</xsl:with-param> + </xsl:call-template> + </xsl:with-param> + </xsl:call-template> + + + + + + + + + + + + + + + + + + + + + + abcd + + + + + + + + + + + + + + + + abcd + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + bc + + + + + + + + + + + + + + + + + + + + + + + + + + + yes + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Arabic + Latin + Chinese, Japanese, Korean + Cyrillic + Hebrew + Greek + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + summary or subtitle + sung or spoken text + libretto + table of contents + accompanying material + translation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + summary or subtitle + sung or spoken text + libretto + table of contents + accompanying material + translation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + abcdefghijklmnopqrstuvwxyz + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + .:,;/ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + .:,;/] + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Warning: string contains a character + that is out of range! Substituting "?". + 63 + + + + + + + + + + + + + + + + +$XXXX$ where name = $$mods33$$; + +COMMIT; + diff --git a/Open-ILS/src/sql/Pg/upgrade/YYYY.data.virtual_index_defs.sql b/Open-ILS/src/sql/Pg/upgrade/YYYY.data.virtual_index_defs.sql new file mode 100644 index 0000000000..65e67ed369 --- /dev/null +++ b/Open-ILS/src/sql/Pg/upgrade/YYYY.data.virtual_index_defs.sql @@ -0,0 +1,42 @@ +BEGIN; + +INSERT INTO config.metabib_field (id, field_class, name, label, browse_field) + VALUES (45, 'keyword', 'blob', 'All searchable fields', FALSE); + +INSERT INTO config.metabib_field_virtual_map (real, virtual) + SELECT id, + 45 + FROM config.metabib_field + WHERE search_field + AND id NOT IN (15, 45); + +UPDATE config.metabib_field SET xpath=$$//mods32:mods/mods32:subject[not(descendant::mods32:geographicCode)]$$ WHERE id = 16; + +COMMIT; + +\qecho +\qecho Reingesting all records. This may take a while. +\qecho This command can be stopped (control-c) and rerun later if needed: +\qecho +\qecho DO $FUNC$ +\qecho DECLARE +\qecho same_marc BOOL; +\qecho BEGIN +\qecho SELECT INTO same_marc enabled FROM config.internal_flag WHERE name = 'ingest.reingest.force_on_same_marc'; +\qecho UPDATE config.internal_flag SET enabled = true WHERE name = 'ingest.reingest.force_on_same_marc'; +\qecho UPDATE biblio.record_entry SET id=id WHERE not deleted AND id > 0; +\qecho UPDATE config.internal_flag SET enabled = same_marc WHERE name = 'ingest.reingest.force_on_same_marc'; +\qecho END; +\qecho $FUNC$; + +DO $FUNC$ +DECLARE + same_marc BOOL; +BEGIN + SELECT INTO same_marc enabled FROM config.internal_flag WHERE name = 'ingest.reingest.force_on_same_marc'; + UPDATE config.internal_flag SET enabled = true WHERE name = 'ingest.reingest.force_on_same_marc'; + UPDATE biblio.record_entry SET id=id WHERE not deleted AND id > 0; + UPDATE config.internal_flag SET enabled = same_marc WHERE name = 'ingest.reingest.force_on_same_marc'; +END; +$FUNC$; + diff --git a/Open-ILS/src/support-scripts/test-scripts/query_parser.pl b/Open-ILS/src/support-scripts/test-scripts/query_parser.pl index 6f835dedc7..ebc0777d8e 100755 --- a/Open-ILS/src/support-scripts/test-scripts/query_parser.pl +++ b/Open-ILS/src/support-scripts/test-scripts/query_parser.pl @@ -80,6 +80,11 @@ if (!$noconnect) { 'open-ils.cstore.direct.config.metabib_field.search.atomic', { id => { "!=" => undef } } )->gather(1), + config_metabib_field_virtual_map => + $cstore->request( + 'open-ils.cstore.direct.config.metabib_field_virtual_map.search.atomic', + { id => { "!=" => undef } } + )->gather(1), config_metabib_search_alias => $cstore->request( 'open-ils.cstore.direct.config.metabib_search_alias.search.atomic', @@ -100,10 +105,11 @@ if (!$noconnect) { } $parser->parse; - -print "Parsed query tree:\n" . Dumper($parser->parse_tree) unless $quiet; - my $sql = $parser->toSQL; $sql =~ s/^\s*$//gm; + +print "Parsed query tree:\n" . Dumper($parser) unless $quiet; +print "Abstract query:\n" . Dumper($parser->parse_tree->to_abstract_query) unless $quiet; + print "SQL:\n$sql\n\n" unless $quiet; diff --git a/Open-ILS/src/templates/conify/global/config/metabib_field.tt2 b/Open-ILS/src/templates/conify/global/config/metabib_field.tt2 index 95fec5af69..39dcf26818 100644 --- a/Open-ILS/src/templates/conify/global/config/metabib_field.tt2 +++ b/Open-ILS/src/templates/conify/global/config/metabib_field.tt2 @@ -17,7 +17,13 @@ autoHeight='true' editOnEnter='true'> - + + + [% l('Data Suppliers') %] + + + + @@ -25,6 +31,22 @@ [% END %] diff --git a/Open-ILS/src/templates/conify/global/config/metabib_field_virtual_map.tt2 b/Open-ILS/src/templates/conify/global/config/metabib_field_virtual_map.tt2 new file mode 100644 index 0000000000..a19e370b0e --- /dev/null +++ b/Open-ILS/src/templates/conify/global/config/metabib_field_virtual_map.tt2 @@ -0,0 +1,77 @@ +[% WRAPPER base.tt2 %] +

[% l('Virtual Field Data Providers') %]


+ +
+
[% l('Virtual Field Data Providers') %]
+
+ + + +
+
+ +
+ [% l('Record Attribute Type: ') %]
+
+ +
+ +
+
+ + + +[% END %] diff --git a/Open-ILS/src/templates/opac/css/style.css.tt2 b/Open-ILS/src/templates/opac/css/style.css.tt2 index aade957dc9..e176ed712a 100644 --- a/Open-ILS/src/templates/opac/css/style.css.tt2 +++ b/Open-ILS/src/templates/opac/css/style.css.tt2 @@ -3333,3 +3333,13 @@ label[for*=expert_] [id^="toggled-inline-"]:target{ display: inline; } + +.oils_SH { + font-weight: bolder; + background-color: #99ff99; +} + +.oils_SH.identifier { + font-weight: bolder; + background-color: #42b0f4; +} diff --git a/Open-ILS/src/templates/opac/parts/misc_util.tt2 b/Open-ILS/src/templates/opac/parts/misc_util.tt2 index 97cbd1e7c8..2d48ccc53f 100644 --- a/Open-ILS/src/templates/opac/parts/misc_util.tt2 +++ b/Open-ILS/src/templates/opac/parts/misc_util.tt2 @@ -84,10 +84,50 @@ END; # Extract MARC fields from XML - # get_marc_attrs( { marc_xml => doc } ) + # get_marc_attrs( args = { marc_xml => doc } ) BLOCK get_marc_attrs; + USE Dumper; xml = args.marc_xml; + args.bibid = []; + FOR bibid IN xml.findnodes('//*[@tag="901"]/*[@code="c"]'); + args.bibid.push(bibid.textContent); + END; + + args.df_bib_list = args.bibid; + args.bibid = args.bibid.0; + + IF args.mr_constituent_ids.size && !args.df_bib_list.size; + args.df_bib_list = args.mr_constituent_ids; + END; + + + # Gather display field data for this record and map it + # to a display field map. Hopefully, one day, this can + # replace the XPath below entirely. + + args.display_fields = {}; + args.hl = {}; + IF !CGI.param('no_highlight'); + args.display_field_list = ctx.fetch_display_fields(args.df_bib_list.list); + + FOR df IN args.display_field_list; + df_map = ctx.search_cdfm('field', df.field).0; + df_name = df_map.name(); + IF df_map.multi() == 't'; + IF NOT args.hl_display_fields.$df_name; + args.hl_display_fields.$df_name = []; + args.hl.$df_name = []; + END; + args.hl_display_fields.$df_name.push(df); + args.hl.$df_name.push(df.highlight || df.value); + ELSIF !args.hl_display_fields.$df_name.defined; + args.hl_display_fields.$df_name = df; + args.hl.$df_name = df.highlight || df.value; + END; + END; + END; + # Map item types to schema.org types; impedance mismatch :( args.schema.itemtype = {}; schema_typemap = {}; @@ -726,12 +766,6 @@ END; args.mmr_unique_bib = mmr_unique_bib; - args.bibid = []; - FOR bibid IN xml.findnodes('//*[@tag="901"]/*[@code="c"]'); - args.bibid.push(bibid.textContent); - END; - args.bibid = args.bibid.0; - IF args.ebook_test_id; args.ebook.ebook_id = args.ebook_test_id; args.ebook.vendor = 'ebook_test'; diff --git a/Open-ILS/src/templates/opac/parts/record/authors.tt2 b/Open-ILS/src/templates/opac/parts/record/authors.tt2 index 3f0dd2fdb9..4140e78ecb 100644 --- a/Open-ILS/src/templates/opac/parts/record/authors.tt2 +++ b/Open-ILS/src/templates/opac/parts/record/authors.tt2 @@ -23,8 +23,23 @@ authors = [ } ]; +BLOCK find_hl_value; + outlist = []; + norm_needle = PROCESS normalize_string(unnorm_string=needle); + FOREACH hl IN attrs.display_field_list; + norm_value = PROCESS normalize_string(unnorm_string=hl.value); + outlist.push(hl.highlight) IF norm_value == norm_needle; + END; + + outlist.0; +END; + +BLOCK normalize_string; + unnorm_string.replace('[#"^$\+\-,\.:;&|\[\]()]', ' ').replace('\s+',' ').replace('^\s+','').replace('\s+$',''); +END; + BLOCK normalize_qterm; - subfield.textContent.replace('[#"^$\+\-,\.:;&|\[\]()]', ' '); + PROCESS normalize_string(unnorm_string=subfield.textContent); END; BLOCK normalize_authors; @@ -120,14 +135,23 @@ BLOCK build_author_links; END; iprop = iprop _ '"'; END; + + link_term = link_term.replace('^\s+', ''); + match_term = link_term _ ' ' _ birthdate _ ' ' _ deathdate; + matching_author_hl = PROCESS find_hl_value needle=match_term; + authtml = ' '; IF iprop; authtml = authtml _ ''; END; - authtml = authtml _ link_term.replace('^\s+', ''); + IF matching_author_hl; + authtml = authtml _ matching_author_hl; + ELSE; + authtml = authtml _ link_term; + END; IF iprop; authtml = authtml _ ''; END; - IF birthdate; + IF birthdate AND !matching_author_hl; authtml = authtml _ ' ' _ birthdate _ '-'; END; - IF deathdate; + IF deathdate AND !matching_author_hl; authtml = authtml _ '' _ deathdate _ ''; END; authtml = authtml _ ''; # End search link diff --git a/Open-ILS/src/templates/opac/parts/record/contents.tt2 b/Open-ILS/src/templates/opac/parts/record/contents.tt2 index 29fc33b4cd..4f7897e775 100644 --- a/Open-ILS/src/templates/opac/parts/record/contents.tt2 +++ b/Open-ILS/src/templates/opac/parts/record/contents.tt2 @@ -13,6 +13,7 @@ contents = [ label => l('Bibliography, etc. Note: '), xpath => '//*[@tag="504"]' }, { + display_field => 'toc', label => l('Formatted Contents Note: '), xpath => '//*[@tag="505"]' }, { @@ -46,6 +47,7 @@ contents = [ label => l('Date/Time and Place of an Event Note: '), xpath => '//*[@tag="518"]' }, { + display_field => 'abstract', label => l('Summary, etc.: '), xpath => '//*[@tag="520"]' }, { @@ -178,11 +180,17 @@ BLOCK render_contents; ''; END; END; -END -%] -[% BLOCK render_all_contents; +END; + +BLOCK render_all_contents; FOREACH cont IN contents; - content = PROCESS render_contents(xpath=cont.xpath); + content = ''; + df = cont.display_field; + IF df AND attrs.hl.$df.size; + content = attrs.hl.$df.join('
'); + ELSE; + content = PROCESS render_contents(xpath=cont.xpath); + END; IF content.match('\S'); -%] diff --git a/Open-ILS/src/templates/opac/parts/record/subjects.tt2 b/Open-ILS/src/templates/opac/parts/record/subjects.tt2 index b0702a427d..ad028561b0 100644 --- a/Open-ILS/src/templates/opac/parts/record/subjects.tt2 +++ b/Open-ILS/src/templates/opac/parts/record/subjects.tt2 @@ -1,6 +1,7 @@ [% subjects = [ { + display_field => 'subject', label => l('Subject: '), xpath => '//*[@tag="600" or @tag="610" or @tag="611" or @tag="630" or @tag="650" or @tag="651"]' }, { @@ -78,13 +79,42 @@ END; ''; END; - END + END; + + BLOCK render_hl_subject; + ''; + %][% s.highlight %] [%- + ''; + END; %] [% BLOCK render_all_subjects; FOREACH subj IN subjects; - content = PROCESS render_subject(s=subj); - IF content.match('\S'); + content = ''; + df = subj.display_field; + IF df AND attrs.hl_display_fields.$df.size; + content = []; + FOREACH hl_s IN attrs.hl_display_fields.$df; + next_s = PROCESS render_hl_subject(s=hl_s); + content.push(next_s); + END; + + content = content.join('
'); +%] + + + + + + + +
[% subj.label %][% content %]
+[% + ELSE; + content = PROCESS render_subject(s=subj); + IF content.match('\S'); %] @@ -94,6 +124,7 @@
+ [%- END; %] [%- END; %] [%- END; %] [%- END %] diff --git a/Open-ILS/src/templates/opac/parts/record/summary.tt2 b/Open-ILS/src/templates/opac/parts/record/summary.tt2 index d26810dec7..20b7d368a3 100644 --- a/Open-ILS/src/templates/opac/parts/record/summary.tt2 +++ b/Open-ILS/src/templates/opac/parts/record/summary.tt2 @@ -11,7 +11,7 @@ [%-# This holds the record summary information %]
-

[% attrs.title_extended | html %]

+

[% IF attrs.hl.title; attrs.hl.title; ELSE; attrs.title_extended | html; END %]

[%- FOR link880 IN attrs.graphic_titles; FOR alt IN link880.graphic; diff --git a/Open-ILS/src/templates/opac/parts/result/table.tt2 b/Open-ILS/src/templates/opac/parts/result/table.tt2 index 5ab8e9a087..df787f74aa 100644 --- a/Open-ILS/src/templates/opac/parts/result/table.tt2 +++ b/Open-ILS/src/templates/opac/parts/result/table.tt2 @@ -92,7 +92,7 @@ - [% attrs.title | html %] + [% IF attrs.hl.title; attrs.hl.title; ELSE; attrs.title | html; END %] [% IF rec.mr_constituent_count.defined && rec.mr_constituent_count > 1 %] @@ -122,7 +122,7 @@ END; href="[%- authorquery = attrs.author | replace('[#"^$\+\-,\.:;&|\[\]()]', ' '); mkurl(ctx.opac_root _ '/results', {qtype => 'author', query => authorquery}, general_search_parms.merge(expert_search_parms, browse_search_parms, facet_search_parms)) - -%]" rel="nofollow" vocab="">[% attrs.author | html %] + -%]" rel="nofollow" vocab="">[% IF attrs.hl.author; attrs.hl.author; ELSE; attrs.author | html; END %] [%- FOR entry IN attrs.graphic_authors; FOR alt IN entry.graphic; diff --git a/Open-ILS/src/templates/staff/admin/server/config/metabib_field.tt2 b/Open-ILS/src/templates/staff/admin/server/config/metabib_field.tt2 new file mode 100644 index 0000000000..97f4031dac --- /dev/null +++ b/Open-ILS/src/templates/staff/admin/server/config/metabib_field.tt2 @@ -0,0 +1,64 @@ +[% + WRAPPER "staff/base.tt2"; + ctx.page_title = l("Metabib Fields"); + ctx.page_app = "egAdminConfig"; + ctx.page_ctrl = 'MetabibField'; +%] + +[% BLOCK APP_JS %] + + + + + +[% END %] + +
+
+ [% l('Metabib Fields') %] +
+
+ +
+
+ +
+
+ +
+
+ + + + + + + + + + + + + + + + + [% l('Manage'); %] + + + +[% END %] diff --git a/Open-ILS/src/templates/staff/admin/server/config/metabib_field_virtual_map.tt2 b/Open-ILS/src/templates/staff/admin/server/config/metabib_field_virtual_map.tt2 new file mode 100644 index 0000000000..4959c66cc1 --- /dev/null +++ b/Open-ILS/src/templates/staff/admin/server/config/metabib_field_virtual_map.tt2 @@ -0,0 +1,41 @@ +[% + WRAPPER "staff/base.tt2"; + ctx.page_title = l("Virtual Field Data Providers"); + ctx.page_app = "egAdminConfig"; + ctx.page_ctrl = 'MetabibFieldVirtualMap'; +%] + +[% BLOCK APP_JS %] + + + + + +[% END %] + +
+
+ [% l('Virtual Field Data Providers') %] +
+
+ + + + + + + + + + + + + +[% END %] diff --git a/Open-ILS/web/js/ui/default/staff/admin/server/config/metabib_field.js b/Open-ILS/web/js/ui/default/staff/admin/server/config/metabib_field.js new file mode 100644 index 0000000000..3f28e492e0 --- /dev/null +++ b/Open-ILS/web/js/ui/default/staff/admin/server/config/metabib_field.js @@ -0,0 +1,93 @@ +angular.module('egAdminConfig', + ['ngRoute','ui.bootstrap','egCoreMod','egUiMod','egGridMod','egFmRecordEditorMod']) + +.controller('MetabibField', + ['$scope','$q','$timeout','$location','$window','$uibModal','egCore','egGridDataProvider', + 'egConfirmDialog', +function($scope , $q , $timeout , $location , $window , $uibModal , egCore , egGridDataProvider , + egConfirmDialog) { + + egCore.startup.go(); // standalone mode requires manual startup + + $scope.search_class = ''; + $scope.$watch('search_class', function(newVal, oldVal) { + if (newVal != oldVal) { + $scope.gridControls.setQuery(generateQuery($scope.search_class)); + $scope.gridControls.refresh(); + } + }); + + $scope.new_record = function() { + spawn_editor(); + } + + $scope.edit_record = function(items) { + if (items.length != 1) return; + spawn_editor(items[0].id); + } + + spawn_editor = function(id) { + var templ; + if (arguments.length == 1) { + templ = ''; + } else { + templ = ''; + } + gridControls = $scope.gridControls; + $uibModal.open({ + template : templ, + backdrop: 'static', + controller : [ + '$scope', '$uibModalInstance', + function($scope , $uibModalInstance) { + $scope.id = id; + + $scope.ok = function($event) { + $uibModalInstance.close(); + gridControls.refresh(); + } + + $scope.cancel = function($event) { + $uibModalInstance.dismiss(); + } + } + ] + }); + } + + $scope.delete_record = function(selected) { + if (!selected || !selected.length) return; + + egCore.pcrud.retrieve('cmf', selected[0].id).then(function(rec) { + egConfirmDialog.open( + egCore.strings.EG_CONFIRM_DELETE_RECORD_TITLE, + egCore.strings.EG_CONFIRM_DELETE_RECORD_BODY, + { id : rec.id() } // TODO replace with selector if available? + ).result.then(function() { + egCore.pcrud.remove(rec).then(function() { + $scope.gridControls.refresh(); + }); + }); + }); + } + + function generateQuery(search_class) { + var q = { 'id' : { '!=' : null } }; + + if (search_class) { + q.field_class = search_class; + } + + return q; + } + + $scope.gridControls = { + activateItem : function (i) { return $scope.edit_record([i]) }, + setQuery : function() { + return generateQuery($scope.search_class); + }, + setSort : function() { + return ['label']; + } + } +}]) diff --git a/Open-ILS/web/js/ui/default/staff/admin/server/config/metabib_field_virtual_map.js b/Open-ILS/web/js/ui/default/staff/admin/server/config/metabib_field_virtual_map.js new file mode 100644 index 0000000000..1c2ffb2d84 --- /dev/null +++ b/Open-ILS/web/js/ui/default/staff/admin/server/config/metabib_field_virtual_map.js @@ -0,0 +1,98 @@ +angular.module('egAdminConfig', + ['ngRoute','ui.bootstrap','egCoreMod','egUiMod','egGridMod','egFmRecordEditorMod']) + +.controller('MetabibFieldVirtualMap', + ['$scope','$q','$timeout','$location','$window','$uibModal','egCore','egGridDataProvider', + 'egConfirmDialog', +function($scope , $q , $timeout , $location , $window , $uibModal , egCore , egGridDataProvider , + egConfirmDialog) { + + egCore.startup.go(); // standalone mode requires manual startup + + $scope.cmf = null; + + $scope.virt_field = $location.search().cmf || ''; + if ($scope.virt_field) egCore.pcrud.retrieve('cmf', $scope.virt_field).then(function(c) { $scope.cmf = c }); + + $scope.$watch('virt_field', function(newVal, oldVal) { + if (newVal != oldVal) { + egCore.pcrud.retrieve('cmf', newVal).then(function(c) { $scope.cmf = c }); + $scope.gridControls.setQuery(generateQuery($scope.virt_field)); + $scope.gridControls.refresh(); + } + }); + + $scope.new_record = function() { + spawn_editor(); + } + + $scope.edit_record = function(items) { + if (items.length != 1) return; + spawn_editor(items[0].id); + } + + spawn_editor = function(id) { + var templ; + if (arguments.length == 1) { + templ = ''; + } else { + templ = ''; + } + gridControls = $scope.gridControls; + $uibModal.open({ + template : templ, + backdrop: 'static', + controller : [ + '$scope', '$uibModalInstance', + function($scope , $uibModalInstance) { + $scope.id = id; + + $scope.ok = function($event) { + $uibModalInstance.close(); + gridControls.refresh(); + } + + $scope.cancel = function($event) { + $uibModalInstance.dismiss(); + } + } + ] + }); + } + + $scope.delete_record = function(selected) { + if (!selected || !selected.length) return; + + egCore.pcrud.retrieve('cmfvm', selected[0].id).then(function(rec) { + egConfirmDialog.open( + egCore.strings.EG_CONFIRM_DELETE_RECORD_TITLE, + egCore.strings.EG_CONFIRM_DELETE_RECORD_BODY, + { id : rec.id() } // TODO replace with selector if available? + ).result.then(function() { + egCore.pcrud.remove(rec).then(function() { + $scope.gridControls.refresh(); + }); + }); + }); + } + + function generateQuery(virt_field) { + var q = { 'id' : { '!=' : null } }; + + if (virt_field) { + q.virtual = virt_field; + } + + return q; + } + + $scope.gridControls = { + activateItem : function (i) { return $scope.edit_record([i]) }, + setQuery : function() { + return generateQuery($scope.virt_field); + }, + setSort : function() { + return ['label']; + } + } +}]) diff --git a/Open-ILS/web/js/ui/default/staff/services/fm_record_editor.js b/Open-ILS/web/js/ui/default/staff/services/fm_record_editor.js index 4cb38e5a2d..a7ca7c68cb 100644 --- a/Open-ILS/web/js/ui/default/staff/services/fm_record_editor.js +++ b/Open-ILS/web/js/ui/default/staff/services/fm_record_editor.js @@ -139,21 +139,12 @@ angular.module('egFmRecordEditorMod', function flatten_linked_values(cls, list) { var results = []; - var fields = egCore.idl.classes[cls].fields; - var id_field; - var selector; - angular.forEach(fields, function(fld) { - if (fld.datatype == 'id') { - id_field = fld.name; - selector = fld.selector ? fld.selector : id_field; - return; - } - }); + var id_field = egCore.idl.classes[cls].pkey; + var selector = egCore.idl.classes[cls].field_map[id_field].selector || id_field; angular.forEach(list, function(item) { - var rec = egCore.idl.toHash(item); results.push({ - id : rec[id_field], - name : rec[selector] + id : item[id_field](), + name : item[selector]() }); }); return results; @@ -175,7 +166,7 @@ angular.module('egFmRecordEditorMod', } } if (field.datatype == 'link') { - egCore.pcrud.retrieveAll( + egCore.pcrud.retrieveAll( field.class, {}, {atomic : true} ).then(function(list) { field.linked_values = flatten_linked_values(field.class, list); diff --git a/Open-ILS/web/js/ui/default/staff/services/grid.js b/Open-ILS/web/js/ui/default/staff/services/grid.js index ed21ae0aa0..fc3a05b322 100644 --- a/Open-ILS/web/js/ui/default/staff/services/grid.js +++ b/Open-ILS/web/js/ui/default/staff/services/grid.js @@ -1650,6 +1650,8 @@ angular.module('egGridMod', // idlClass as the base. cols.idlFieldFromPath = function(dotpath) { var class_obj = egCore.idl.classes[cols.idlClass]; + if (!dotpath) return null; + var path_parts = dotpath.split(/\./); var idl_parent; @@ -1913,7 +1915,7 @@ angular.module('egGridMod', angular.forEach(provider.columnsProvider.columns, function(col) { // only query IDL-tracked columns - if (!col.adhoc && (col.required || col.visible)) + if (!col.adhoc && col.name && col.path && (col.required || col.visible)) queryFields[col.name] = col.path; } ); -- 2.43.2