1 Migrating from a legacy system
2 ==============================
4 When you migrate to Evergreen, you generally want to migrate the bibliographic
5 records and copy information that existed in your previous library system. For
6 anything more than a few thousand records, you should import the data directly
7 into the database rather than use the tools in the staff client. While the data
8 that you can extract from your legacy system varies widely, this section
9 assumes that you or members of your team have the ability to write scripts and
10 are comfortable working with SQL to manipulate data within PostgreSQL. If so,
11 then the following section will guide you towards a method of generating common
12 data formats so that you can then load the data into the database in bulk.
14 Making electronic resources visible in the catalog
15 --------------------------------------------------
16 Electronic resources generally do not have any call number or copy information
17 associated with them, and Evergreen enables you to easily make bibliographic
18 records visible in the public catalog within sections of the organizational
19 unit hierarchy. For example, you can make a set of bibliographic records
20 visible only to specific branches that have purchased licenses for the
21 corresponding resources, or you can make records representing publicly
22 available electronic resources visible to the entire consortium.
24 Therefore, to make a record visible in the public catalog, modify the records
25 using your preferred MARC editing approach to ensure the 856 field contains the
26 following information before loading records for electronic resources into
29 .856 field for electronic resources: indicators and subfields
30 [width="100%",options="header"]
31 |=============================================================================
32 |Attribute | Value | Note
34 |Indicator 2 |0 or 1 |
35 |Subfield u |URL for the electronic resource |
36 |Subfield y |Text content of the link |
37 |Subfield z |Public note | Normally displayed after the link
38 |Subfield 9 |Organizational unit short name | The record will be visible when
39 a search is performed specifying this organizational unit or one of its
40 children. You can repeat this subfield as many times as you need.
41 |=============================================================================
43 Once your electronic resource bibliographic records have the required
44 indicators and subfields for each 856 field in the record, you can proceed to
45 load the records using either the command-line bulk import method or the MARC
46 Batch Importer in the staff client.
48 Migrating your bibliographic records
49 ------------------------------------
50 Convert your MARC21 binary records into the MARCXML format, with one record per
51 line. You can use the following Python script to achieve this goal; just
52 install the _pymarc_ library first, and adjust the values of the _input_ and
53 _output_ variables as needed.
56 ------------------------------------------------------------------------------
58 # -*- coding: utf-8 -*-
62 input = 'records_in.mrc'
63 output = 'records_out.xml'
65 reader = pymarc.MARCReader(open(input, 'rb'), to_unicode=True)
66 writer = codecs.open(output, 'w', 'utf-8')
68 record.leader = record.leader[:9] + 'a' + record.leader[10:]
69 writer.write(pymarc.record_to_xml(record) + "\n")
70 ------------------------------------------------------------------------------
72 Once you have a MARCXML file with one record per line, you can load the records
73 into your Evergreen system via a staging table in your database.
75 . Connect to the PostgreSQL database using the _psql_ command. For example:
77 ------------------------------------------------------------------------------
78 psql -U <user-name> -h <hostname> -d <database>
79 ------------------------------------------------------------------------------
81 . Create a staging table in the database. The staging table is a temporary
82 location for the raw data that you will load into the production table or
83 tables. Issue the following SQL statement from the _psql_ command line,
84 adjusting the name of the table from _staging_records_import_, if desired:
87 ------------------------------------------------------------------------------
88 CREATE TABLE staging_records_import (id BIGSERIAL, dest BIGINT, marc TEXT);
89 ------------------------------------------------------------------------------
91 . Create a function that will insert the new records into the production table
92 and update the _dest_ column of the staging table. Adjust
93 "staging_records_import" to match the name of the staging table that you plan
94 to create when you issue the following SQL statement:
97 ------------------------------------------------------------------------------
98 CREATE OR REPLACE FUNCTION staging_importer() RETURNS VOID AS $$
101 FOR stage IN SELECT * FROM staging_records_import ORDER BY id LOOP
102 INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES (stage.marc, 'IMPORT');
103 UPDATE staging_records_import SET dest = currval('biblio.record_entry_id_seq')
108 ------------------------------------------------------------------------------
110 . Load the data from your MARCXML file into the staging table using the COPY
111 statement, adjusting for the name of the staging table and the location of
115 ------------------------------------------------------------------------------
116 COPY staging_records_import (marc) FROM '/tmp/records_out.xml';
117 ------------------------------------------------------------------------------
119 . Load the data from your staging table into the production table by invoking
120 your staging function:
123 ------------------------------------------------------------------------------
124 SELECT staging_importer();
125 ------------------------------------------------------------------------------
127 When you leave out the _id_ value for a _BIGSERIAL_ column, the value in the
128 column automatically increments for each new record that you add to the table.
130 Once you have loaded the records into your Evergreen system, you can search for
131 some known records using the staff client to confirm that the import was
134 Migrating your call numbers, copies, and parts
135 ----------------------------------------------
136 'Holdings', comprised of call numbers, copies, and parts, are the set of
137 objects that enable users to locate and potentially acquire materials from your
140 'Call numbers' connect libraries to bibliographic records. Each call number has a
141 'label' associated with a classification scheme such as a the Library of Congress
142 or Dewey Decimal systems, and can optionally have either or both a label prefix
143 and a label suffix. Label prefixes and suffixes do not affect the sort order of
146 'Copies' connect call numbers to particular instances of that resource at a
147 particular library. Each copy has a barcode and must exist in a particular copy
148 location. Other optional attributes of copies include circulation modifier,
149 which may affect whether that copy can circulate or for how long it can
150 circulate, and OPAC visibility, which controls whether that particular copy
151 should be visible in the public catalog.
153 'Parts' provide more granularity for copies, primarily to enable patrons to
154 place holds on individual parts of a set of items. For example, an encyclopedia
155 might be represented by a single bibliographic record, with a single call
156 number representing the label for that encyclopedia at a given library, with 26
157 copies representing each letter of the alphabet, with each copy mapped to a
158 different part such as _A, B, C, ... Z_.
160 To migrate this data into your Evergreen system, you will create another
161 staging table in the database to hold the raw data for your materials from
162 which the actual call numbers, copies, and parts will be generated.
164 Begin by connecting to the PostgreSQL database using the _psql_ command. For
167 ------------------------------------------------------------------------------
168 psql -U <user-name> -h <hostname> -d <database>
169 ------------------------------------------------------------------------------
171 Create the staging materials table by issuing the following SQL statement:
174 ------------------------------------------------------------------------------
175 CREATE TABLE staging_materials (
176 bibkey BIGINT, -- biblio.record_entry_id
177 callnum TEXT, -- call number label
178 callnum_prefix TEXT, -- call number prefix
179 callnum_suffix TEXT, -- call number suffix
180 callnum_class TEXT, -- classification scheme
182 location TEXT, -- shelving location code
183 item_type TEXT, -- circulation modifier code
184 owning_lib TEXT, -- org unit code
185 barcode TEXT, -- copy barcode
188 ------------------------------------------------------------------------------
190 For the purposes of this example migration of call numbers, copies, and parts,
191 we assume that you are able to create a tab-delimited file containing values
192 that map to the staging table properties, with one copy per line. For example,
193 the following 5 lines demonstrate how the file could look for 5 different
194 copies, with non-applicable attribute values represented by _\N_, and 3 of the
195 copies connected to a single call number and bibliographic record via parts:
197 ------------------------------------------------------------------------------
198 1 QA 76.76 A3 \N \N LC 2012-12-05 STACKS BOOK BR1 30007001122620 \N
199 2 GV 161 V8 Ref. Juv. LC 2010-11-11 KIDS DVD BR2 30007005197073 \N
200 3 AE 5 E363 1984 \N \N LC 1984-01-10 REFERENCE BOOK BR1 30007006853385 A
201 3 AE 5 E363 1984 \N \N LC 1984-01-10 REFERENCE BOOK BR1 30007006853393 B
202 3 AE 5 E363 1984 \N \N LC 1984-01-10 REFERENCE BOOK BR1 30007006853344 C
203 ------------------------------------------------------------------------------
205 Once your holdings are in a tab-delimited format--which, for the purposes of
206 this example, we will name _holdings.tsv_--you can import the holdings file
207 into your staging table. Copy the contents of the holdings file into the
208 staging table using the _COPY_ SQL statement:
211 ------------------------------------------------------------------------------
212 COPY staging_items (bibkey, callnum, callnum_prefix,
213 callnum_suffix, callnum_class, create_date, location,
214 item_type, owning_lib, barcode, part) FROM 'holdings.tsv';
215 ------------------------------------------------------------------------------
217 Generate the copy locations you need to represent your holdings:
220 ------------------------------------------------------------------------------
221 INSERT INTO asset.copy_location (name, owning_lib)
222 SELECT DISTINCT location, 1 FROM staging_materials
224 SELECT 1 FROM asset.copy_location
225 WHERE name = location
227 ------------------------------------------------------------------------------
229 Generate the circulation modifiers you need to represent your holdings:
232 ------------------------------------------------------------------------------
233 INSERT INTO config.circ_modifier (code, name, description, sip2_media_type)
234 SELECT DISTINCT circmod, circmod, circmod, '001'
235 FROM staging_materials
237 SELECT 1 FROM config.circ_modifier
240 ------------------------------------------------------------------------------
242 Generate the call number prefixes and suffixes you need to represent your
246 ------------------------------------------------------------------------------
247 INSERT INTO asset.call_number_prefix (owning_lib, label)
248 SELECT DISTINCT aou.id, callnum_prefix
249 FROM staging_materials sm
250 INNER JOIN actor.org_unit aou
251 ON aou.shortname = sm.owning_lib
253 SELECT 1 FROM asset.call_number_prefix acnp
254 WHERE callnum_prefix = acnp.label
255 AND aou.id = acnp.owning_lib
256 ) AND callnum_prefix IS NOT NULL;
258 INSERT INTO asset.call_number_suffix (owning_lib, label)
259 SELECT DISTINCT aou.id, callnum_suffix
260 FROM staging_materials sm
261 INNER JOIN actor.org_unit aou
262 ON aou.shortname = sm.owning_lib
264 SELECT 1 FROM asset.call_number_suffix acns
265 WHERE callnum_suffix = acns.label
266 AND aou.id = acns.owning_lib
267 ) AND callnum_suffix IS NOT NULL;
268 ------------------------------------------------------------------------------
270 Generate the call numbers for your holdings:
273 ------------------------------------------------------------------------------
274 INSERT INTO asset.call_number (
275 creator, editor, record, owning_lib, label, prefix, suffix, label_class
277 SELECT DISTINCT 1, 1, bibkey, aou.id, callnum, acnp.id, acns.id,
278 CASE WHEN callnum_class = 'LC' THEN 1
279 WHEN callnum_class = 'DEWEY' THEN 2
281 FROM staging_materials sm
282 INNER JOIN actor.org_unit aou
283 ON aou.shortname = owning_lib
284 INNER JOIN asset.call_number_prefix acnp
285 ON COALESCE(acnp.label, '') = COALESCE(callnum_prefix, '')
286 INNER JOIN asset.call_number_suffix acns
287 ON COALESCE(acns.label, '') = COALESCE(callnum_suffix, '')
289 ------------------------------------------------------------------------------
291 Generate the copies for your holdings:
294 ------------------------------------------------------------------------------
295 INSERT INTO asset.copy (
296 circ_lib, creator, editor, call_number, location,
297 loan_duration, fine_level, barcode
299 SELECT DISTINCT aou.id, 1, 1, acn.id, acl.id, 2, 2, barcode
300 FROM staging_materials sm
301 INNER JOIN actor.org_unit aou
302 ON aou.shortname = sm.owning_lib
303 INNER JOIN asset.copy_location acl
304 ON acl.name = sm.location
305 INNER JOIN asset.call_number acn
306 ON acn.label = sm.callnum
307 WHERE acn.deleted IS FALSE
309 ------------------------------------------------------------------------------
311 Generate the parts for your holdings. First, create the set of parts that are
312 required for each record based on your staging materials table:
315 ------------------------------------------------------------------------------
316 INSERT INTO biblio.monograph_part (record, label)
317 SELECT DISTINCT bibkey, part
318 FROM staging_materials sm
319 WHERE part IS NOT NULL AND NOT EXISTS (
320 SELECT 1 FROM biblio.monograph_part bmp
321 WHERE sm.part = bmp.label
322 AND sm.bibkey = bmp.record
324 ------------------------------------------------------------------------------
326 Now map the parts for each record to the specific copies that you added:
329 ------------------------------------------------------------------------------
330 INSERT INTO asset.copy_part_map (target_copy, part)
331 SELECT DISTINCT acp.id, bmp.id
332 FROM staging_materials sm
333 INNER JOIN asset.copy acp
334 ON acp.barcode = sm.barcode
335 INNER JOIN biblio.monograph_part bmp
336 ON bmp.record = sm.bibkey
337 WHERE part IS NOT NULL
339 AND acp.deleted IS FALSE
341 SELECT 1 FROM asset.copy_part_map
342 WHERE target_copy = acp.id
345 ------------------------------------------------------------------------------
347 At this point, you have loaded your bibliographic records, call numbers, call
348 number prefixes and suffixes, copies, and parts, and your records should be
349 visible to searches in the public catalog within the appropriate organization