Evergreen ILS Website

IRC log for #evergreen, 2021-08-20

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
06:01 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
07:17 JBoyer Bmagic, check out https://www.loc.gov/marc/authority/ad008.html to see what's what for Auth records. Positions 14-16 in the 008 say what a record is used for
07:19 rjackson_isl_hom joined #evergreen
07:42 collum joined #evergreen
08:36 Dyrcona joined #evergreen
08:43 Dyrcona We have a vendor asking us to send them records with correct length field values. I'm looking into why we're not generating records with correct lengths on export. I wonder if anyone else has noticed this?
08:44 rfrasur joined #evergreen
08:45 Dyrcona They also sent a report with some example bad fields, including this gem: 852-  $y:  Subfield cannot be defined in this field.
08:47 rfrasur oh my.  I didn't see the beginning of this thread, Dyrcona, but that doesn't sound good.
08:47 rhamby_ ewww
08:48 rhamby_ Dyrcona - my brain is not fully in gear this morning but is it this issue? https://bugs.launchpad.net/evergreen/+bug/1671845
08:48 pinesol Launchpad bug 1671845 in Evergreen "marc_export creating MARC data that yaz-marcdump dislikes" [Medium,Confirmed]
08:50 Dyrcona rfrasur: 852 is what we use for holdings export. I'll have to check at LoC, but I'm not sure $y is really a problem.
08:50 tlittle joined #evergreen
08:50 rfrasur ++
08:51 Dyrcona rhamby++
08:52 Dyrcona I'll run my output through yaz-marcdump and see what it says. Most of the length errors seem to involve records with multibyte characters and we say the record is two to four bytes longer than the vendor does. I supsect that they might not be counting lengths correctly.
08:53 Dyrcona I know we do have a bunch of garbage records with bad indicators and other junk.
08:55 mmorgan joined #evergreen
08:58 Dyrcona I've also asked this vendor if they can accept records as UTF-8 and not MARC-8.
09:04 Bmagic JBoyer++
09:07 Dyrcona yaz-marcdump's messages are next to useless.
09:08 Dyrcona "Separator but not at end of field...."
09:09 Dyrcona Here's another queition: They're saying MARCEdit is reporting the record length as being wrong. Is MARCEdit known to have issues with multi-byte characters?
09:14 Bmagic JBoyer: That helps tons. I was running in circles trying to find the "glue". I should have known it was articulated in the marc somewhere. Evergreen respects that marc record in pretty much every way possible. Authority control is no different :)
09:15 JBoyer Well, that's what it *should* do, I haven't looked that closely at what our auth stuf *actually* does. ;)
09:15 JBoyer But it's probably using those positions correctly.
09:16 Bmagic follow-up question: What's the magic to make Evergreen provide the "See Also" references when performing a browse search? I have plenty of examples where it gives us "See" (which is the 4XX search term -> 1XX) but never 5XX ->1XX
09:17 JBoyer And from here we're on equal footing as I don't know either.
09:18 Bmagic darn
09:18 * Bmagic greps the code for "see also"
09:21 Dyrcona Great. The records with bad lengths don't show up in the validation report. The validation report includes the 001, so I could easily find the record in the database to try exporting it in MARC8 and UTF8 for comparison.
09:31 Dyrcona I'm starting to wonder if piping a list of ids into marc_export actually works. I'm trying to export 1 record that way, and it looks like it will try to export everything.
09:33 jvwoolf joined #evergreen
09:37 Dyrcona So, it looks like the --library option interferes with that.
09:39 Dyrcona Hmm. Maybe 'cause the record I chose is deleted?
09:41 Dyrcona Nope. If I specify a --library option is ignores the record id....
09:42 Dyrcona You'd think I would know this....
09:44 Dyrcona Complicated software with overloaded options....
09:49 Dyrcona How to fix this? I think we need yet another option or to change one that I suspect no one uses: --location.
09:51 Dyrcona Did berick one propose a --pipe option to marc_export? I swear I remember that.
09:51 Bmagic JBoyer: I discovered a global flag: "opac.show_related_headings_in_browse" but that's enabled in my case and I'm still not getting "see also" - but it's a clue
09:52 Bmagic Dyrcona: I think I remember that too
09:52 JBoyer I suspect it may also depend on certain things being linked correctly by the auth linker, but it's hard to say.
09:59 Dyrcona Bmagic: I couldn't find a Lp bug for it, so I made one: Lp 1940662
09:59 pinesol Launchpad bug 1940662 in Evergreen "marc_export should have a --pipe option to force acceptance of a list of ids on standard input" [Undecided,New] https://launchpad.net/bugs/1940662
10:06 Dyrcona Should --pipe conflict with --since?
10:06 Dyrcona Maybe not...
10:08 Bmagic JBoyer: yeah, that's what I was thinking to begin with. Which makes me wonder if there are 2 processes that need to run on the server. One for bibs -> authority (which is running nightly) and another for authority -> Bibs (which is not running at all)
10:19 Bmagic I guess I can just run authority_control_fields.pl for the whole DB (when new auth records are added?)
10:22 Dyrcona Bmagic: We send records out every quarter to a vendor for cleanup and authority work. I run authority_control_fields.pl after we load the returned records.
10:22 Bmagic but for the full DB?
10:23 Bmagic as opposed to days-back=1 or whatever
10:27 Dyrcona That was easy.
10:32 Dyrcona So, with this one record, it does look like we export a bad record length when export a MARC8 record with diacritics.
10:40 Bmagic that makes sense
10:47 Dyrcona Already using --pipe in production.
10:49 Dyrcona I also think the --location option, which I'm pretty sure nobody else uses, could stand to be more complex.
10:50 Dyrcona I wonder if it would be useful to store a National Library Code in org unit settings and have an option to replace shortnames with that code in a holdings export?
10:57 Dyrcona Bmagic: There are two authority linker scripts in support scripts.
10:57 Bmagic one for linking bibs to authority and one linking authority to authority
10:58 Bmagic this was helpful: https://docs.evergreen-ils.org/eg/d​ocs/latest/development/support_scri​pts.html#authority_control_fields
10:58 Dyrcona Right.
10:58 Bmagic but no mention of the authority_authority one
10:59 Dyrcona When we do our quarterly updates, we actually run them via eg_staged_bib_overlay from the EOLI migration-tools repo: https://github.com/EquinoxOpenLi​braryInitiative/migration-tools
10:59 Bmagic Me too :)
11:00 Dyrcona Well, then, looks like i have nothing new to tell ya. :)
11:00 Bmagic Dyrcona: do you get "See also" references in browse search results?
11:01 Dyrcona I'm not sure. I'll have to take a look.
11:04 Dyrcona I get See references in the browse list.
11:05 Dyrcona I don't see any "See also."
11:08 Bmagic ok, same here
11:09 Bmagic though, I think* the wording is still "See" even when it's referring to 5XX's in Authority records. As per browse.tt2 heading_use_label(use=h.type)
11:10 Bmagic CASE 'variant';
11:11 Dyrcona I don't know. I've never been super interested in authorities.
11:27 Bmagic This is the call that would yeild what I want: open-ils.supercat.authority.author.refs.browse
11:28 Bmagic greping the code for that call isn't getting me anywhere, but it's probably not written out like that in the calling code
11:33 rfrasur_v2 joined #evergreen
12:06 Dyrcona Bmagic: I found it on line 1,226 of Supercat.pm. It goes through the general_authority_browse function.
12:06 Bmagic looking
12:06 Dyrcona You can't always find method names via grep because some methods names are built from code.
12:06 Dyrcona This isn't one of those, however.
12:07 jihpringle joined #evergreen
12:08 Bmagic I found lots of " if ($self->api_name =~ /\.refs\./)"  but where is the code that includes "refs" in the api call?
12:09 Bmagic I'm aware of the definition of the api call, just can't find the code that calls it
12:11 Bmagic It's gotta be driven by the template toolkit stuff somewhere. I was thinking misc_utils
12:19 Dyrcona Bmagic: The $self->api_name is the open-ils.supercat.authority.author.refs.brosse bit. It is "magically" passed by OpenSRF.
12:19 Dyrcona Well, browse, not brosse.
12:20 Bmagic right, but something needs to call it
12:20 Dyrcona You'll notice that function is used by a number of different APIs. It's common to see things like that in the Evergreen Perl code.
12:21 Dyrcona Well, sure, someone needs to call it.
12:21 Bmagic that's where I'm stuck. Where is it called
12:22 Dyrcona It's going be under OpenILS/WWW/EGcatLoader somewhere, and it may be built programmatically, i.e. a variable could be set up to call an appropriate API depending on if you're looking at authors, subjects, etc.
12:22 Bmagic looking
12:24 Dyrcona Bmagic: Open-ILS/src/perlmods/lib/OpenILS/WWW/SuperCat.pm
12:24 Bmagic :) I was there
12:24 Bmagic as you typed
12:24 Bmagic line 1714!
12:24 Dyrcona I'm not sure that comes into play for authority stuff, though.
12:26 Dyrcona I'm not really expert in that, though.
12:27 Bmagic well, 1714 isn't exactly right. That's a different api call
12:27 Bmagic but we are warm (I think)
12:27 Bmagic at least there is something in here that looks like it's generating the api call with "ref" in the call
12:52 jeffdavis We're once again seeing excessive duplicate open-ils.search.z3950.search_class calls - most recently 2000 of them over an 11-minute period, all searching the same targets for the same ISBN. It's enough to max out open-ils.search drones.
12:54 jeffdavis (I think it's specifically the resulting native EG catalog ISBN searches that are exhausting drones.)
13:01 Dyrcona jeffdavis: What Eg version?
13:01 jeffdavis 3.7 beta-ish
13:02 Dyrcona Ok. I've not seen that on 3.5.3, but we may also have very different Z39.50 use patterns.
13:09 Dyrcona I wonder if I should open this bug on Evergreen or on marc-perl on github? I've got a bib that consistently produces a record with a bad length when exported in MARC8.
13:10 jeffdavis bug 1940698 for the Z39.50 thing
13:10 pinesol Launchpad bug 1940698 in Evergreen "Duplicate open-ils.search.z3950.search_class calls lead to drone exhaustion" [Undecided,New] https://launchpad.net/bugs/1940698
13:17 Dyrcona FYI: I just used the --pipe option on marc_export in production and it did exactly what I expected: Lp 1940662.
13:17 pinesol Launchpad bug 1940662 in Evergreen "marc_export should have a --pipe option to force acceptance of a list of ids on standard input" [Wishlist,New] https://launchpad.net/bugs/1940662
13:22 collum joined #evergreen
13:58 jeff Dyrcona: do you think that bug 1940702 is a different issue than the existing bug 1671845?
13:58 pinesol Launchpad bug 1940702 in Evergreen "MARC8 records with diacritics are exported with incorrect record length" [Undecided,New] https://launchpad.net/bugs/1940702
13:58 pinesol Launchpad bug 1671845 in Evergreen "marc_export creating MARC data that yaz-marcdump dislikes" [Medium,Confirmed] https://launchpad.net/bugs/1671845
13:59 Dyrcona jeff: I think so, but I can't say for certain because the yaz-marcdump output is cryptic. I haven't been able to find anything that explains the mssages.
13:59 jeff got it.
14:02 Dyrcona It could be a bunch of different issues.
14:02 rhamby_ It's been a long week.  I read Dyrcona's 'mssages' and my brain went 'massages' instead of 'messages' and I was really confused for a second
14:03 Dyrcona :)
17:08 mmorgan left #evergreen
18:01 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat