| Time |
Nick |
Message |
| 06:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:17 |
JBoyer |
Bmagic, check out https://www.loc.gov/marc/authority/ad008.html to see what's what for Auth records. Positions 14-16 in the 008 say what a record is used for |
| 07:19 |
|
rjackson_isl_hom joined #evergreen |
| 07:42 |
|
collum joined #evergreen |
| 08:36 |
|
Dyrcona joined #evergreen |
| 08:43 |
Dyrcona |
We have a vendor asking us to send them records with correct length field values. I'm looking into why we're not generating records with correct lengths on export. I wonder if anyone else has noticed this? |
| 08:44 |
|
rfrasur joined #evergreen |
| 08:45 |
Dyrcona |
They also sent a report with some example bad fields, including this gem: 852- $y: Subfield cannot be defined in this field. |
| 08:47 |
rfrasur |
oh my. I didn't see the beginning of this thread, Dyrcona, but that doesn't sound good. |
| 08:47 |
rhamby_ |
ewww |
| 08:48 |
rhamby_ |
Dyrcona - my brain is not fully in gear this morning but is it this issue? https://bugs.launchpad.net/evergreen/+bug/1671845 |
| 08:48 |
pinesol |
Launchpad bug 1671845 in Evergreen "marc_export creating MARC data that yaz-marcdump dislikes" [Medium,Confirmed] |
| 08:50 |
Dyrcona |
rfrasur: 852 is what we use for holdings export. I'll have to check at LoC, but I'm not sure $y is really a problem. |
| 08:50 |
|
tlittle joined #evergreen |
| 08:50 |
rfrasur |
++ |
| 08:51 |
Dyrcona |
rhamby++ |
| 08:52 |
Dyrcona |
I'll run my output through yaz-marcdump and see what it says. Most of the length errors seem to involve records with multibyte characters and we say the record is two to four bytes longer than the vendor does. I supsect that they might not be counting lengths correctly. |
| 08:53 |
Dyrcona |
I know we do have a bunch of garbage records with bad indicators and other junk. |
| 08:55 |
|
mmorgan joined #evergreen |
| 08:58 |
Dyrcona |
I've also asked this vendor if they can accept records as UTF-8 and not MARC-8. |
| 09:04 |
Bmagic |
JBoyer++ |
| 09:07 |
Dyrcona |
yaz-marcdump's messages are next to useless. |
| 09:08 |
Dyrcona |
"Separator but not at end of field...." |
| 09:09 |
Dyrcona |
Here's another queition: They're saying MARCEdit is reporting the record length as being wrong. Is MARCEdit known to have issues with multi-byte characters? |
| 09:14 |
Bmagic |
JBoyer: That helps tons. I was running in circles trying to find the "glue". I should have known it was articulated in the marc somewhere. Evergreen respects that marc record in pretty much every way possible. Authority control is no different :) |
| 09:15 |
JBoyer |
Well, that's what it *should* do, I haven't looked that closely at what our auth stuf *actually* does. ;) |
| 09:15 |
JBoyer |
But it's probably using those positions correctly. |
| 09:16 |
Bmagic |
follow-up question: What's the magic to make Evergreen provide the "See Also" references when performing a browse search? I have plenty of examples where it gives us "See" (which is the 4XX search term -> 1XX) but never 5XX ->1XX |
| 09:17 |
JBoyer |
And from here we're on equal footing as I don't know either. |
| 09:18 |
Bmagic |
darn |
| 09:18 |
* Bmagic |
greps the code for "see also" |
| 09:21 |
Dyrcona |
Great. The records with bad lengths don't show up in the validation report. The validation report includes the 001, so I could easily find the record in the database to try exporting it in MARC8 and UTF8 for comparison. |
| 09:31 |
Dyrcona |
I'm starting to wonder if piping a list of ids into marc_export actually works. I'm trying to export 1 record that way, and it looks like it will try to export everything. |
| 09:33 |
|
jvwoolf joined #evergreen |
| 09:37 |
Dyrcona |
So, it looks like the --library option interferes with that. |
| 09:39 |
Dyrcona |
Hmm. Maybe 'cause the record I chose is deleted? |
| 09:41 |
Dyrcona |
Nope. If I specify a --library option is ignores the record id.... |
| 09:42 |
Dyrcona |
You'd think I would know this.... |
| 09:44 |
Dyrcona |
Complicated software with overloaded options.... |
| 09:49 |
Dyrcona |
How to fix this? I think we need yet another option or to change one that I suspect no one uses: --location. |
| 09:51 |
Dyrcona |
Did berick one propose a --pipe option to marc_export? I swear I remember that. |
| 09:51 |
Bmagic |
JBoyer: I discovered a global flag: "opac.show_related_headings_in_browse" but that's enabled in my case and I'm still not getting "see also" - but it's a clue |
| 09:52 |
Bmagic |
Dyrcona: I think I remember that too |
| 09:52 |
JBoyer |
I suspect it may also depend on certain things being linked correctly by the auth linker, but it's hard to say. |
| 09:59 |
Dyrcona |
Bmagic: I couldn't find a Lp bug for it, so I made one: Lp 1940662 |
| 09:59 |
pinesol |
Launchpad bug 1940662 in Evergreen "marc_export should have a --pipe option to force acceptance of a list of ids on standard input" [Undecided,New] https://launchpad.net/bugs/1940662 |
| 10:06 |
Dyrcona |
Should --pipe conflict with --since? |
| 10:06 |
Dyrcona |
Maybe not... |
| 10:08 |
Bmagic |
JBoyer: yeah, that's what I was thinking to begin with. Which makes me wonder if there are 2 processes that need to run on the server. One for bibs -> authority (which is running nightly) and another for authority -> Bibs (which is not running at all) |
| 10:19 |
Bmagic |
I guess I can just run authority_control_fields.pl for the whole DB (when new auth records are added?) |
| 10:22 |
Dyrcona |
Bmagic: We send records out every quarter to a vendor for cleanup and authority work. I run authority_control_fields.pl after we load the returned records. |
| 10:22 |
Bmagic |
but for the full DB? |
| 10:23 |
Bmagic |
as opposed to days-back=1 or whatever |
| 10:27 |
Dyrcona |
That was easy. |
| 10:32 |
Dyrcona |
So, with this one record, it does look like we export a bad record length when export a MARC8 record with diacritics. |
| 10:40 |
Bmagic |
that makes sense |
| 10:47 |
Dyrcona |
Already using --pipe in production. |
| 10:49 |
Dyrcona |
I also think the --location option, which I'm pretty sure nobody else uses, could stand to be more complex. |
| 10:50 |
Dyrcona |
I wonder if it would be useful to store a National Library Code in org unit settings and have an option to replace shortnames with that code in a holdings export? |
| 10:57 |
Dyrcona |
Bmagic: There are two authority linker scripts in support scripts. |
| 10:57 |
Bmagic |
one for linking bibs to authority and one linking authority to authority |
| 10:58 |
Bmagic |
this was helpful: https://docs.evergreen-ils.org/eg/docs/latest/development/support_scripts.html#authority_control_fields |
| 10:58 |
Dyrcona |
Right. |
| 10:58 |
Bmagic |
but no mention of the authority_authority one |
| 10:59 |
Dyrcona |
When we do our quarterly updates, we actually run them via eg_staged_bib_overlay from the EOLI migration-tools repo: https://github.com/EquinoxOpenLibraryInitiative/migration-tools |
| 10:59 |
Bmagic |
Me too :) |
| 11:00 |
Dyrcona |
Well, then, looks like i have nothing new to tell ya. :) |
| 11:00 |
Bmagic |
Dyrcona: do you get "See also" references in browse search results? |
| 11:01 |
Dyrcona |
I'm not sure. I'll have to take a look. |
| 11:04 |
Dyrcona |
I get See references in the browse list. |
| 11:05 |
Dyrcona |
I don't see any "See also." |
| 11:08 |
Bmagic |
ok, same here |
| 11:09 |
Bmagic |
though, I think* the wording is still "See" even when it's referring to 5XX's in Authority records. As per browse.tt2 heading_use_label(use=h.type) |
| 11:10 |
Bmagic |
CASE 'variant'; |
| 11:11 |
Dyrcona |
I don't know. I've never been super interested in authorities. |
| 11:27 |
Bmagic |
This is the call that would yeild what I want: open-ils.supercat.authority.author.refs.browse |
| 11:28 |
Bmagic |
greping the code for that call isn't getting me anywhere, but it's probably not written out like that in the calling code |
| 11:33 |
|
rfrasur_v2 joined #evergreen |
| 12:06 |
Dyrcona |
Bmagic: I found it on line 1,226 of Supercat.pm. It goes through the general_authority_browse function. |
| 12:06 |
Bmagic |
looking |
| 12:06 |
Dyrcona |
You can't always find method names via grep because some methods names are built from code. |
| 12:06 |
Dyrcona |
This isn't one of those, however. |
| 12:07 |
|
jihpringle joined #evergreen |
| 12:08 |
Bmagic |
I found lots of " if ($self->api_name =~ /\.refs\./)" but where is the code that includes "refs" in the api call? |
| 12:09 |
Bmagic |
I'm aware of the definition of the api call, just can't find the code that calls it |
| 12:11 |
Bmagic |
It's gotta be driven by the template toolkit stuff somewhere. I was thinking misc_utils |
| 12:19 |
Dyrcona |
Bmagic: The $self->api_name is the open-ils.supercat.authority.author.refs.brosse bit. It is "magically" passed by OpenSRF. |
| 12:19 |
Dyrcona |
Well, browse, not brosse. |
| 12:20 |
Bmagic |
right, but something needs to call it |
| 12:20 |
Dyrcona |
You'll notice that function is used by a number of different APIs. It's common to see things like that in the Evergreen Perl code. |
| 12:21 |
Dyrcona |
Well, sure, someone needs to call it. |
| 12:21 |
Bmagic |
that's where I'm stuck. Where is it called |
| 12:22 |
Dyrcona |
It's going be under OpenILS/WWW/EGcatLoader somewhere, and it may be built programmatically, i.e. a variable could be set up to call an appropriate API depending on if you're looking at authors, subjects, etc. |
| 12:22 |
Bmagic |
looking |
| 12:24 |
Dyrcona |
Bmagic: Open-ILS/src/perlmods/lib/OpenILS/WWW/SuperCat.pm |
| 12:24 |
Bmagic |
:) I was there |
| 12:24 |
Bmagic |
as you typed |
| 12:24 |
Bmagic |
line 1714! |
| 12:24 |
Dyrcona |
I'm not sure that comes into play for authority stuff, though. |
| 12:26 |
Dyrcona |
I'm not really expert in that, though. |
| 12:27 |
Bmagic |
well, 1714 isn't exactly right. That's a different api call |
| 12:27 |
Bmagic |
but we are warm (I think) |
| 12:27 |
Bmagic |
at least there is something in here that looks like it's generating the api call with "ref" in the call |
| 12:52 |
jeffdavis |
We're once again seeing excessive duplicate open-ils.search.z3950.search_class calls - most recently 2000 of them over an 11-minute period, all searching the same targets for the same ISBN. It's enough to max out open-ils.search drones. |
| 12:54 |
jeffdavis |
(I think it's specifically the resulting native EG catalog ISBN searches that are exhausting drones.) |
| 13:01 |
Dyrcona |
jeffdavis: What Eg version? |
| 13:01 |
jeffdavis |
3.7 beta-ish |
| 13:02 |
Dyrcona |
Ok. I've not seen that on 3.5.3, but we may also have very different Z39.50 use patterns. |
| 13:09 |
Dyrcona |
I wonder if I should open this bug on Evergreen or on marc-perl on github? I've got a bib that consistently produces a record with a bad length when exported in MARC8. |
| 13:10 |
jeffdavis |
bug 1940698 for the Z39.50 thing |
| 13:10 |
pinesol |
Launchpad bug 1940698 in Evergreen "Duplicate open-ils.search.z3950.search_class calls lead to drone exhaustion" [Undecided,New] https://launchpad.net/bugs/1940698 |
| 13:17 |
Dyrcona |
FYI: I just used the --pipe option on marc_export in production and it did exactly what I expected: Lp 1940662. |
| 13:17 |
pinesol |
Launchpad bug 1940662 in Evergreen "marc_export should have a --pipe option to force acceptance of a list of ids on standard input" [Wishlist,New] https://launchpad.net/bugs/1940662 |
| 13:22 |
|
collum joined #evergreen |
| 13:58 |
jeff |
Dyrcona: do you think that bug 1940702 is a different issue than the existing bug 1671845? |
| 13:58 |
pinesol |
Launchpad bug 1940702 in Evergreen "MARC8 records with diacritics are exported with incorrect record length" [Undecided,New] https://launchpad.net/bugs/1940702 |
| 13:58 |
pinesol |
Launchpad bug 1671845 in Evergreen "marc_export creating MARC data that yaz-marcdump dislikes" [Medium,Confirmed] https://launchpad.net/bugs/1671845 |
| 13:59 |
Dyrcona |
jeff: I think so, but I can't say for certain because the yaz-marcdump output is cryptic. I haven't been able to find anything that explains the mssages. |
| 13:59 |
jeff |
got it. |
| 14:02 |
Dyrcona |
It could be a bunch of different issues. |
| 14:02 |
rhamby_ |
It's been a long week. I read Dyrcona's 'mssages' and my brain went 'massages' instead of 'messages' and I was really confused for a second |
| 14:03 |
Dyrcona |
:) |
| 17:08 |
|
mmorgan left #evergreen |
| 18:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
