| Time |
Nick |
Message |
| 06:02 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:12 |
|
rjackson_isl_hom joined #evergreen |
| 08:11 |
|
collum joined #evergreen |
| 08:25 |
|
mantis joined #evergreen |
| 08:28 |
|
rfrasur joined #evergreen |
| 08:30 |
|
rjackson_isl_hom joined #evergreen |
| 08:34 |
|
mmorgan joined #evergreen |
| 08:48 |
|
Dyrcona joined #evergreen |
| 09:06 |
|
jvwoolf joined #evergreen |
| 09:12 |
|
collum joined #evergreen |
| 09:15 |
|
alynn26 joined #evergreen |
| 09:18 |
Dyrcona |
Poor Bilbio Recordentry.... Not as famous as as his cousin Biblo Baggins.... :P |
| 09:18 |
Dyrcona |
Hah! I even typoed the joke about the typo... Seems appropriate, though. :P |
| 09:38 |
mmorgan |
:) |
| 09:42 |
Dyrcona |
If I had noticed, I would have gone all the way with Biblio Baggins. :) |
| 10:06 |
csharp |
@band add Bilbio Recordentry |
| 10:06 |
pinesol |
csharp: Band 'Bilbio Recordentry' added to list |
| 10:09 |
Dyrcona |
@band add Biblio Baggins |
| 10:09 |
pinesol |
Dyrcona: Band 'Biblio Baggins' added to list |
| 10:09 |
Dyrcona |
:) |
| 11:00 |
|
rjackson_isl_hom joined #evergreen |
| 11:18 |
|
collum_ joined #evergreen |
| 12:23 |
|
collum joined #evergreen |
| 12:45 |
|
jihpringle joined #evergreen |
| 12:55 |
pastebot |
"jeffdavis" at 168.25.130.30 pasted ""INSERT INTO biblio.record_entry" performance on an upgraded 3.7 system" (37 lines) at http://paste.evergreen-ils.org/10138 |
| 12:57 |
jeffdavis |
^ In my 3.7 test environment, once search.symspell_dictionary is populated, the maintain_symspell_entries_tgr triggers on metabib tables make creating a bib record unusably slow. Not sure if this is an issue with the environment or a performance problem with the symspell stuff. |
| 12:59 |
jeff |
the aaa_indexing_ingest_or_delete line is unfortunately a little opaque there. :-( |
| 13:05 |
jeff |
(though we can probably safely assume all the extra time is search.symspell_maintain_entries) |
| 13:07 |
jeffdavis |
yeah, the only difference between the two results is that I disabled the symspell triggers before re-running the same insert |
| 13:08 |
jeff |
statistics up to date, in terms of VACUUM ANALYZE on relevant (or all) tables? |
| 13:11 |
|
jvwoolf joined #evergreen |
| 13:17 |
jeffdavis |
I've run VACUUM ANALYZE on search.symspell_dictionary at least |
| 13:19 |
jeff |
profiling/debugging the symspell triggers using RAISE LOG seems crude, but it might be a start, unless you already have a debugger that you're familiar with installed. |
| 13:20 |
jeff |
i'm looking at parts of these functions: search.symspell_build_and_merge_entries, evergreen.text_array_merge_unique, search.symspell_build_entries |
| 13:21 |
jeff |
ah, and there's the perl... |
| 13:21 |
jeff |
jeffdavis: you might be better served by someone else speaking up who may have already encountered this. I'm looking at most of the symspell stuff for the first time now. |
| 13:31 |
|
rjackson_isl_hom joined #evergreen |
| 13:39 |
Dyrcona |
jeff | jeffdavis: I use pl_profiler https://github.com/bigsql/plprofiler |
| 13:39 |
Dyrcona |
Not sure how much it would help with the Perl in the database, but it has helped me improve some queries. |
| 13:44 |
jeff |
i think most of the symspell perl db functions are wrapping single perl function calls, like: use String::KeyboardDistance qw(:all); return qwerty_keyboard_distance(@_); |
| 13:45 |
Dyrcona |
Yeah, that's what it looked like when I perused the code before. I suspect the issue is that we've added yet another ingest when updating a bib record. |
| 13:47 |
Dyrcona |
This should also be faster on newer releases of Pg, since a lot of that work can be done in parallel. I've seen an auth record update go from taking over 6 minutes on Pg 9.6 down to a couple of seconds of Pg 12. |
| 13:47 |
Dyrcona |
Same hardare/same optimization settings. |
| 13:48 |
Dyrcona |
Same data.... as if I needed to add that. |
| 13:48 |
Dyrcona |
That said, records with located URIs take longer to update on Pg 12. |
| 13:50 |
|
jihpringle joined #evergreen |
| 13:57 |
Dyrcona |
*thinking aloud* Hm. Guess I'll write this program in Python. I need to search several directories of code looking for references to custom tables and views to see if we're still using them. |
| 14:07 |
jeffdavis |
Running search.symspell_build_and_merge_entries() directly seems fast, consistently under 50ms (usually well under) except on a large integer as an identifier, which is more like 400ms. |
| 14:22 |
|
jihpringle joined #evergreen |
| 14:22 |
Dyrcona |
jeffdavis: If you install plprofiler and do an update using a script, you can get output that breaks down every function that runs with how long each takes. |
| 14:31 |
Bmagic |
I stumbled into a conundrum. When I updatea MARC record via the staff client, I see that it creates the 035, moving the 001 down. But when I update the database on the exact same record via "UPDATE biblio.record_entry SET marc=$$....$$ where id=xxxxx" - The record does NOT get it's 001 moved to a 035 |
| 14:32 |
Bmagic |
I know it's a DB trigger. So it should* be executed whether the driving update was via staff client or from psql. The setting in config.global_flag cat.maintain_control_numbers is enabled |
| 14:35 |
Dyrcona |
Bmagic: It's a different flag: cat.bib.use_id_for_tcn |
| 14:36 |
Bmagic |
that one is enabled as well |
| 14:36 |
Bmagic |
it works when I update the record in the staff client |
| 14:41 |
Dyrcona |
It moves the 001 to 003, not 035. |
| 14:41 |
Dyrcona |
The client *may* be doing something with 035. IDK. |
| 14:43 |
jeff |
Bmagic: UPDATE vs INSERT, I believe. |
| 14:43 |
jeff |
oh, but you said when you update in the staff client, not create. |
| 14:43 |
Bmagic |
right, it's an UPDATE in both cases |
| 14:44 |
Dyrcona |
Heh, this comment: # Stupid VIM syntax highlighting" |
| 14:44 |
Dyrcona |
The trigger fires before update or insert in my db. |
| 14:44 |
Bmagic |
I noticed that comment! |
| 14:44 |
Dyrcona |
c_maintain_control_numbers BEFORE INSERT OR UPDATE ON biblio.record_entry FOR EACH ROW EXECUTE PROCEDURE maintain_control_numbers() |
| 14:45 |
Bmagic |
so the issue is that it happens BEFORE ? |
| 14:47 |
Dyrcona |
I was wrong. It does put data in the 035. I had barely skimmed the code when I said that. |
| 14:47 |
JBoyer |
The staff client is calling an opensrf api call to update the marc which does more things than just running an UPDATE biblio.record_entry... I don't recall what it uses but I know that's the case because when the staff client used pcrud there was a bug filed about it missing something. |
| 14:47 |
Dyrcona |
Bmagic: No. That isn't the problem at all. If it didnt do it before, the reocrd couldn't be updated. |
| 14:47 |
jeff |
Bmagic: reading evergreen.maintain_control_numbers source should answer your question, but I think it boils down to your second edit doesn't meet the requirements for munging the 001, AND/OR the "don't create a duplicate 035" is coming into play |
| 14:48 |
jeff |
ah, JBoyer may be on it. |
| 14:48 |
Bmagic |
jerff: I read that function several times. And concluded that it should be making that 035 |
| 14:48 |
jeff |
Bmagic: hard to guess without knowing the circumstances of the record, which is tricky to extract and share. :-) |
| 14:48 |
Bmagic |
but I agree, something in there is causing it to not* perform it's stuff |
| 14:49 |
Dyrcona |
Bmagic: You're updating, not inserting, right? |
| 14:49 |
Bmagic |
right |
| 14:49 |
Bmagic |
maybe it's a search path thing? |
| 14:49 |
Dyrcona |
Does the 035 already exist in the record? |
| 14:51 |
Bmagic |
no, I deleted all the 035 prior to my test |
| 14:51 |
Dyrcona |
Is the 001 the same as the biblio.record_entry.id? |
| 14:52 |
Bmagic |
no |
| 14:53 |
Bmagic |
in my SQL statement, I have my marc xml setup so that the 001 is something totally different. And I would expect it to land in the database in the 035, and the 001 would be assigned the bib ID |
| 14:55 |
jeff |
regarding what JBoyer said, if the staff client update is being done with open-ils.cat.biblio.record.xml.update, take a look at OpenILS::Application::Cat::BibCommon::biblio_record_replace_marc and its tcn logic and tag stripping to see if you find an answer there. |
| 14:56 |
JBoyer |
Oh, that bug may have just been because of the tcn value; that function may or may not have any affect on the 001 / 035 then. |
| 14:57 |
Bmagic |
looking |
| 14:57 |
Dyrcona |
THAT, and taking a closer look, the maintain_control_numbers function doesn't look at cat.bib.use_id_for_tcn. It also doesn't set the 001. So, database inserts do that stuff for you. |
| 14:59 |
Dyrcona |
My first hunch when Bmagic said it was a database trigger was, "that doesn't do what you think it does," but I ignored that and went looking. |
| 14:59 |
Dyrcona |
Finally, perl-- :) |
| 15:00 |
Bmagic |
right, I think you're caught up with where I am now |
| 15:00 |
Bmagic |
that function - maintain_control_numbers() is where the magic happens |
| 15:00 |
Dyrcona |
What I meant to say was: If you update bibs in the database, you have to move the 001, etc., yourself. |
| 15:00 |
Dyrcona |
Not the magic you want, though. |
| 15:01 |
Bmagic |
it's all* there. I don't think it happens in the server-side Perl |
| 15:01 |
Bmagic |
no? |
| 15:01 |
Dyrcona |
To reiterate: |
| 15:02 |
* jeff |
hopes that nobody involved in the conversation has a public.maintain_control_numbers and an evergreen.maintain_control_numbers :-) |
| 15:02 |
Dyrcona |
maintain_control_numbers DOES NOT reference cat.bib.use_id_for_tcn and DOES NOT set the 001 to the record ID. |
| 15:02 |
Bmagic |
oh boy |
| 15:02 |
jeff |
Dyrcona: looking at master, evergreen.maintain_control_numbers appears to set an 001 if certain conditions are met. |
| 15:03 |
Dyrcona |
I'm on 3.5, and I only have evergreen.maintain_control_numbers. |
| 15:04 |
jeff |
https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/sql/Pg/002.functions.config.sql;h=9a0d549cbb01cc0ae3a92c33115bfe0a1fd2c943;hb=HEAD#l559 |
| 15:04 |
Bmagic |
evergreen.maintain_control_numbers refrences the setting "cat.maintain_control_numbers" (which is enabled) - and if it is enabled, the function should be moving the 001 to the 035. |
| 15:04 |
Dyrcona |
Yeah, just ignore everything that I've said. I'm trying to do too many things at once... |
| 15:04 |
jeff |
Dyrcona++ |
| 15:05 |
Dyrcona |
It doesn't look at cat.bib.use_id_for_tcn, so that's possibly a bug. |
| 15:06 |
Dyrcona |
Now, what was *I* trying to do? |
| 15:09 |
Dyrcona |
*mumbles* We do to much in the database. |
| 15:19 |
|
jvwoolf joined #evergreen |
| 15:22 |
Dyrcona |
I guess the question is which Python module do I use to parse fm_IDL.xml and search it with XPath? |
| 15:23 |
|
sandbergja joined #evergreen |
| 15:24 |
Bmagic |
I found the problem |
| 15:26 |
Bmagic |
Dyrcona, jeff: My SQL inserted XML contained a 001 but not a 003. The lack of the 003 caused the variable $create to become true as well as $munge to become true. In the case where both $munge and $create are true, this if statement evaluates as false if ($munge and not $create) { |
| 15:27 |
Bmagic |
to figure that out, I replaced the function a few times with some alterations, writing out debugging info |
| 15:28 |
Bmagic |
the logic would lead us to the conclusion: "If I have to create either a 001 or a 003, then you don't get a 035" |
| 15:29 |
Bmagic |
it would make perfect sense to not* create a 035 if a 001 didn't exist. But does it make sense to not* create the 035 when the 003 doesn't exist? |
| 15:37 |
jeff |
I suspect the logic is "we don't have all the required info to create the 035" |
| 15:38 |
Dyrcona |
Grr... xml namespaces........ |
| 15:38 |
jeff |
I'm not sure if an "unknown" value is defined / suitable. |
| 15:39 |
|
mantis left #evergreen |
| 15:39 |
jeff |
of course, we probably have lots of folk who aren't putting a registered organization code in there anyway... |
| 15:40 |
jeff |
no strong opinion, but others may have one. bug it for discussion? |
| 15:40 |
Dyrcona |
Also, my fingers are full of typos today.... |
| 16:07 |
|
dbwells joined #evergreen |
| 16:26 |
|
jvwoolf1 joined #evergreen |
| 16:36 |
|
dbwells joined #evergreen |
| 17:17 |
|
dbwells joined #evergreen |
| 17:31 |
|
mmorgan left #evergreen |
| 17:31 |
|
dbwells joined #evergreen |
| 17:33 |
|
jvwoolf1 left #evergreen |
| 17:50 |
|
dbwells joined #evergreen |
| 18:02 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:15 |
|
dbwells joined #evergreen |
| 18:35 |
|
sandbergja joined #evergreen |
| 19:49 |
|
jihpringle joined #evergreen |
| 21:31 |
|
dbwells joined #evergreen |
| 21:48 |
|
dbwells joined #evergreen |
