| Time |
Nick |
Message |
| 08:01 |
|
BDorsey joined #evergreen |
| 08:31 |
|
redavis joined #evergreen |
| 08:32 |
|
jonadab joined #evergreen |
| 08:48 |
|
sleary joined #evergreen |
| 09:00 |
|
Dyrcona joined #evergreen |
| 09:06 |
|
mantis1 joined #evergreen |
| 09:08 |
|
dguarrac joined #evergreen |
| 09:10 |
redavis |
Stompro++ |
| 09:27 |
|
mantis1 left #evergreen |
| 09:37 |
Bmagic |
I'm having issues with edi fetcher getting stuck. The last log entry is Loading events xml file /openils/var/data/ils_events.xml. There are a few FTP login issues throughout the process but that doesn't seem to stop it from moving on. But it's still running right now, with nothing in the log. It's just hung |
| 09:38 |
Dyrcona |
Stompro: Yeah, I've noticed that some additional fields are "missing" from SuperCat item exports. I was planning to add copy notes and tags as part of Lp 2045440. I wonder if I shouldn't just open another bug to get parity between SuperCat and marc_export? There are features of marc_export that might be difficult to implement in SuperCat. |
| 09:38 |
pinesol |
Launchpad bug 2045440 in Evergreen "Add Public Copy Notes to MARC export when items requested" [Wishlist,New] https://launchpad.net/bugs/2045440 - Assigned to Jason Stephenson (jstephenson) |
| 09:42 |
Stompro |
Dyrcona, the Prefix and Suffix data are there I believe, the aspen fetcher just isn't grabbing them. I kind of want to make use of the Aspen import features to send things like audience and literary form from our stat cats.. but since the fetcher wouldn't be able to do the same thing it seems messy. |
| 09:42 |
Stompro |
I want to avoid cleaning up our bibs... but I think I have to. |
| 09:43 |
Dyrcona |
We found numerous "bad" bibs as a part of this project. |
| 09:44 |
Dyrcona |
Also: https://bugs.launchpad.net/evergreen/+bug/2045440/comments/7 |
| 09:44 |
pinesol |
Launchpad bug 2045440 in Evergreen "Add Public Copy Notes to MARC export when items requested" [Wishlist,New] - Assigned to Jason Stephenson (jstephenson) |
| 09:45 |
Dyrcona |
Stompro: Audience and literary form should be properly encoded in the MARC. I think it would be bad to have different Evergreen sites with customized exports for Aspen, etc. We really should be doing "standard" things as much as possible. |
| 09:59 |
|
sleary joined #evergreen |
| 10:06 |
|
mantis1 joined #evergreen |
| 10:21 |
|
mantis1 joined #evergreen |
| 10:50 |
Dyrcona |
And, I forgot that SuperCat doesn't export items via 852. It uses its own XML format. |
| 10:50 |
Dyrcona |
And, it already exports copy notes, but not tags. |
| 10:51 |
* Dyrcona |
checks 3.7 for that code. |
| 10:52 |
|
briank joined #evergreen |
| 10:56 |
Dyrcona |
Stompro++ |
| 10:56 |
Dyrcona |
You are right. The prefix and suffix are there in the SuperCat output. So are copy notes, but not copy tags. |
| 10:57 |
Dyrcona |
Guess Apen needs to be taught about that data. |
| 11:37 |
|
kmlussier joined #evergreen |
| 11:41 |
|
sandbergja joined #evergreen |
| 13:01 |
Bmagic |
I want to reingest all bibs and truncate the tables ahead of time. I just wanted to verify the list of tables that I can truncate: |
| 13:01 |
Bmagic |
metabib.keyword_field_entry; metabib.identifier_field_entry; metabib.display_entry; metabib.facet_entry; metabib.browse_entry_def_map; metabib.real_full_rec; metabib.series_field_entry; metabib.subject_field_entry; metabib.title_field_entry; metabib.author_field_entry |
| 13:01 |
Bmagic |
maybe I should hit the "combined" tables as well? |
| 13:06 |
jeffdavis |
Yes, I think you'd want to truncate the combined tables too in that case. |
| 13:07 |
jeffdavis |
You might want to truncate metabib.browse_entry as well? |
| 13:08 |
jeffdavis |
(I have no informed opinion on truncate-before-reingest as an overall strategy but if you're doing it it would make sense to include those tables I think.) |
| 13:13 |
Bmagic |
jeffdavis: sweet, thanks |
| 13:13 |
Bmagic |
I just want to feel better that there's no cruft |
| 13:20 |
berick |
Bmagic: consider me curious about how it goes |
| 13:21 |
Bmagic |
berick: you think it's not going to rebuild those tables? |
| 13:23 |
berick |
Bmagic: no, I'm just curious how it all works out. |
| 13:23 |
Dyrcona |
Truncating the tables won't have prevent the tables from being rebuilt. It might help slightly with speed as it will probably cut down on autovacuum tasks. |
| 13:23 |
Bmagic |
there is something that's making search slower than normal. To the point where the database server is having dogpiling queries. It's a bit of a coincidence that it's happening now that queued ingest is on (and we overwrote a ton of bibs), so the ingester is working hard. the bricks aren't get responses fast enough, and the whole thing is coming down |
| 13:24 |
Bmagic |
so, I turned off the ingester, and I'm thinking I'd like to "start over" - truncate all of it and rebuild |
| 13:25 |
Bmagic |
those tables could probably use some special tuning, so if you have any magic that you use, I'm open |
| 13:26 |
Bmagic |
2,021,909 non-deleted bibs atm |
| 13:27 |
Dyrcona |
Me? I'm fresh out of mana.... Have to cool down on the next turn. :) |
| 13:28 |
Bmagic |
search is slow as you might imagine, and I'm thinking that theoretically a fresh install of those metabib tables would help |
| 13:30 |
Dyrcona |
It won't hurt. |
| 13:30 |
Bmagic |
:) |
| 13:32 |
Dyrcona |
If the tables are badly fragmented, then a truncate and rebuild can only help. |
| 13:35 |
jeffdavis |
Bmagic: does vacuum help at all (if only temporarily)? What autovacuum settings are you using? (sorry if I've missed previous discussion about this) |
| 13:36 |
Bmagic |
nightly vacuum analyze; |
| 13:38 |
Bmagic |
but, now that you mention it, I could vacuum analyze those tables and see |
| 13:40 |
Bmagic |
this database server has 1TB memory, no swap. PG10, 1000 max_connections |
| 13:41 |
Bmagic |
maintenance_work_mem = 1GB constraint_exclusion = on checkpoint_completion_target = 0.9 effective_cache_size = 700GB work_mem = 512MB wal_buffers = 8MB shared_buffers = 32GB |
| 13:42 |
Dyrcona |
Bmagic: Did-you-mean is turned on? |
| 13:42 |
Bmagic |
it was*, that was the first thing to go |
| 13:43 |
Bmagic |
I disabled the triggers, for example: ALTER TABLE metabib.title_field_entry DISABLE TRIGGER maintain_symspell_entries_tgr; |
| 13:43 |
Bmagic |
and then truncated: TRUNCATE search.symspell_dictionary; |
| 13:56 |
|
sleary joined #evergreen |
| 13:59 |
Dyrcona |
Yeah, we disabled the symspell triggers, too. |
| 13:59 |
* Dyrcona |
opines about trying to summarize the release discussion from last Tuesday. It's proving more challenging than it sounds like it should be. |
| 14:18 |
|
jvwoolf joined #evergreen |
| 14:52 |
|
dguarrac_ joined #evergreen |
| 16:01 |
kmlussier |
Dyrcona++ |
| 16:03 |
Dyrcona |
Thank you. I hope my summary is accurate. |
| 16:04 |
Dyrcona |
I cut and rewrote whole paragraphs down to one or two sentences trying to condense it as much as possible. |
| 16:09 |
|
jvwoolf left #evergreen |
| 17:17 |
|
kmlussier left #evergreen |
