| Time |
Nick |
Message |
| 07:03 |
|
rjackson_isl joined #evergreen |
| 07:34 |
|
Dyrcona joined #evergreen |
| 07:36 |
|
bdljohn joined #evergreen |
| 08:24 |
|
jvwoolf joined #evergreen |
| 08:43 |
|
bos20k joined #evergreen |
| 08:43 |
|
mmorgan joined #evergreen |
| 08:46 |
Dyrcona |
Crazy. Doing a marc_export and the query only spends about 10 seconds in the database. The rest of the time seems to b I/O and manipulation of the output. |
| 08:49 |
|
terran joined #evergreen |
| 09:02 |
|
mdriscoll joined #evergreen |
| 09:03 |
rhamby |
seems about right |
| 09:13 |
|
aabbee joined #evergreen |
| 09:16 |
terran |
general question: I'm seeing the title from simple record extracts normalized in production but not in Concerto - anyone else have this problem? |
| 09:31 |
Dyrcona |
terran: I can't say that I have noticed that. |
| 09:32 |
terran |
Okay - we may have done something locally with normalization :/ |
| 09:34 |
Dyrcona |
terran: You're talking about data from reporter.marc_simple_record? |
| 09:36 |
terran |
Dyrcona: yes - when we're creating reports and also in the web staff client, all of our titles and authors are normalized (but not in the opac, which is fine) |
| 09:37 |
Dyrcona |
OK, just a correction: reporter.materialized_simple_record... but you understood me. :) |
| 09:38 |
Dyrcona |
So, I just fired up a database of recent concerto data, master as of a month ago or so, and I'm seeing it. |
| 09:39 |
terran |
Seeing it capitalized or seeing it normalized? |
| 09:39 |
Dyrcona |
Yeah. In a 3.0.13 database of production data, titles are all lower case and normalized. |
| 09:40 |
terran |
Oooh, so it's not just us |
| 09:40 |
terran |
Thank you for verifying |
| 09:40 |
Dyrcona |
In the master concerto database titles are NOT normalized. |
| 09:40 |
Dyrcona |
I seem to recall this possibly being a deliberate change, but can't think of a bug # of the top of my head. |
| 09:42 |
terran |
I'm running 3.2.2 on my test server and the concerto data is coming in properly capitalized. But our 3.2.2 server with a copy of production data is normalized. |
| 09:43 |
Dyrcona |
terran: Well, I would expect that until you drop and recreate the reporter.materialized_simple_record table. A good test is to edit a record to see what happens to it in rmsr. |
| 09:43 |
terran |
Good idea, thank you! |
| 09:44 |
* Dyrcona |
checks the training database that was upgraded to 3.2.2. |
| 09:47 |
Dyrcona |
Well, I don't find any not normalized titles in rmsr in training, but possibly no one updated records. |
| 09:47 |
Dyrcona |
This could also be something that missed getting into an upgrade script. /me checks |
| 09:50 |
Dyrcona |
It looks like 1059 changes a view, but does not touch the trigger or reporter.materialized_simple_record table. |
| 09:50 |
terran |
Editing and resaving the record did indeed fix the normalization for that record |
| 09:52 |
Dyrcona |
Looks like 1059 was applied to 3.0. |
| 09:52 |
Dyrcona |
OK. |
| 09:54 |
Dyrcona |
It looks like running 'SELECT reporter.disable_materialized_simple_record_trigger();' followed by 'SELECT reporter.enable_materialized_simple_record_trigger();' will rebuild the table. |
| 09:55 |
Dyrcona |
And the latter could be made more efficient by using TRUNCATE instead of DELETE FROM. |
| 09:56 |
csharp |
and reporter.refresh_materialized_simple_record() appears to combine those two |
| 09:57 |
csharp |
why wouldn't a reingest have done this? |
| 09:57 |
csharp |
I just ran pingest (--skip-browse) last week |
| 09:58 |
miker |
3.0 did not use display fields in rmsr, but normalized values from mrfr. 3.2 does use display fields, but it's a materialized view that's only updated when the record is reingested, which is why an edit (read: reingest) changes that |
| 09:58 |
Dyrcona |
Looks like 1085 changes the reporter.simple_rec_trigger. |
| 09:59 |
Dyrcona |
No, pingest will not catch this, though the brute force update might. |
| 09:59 |
* miker |
disappears to a meeting |
| 10:00 |
Dyrcona |
csharp: Just running the two selects I mentioned earlier will fix it. |
| 10:00 |
terran |
Dyrcona++ thank you so much! |
| 10:02 |
Dyrcona |
"materialized views"-- |
| 10:04 |
csharp |
so now I'm on a search for the display fields miker just mentioned... |
| 10:04 |
csharp |
not sure we have that |
| 10:06 |
* Dyrcona |
was wrong about 1085. It doesn't change the trigger directly, just some of the things used by the trigger. |
| 10:08 |
* Dyrcona |
will add a bug to add a flag to pingest to update the rmsr as well. |
| 10:08 |
csharp |
Dyrcona++ |
| 10:08 |
Dyrcona |
Well, I'll add code to pingest.pl to update rmsr, probably not in parallel with itself, though. |
| 10:09 |
Dyrcona |
The flag will skip it. |
| 10:09 |
csharp |
bug 1048822 |
| 10:09 |
Dyrcona |
Hmm... Maybe the flag will enable it. |
| 10:09 |
pinesol |
Launchpad bug 1048822 in Evergreen "Simplified Pull List - Need fuller title" [Wishlist,Fix released] https://launchpad.net/bugs/1048822 |
| 10:09 |
csharp |
that's not it |
| 10:12 |
Dyrcona |
Looks like 1073 and 1074 are more relevant than 1085. |
| 10:19 |
* Dyrcona |
scratches his head and moves on. :) |
| 10:21 |
csharp |
yeah, I don't see where it changed |
| 10:42 |
* jeff |
scrolls up to confirm a suspicion |
| 10:42 |
Dyrcona |
Well, 1059 changed the view that is used to populate the table, and I think the other upgrades may have changed some things that the view uses. |
| 10:45 |
jeff |
we (and others) ran into interesting issues with display field definitions being out of sync, leading to blank titles in rmsr. |
| 10:46 |
terran |
We were seeing blank titles for a time, but it seemed that the reingest that we did resolved them |
| 10:46 |
miker |
csharp / Dyrcona: 1100 changes rmsr to use metabib.wide_display_entry via a change to reporter.old_super_simple_record, which is the view used to update the materialized version (we materialize for speed of reports and other batch reads. one bib is fast, many is ... not) |
| 10:47 |
jeff |
I'd start by comparing config.metabib_field, config.display_field_map between the two databases you're comparing. |
| 10:49 |
csharp |
miker: thanks for the pointer - we can see it working, so I'm sure the rmsr refresh function will do the job |
| 10:50 |
csharp |
miker: also curious, do you agree with Dyrcona that TRUNCATE would be better than DELETE FROM for the enable trigger function? |
| 10:50 |
* csharp |
was considering changing that locally |
| 10:51 |
Dyrcona |
miker: TBH, it isn't that slow on all of our records, ran in less than 20 minutes on my training database. I didn't time it. |
| 10:52 |
miker |
probably. it uses delete because, as a project, we relied on slony for replication many moons ago, and that did not handle truncate originally |
| 10:52 |
miker |
so there are several places where we delete instead of truncate, esp in upgrade scripts, but that function is one I specifically recall writing in that way for that reason |
| 10:53 |
miker |
now with built-in replication that obv handles truncate, that's no longer an issue |
| 10:57 |
csharp |
miker: thanks! |
| 11:01 |
JBoyer |
Modern slony also handles truncate correctly so there's precious little reason to avoid it anymore. (We're happily on PG Streaming these days, but I've paid special attention to slony's truncation support since The Incident. ;) ) |
| 11:08 |
|
terran joined #evergreen |
| 11:11 |
|
Christineb joined #evergreen |
| 11:20 |
|
khuckins joined #evergreen |
| 11:54 |
|
sandbergja joined #evergreen |
| 12:35 |
|
terran joined #evergreen |
| 12:40 |
|
bdljohn joined #evergreen |
| 12:41 |
|
jamesrf joined #evergreen |
| 12:49 |
|
jihpringle joined #evergreen |
| 13:07 |
|
khuckins_ joined #evergreen |
| 13:21 |
csharp |
heh - The Incident |
| 13:21 |
* csharp |
is re-watching Lost with his 12-year-old |
| 13:36 |
|
jvwoolf joined #evergreen |
| 13:36 |
berick |
csharp: oh good you explain Hatch to them |
| 13:41 |
* jeff |
laughs |
| 13:47 |
|
jamesrf joined #evergreen |
| 13:48 |
jeff |
"And here's our new monitoring system dashboard, or as we like to call it, 'The Pearl'." |
| 13:56 |
csharp |
berick++ jeff++ |
| 14:17 |
|
dkyle1 left #evergreen |
| 14:19 |
Bmagic |
Anyone know off hand which receipt template the self-check web interface uses? |
| 14:25 |
Bmagic |
oh! It's an AT? |
| 14:31 |
Bmagic |
@weather 65203 |
| 14:31 |
pinesol |
Bmagic: Columbia, MO :: Snow :: 33F/1C | Wind Chill: 26F/-4C | Friday: Periods of snow. Some sleet or freezing rain possible. Temps nearly steady in the low to mid 30s. Winds SE at 5 to 10 mph. Chance of snow 100%. 1 to 3 inches of snow expected. Friday Night: Snow likely. Low 31F. Winds ESE at 10 to 15 mph. Chance of snow 100%. Snow accumulating 3 to 5 inches. | Updated: 37m ago |
| 14:31 |
Bmagic |
Columbia is CLOSING. Schools are out, the university is closing, and MOBIUS is going home! |
| 14:32 |
Bmagic |
later all, have a good weekend |
| 14:32 |
remingtron |
Bmagic: stay warm and safe! |
| 14:32 |
Bmagic |
Go go gadget Prius |
| 14:34 |
berick |
:) |
| 14:46 |
csharp |
@weather 30084 |
| 14:46 |
pinesol |
csharp: Tucker, GA :: Clear :: 50F/10C | Friday: Sunshine and clouds mixed. High 53F. Winds light and variable. Friday Night: Partly cloudy skies early will give way to cloudy skies late. Low 33F. Winds E at 5 to 10 mph. |
| 14:52 |
terran |
Bmagic: 1) have fun and 2) be safe (I know that's secondary, and sometimes contradictory) |
| 14:58 |
rjackson_isl |
@weather 46204 |
| 14:58 |
pinesol |
rjackson_isl: Indianapolis, IN :: Clear :: 31F/-1C | Wind Chill: 23F/-5C | Friday: Generally cloudy. Temps nearly steady in the low to mid 30s. Winds SE at 5 to 10 mph. Friday Night: Cloudy with snow developing after midnight. Low 29F. Winds ESE at 5 to 10 mph. Chance of snow 100%. Snow accumulating 1 to 3 inches. | Updated: 4m ago |
| 15:00 |
rjackson_isl |
well that isn't too impressive! forecast for tonight: 1-3 tomorrow 3-5 and tomorrow night 1-3 |
| 15:01 |
rjackson_isl |
at least the wind isn't supposed to howl |
| 16:23 |
|
Dyrcona joined #evergreen |
| 17:11 |
|
khuckins joined #evergreen |
| 17:18 |
|
khuckins_ joined #evergreen |
| 17:18 |
|
mmorgan left #evergreen |
| 17:22 |
|
yboston joined #evergreen |
| 17:46 |
csharp |
971eac9a |
| 17:46 |
pinesol |
csharp: [evergreen|miker] some new management functions for the materialized reporting view - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=971eac9> |
| 17:50 |
* Dyrcona |
blinks. |
| 17:53 |
* Dyrcona |
types "sudo shutdown -r now" on a remote server and crosses his fingers. |
| 18:03 |
Dyrcona |
Well, we'll see if training is any faster after this. |
| 18:14 |
Dyrcona |
Anyone present remember the trick for priming the database in memory? |
| 18:17 |
Dyrcona |
This is cool, since 9.4: https://www.postgresql.org/docs/current/pgprewarm.html |
| 18:18 |
|
jvwoolf left #evergreen |
| 18:23 |
Dyrcona |
Oh, well. Looks like I could try using cat on all the db files. "Ain't nobody got time for that!" |
| 18:24 |
Dyrcona |
It is using hugepages, though, so that should be an improvement. |
| 18:55 |
|
Dyrcona joined #evergreen |
| 19:10 |
Dyrcona |
Stupid wifi... |
| 20:20 |
|
beanjammin joined #evergreen |
