| Time |
Nick |
Message |
| 00:19 |
|
bshum joined #evergreen |
| 02:06 |
|
StomproJ joined #evergreen |
| 02:09 |
|
Bmagic joined #evergreen |
| 02:09 |
|
dluch joined #evergreen |
| 07:30 |
|
rjackson_isl_hom joined #evergreen |
| 07:36 |
|
collum joined #evergreen |
| 07:55 |
|
rfrasur joined #evergreen |
| 08:34 |
|
Dyrcona joined #evergreen |
| 08:39 |
|
mmorgan joined #evergreen |
| 09:09 |
|
Stompro joined #evergreen |
| 09:28 |
Stompro |
Baker and Taylor support notified me this morning that PASV mode should work again on the normal ftp site. But I haven't tested it yet. |
| 09:33 |
mmorgan |
Offline circ issue: A library can't record checkouts, likely because the "Working Location" field isn't filled in. But the dropdown for that field is not populated. Any suggestions? |
| 09:34 |
Stompro |
Seems to be working when I test. Public IP returned when passive commands are sent using netkit-ftp, which doesn't have special handling I believe. |
| 09:45 |
* mmorgan |
goes offline to test offline circ for real |
| 09:49 |
Dyrcona |
mmorgan: You have to be completely offline. If it can see the server at all, offline doesn't work. |
| 09:50 |
Dyrcona |
:) |
| 09:50 |
Dyrcona |
I figured she knew, but..."Won't someone please think of the logs!?" |
| 09:52 |
|
mmorgan joined #evergreen |
| 09:54 |
Dyrcona |
mmorgan: AFAIK, there's no easy way to change the offline data. It's in local storage somewhere. You might be able to do it with some fancy JavaScript. |
| 09:56 |
mmorgan |
Dyrcona: Thanks! |
| 09:57 |
mmorgan |
Once I was really offline, my working location was filled in. |
| 09:57 |
mmorgan |
I cleared cache, JUST cache, and my working location went blank with no dropdown. I suspect they may have cleared cache. |
| 10:01 |
JBoyer |
I believe there is still an option to Export the offline data, so while it's not ideal it can be edited as needed. |
| 10:04 |
mmorgan |
JBoyer: Unfortunately, they can't record the offline data. Clicking "Checkout" doesn't create a transaction. |
| 10:04 |
JBoyer |
Ah, can't even get to that point, forgot that. :( |
| 10:09 |
Dyrcona |
Am I hearing votes for a standalone offline circulation binary? |
| 10:10 |
mmorgan |
Our library would appreciate that right now :) |
| 10:15 |
mmorgan |
lightning-- |
| 10:16 |
Dyrcona |
mmorgan: Is the lightning related to the offline circ issue? |
| 10:18 |
mmorgan |
Well, not directly, but the lightning is very likely related to the need for offline circ :) |
| 10:18 |
Dyrcona |
It was bad last night. |
| 10:20 |
mmorgan |
Indeed! And Comcast is very busy repairing storm damage in the area. |
| 10:23 |
miker |
Dyrcona: I don't recall if pgingest gives you regular progress reports, or just for completed batches, but I'd be interested to know what the average record ingest time is, what your batch size is, and especially if pg_stat_activity shows lock waits. (top would also show "(waiting)" in the process name if there's lock blocking.) |
| 10:25 |
miker |
because with small batch sizes on slow disks I was seeing per record averages of under a second (in parallel) with symspell reification delayed |
| 10:25 |
Dyrcona |
miker: I'm keeping notes to update the bug. I'll check for locks. It seems to be exponentially slower at each set of 7/8 batches. This is what I saw before the changes, IIRC. |
| 10:25 |
miker |
sounds like maybe reification isn't being delayed, then |
| 10:25 |
Dyrcona |
miker: Please define "small batch sizes." I'm using the default of 10,000. |
| 10:25 |
miker |
uh, like, 10 |
| 10:26 |
Dyrcona |
I'll give 10 a shot next week. I'm going to let the current batch finish at least. |
| 10:28 |
miker |
Dyrcona: does the unlogged table have visible rows in it? select count(*) from search.symspell_dictionary_updates; |
| 10:29 |
Dyrcona |
I have 6 postgres processes that are "SELECT waiting" |
| 10:31 |
* Dyrcona |
waits on the select count(*) to finish. |
| 10:32 |
Dyrcona |
On this hardware, 8 batches of 10,000 records each usually finish in about 4 hours. It takes almost exactly 4 days to do a full ingest on our production data, 2.2 million or so bibs. |
| 10:34 |
Dyrcona |
Load is only 5.98, but I suspect the database is I/O bound. It's an array of spinning rust that wasn't originally intended PostgreSQL. The server previously ran MS Exchange. |
| 10:35 |
* Dyrcona |
goes to check the health of the array. We replaced a drive last year, but we might be out of spares. |
| 10:38 |
Dyrcona |
Well, this is nice. One drive in the middle of the array is flashing an amber light. The box that the spares came in is empty. |
| 10:39 |
Dyrcona |
It may be time to retire Dumbo. |
| 10:47 |
|
stephengwills joined #evergreen |
| 10:48 |
|
mmorgan joined #evergreen |
| 10:48 |
Dyrcona |
I assume that there is something in search.symspell_dictionary_updates because my select count(*) still hasn't returned. It's also not waiting. |
| 10:58 |
miker |
Dyrcona: you could `vaccum analyze verbose` it to see if it's full of visible rows or not. if it's all visible rows then the delay code is "working", if it's mostly dead rows, there's something wrong. |
| 10:59 |
Dyrcona |
Sure. I'll just cancel the select, first. :) |
| 11:02 |
Dyrcona |
There are also 3 autovacuum worker processes running in the database. I've been meaning to look into our autovacuum settings. |
| 11:16 |
Dyrcona |
INFO: scanned index "symspell_dictionary_updates_tid_idx" to remove 126549937 row versions |
| 11:22 |
Dyrcona |
miker: https://pastebin.com/E4iNzSzw |
| 11:23 |
miker |
yeah, the delayed reification isn't actually being tested... wheeee |
| 11:23 |
miker |
you're just testing the deadlock code (again) |
| 11:23 |
Dyrcona |
Did I miss a flag to turn that on? |
| 11:24 |
miker |
the pingest param is supposed to casue a function to be called that turns that delay feature on, and then at the end it calls a separate "clean it all up" function |
| 11:24 |
miker |
you know pingest better than me, so I'd appreciate eyes on that logic change |
| 11:26 |
Dyrcona |
OK. So, I missed using that option. |
| 11:27 |
Dyrcona |
Can I ask why that is optional? Shouldn't it just work that way, all the time? |
| 11:28 |
Dyrcona |
I simply forgot about the option when I started the test run. This is what happens when you a) get older, b) work on many things, and c) have to put one thing down for a week or more before coming back to it. |
| 11:33 |
miker |
because for non-batch ingest it just creates headaches. human bib editing doesn't interact often enough to matter, but because it does cause serialization (INSERT ... ON CONFLICT does that to prevent deadlocks) with intentionally parallel updates the delay makes it overall faster. same thing as with the browse part of ingest |
| 11:34 |
miker |
now, I could certainly see making it the default for pingest! but I didn't want to change the default behavior on my own |
| 11:34 |
miker |
default behavior of pingest, I mean |
| 11:39 |
|
jihpringle joined #evergreen |
| 11:40 |
Dyrcona |
If the performance without the option is so bad, I don't know why anyone would not want to use the option, so it should probably not even be an option. It should just be how it works. That said, I'll look at this some more and see how it performs. |
| 11:46 |
Dyrcona |
heh. I was just thinking of changing the option from --delay-symspell to --no-delay-symspell and inverting what it does. Then, I thought it should be changed to --make-it-slow. :) |
| 11:46 |
* Dyrcona |
eats lunch. |
| 11:56 |
* Dyrcona |
mumbles: Too much rice. |
| 12:26 |
|
Stompro joined #evergreen |
| 12:44 |
csharp_ |
@band add [ana too much rice] |
| 12:44 |
pinesol |
csharp_: Band 'Choice tumor' added to list |
| 12:45 |
csharp_ |
pinesol: ick |
| 12:45 |
pinesol |
csharp_: Zoia knows how to make fusilli. |
| 12:51 |
Dyrcona |
Ha! |
| 12:59 |
mmorgan |
:) |
| 13:04 |
berick |
@ana it's not a choice tumor |
| 13:04 |
pinesol |
berick: Im toothier accounts |
| 13:04 |
berick |
pinesol: i gave you an apostrophe |
| 13:04 |
pinesol |
berick: NOBODY EXPECTS THE SPANISH INQUISITION! |
| 13:05 |
mmorgan |
:) |
| 13:09 |
csharp_ |
"amongst our weaponry, are surprise! fear! oh, I'll come in again :-/" |
| 13:10 |
* mmorgan |
opts for soft cushions. |
| 13:10 |
|
collum joined #evergreen |
| 13:13 |
|
collum joined #evergreen |
| 14:05 |
Dyrcona |
@ana Nobody expects the Spanish Inquisition |
| 14:05 |
pinesol |
Dyrcona: Qoph bonds cheapest sixty-nine |
| 14:19 |
berick |
huh, learning. https://en.wikipedia.org/wiki/Qoph |
| 14:22 |
Dyrcona |
Yeah, great for Scrabble. |
| 14:35 |
|
jihpringle joined #evergreen |
| 14:52 |
|
jvwoolf joined #evergreen |
| 15:00 |
|
Stompro joined #evergreen |
| 16:15 |
|
mmorgan joined #evergreen |
| 16:46 |
|
jvwoolf left #evergreen |
| 17:15 |
|
mmorgan left #evergreen |
| 17:19 |
|
stephengwills left #evergreen |
| 17:19 |
csharp_ |
@blame [band] for [praise [someone]] |
| 17:19 |
* pinesol |
The Six Brick Situation is really just another name for autogen for degraafk is very kind and good-looking and always does what's best for the project |
| 17:20 |
degraafk |
heh |
| 17:21 |
berick |
that's a pretty good one |
| 17:22 |
csharp_ |
@who is here all week, so tip your servers while we watch [band] |
| 17:22 |
pinesol |
DaMobi is here all week, so tip my servers while we watch Belated Karma. |
| 17:58 |
berick |
@who camped out all night to see [band] |
| 17:58 |
pinesol |
phasefx camped out all night to see Failed Vegetarians. |
| 19:51 |
|
Stompro joined #evergreen |
