| 12:33 |
Bmagic |
I'm troubleshooting a database issue. explain analyze on an insert into biblio.record_entry results in 6+ seconds. The analyze shows me that it spends the majority of the time in reporter.simple_rec_trigger(). Which is odd, because if it's an INSERT, it should just skip down to RETURN. Weird right? |
| 12:33 |
Dyrcona |
Bmagic: No. It still builds the rmsr entry. |
| 12:34 |
Dyrcona |
Wait until you update an authority with hundreds of bibs "attached." :) |
| 12:35 |
Bmagic |
this is EG 3.11.1 with queued ingest. I updated the global flag to turn off queued ingest, and tested it. and visa versa. it's 13 seconds for an insert with it turned off, and 6 seconds with it turned on. But still, 6 seconds seems insane for an insert on biblio.record_entry, especially if we're deferring all of the metabib stuff for the ingester process to come later |
| 12:35 |
Dyrcona |
Updating/inserting bibs and auths can be slow because of all the triggers. |
| 12:36 |
Bmagic |
I got here because it's breaking the course reserves interface. Taking too long |
| 12:37 |
* Dyrcona |
colour me unsurprised. |
| 16:38 |
Bmagic |
Maybe I can change PG configuration to eliminate the secondary DB to prove it has something to do with that (or not) |
| 16:38 |
Dyrcona |
Nope. None of them checked in. |
| 16:40 |
Dyrcona |
Bmagic: Could be. I often have queries fail on the secondary DB, 'cause changed rows from replication are needed. I know that's not your issue, but you shouldn't be inserting into a replicated db, so I'm not sure what you're checking. |
| 16:40 |
Bmagic |
This is my test query: explain analyze select * from reporter.old_super_simple_record where id=671579 |
| 16:41 |
Bmagic |
6+ seconds on db1. and less than one second on db2 |
| 16:42 |
Dyrcona |
OK. Das stimmt. |
| 16:43 |
Dyrcona |
Hm... I need to find the circulations. The copies are all Lost status.... They're also deleted, and I thought using the copy_id would resolve the latter. |
| 10:03 |
Dyrcona |
This could be a case where joins are slower than subqueries for .... reasons. |
| 10:04 |
Dyrcona |
It could also be completely different on a newer Pg release. |
| 10:04 |
Stompro |
I started using NYTProf now, it is much heavier weight, but the output much more detailed. https://metacpan.org/pod/Devel::NYTProf |
| 10:08 |
Stompro |
Devel::Profile is much faster for quickly seeing results. NYTProf generates a 2G output file in my testing that then has to be processed into an HTML report. |
| 10:08 |
Dyrcona |
I think I'll go for fiaster/lighter weight. I'll read the docs. |
| 10:09 |
Dyrcona |
My code to log subroutine names with timestamps as they're called produces a largeish output and is likely less accurate since it spends time generating timestamps and printing them. |
| 10:10 |
Stompro |
All I had to do was run marc_export with "perl -d:Profile ./marc_export_testing" and it generates the profile log prof.out |
| 10:10 |
Dyrcona |
I'll make a patch and throw it on the LP. |
| 10:11 |
Dyrcona |
I mean a patch for my logging code. |
| 10:11 |
Dyrcona |
So, you think we should just switch from insert_grouped_field to insert_field? |
| 10:13 |
Stompro |
I put a diff of the changes I was testing with at https://gitlab.com/LARL/evergreen-larl/-/snippets/3615366 |
| 10:14 |
Stompro |
Put all the 852s in an array, then once they are all added, call insert_grouped_field for the first one to get the same ordering, and use insert_fields_after for the rest with one call. |
| 10:18 |
Dyrcona |
Stompro: There's a simpler way to do the insert: push the fields to an array, then do the first one with shift and the rest of the array after that. |
| 10:19 |
Stompro |
I figured, my perl array skills need work. :-) |
| 10:20 |
Dyrcona |
I wonder if the first one even needs to be grouped? |
| 10:20 |
Dyrcona |
I'm going to look at MARC::Record again. |
| 10:21 |
Dyrcona |
Stompro++ # For the notes in the snippets. |
| 10:21 |
Stompro |
In my test data, the 901 tag would be placed before the 852 without using the insert_grouped_field for the first. |
| 10:23 |
Stompro |
I don't think MARC::Record re-orders the fields. |
| 10:24 |
Stompro |
If I'm understanding where you are going with that. |
| 10:33 |
Dyrcona |
OK. LoC says the records are supposed to be grouped by hundreds, they don't have to be in order. |
| 10:41 |
|
briank joined #evergreen |
| 10:44 |
Dyrcona |
Oh! That patch I threw on Lp is missing a local change to format the microsecond timestamps to %06d..... |
| 10:53 |
Dyrcona |
Heh. This branch is a mess.... |
| 10:54 |
Dyrcona |
So, I was testing with a dump of 1 library's records. It took about 1 hour 4 minutes to run. I'll make a change based on Stompro's bug description and see what happens. |
| 11:06 |
Dyrcona |
OK. Here goes.... |
| 11:11 |
Stompro |
Dyrcona, does this library have some of the bibs with large numbers of copies? |
| 11:20 |
Dyrcona |
I don't know. I doubt it. It doesn't seem to have made much difference so far. I'll try a larger library or the whole consortium next. |
| 11:20 |
Dyrcona |
It's one we do a weekly export for, so that's why I chose it to test. |
| 11:28 |
Dyrcona |
It does use slightly less (~3%) CPU |
| 11:30 |
Stompro |
In my testing, with our production data it had only a slight improvement. But it really improved the run that was stacked with bibs with lots of copies. |
| 11:36 |
Stompro |
acps_for_bre needs to be reworked to improve the --items performance in general. Maybe the first call just pulls in all call numbers and copies and caches them in a hash... |
| 11:37 |
Stompro |
Or go with the rust version that is already better :-) |
| 11:43 |
Dyrcona |
When I ran the queries through EXPLAIN ANALYZE, none of them were particularly slow. The slowest was the acp_for_bres query. On one particular record, it spent 40ms on a seq scan of copy.cp_cn_idx. I'm not sure how to improve a seq scan on an index, unless it can be coerced to a heap scan somehow. |
| 12:34 |
|
collum joined #evergreen |
| 12:38 |
Dyrcona |
Heh. Almost 1 minute longer..... |
| 12:50 |
|
collum joined #evergreen |
| 13:02 |
Dyrcona |
I am testing this now: time marc_export --all -e UTF-8 --items > all.mrc 2>all.err |
| 13:14 |
Dyrcona |
The Rust marc export does batching of the queries by adding limit and offset. I wonder if we should do the same? I've noticed that the CPU usage goes up over time, which implies that something is struggling through the records. The memory use stays mostly constant once all of the records are retrieved from the database. |
| 13:20 |
Stompro |
Dyrcona, if you use gnu time, it gets you max memory usage also. /usr/bin/time -v... so you don't have to check that separately. |
| 13:25 |
Stompro |
Dyrcona, I'm surprised the execution time increased for you... hmm. |
| 14:00 |
csharp_ |
berick: after installing the default concerto set, notes work - everything is speedy under redis - no errors yet |
| 14:07 |
|
sleary joined #evergreen |
| 14:16 |
berick |
csharp_: woohoo |
| 14:35 |
Dyrcona |
I've been testing Redis with production data, but not much lately. I need to write an email to ask the relevant staff here if they'd like me to update the dev/training server to use Redis on the backend. |
| 14:37 |
Dyrcona |
Current calculation puts it at only 20% faster, i.e. -1 day. |
| 14:38 |
Dyrcona |
I'm going to add a --batch-size option. If it is specified the main query will retrieve that number of records per request. I don't know if I'll get that implemented today. |
| 14:51 |
Dyrcona |
Looks like adding tags isn't my bottleneck My current estimate is minimal difference in performance. I'm going to let this run over the weekend to see if I'm wrong. On Monday, I'll add a batching/paging option to the main query and see if that helps. |
| 09:08 |
Dyrcona |
berick: I did `cargo build --release --package evergreen` then copied eg-marc-export to /openils/bin/. I missed the password on of the two lines for eg-marc-export in my script, so I don't know if it is faster, but the binary is certainly smaller without the debugging symbols, etc. |
| 09:14 |
|
redavis joined #evergreen |
| 09:18 |
|
terranm joined #evergreen |
| 09:18 |
Dyrcona |
FWIW, I haven't used --release on my test system. I did that for the production server. |
| 09:40 |
csharp_ |
one of the Supy/Limnoria bots in another IRC channel I was in had a @botsnack command and the bot would reply "YUM!" |
| 09:41 |
csharp_ |
actually in the Ubuntu channel, it would say "YUM!.. I mean, er, APT!" |
| 09:41 |
bshum |
Put that on the wishlist for bot development tasks... got it :) |
| 15:18 |
kmlussier1 |
I don't know what the harlequin/silhouette subscription thing was, but I was just talking fondly this week about my Columbia House mail order subscription. Those were good times! |
| 15:19 |
kmlussier1 |
Oh, my nick is messed up. How sad. Anyway, have a good weekend #evergreen! Safe travels to all who will be going to Indianapolis! |
| 15:19 |
|
kmlussier1 left #evergreen |
| 15:21 |
terranm |
The true Gen Xer test is the exposure to the Flowers in the Attic books |
| 15:21 |
terranm |
Those were passed around like contraband in my junior high |
| 15:22 |
redavis |
Oh, full exposure here. I was just thinking about those the other day and how wrong it all was. |
| 15:23 |
redavis |
hmm, I'm not sure how I got ahold of them. Kinda thing my mom might have accidentally bought them at a garage sale or something like that. The paperbacks swirled. Same with Stephen King. |
| 15:24 |
terranm |
So much Stephen King! |
| 10:28 |
mantis1 |
was missing one of those thank yo! |
| 10:32 |
jeff |
recall holds are a thing that requires some additional A/T setup. I don't know if the defaults are suitable / functional out of the box. Org unit settings likely also. |
| 10:33 |
jeff |
I was actually just wondering if it made sense to have an option to hide recall as a hold option, for those reasons and more. :-) |
| 10:39 |
Dyrcona |
berick: My test of the Rust marc export finish in 9 hours 23 minutes, and exported a 3.4GB binary file with 1,737,349 records. There are 411 records in the error output. That looks good to me, compared to what I'm getting from Perl. |
| 10:46 |
csharp_ |
rs++ |
| 10:57 |
Dyrcona |
csharp_: Do you think the marc_export is slow with the --items option? |
| 10:57 |
|
briank joined #evergreen |
| 10:59 |
berick |
Dyrcona: cool, good to know. |
| 11:00 |
berick |
re: errors, I saw a record or 2 in my tests that had subfield codes of "" (zero bytes) |
| 11:00 |
berick |
could add a flag or something to massage those into " " or some such |
| 11:01 |
csharp_ |
Dyrcona: yes it is |
| 11:01 |
Dyrcona |
I see some of those. I'll look through it later. I know we have bad records, because the Perl also complains. |
| 11:02 |
Dyrcona |
csharp_: Thanks. I felt like I'm going crazy because others have said it's not that slow. It was taking 5 days to the same export as I mentioned above. |
| 11:19 |
kmlussier |
Dyrcona: Wow, that's great that you were able to see such a big improvement! |
| 11:21 |
Dyrcona |
kmlussier: Yeah, I think Rust is more efficient than Perl, at least in my situation. I'm doing a test in produciton this evening. |
| 11:27 |
Dyrcona |
Y'know....It would be cool in psql if you could use \i for a subquery or in a CTE. That would save having two copies of the same code. (I know.... "Find another way to do it.") |
| 11:56 |
Dyrcona |
@decide Redgum or Rimu |
| 11:56 |
pinesol |
Dyrcona: go with Redgum |
| 15:28 |
sharpsie |
Burn It Down and Rebuild™ |
| 15:32 |
|
book` joined #evergreen |
| 15:40 |
JBoyer |
Sometimes a forest fire is responsible forest management! :p |
| 15:43 |
jeff |
okay, about to bug this unless it rings a bell for someone and they point out it's already got a bug: Angular MARC editor stashes field data in current_tag_table_marc21_biblio in a format that's just different enough to break context menus for the AngularJS MARC editor (which you may encounter in at least the Z39.50 Edit then Import interface). |
| 15:44 |
jeff |
for the Angular MARC editor, the way it creates the local data structure results in an array with element 100 being tag 100, etc. The older AngularJS MARC editor created an object with key 100, not array index 100, etc. |
| 15:45 |
jeff |
and of course, it's technically the tagtable service (old and new) that are doing this, not the "MARC editor"... |
| 15:47 |
jeff |
looks like it affects 3.10 and 3.11 at least, haven't tested main or looked too closely to see if the Z39.50 Edit then Import has moved to Angular yet. |
| 15:52 |
jeff |
steps to reproduce: clear (at least) current_tag_table_marc21_biblio from localStorage, then open the Angular MARC editor followed by Edit then Create from a Z39.50 search (which uses the older AngularJS MARC editor), and try to open the context menu on a tag value (other than the leader). |
| 15:52 |
jeff |
If you do the reverse, it works okay (the newer editor can tolerate the data in both formats, possibly by design, possibly by happenstance). |
| 15:53 |
jeff |
(or happy accident?) :shrug: :-) |
| 16:16 |
sleary |
Z39.50 in Angular and a revised MARC editor that relies less on context menus are both under construction now |
| 16:19 |
jeff |
the latter is bug 2006969, I think. |
| 16:19 |
pinesol |
Launchpad bug 2006969 in Evergreen "wishlist: enhanced MARC editor rewrite" [Wishlist,Confirmed] https://launchpad.net/bugs/2006969 - Assigned to Stephanie Leary (stephanieleary) |
| 17:01 |
jeff |
sleary++ jihpringle++ |
| 17:02 |
jeff |
(I'll still create a bug so we can fix 3.10/3.11) |
| 17:05 |
|
mmorgan left #evergreen |
| 17:07 |
abneiman |
jeff: I am not finding an LP for sprint A (which is on me, and I will rectify shortly) - but the spec you linked is the one we're working from. It's opening testing this week, and MARC Editor just opened testing so look for both on the spring release roadmap! |
| 17:08 |
abneiman |
also, apologies in advance for the flood of commits about to ensue ... next time I will learn to squash |
| 17:11 |
pinesol |
News from commits: Docs: LP2022100 updates to Item Status docs <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=e36ddd543aba55fd6ce4aeb5dd968b6f26eec4a4> |
| 17:11 |
pinesol |
News from commits: Docs: LP2022100 updates to Item Status docs <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=483d2f78eda53393c02a05eb54f5b91baf503c26> |
| 17:11 |
pinesol |
News from commits: Docs: LP2022100 updates to Item Status docs <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=c2ce3c25973f9a2128970cef0e1b2e0a85f1a022> |
| 10:41 |
redavis |
Thank you for your many hours of coffee roasting sacrifice. |
| 10:47 |
|
smayo joined #evergreen |
| 10:57 |
|
briank joined #evergreen |
| 11:00 |
Dyrcona |
berick: I don't know what the size of the file was at 143,000 records. The final file is 15GB with 2,251,864 bib records and 7,775,264 items. It took 18.3 hours to complete on my test system. |
| 11:00 |
Dyrcona |
berick++ |
| 11:04 |
Dyrcona |
Hmm. I guess that export running on production explains the "processor load too high on utility server emails." It keeps 1 CPU busy all the time, and there are only 8 cpus. Load was 11.5 when I just checked. |
| 11:04 |
Dyrcona |
Bmagic: I'm installing Rust on the CW MARS utility server today. |
| 14:53 |
Dyrcona |
Currently, it's actually returning acn.record, bre.marc, and the count on acp.id. |
| 14:56 |
Dyrcona |
Dude..... I just noticed the file ends with two semicolons.....I'll bet that's it. |
| 14:56 |
Dyrcona |
Still, I think I'll do the CTE. |
| 14:57 |
berick |
my test file contain: SELECT id, marc FROM biblio.record_entry WHERE NOT deleted (no semicolons needed) |
| 14:58 |
berick |
afk for a bit |
| 14:59 |
Dyrcona |
Yeah, ; is a habit from writing stuff for psql. |
| 15:07 |
pinesol |
News from commits: LP2035287: Update selectors in e2e tests <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=f562b3ac30a3753d63d565c2d7be4d3a7121a2fb> |
| 15:11 |
Dyrcona |
Now, I'm cooking with gas. I got the "large" bibs file (85 records with 87,296 items) dumped to XML in under a minute. That took almost 2 hours with the Perl program the other day. |
| 15:18 |
Dyrcona |
Using query-file to feed the eg-marc-export, it is using a lot more RAM than before, about the same as the Perl export was using. It still uses less CPU. We'll see if that changes over time. |
| 15:28 |
jeff |
you have 87,296 items that are spread across only 85 bibs? |
| 15:55 |
|
brianmk joined #evergreen |
| 15:56 |
berick |
Dyrcona: --pipe option pushed. i'm also happy to see any errors you encounter. |
| 15:57 |
Dyrcona |
berick++ |
| 15:57 |
Stompro |
Do the test email / sms features require that the app servers be able to send email? |
| 15:57 |
Dyrcona |
I might work up the nerve to submit some PRs later. |
| 15:58 |
Dyrcona |
Stompro: I don't remember, but there is something that does. |
| 15:59 |
Stompro |
I'm so used to only our utility server sending out email... |
| 16:01 |
Dyrcona |
Stompro: I think that's bug worthy if that's the case. |
| 16:02 |
Stompro |
I think I've been under the incorrect impression that action_trigger_runner had to be called to send email.... |
| 16:07 |
Dyrcona |
I could be wrong, but I swear I remember there being something that required the app servers to send email. Test emails and SMS go through action trigger. |
| 16:08 |
berick |
Stompro: open-ils.actor.event.test_notification ? looks like that fires the A/T event in real time from whatever server the API is called at. |
| 16:08 |
berick |
it doesn't wait for a_t_runner |
| 16:09 |
berick |
a variation on that API that creates but does not fire the event would be useful |
| 16:10 |
Dyrcona |
berick++ |
| 16:13 |
Stompro |
Hmm, my flowroute sms reactor is only setup on our utility server... I'll have to figure out how to get those test messages working. |
| 16:14 |
Stompro |
There probably is no reason it couldn't be on the app servers also. |
| 16:16 |
Dyrcona |
Stompro: I think a setting could be added to make the test notification API NOT fire the event, just leave it pending, then the pending a/t runner could pick it up, or it could be given the same granularity and the password reset emails. |
| 16:16 |
Dyrcona |
It would require development, of course. |
| 16:16 |
Stompro |
I was just stumped when I looked at the event def 155 entries and saw that they were processed 1 second after they were created. |
| 16:17 |
Dyrcona |
Yeah, in my case they're 523 and 524... |
| 10:40 |
mmorgan |
Stompro: extend_reporter.legacy_circ_count |
| 10:40 |
Stompro |
mmorgan++ thank you!! |
| 10:41 |
mmorgan |
YW! |
| 10:43 |
Stompro |
Dyrcona, I was conversing with Brian about his marc_export issues. And I did a test export or our DB. 200K bibs took 5 minutes, used 1.2GB of RAM, and created a 1GB uncompressed xml file, which compressed to 115MB. |
| 10:43 |
Dyrcona |
user_ingest_name_keywords_tgr BEFORE INSERT OR UPDATE ON actor.usr FOR EACH ROW EXECUTE PROCEDURE actor.user_ingest_name_keywords() <- I think that should maybe be an AFTER, but I'll play with it. |
| 10:45 |
Stompro |
That was without items... I should try it again with items. |
| 10:45 |
Dyrcona |
Stompro: I still have to do this in production, but it has always taken longer than that. Are you feeding IDs to marc_export, and what marc_export options are you using? |
| 10:48 |
|
briank joined #evergreen |
| 10:49 |
Dyrcona |
Also, I realize my comment about the user_ingest_name_keywords_tgr needing to be AFTER is bogus. I pulled an extra couple of fields in my query and discovered why I was seeing what I thought was anomalous. |
| 10:50 |
|
kworstell_isl joined #evergreen |
| 10:51 |
Dyrcona |
Some test accounts have weird names. :) |
| 10:52 |
Dyrcona |
The problem could be the extra query to grab item data combined with the massive amount of data. |
| 10:54 |
Stompro |
Dyrcona, --items didn't change the memory usage, still 1.2G for 194432 bibs.. run time seems longer.. I'll report back when done. |
| 10:56 |
Dyrcona |
I think running that select for items in a loop is the real issue. I should refactor this to grab the items at the time the query runs. That complicates the main loop though. |
| 11:13 |
Dyrcona |
I estimate it will export about 1.78 million records. |
| 11:14 |
Stompro |
Dyrcona, Nevermind about the cpu usage, that was me seeing the 9% memory used by mistake. |
| 11:15 |
Stompro |
Using your method I see 67%cpu |
| 11:16 |
Dyrcona |
I'm going to try smaller batches in production starting this afternoon to see if it helps. I may or may not stop the one running on a test vm. Maybe it is time to refactor export to speed things up? |
| 11:24 |
pinesol |
News from commits: Docs: LP1845957 Permissions List with Descriptions <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=2680eca9e4dbaa79b2cd00c7fd3373311b85901c> |
| 11:24 |
pinesol |
News from commits: Docs: LP1845957 Part 1 - Update describing_your_people.adoc <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=c7b205fb604e7827843c7a5ea6542ce02c2f72ef> |
| 11:25 |
Dyrcona |
I think I'll break for lunch early and start on that about noon. |
| 12:49 |
eeevil |
smayo: I do not recall ... it's been A While (TM) ;) ... I can look, though |
| 12:51 |
Dyrcona |
hmm... marc_export should exit when a bad command line option is passed in. |
| 12:56 |
Dyrcona |
It's taking a while to export the "large" bibs. It has to be the items query that is slowing things down. These are 90 bibs in our database with > 500 items. |
| 12:57 |
eeevil |
@later tell smayo I do not recall ... it's been A While (TM) ;) ... I can look, though. UPDATE: looks like it was a "just check a week" flag, basically, though the breakout variable is similar (if 15x larger). if skipping dow_count testing makes everything happy, I'm for it. |
| 12:57 |
pinesol |
eeevil: The operation succeeded. |
| 12:58 |
|
smayo joined #evergreen |
| 12:58 |
Dyrcona |
:) |
| 13:10 |
Dyrcona |
18 minutes and 54 seconds and only 13 records exported.... Well, I know where I need to look. |
| 13:34 |
|
smayo joined #evergreen |
| 13:48 |
|
smayo joined #evergreen |
| 13:50 |
Dyrcona |
58 minutes and it is a bit over halfway done with the 90 large records. I have a 'debug' version of marc_export that I'll use to dump the queries on a test system. |
| 13:56 |
|
jihpringle joined #evergreen |
| 14:24 |
pinesol |
News from commits: Docs: updates to Z39.50 documentation <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=bb4d795eb16102c35d26f0d59d14da50b86605e4> |
| 14:54 |
Dyrcona |
Doing batches does not appear to have improved performance. If anything, it seems worse, but maybe my dev database is faster than production. |
| 15:11 |
berick |
./eg-marc-export --items --to-xml --out-file /tmp/recs.xml # --help also works / other options to limit the data set |
| 15:12 |
Dyrcona |
berick++ I'll give it a look. |
| 15:12 |
berick |
Dyrcona++ |
| 15:21 |
jeffdavis |
Interesting performance problem on our test servers - the Items Out tab is very slow to load. Looks like the call to open-ils.pcrud.search.circ is consistently taking 6.5 seconds per item for some reason. |
| 15:21 |
berick |
oof |
| 15:22 |
jeff |
do the items in question have extremely high circulation counts? |
| 15:23 |
jeff |
we have a test item that gets checked in and out multiple times a day via SIP2 for a Nagios/Zabbix style check. |
| 15:23 |
jeff |
I once made the mistake of using that same item to test something unrelated, and it took consistently 6 seconds or more to retrieve in item status. |
| 15:24 |
jeffdavis |
No, these are just randomly selected items - the one I checked has 8 total circs. |
| 15:24 |
jeff |
(It may not apply in this case. I don't think the pcrud search is going to be trying to get a total circ count for the items.) |
| 15:24 |
jeff |
ah, drat. |
| 15:25 |
jeffdavis |
Our production environment is not affected, but a test server running the same version of Evergreen (3.9) is, as is a different test server running 3.11. |
| 15:25 |
|
mmorgan1 joined #evergreen |
| 15:28 |
Stompro |
jeffdavis, are they all on the same Postgres version? |
| 15:29 |
jeffdavis |
Yes, all PG14. The test servers all share the same Postgres server, my guess is that's where the issue lies but not sure what the mechanism would be. |
| 15:30 |
Dyrcona |
jeffdavis: You have all of the latest patches for Pg installed? |
| 15:30 |
Dyrcona |
Meaning Evergreen patches. |
| 15:32 |
jeffdavis |
The affected servers are either 3.9.1-ish or 3.11.1-ish with some additional backports -- not fully up to date but pretty close. Are there specific recent patches you're thinking of? |
| 15:51 |
berick |
IOW, evergreen-universe-rs does a variety of stuff, but when it talks to EG, it assumes Redis is the communication layer |
| 15:51 |
berick |
ah, no, redis required for those actions |
| 15:55 |
pinesol |
News from commits: Docs: Circulation Patron Record Page <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=9d632b3589a263333a187fda59a708fe672f2813> |
| 15:56 |
Dyrcona |
If I'm going to start messing with Rust, I guess I should dust off the VM where I tested the RedisRF branches. |
| 15:56 |
berick |
muahaha i have successfully distracted you :) |
| 15:57 |
Dyrcona |
:) |
| 16:01 |
Dyrcona |
If the Rust export is faster, then I won't consider it a distraction. :) |
| 11:05 |
Dyrcona |
It might need more patches than just that one.... I'll leave it for now. |
| 11:14 |
|
kmlussier joined #evergreen |
| 11:18 |
Dyrcona |
So, going back to yesterday's conversation about MARC export, I wonder if that commit really was the problem. I reverted that one and two others, then started a new export. It has been running for almost 21 hours and only exported about 340,000 records. I estimate it should export about 1.7 million. |
| 11:20 |
Dyrcona |
At that rate, it will still take roughly 5 days to export them all. This is one a test system, but it's an old production database server and it's "configured." The hardware is no slouch. I guess I will have to dump queries and run them through EXPLAIN. |
| 11:28 |
Dyrcona |
Y'know what. I think I'll stop this export, back out the entire feature and go again. |
| 11:29 |
jeff |
if it's similar behavior as yesterday and most of the resource usage appears to be marc_export using CPU, I'd suspect inefficiency in the MARC record manipulation or in dealing with the relatively large amount of data in memory from the use of fetchall_ on such a large dataset. |
| 11:29 |
jeff |
even if it's not swapping, dealing with that large a data structure might be giving Perl / DBI a challenge. |
| 11:30 |
jeff |
(both of those are just guesses, though. I don't have time this week to experiment to test the theories.) :-) |
| 11:30 |
Dyrcona |
jeff: That might be it. Maybe it's actually the other patch that I pushed yesterday to manipulate the 852? |
| 11:30 |
jeff |
I like your idea of next step, though. Especially if you've exported a set this large before without issue. |
| 11:31 |
Dyrcona |
Well, "without issue" is up for debate.... |
| 12:07 |
pinesol |
Launchpad bug 1788680 in Evergreen 3.3 "Null statcats break copy templates" [High,Fix released] https://launchpad.net/bugs/1788680 |
| 12:07 |
|
jihpringle joined #evergreen |
| 12:08 |
Dyrcona |
We're on 3.7, and I think it stores fleshed JSON objects. The Angular client looks like it only stores stat cat ids. |
| 12:08 |
jeff |
I may resort to empirical testing. Copy affected JSON to a 3.10 system and re-test there. :-) |
| 12:08 |
Dyrcona |
I didn't make a not of the line numbers. I didn't think I'd want to look at it again. |
| 12:09 |
Dyrcona |
On 3.7, it looks like it includes everything in the template including alerts. I seem to recall in master, it ONLY looks at stat cats an only puts in the ID. This may come up again on Friday, so I think I'll have another look. |
| 12:14 |
Dyrcona |
I should have made notes. |
| 12:40 |
jihpringle |
the new alert messages have also caused some funky template issues - https://bugs.launchpad.net/evergreen/+bug/2022349 |
| 12:40 |
pinesol |
Launchpad bug 2022349 in Evergreen "Remove old-style copy/item alerts" [Medium,Confirmed] |
| 12:56 |
Dyrcona |
Everyone have a good rest of your day (or night)! I'm taking off early. |
| 13:11 |
mmorgan |
jeff: I'm familiar with 1788680, but have seen other issues with stat cats in templates, too. |
| 13:11 |
mmorgan |
Mostly templates trying to apply stat cat entries that no longer exist. |
| 13:11 |
|
collum joined #evergreen |
| 13:13 |
mmorgan |
I think I've seen the -1 once or twice, too. Maybe that was an attempt to remove the stat cat value? |
| 13:13 |
|
jihpringle joined #evergreen |
| 13:17 |
mmorgan |
We're not yet using the angular holdings editor, but in testing, are becoming aware of things lurking in templates that get applied and saved in items in Angular, but were ignored in angularjs. Like the copy alerts in 2022349 |
| 13:28 |
|
redavis joined #evergreen |
| 13:32 |
eby |
https://docs.paradedb.com/blog/introducing_bm25 |
| 13:34 |
jeff |
I tested one of the "statcats":{"24":-1} templates on a 3.10 system and while it did set the stat cat in question to "unset" (likely the original template's intent/origin), it did so in a way that was successful, and allowed the item to be saved. On 3.7, the behavior was that the stat cat was invalid and failed to save. |
| 13:36 |
jeff |
eby: saw that ("pg_bm25: Elastic-Quality Full Text Search Inside Postgres"). Also saw some comments at https://news.ycombinator.com/item?id=37809126 from a few days ago that I bookmarked for later. |
| 13:37 |
jeff |
(the "24" in my template fragment isn't significant, it's just the ID of a copy stat cat here) |
| 13:56 |
|
mantis1 left #evergreen |
| 14:24 |
berick |
eby: neat. |
| 14:24 |
berick |
also neat https://github.com/quickwit-oss/tantivy |
| 10:17 |
jeff |
That's what, 3.2 records per second? That seems exceptionally slow. Any ideas/suspects, or are you just noticing and looking into it? |
| 10:22 |
jeff |
5.9 kb average record size seems reasonable, especially with holdings data added. Our non-deleted bibs (excluding a bunch of deleted stubby ILL bibs) are about 4.5 kb on average. |
| 10:23 |
jeff |
anyway, if you had something looping or some kind of unintentionally additive loop/array, i'd expect you'd have a lot more than 7.7 GB of output. |
| 10:26 |
Dyrcona |
jeff: I'm not really looking into it. I'm testing some patches on a test system, and also doing a test run for a big Aspen export. I thought I'd just run marc_export like this and time it: /usr/bin/perl /openils/bin/marc_export --all --items --exclude-hidden --852b circ_lib --format XML |
| 10:27 |
Dyrcona |
`ps -o etime 24586` said 4-19:00:01 just a few seconds ago. |
| 10:28 |
Dyrcona |
I'm running it with time, but was curious how long it has been going so far. |
| 10:29 |
Dyrcona |
Adding --items seems to really slow it down on this setup. |
| 10:50 |
Dyrcona |
I'd like to run it through explain. It's probably the queries to grab item information to add to the MARC, so I'll have to dump an example of that, too. |
| 10:52 |
Dyrcona |
Guess I will be looking into it later.... *sigh* |
| 10:52 |
jeff |
or tweak log_min_duration_statement just long enough to capture some sample queries. depends on how otherwise loaded your db server is, if this is prod. |
| 10:56 |
Dyrcona |
This is a test system that hosts multiple databases, but this is the only instance currently doing anything. |
| 10:56 |
jeff |
ah, that makes many things easier. |
| 10:56 |
Dyrcona |
I wonder if the number of page faults in marc_export matters.... |
| 10:57 |
jeff |
if you're trying to hold that whole resultset in memory... I haven't looked at how marc_export does paging, if at all. |
| 11:03 |
Dyrcona |
It's using 98.2% CPU, which nothing because the VM is configured with 16. The hardware has 32 with HTT enabled. |
| 11:03 |
Dyrcona |
It's not running on the same server as the DB either. |
| 11:06 |
Dyrcona |
I'll have to do some investigation to see where the problem lies. Maybe I can get some improvements for everyone out of this. |
| 11:20 |
Dyrcona |
jeff++ # I may just crank the logging up for a test run later. I suspect this one will finish sometime later today, but I also thought that it would have done by yesterday to start with. |
| 11:24 |
Dyrcona |
FWIW, I'm dumping XML because it's "easier" to work with than binary MARC, but when a file is about 8GB in size, the format doesn't really matter any longer, does it? :) |
| 11:25 |
|
collum joined #evergreen |
| 11:33 |
|
kmlussier joined #evergreen |
| 11:39 |
|
jihpringle joined #evergreen |
| 13:00 |
Dyrcona |
Y'know. I think I'll also do a similar extract with the --items option and time that. I also want to check what sandbergja reported on Lp 2015484. |
| 13:00 |
pinesol |
Launchpad bug 2015484 in Evergreen "marc_export: provide way to exclude OPAC-suppressed items" [Wishlist,Confirmed] https://launchpad.net/bugs/2015484 - Assigned to Jason Stephenson (jstephenson) |
| 13:41 |
Dyrcona |
jeff: I think some of the patches that I am testing are responsible for the slow down, particularly the one for the above Lp bug. |
| 13:45 |
Dyrcona |
I think I'll revert a couple of commits before I say much more. |
| 14:21 |
Dyrcona |
Hmm... Looks like I have somewhere in the vicinity of 400,000 records left to export. I think I'll stop this one and try again with the suspected commits reverted. |
| 14:25 |
Dyrcona |
Think I'll export to a binary MARC file this time. At least the file will be smaller. |
| 15:18 |
Bmagic |
#info Pullrequest tag Added - 19 |
| 15:18 |
Bmagic |
#info Signedoff tag Added - 8 |
| 15:18 |
Bmagic |
#info Fix Committed - 12 |
| 15:18 |
Bmagic |
#topic New Business - Should we interfile nightwatch tests into the typical eg2/src/app directory (bug 2036831)? - Jane |
| 15:18 |
pinesol |
Launchpad bug 2036831 in Evergreen "Move nightwatch tests into the same directory as the code they test" [Wishlist,Confirmed] https://launchpad.net/bugs/2036831 |
| 15:18 |
mmorgan |
Lots to test, lots to commit :) |
| 15:18 |
Bmagic |
#link https://bugs.launchpad.net/evergreen/+bug/2036831 |
| 15:19 |
sandbergja |
Ooh, This came up in the newdevs group! Especially from sleary, so please jump in if I get something wrong :-) |
| 15:19 |
Bmagic |
mmorgan++ # thanks for doing the stats! it's nice to see them like that |
| 15:20 |
sandbergja |
the idea was to reduce some of the maintenance burden of the nightwatch tests by making it clear when a particular UI has some nightwatch tests that exercise it |
| 15:20 |
sandbergja |
And one easy way to do that is to just house them in the same directories |
| 15:20 |
sandbergja |
like we already do with the unit tests |
| 15:20 |
sleary |
ah yes! I am 100% on board with doing more Nightwatch tests, but (as I am also rearranging our markup on a daily basis) I want to make sure it's easy to find the test that corresponds to the file you're looking at |
| 15:20 |
sandbergja |
So what do you think? |
| 15:21 |
Bmagic |
I've been confused by this before, but I figured that there was a reason for it that predates me |
| 15:21 |
berick |
any concerns about moving them? |
| 15:21 |
berick |
certainly makes sense |
| 15:22 |
berick |
having the .html and .ts files side by side has been a huge help |
| 15:22 |
berick |
just as an example |
| 15:22 |
sleary |
sandbergja: it's relatively easy to change the command to run all the tests from a directory to a filename pattern, right? |
| 15:22 |
terranm |
It will also be easier to see if there are things that should have Nightwatch tests that do not |
| 15:22 |
Bmagic |
I'm all about file folder organization, and it makes sense to me to have related things together. |
| 15:23 |
shulabear |
+1 to having tests in the same directory as the code being tested |
| 15:23 |
sandbergja |
sleary: theoretically, yes, it should be just changing the config file. I haven't actually tried it though |
| 15:23 |
sleary |
sandbergja++ |
| 15:24 |
sandbergja |
Cool! I'm not hearing any objections. Anybody interested in taking this one on? |
| 15:24 |
Bmagic |
do we vote maybe? |
| 15:24 |
sandbergja |
ooh, vote sounds fun |
| 15:25 |
Bmagic |
hah, lets see if I can pull that off |
| 15:26 |
Bmagic |
#startvote Shall we move the tests to related places in the Evergreen codebase yes, or no |
| 15:26 |
pinesol |
Unable to parse vote topic and options. |
| 15:26 |
Bmagic |
lol |
| 15:27 |
Bmagic |
#startvote Shall we move the tests to related places in the Evergreen codebase? yes, no |
| 15:27 |
pinesol |
Begin voting on: Shall we move the tests to related places in the Evergreen codebase? Valid vote options are yes, no. |
| 15:27 |
pinesol |
Vote using '#vote OPTION'. Only your last vote counts. |
| 15:27 |
Bmagic |
#vote yes |
| 15:27 |
shulabear |
#vote yes |
| 15:27 |
Dyrcona |
#vote yes |
| 15:27 |
pinesol |
yes (6): shulabear, sandbergja, mmorgan, Bmagic, Dyrcona, collum |
| 15:27 |
Bmagic |
voting is still open |
| 15:27 |
Bmagic |
#endvote |
| 15:27 |
pinesol |
Voted on "Shall we move the tests to related places in the Evergreen codebase?" Results are |
| 15:27 |
pinesol |
yes (6): shulabear, sandbergja, mmorgan, Bmagic, Dyrcona, collum |
| 15:28 |
Bmagic |
that was fun |
| 15:28 |
berick |
heh, "the tests" to "the places". we're still talking about nightwatch yes? |
| 15:28 |
terranm |
Sorry, "yes" - heh |
| 15:28 |
dluch |
lol |
| 15:28 |
berick |
#vote yes |
| 15:35 |
sandbergja |
if not, I'm happy to be on hand with any questions you have shulabear |
| 15:35 |
sandbergja |
^on hand to answer |
| 15:36 |
shulabear |
Assuming my girlfriend's surgical consult that morning doesn't run long, I should be free |
| 15:36 |
Bmagic |
#action sandbergja will go over the Nightwatch test reorg with folks at the Monday at 2pm ET meeting or another time as available |
| 15:36 |
shulabear |
Put it on my calendar |
| 15:36 |
shulabear |
Sandbergja++ |
| 15:37 |
Bmagic |
sandbergja++ |
| 15:39 |
dluch |
mmorgan++ |
| 15:39 |
Bmagic |
#info Hack-a-way is coming up! |
| 15:39 |
Bmagic |
#link https://wiki.evergreen-ils.org/doku.php?id=hack-a-way:hack-a-way-2023 |
| 15:39 |
sandbergja |
it occured to me that we should probably merge bug 2035287 before moving the nightwatch tests. Make them work first before messing with them hahaha |
| 15:39 |
pinesol |
Launchpad bug 2035287 in Evergreen "e2e tests are failing" [Undecided,New] https://launchpad.net/bugs/2035287 |
| 15:39 |
Bmagic |
mmorgan++ |
| 15:41 |
|
jihpringle joined #evergreen |
| 15:41 |
Bmagic |
thanks sandbergja, that looks like a good one to take care of |
| 11:53 |
|
redavis joined #evergreen |
| 12:01 |
|
jihpringle joined #evergreen |
| 14:02 |
|
redavis_ joined #evergreen |
| 14:07 |
jeffdavis |
I've got a test server where generating Did You Mean suggestions is adding 23 seconds to the runtime of a keyword search, because the call to search.symspell_lookup has to process 62,000 potential suggestions. |
| 14:08 |
jeffdavis |
This is a 3.11 server. |
| 14:11 |
|
redavis joined #evergreen |
| 14:13 |
jeffdavis |
symspell dictionary was generated using the sideload script with the default values for internal flags (symspell.prefix_length = 6, symspell.max_edit_distance = 3) |
| 14:23 |
|
Dhruv_Fumtiwala joined #evergreen |
| 14:23 |
Dhruv_Fumtiwala |
copy_vis_attr_cache |
| 14:23 |
Dhruv_Fumtiwala |
Can you please help me as to what this table has information about. |
| 15:43 |
jeff |
jeffdavis: possibly reading too much into your words, by "slow in 3.11" do you mean that there is a version of Evergreen prior to 3.11 where search suggestions did not result in very slow searches? |
| 15:45 |
|
kworstell-isl joined #evergreen |
| 15:45 |
Stompro |
In the angular catalog search, the opac hidden marc coded value maps that are not marked opac visible are showing. Is that a config option? |
| 15:45 |
jihpringle |
jeff: when I do a search on our 3.11 test server public catalogue it takes 52 seconds to give me any results, a search on PINES' catalogue takes seconds and they have DYM running in 3.10 |
| 15:50 |
jihpringle |
Strompo: it's a bug, we deleted the hidden ones off our server since we weren't using them anyways |
| 15:50 |
jihpringle |
Stompro ^^ |
| 15:51 |
jihpringle |
here's the original bug - https://bugs.launchpad.net/evergreen/+bug/1872867 |
| 15:51 |
pinesol |
Launchpad bug 1872867 in Evergreen "Angular Staff Catalog Format Drop Down List Not Using Search Labels or Observing OPAC Visible Value" [Undecided,Fix released] |
| 15:52 |
Stompro |
jihpringle++ thank you very much. |
| 15:52 |
jihpringle |
np :) |
| 09:33 |
Bmagic |
CONS is there |
| 09:34 |
Dyrcona |
Errors from Socket.pm are a bit surprising if that's the problem, but it could be that some service doesn't run. |
| 09:34 |
Dyrcona |
Last time I tried this, it was for the MVLC migration in 2012... |
| 09:38 |
Bmagic |
just about to test concerto. If I get the same error, then I'll chase a different problem |
| 09:44 |
Bmagic |
same prob |
| 09:44 |
Bmagic |
I'll figure it out. This should* work. I setup concerto machines all the time |
| 09:46 |
Dyrcona |
I'd osrf_control --diagnostic and check that ejabberd, memcached, etc., are running. |
| 09:56 |
berick |
Bmagic++ |
| 09:56 |
Bmagic |
at least at first :) |
| 09:56 |
Dyrcona |
Bmagic++ |
| 09:57 |
Dyrcona |
I've been using Pg 15 with test VMs for awhile. I've even got one with produciton data. It's faster at some things than Pg 10, even when Pg 15 isn't optimized. |
| 09:57 |
Dyrcona |
We should start looking at Pg 16. It was released already. |
| 09:57 |
Bmagic |
I figured this would be an opportunity to see if Evergreen (with nothing and no history) can setup and work on PG15. I'll be watching the PG logs like a hawk |
| 09:58 |
Dyrcona |
Bmagic: It should. |
| 09:58 |
Bmagic |
I'm sure it will look* like it's working |
| 09:58 |
Bmagic |
I'll be expecting to see some nasy stuff in the logs, but hey! maybe not |
| 09:58 |
Dyrcona |
I told you it will* work. What more do you need? |
| 09:59 |
Bmagic |
running through the migration of 500k bibs into an empty DB with PG15 should be a good test |
| 10:00 |
* Dyrcona |
loads dumps and has imported bibs into Pg 15 with production data. It should work. |
| 10:01 |
Bmagic |
wouldn't it be funny if PG15 was the issue with this socket error |
| 10:01 |
Dyrcona |
Did you set pg_hba.conf properly? |
| 10:24 |
Dyrcona |
yeah, I think it does. |
| 10:24 |
Dyrcona |
If it has the installation for Pg 15, then it should have all of the commits. |
| 10:25 |
* Dyrcona |
makes a Lp bug.... |
| 10:27 |
Bmagic |
I'm testing my theory that my socket issue was due to the Linux host being 22.04 and the container was 20.04 |
| 10:28 |
Dyrcona |
Maybe..... |
| 10:28 |
Dyrcona |
You can run Evergreen on 22.04. It works there, too. |
| 10:28 |
Bmagic |
yeah, I might target 22.04 for this. I can't remember why I shy away from that one. There was something about it but I can't remember |
| 10:28 |
Dyrcona |
I do most of my testing on Ubuntu 22.04. |
| 10:29 |
Dyrcona |
Lp 2037656 |
| 10:29 |
pinesol |
Launchpad bug 2037656 in Evergreen "Add Support for PostgreSQL 16" [Wishlist,New] https://launchpad.net/bugs/2037656 |
| 10:29 |
Bmagic |
Dyrcona++ |
| 11:58 |
Bmagic |
the Evergreen application is installed on the same machine with PG? |
| 11:58 |
Bmagic |
when you said 1 machine (out of 5) I assumed each one had a single-purpose dedicated OS |
| 12:01 |
Bmagic |
how about this: what if you setup a new brick from scratch. Does the problem occur there? |
| 12:01 |
jmurray-isl |
Yes, we have two non-prod bare metal machines that use a full copy of the prod database, one for migration testing, and the other for practice. I eventually plan to move practicing over to the migration testing server, but currently it helps to have them separate. The others would be training/testing VMs, which work fine. |
| 12:03 |
jmurray-isl |
At this point, it would probably save time just to rebuild the server in question, but I wasn't sure if there'd be a quicker solution. |
| 12:04 |
Bmagic |
if you can't find a difference between the working machine and the non-working machine. A difference in any of the meaningful Evergreen configs and Apache configs, then it's gotta be OS |
| 12:04 |
jmurray-isl |
One would think. The OS should be the same (both Ubuntu Focal). |
| 12:05 |
Dyrcona |
jmurray-isl: Does the too many redirects message prevent it from working, or does it go away and then things work? |
| 08:45 |
|
sandbergja joined #evergreen |
| 09:12 |
|
Dyrcona joined #evergreen |
| 09:21 |
|
collum joined #evergreen |
| 09:32 |
Dyrcona |
Today's test is going better than yesterday's. It is still chugging away at the events, and I apparently have not run out of drones. |
| 09:34 |
Dyrcona |
I'm using our production server's configuration for this test. |
| 09:35 |
Dyrcona |
Modulo the database connection parameters, etc., of course. |
| 09:36 |
Stompro |
Morning all, I'm looking into cleaning up our /openils/var/web/reporter data this morning. Every report generated since we migrated in 2015 exists in there. Is there an existing evergreen method for purging those? |
| 09:47 |
Stompro |
Bug 1835317 talks about adding a retention interval to report output. |
| 12:12 |
Dyrcona |
Is the editor always_xact? |
| 12:12 |
Dyrcona |
Oh, never mind. It is. |
| 12:12 |
Dyrcona |
:P |
| 12:13 |
Dyrcona |
I'm having one of those days. I have to rerun a test of marc_export because I overwrote one of the output files with the data that I wanted to compare it to. |
| 12:15 |
berick |
Bmagic: but also... working code wins, so don't let me mood derail everything ;) feeling ranty this a.m. |
| 12:16 |
berick |
also, apparently, talking like a pirate |
| 12:17 |
Bmagic |
berick: I think I agree that the API shouldn't crash |
| 12:31 |
berick |
Bmagic: that means those try/catch'es do nothing. that crash is happening in a whole different OS process |
| 12:32 |
berick |
(the api should also be fixed, of course, but on the topic of try/catching...) |
| 12:32 |
Bmagic |
we'll dig |
| 13:01 |
Dyrcona |
My other test turns out to have a bad template... |
| 13:07 |
Dyrcona |
Ah, found it. Though the error message didn't really help.... It complained about an unexpected token (END) when it looks like the problem was an extra %] higher up. |
| 13:57 |
|
jihpringle joined #evergreen |
| 14:29 |
Stompro |
Great... testing EDI is fun. Just sent in an order because "edi_pusher.pl --debug-only=1" is not the correct format for that flag, so it just runs like normal. |
| 14:30 |
jeffdavis |
:( |
| 14:35 |
Stompro |
I guess I'll get to see what happens when PO and line item id numbers get reused does to the B&T system and to our system if they send us a response message. |
| 14:37 |
Stompro |
The test seems to be showing that EDI generates the same message on 3.11.1, with Ruby 3.1 and the Ruby EDI translator on Debian 12 as it did on our old system. |
| 14:38 |
jeff |
later that same day, From: Baker & Taylor / Subject: Server Outage Impacting Systems |
| 14:44 |
Stompro |
Ha, but maybe it will fix the content cafe issue... if they have to restart their redis servers. Ever hopeful. |
| 14:52 |
|
jihpringle joined #evergreen |
| 08:28 |
|
kworstell-isl joined #evergreen |
| 08:33 |
|
mmorgan joined #evergreen |
| 09:10 |
|
Dyrcona joined #evergreen |
| 09:15 |
Dyrcona |
So, the action_trigger_runner blew up processing the modified autorenew events in my test last night. There were 55,706 of them. |
| 09:17 |
Dyrcona |
Most of the events last night (i.e. all of the events that run) are collected or collecting. Only a few event completed. There was nothing else going on in the database, only this one vm running this test, so it's easy to overwhelm action trigger. |
| 09:19 |
Dyrcona |
Time for some spelunking in the logs. |
| 09:21 |
Bmagic |
why is authority_control_fields.pl so expensive? |
| 09:21 |
Dyrcona |
The open-ils.trgger stderr log has a new one on me: Caught error from 'run' method: Unable to update event state at /usr/local/share/perl/5.34.0/OpenILS/Application/Trigger/Event.pm line 247. That might have something to do with a not connected to the network message a few errors higher up. Too bad the stderr log lacks timestamps. |
| 09:51 |
Dyrcona |
Yeah, but I've got 4 total certs for the chain file. |
| 09:56 |
Dyrcona |
I suppose I can figure it out by inspecting the certs. Our cert goes first, then I think they go in order from the one that sigend our cert down to the root. |
| 09:57 |
Dyrcona |
And, I can omit the self-signed root authority assuming that the recipient has it anyway. |
| 10:01 |
Dyrcona |
Right. Easy-peasy... Now to test it. |
| 10:06 |
jeff |
for intermediate certs, apache (starting with 2.4.8) is leaf-to-root, nginx needs the leaf to be first but may tolerate a different order on the others (but why complicate things -- leaf-to-root works here too). pound uses leaf to root followed by the private key in the same PEM file. in most cases (with the possible exclusion of some weird cross-signing), the self-signed actual root cert found in the |
| 10:06 |
jeff |
browser's trust store doesn't need to be included in the file or sent. |
| 10:07 |
jeff |
exim and dovecot and friends i usually look in the docs, or follow whatever my deployment scripts or existing files contain. :-) |
| 10:07 |
Dyrcona |
exim and dovecot work with whatever certbot does. I can say that. Oh, so does Apache 2.4. |
| 10:08 |
jeff |
happily, with intermediates being more or less a fact of life now, you rarely have to go digging in the source or rely on empirical test-and-hope techniques... since they're so common, they're much more well-documented now. :-) |
| 10:08 |
Dyrcona |
This one is just nginx an apache, and I'm getting a gateway timeout. |
| 10:08 |
jeff |
I am happy to keep certbot far, far away from as many systems as possible. |
| 10:08 |
Dyrcona |
Well, the RFC also specifies the order, and most software follows the RFC. |
| 10:48 |
Dyrcona |
s/(list)/\1en/ |
| 10:55 |
Dyrcona |
Y'know. I'm starting to suspect that the 500 Ineternal Sever Error was caused by the backlog. It wasn't just open-ils.trigger that was overloaded. |
| 10:56 |
Dyrcona |
Yeah. It's fine now that I've restarted services. |
| 10:57 |
Dyrcona |
So, my test failed successfully. I now know that I need to increase resources before trying this again. :) |
| 10:58 |
|
briank joined #evergreen |
| 11:35 |
sharpsie |
@dunno add SYSTEM CONTROL RESTART OPEN SURF |
| 11:35 |
pinesol |
sharpsie: Error: You must be registered to use this command. If you are already registered, you must either identify (using the identify command) or add a hostmask matching your current hostmask (using the "hostmask add" command). |
| 10:28 |
* mmorgan1 |
*believes* the biblio.record_entry.email trigger is for both baskets and individual bib records. |
| 10:29 |
mantis1 |
mmorgan1: thinking the same at this point |
| 10:29 |
Bmagic |
that might be right. I've had to track this down before but I don't remember what the conclusion was |
| 10:30 |
mantis1 |
testing now |
| 10:32 |
mantis1 |
yup that was it |
| 10:32 |
mantis1 |
mmorgan1++ |
| 10:32 |
mantis1 |
Bmagic++ |
| 10:32 |
Bmagic |
matis++ mmorgan1++ |
| 10:32 |
mmorgan |
mantis1++ |
| 10:32 |
Bmagic |
mantis1++ even |
| 14:53 |
Stompro |
Does the vendor (B&T) know what attributes they want if I ask them? |
| 14:55 |
jihpringle |
Stompro: maybe email the acq list? based on the last time we talked about EDI Attributes in the acq interest group very few libraries/consortia have made the switch |
| 14:56 |
Dyrcona |
We've switched but I don't know the details about the attributes. That would be someone else here who doesn't hang out in IRC. |
| 14:57 |
Stompro |
Thanks, I think I have the Ruby translator installed on Debian 12, but I don't know how to test it out. So I was looking at the new generator also. |
| 14:58 |
Dyrcona |
We haven't used the Ruby code in a couple of years. |
| 14:59 |
mantis1 |
We had a problem with the loader before |
| 14:59 |
mantis1 |
turned out it was because we didn't upgrade Ruby |
| 15:03 |
Stompro |
I guess if I don't run the order pusher, and just the A/T I can see if the EDI message gets generated. |
| 15:06 |
|
jihpringle joined #evergreen |
| 15:10 |
Dyrcona |
We've had fun with that and new versions of Ruby in the past. I'll be glad to see it go. |
| 16:11 |
Dyrcona |
I'm going to start another test run of action triggers on a vm with 32GB of RAM and 16 cores. (It has double the number of cores over production.) Should I double the parallel values for action trigger from 3 to 6 for collectors and reactors? |
| 16:15 |
Dyrcona |
All right, I will double the values. We'll see how this goes tomorrow. |
| 16:55 |
pinesol |
News from commits: Docs: DIG Reports Docs Update Project <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=95c08fe464ce0b0686893ba19a87085758cfd5bf> |
| 17:02 |
|
mantis1 left #evergreen |
| 09:08 |
pinesol |
News from commits: LP2034969: Update docs to reference EDI cron jobs <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=d9ec9d3295389ca42b8b0c29b18b6ce40e28cdbd> |
| 09:08 |
pinesol |
News from commits: LP2034969: Add EDI Scripts to Example crontab <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=8f3690d9580bad8d4c1fb9b324811b2287f6dd32> |
| 09:08 |
pinesol |
News from commits: LP2034969: Add EDI Scripts to Makefile.am <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=74d46c4f7b726d611cf0151f8b7ecba9fd7791ad> |
| 09:11 |
sandbergja |
Fun fact (and cause for celebration): we have exactly 100 angular client unit tests in main right now! |
| 09:12 |
sandbergja |
Our safety net is getting stronger! Maybe we can make it 200 before the next big angular or ng-bootstrap upgrade. :-) |
| 09:22 |
Dyrcona |
sandbergja++ |
| 09:39 |
|
kworstell-isl joined #evergreen |
| 09:43 |
berick |
sandbergja++ |
| 10:22 |
sharpsie |
lrwxrwxrwx 1 root root 4 Jul 18 2019 /bin/sh -> dash |
| 10:23 |
sharpsie |
that's on a 20.04 server here |
| 10:24 |
Dyrcona |
sharpsie: The crontabs both do SHELL = /bin/bash |
| 10:25 |
mantis1 |
JBoyer it may be I can give it a shot in our test server then get back to you |
| 10:27 |
sharpsie |
Dyrcona: ah |
| 10:27 |
Dyrcona |
Anyway, that's small potatoes at the moment. I tried replacing our 2-day courtesy notices with a combined template for autorenewals on the "bad" vm last night, and it choked on 62,321 notices.I'll have to dig into the logs, but the a/t runner isn't running and its stuck with 52000+ "collected" events. |
| 10:27 |
sharpsie |
is LYING=true set :-) |
| 10:55 |
Dyrcona |
Heh. Confusing ld for ls is kind of funny: ld:/var/log/syslog.1: file format not recognized; treating as linker script |
| 10:55 |
Dyrcona |
ld:/var/log/syslog.1:1: syntax error |
| 10:56 |
Bmagic |
mantis1: lol, just made it through the backlog, ignore me |
| 10:58 |
Dyrcona |
I'll make sure I have a backup of the crontab that I'm testing, and then wipe this vm and build a replacement. It's probably not worth figuring out the issues. |
| 11:01 |
|
briank joined #evergreen |
| 11:07 |
jeff |
semi offtopic, but also not: anyone with a library card vendor that they can recommend, I'd be interested. Our requirements are fairly typical, but also we supply a list of card numbers (not a range), so the vendor would need to be able to support that also. |
| 11:08 |
pinesol |
News from commits: LP#1840990: The Mark Damaged and Mark Missing dialogs are missing some <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=22a30e43f45a83dfd72b55be17baf5c1bdd0bdb2> |
| 14:11 |
Dyrcona |
mantis2: At the moment, no, but there's a Lp bug for that. |
| 14:12 |
Dyrcona |
Lp 1904737 |
| 14:12 |
pinesol |
Launchpad bug 1904737 in Evergreen "Flag/setting for Items with Specific Statuses to Display on Holds Pull List" [Wishlist,Confirmed] https://launchpad.net/bugs/1904737 |
| 14:13 |
Dyrcona |
Feel free to test it and signoff! |
| 14:16 |
mantis2 |
ooo excellent |
| 14:19 |
* jeff |
reviews the comments to see where the "why not use locations for that?" discussion went |
| 14:20 |
* Dyrcona |
doesn't want to get into it. That horse has been talked to death. |
| 14:26 |
Dyrcona |
All of that being said, I like the idea of the flag on the status to show up on holds. It eliminates the "magic" status of some statuses and makes it obvious. |
| 14:27 |
Dyrcona |
Very often good ideas get shot down or delayed because the discussion gets bogged down in unnecessary detail..... |
| 14:27 |
* Dyrcona |
trips off the soapbox he didn't intend to stand on. |
| 14:39 |
Dyrcona |
mantis2: If you're going to test that, it would be helpful if you could assign the bug to yourself in Launchpad. |
| 14:41 |
jeffdavis |
Upshot of the status vs location discussion seems to be "use status for items that should revert to Available on checkin, otherwise use locations"? |
| 14:52 |
mmorgan |
At the risk of tripping over Dyrcona's soapbox, it's the fluidity of the status that's appealing vs. the rigidity of the location. My observation, anyway. |
| 14:55 |
Dyrcona |
So, I figured out why one of my cron jobs was failing. It was trying to talk a database server that doesn't exist. Turns out I was also clever enough to implement the cron job so that if I set PGHOST and PGDATABASE in the crontab it should be fixed. |
