| Time |
Nick |
Message |
| 07:18 |
|
collum joined #evergreen |
| 07:41 |
|
BDorsey joined #evergreen |
| 08:21 |
|
kworstell-isl joined #evergreen |
| 08:31 |
|
shulabear joined #evergreen |
| 08:31 |
|
bott-grpl joined #evergreen |
| 08:32 |
|
mmorgan joined #evergreen |
| 09:03 |
|
mantis1 joined #evergreen |
| 09:23 |
|
Dyrcona joined #evergreen |
| 09:29 |
|
BDorsey joined #evergreen |
| 09:39 |
|
sandbergja joined #evergreen |
| 10:22 |
Stompro |
gmcharlt++ thanks for the great suggestions for fieldmapper usage. |
| 10:33 |
Dyrcona |
Y'know, the way GMail handles message threads is sometimes nice, sometimes bothersome, and downright dangerous to a person's career. |
| 10:34 |
|
dguarrac joined #evergreen |
| 10:36 |
mmorgan |
:) |
| 10:43 |
Dyrcona |
i have one email chain with 3 different conversations going on, and that is not unusual. |
| 10:44 |
Dyrcona |
Well, different, but related....They're mostly digressions on subtopics of the main chain. |
| 10:44 |
Dyrcona |
I can see how doing a replay all on the wrong email in such a situation could get someone in trouble. :) |
| 10:45 |
Dyrcona |
s/replay/reply/ |
| 10:45 |
Dyrcona |
Anyway, anyone else having a fun morning? We turned on Quipu e-renewals for e-card accounts in production today and found some things that slipped through. |
| 10:46 |
mmorgan |
Yes, those threads can get tangled. I've certainly missed messages a few times because they got sucked into an old or unrelated thread. |
| 10:47 |
Dyrcona |
Yeah. |
| 10:47 |
mmorgan |
Dyrcona: Interesting! |
| 10:47 |
Dyrcona |
In fact, I missed something this morning that way. |
| 10:49 |
jeff |
if only there were a standard way for email clients to tie replies into their related threads, without needing to rely on weak heuristics like subject lines... </sarcasm> |
| 10:50 |
Dyrcona |
Well, the issues were minor and mostly caused by my having 8 to 10 different branches on 3 different Evergreen "versions" scattered in 5 repositories across 3 different machines, and a couple of those repositories are not Evergreen, but for the way MOBIUS does virtual machines. Something was bound to get missed somewhere. |
| 10:50 |
jeff |
I'm not sure which is worse: Gmail completely disregarding the established standards, or clients that used the standards successfully but in ways that were less apparent to the end user, resulting in unintentional mis-threading. Probably a wash at first, though it would have been nice to have reached a better place in the 20 years since. |
| 10:51 |
Dyrcona |
That, and Gmail is a web application. I'm becoming less and less fond of those. ;) |
| 10:52 |
Dyrcona |
I suppose the Gmail app on the phone isn't(?) a web application. It's hard to tell. ;) |
| 10:52 |
mmorgan |
Dyrcona: Glad the issues were minor! Were you missing updates to files? |
| 10:53 |
|
briank joined #evergreen |
| 10:54 |
Dyrcona |
mmorgan: Yes for the most part. Or, rather, the updates were missing parts, or in one case had code that should have been removed. |
| 10:55 |
Dyrcona |
The template fix was easy: put it in the repository, push it, pull it on the server, wait for the hour to tick around for Bmagic's sync script to run. |
| 10:55 |
mmorgan |
Ah, gotcha. Glad the fix was easy. |
| 10:55 |
Dyrcona |
The change to a new file: WWW/EGCatLoader/Ecard.pm will have to wait for new docker images. |
| 10:58 |
Dyrcona |
BTW, most of what I'm talking about is in the working repository if you want to have a look: collab/dyrcona/lp1902937-quipu-integration+erenew, user/dyrcona/cwmars_custom_rel_3_7_4, user/dyrcona/cwmars_custom_rel_3_10_3+erenew, and a couple of others. Then there are things in private repositories. |
| 10:59 |
mmorgan |
Dyrcona++ |
| 10:59 |
mmorgan |
Thanks, we'll have a look! |
| 11:02 |
Dyrcona |
I should check that last branch. It probably needs the patch I made this morning. |
| 11:02 |
Dyrcona |
We will also likely never use it in produciton.... |
| 11:05 |
Dyrcona |
It might need more patches than just that one.... I'll leave it for now. |
| 11:14 |
|
kmlussier joined #evergreen |
| 11:18 |
Dyrcona |
So, going back to yesterday's conversation about MARC export, I wonder if that commit really was the problem. I reverted that one and two others, then started a new export. It has been running for almost 21 hours and only exported about 340,000 records. I estimate it should export about 1.7 million. |
| 11:20 |
Dyrcona |
At that rate, it will still take roughly 5 days to export them all. This is one a test system, but it's an old production database server and it's "configured." The hardware is no slouch. I guess I will have to dump queries and run them through EXPLAIN. |
| 11:28 |
Dyrcona |
Y'know what. I think I'll stop this export, back out the entire feature and go again. |
| 11:29 |
jeff |
if it's similar behavior as yesterday and most of the resource usage appears to be marc_export using CPU, I'd suspect inefficiency in the MARC record manipulation or in dealing with the relatively large amount of data in memory from the use of fetchall_ on such a large dataset. |
| 11:29 |
jeff |
even if it's not swapping, dealing with that large a data structure might be giving Perl / DBI a challenge. |
| 11:30 |
jeff |
(both of those are just guesses, though. I don't have time this week to experiment to test the theories.) :-) |
| 11:30 |
Dyrcona |
jeff: That might be it. Maybe it's actually the other patch that I pushed yesterday to manipulate the 852? |
| 11:30 |
jeff |
I like your idea of next step, though. Especially if you've exported a set this large before without issue. |
| 11:31 |
Dyrcona |
Well, "without issue" is up for debate.... |
| 11:31 |
jeff |
heh. |
| 11:32 |
Dyrcona |
I don't think I'll revert the commit. I'm going to change the options to avoid both the 852 manipulation and the exclusion of hidden items. It might just be my hardware. |
| 11:32 |
jeff |
improving marc_export so that it using paging instead of using fetchall_ DBI methods would probably count as an "improve it for everyone" thing as you mentioned before. Probably would help Calvin's issue as reported on the mailing list recently, as well. |
| 11:33 |
jeff |
s/it using/it's using/ or s/it using/it uses/ :-P |
| 11:33 |
jeff |
(you get the idea) |
| 11:34 |
Dyrcona |
The main loop(s) use fetchrow_hashref. It uses fetchall_* in a couple of places when getting items, etc. |
| 11:36 |
Dyrcona |
I wonder if DBI isn't grabbing everything all at once anyway. My latest run had RSS of 9546368, 29.0% of RAM. I think RSS is reported in kb (i.e. 1000 bytes). |
| 11:37 |
|
bott-grpl joined #evergreen |
| 11:37 |
jeff |
Yeah. It's been a while since I looked at that DBI behavior, I might be wrong in placing all the blame on fetchall_ methods. |
| 11:39 |
jeff |
I know I've been surprised by Perl and/or Python memory usage when dealing with large resultsets before, only to have the docs explain why what I was seeing was perfectly expected. :-) |
| 11:39 |
Dyrcona |
:) |
| 11:41 |
Dyrcona |
I wonder if some kind of in-database procedure or cursor might help? I remember back in Sybase days being told "Don't user cursors; they're slow." Only to find out that read only cursors were pretty efficient. It was update cursors one wanted to avoid. |
| 11:42 |
Dyrcona |
Anyway, I started another run. I'll it go overnight and see what happens. |
| 11:51 |
Dyrcona |
jeff: I do think DBI is getting all the data into RAM. I'm watching the RAM usage go up, and the file is still zero bytes. |
| 11:52 |
Dyrcona |
I'm using `ps -o etime,pcpu,rss,pmem $pid` to monitor. |
| 12:00 |
jeff |
Yeah, I'm pretty sure that the DBD::Pg driver reads results into memory on execute(). Solution can be to use your own paging (with ORDER BY and LIMIT and some logic in the WHERE to start where the last page left off (like id > last id)), or there's a nested while() loop example using a declared cursor in the DBD::Pg docs, but I don't know if there's pitfalls lurking there. |
| 12:02 |
jeff |
I think that there have been a few copy template conversion bugs, including at least one dealing with stat cats. I'm looking for one where converted templates attempt to set an invalid stat cat due to the template containing JSON like: "statcats":{"24":-1} |
| 12:04 |
jeff |
probably affects at least 3.7, and while it's likely (but not certainly) fixed, I'm curious to see the discussion and see if it's a fix requiring manual updating of templates or not. Ring any bells for anyone here? |
| 12:06 |
Dyrcona |
We had copy template issues going from XUL to 3.0 and the web staff client. |
| 12:07 |
Dyrcona |
I was looking at the code for Angular and AngularJS the other day, and it looks to me like they store them differently. |
| 12:07 |
jeff |
it's similar to bug 1788680 but not quite. |
| 12:07 |
pinesol |
Launchpad bug 1788680 in Evergreen 3.3 "Null statcats break copy templates" [High,Fix released] https://launchpad.net/bugs/1788680 |
| 12:07 |
|
jihpringle joined #evergreen |
| 12:08 |
Dyrcona |
We're on 3.7, and I think it stores fleshed JSON objects. The Angular client looks like it only stores stat cat ids. |
| 12:08 |
jeff |
I may resort to empirical testing. Copy affected JSON to a 3.10 system and re-test there. :-) |
| 12:08 |
Dyrcona |
I didn't make a not of the line numbers. I didn't think I'd want to look at it again. |
| 12:09 |
Dyrcona |
On 3.7, it looks like it includes everything in the template including alerts. I seem to recall in master, it ONLY looks at stat cats an only puts in the ID. This may come up again on Friday, so I think I'll have another look. |
| 12:14 |
Dyrcona |
I should have made notes. |
| 12:24 |
Dyrcona |
jeff: Line 2574 in Open-ILS/web/js/ui/default​/staff/cat/volcopy/app.js. It looks like th AngularJS editor saves any fleshed object as a fleshed object, despite the comment about call numbers and stat cats. |
| 12:29 |
Dyrcona |
jeff: Also the if block at line 641 in Open-ILS/src/eg2/src/app/staff/cat​/volcopy/copy-attrs.component.ts. This looks like it stores fleshed stat cats and the id for anything else. |
| 12:32 |
Dyrcona |
Hope that helps! |
| 12:32 |
|
kmlussier joined #evergreen |
| 12:37 |
jihpringle |
jeff: we've had to have our libraries remake their templates a few times since over the years |
| 12:38 |
jihpringle |
there were some recent bugs/fixes around templates and stat cats |
| 12:38 |
jihpringle |
https://bugs.launchpad.net/evergreen/+bug/1999696 |
| 12:38 |
pinesol |
Launchpad bug 1999696 in Evergreen 3.9 "Holdings Templates not working with stat cat data" [Undecided,Fix released] |
| 12:38 |
Dyrcona |
We had lots of issues going from XUL to the web client. |
| 12:39 |
jihpringle |
same, and we also had issues going from angularjs to angular |
| 12:39 |
Dyrcona |
Yeah. We have not upgraded because of those issues. |
| 12:40 |
jihpringle |
the new alert messages have also caused some funky template issues - https://bugs.launchpad.net/evergreen/+bug/2022349 |
| 12:40 |
pinesol |
Launchpad bug 2022349 in Evergreen "Remove old-style copy/item alerts" [Medium,Confirmed] |
| 12:56 |
Dyrcona |
Everyone have a good rest of your day (or night)! I'm taking off early. |
| 13:11 |
mmorgan |
jeff: I'm familiar with 1788680, but have seen other issues with stat cats in templates, too. |
| 13:11 |
mmorgan |
Mostly templates trying to apply stat cat entries that no longer exist. |
| 13:11 |
|
collum joined #evergreen |
| 13:13 |
mmorgan |
I think I've seen the -1 once or twice, too. Maybe that was an attempt to remove the stat cat value? |
| 13:13 |
|
jihpringle joined #evergreen |
| 13:17 |
mmorgan |
We're not yet using the angular holdings editor, but in testing, are becoming aware of things lurking in templates that get applied and saved in items in Angular, but were ignored in angularjs. Like the copy alerts in 2022349 |
| 13:28 |
|
redavis joined #evergreen |
| 13:32 |
eby |
https://docs.paradedb.com/blog/introducing_bm25 |
| 13:34 |
jeff |
I tested one of the "statcats":{"24":-1} templates on a 3.10 system and while it did set the stat cat in question to "unset" (likely the original template's intent/origin), it did so in a way that was successful, and allowed the item to be saved. On 3.7, the behavior was that the stat cat was invalid and failed to save. |
| 13:36 |
jeff |
eby: saw that ("pg_bm25: Elastic-Quality Full Text Search Inside Postgres"). Also saw some comments at https://news.ycombinator.com/item?id=37809126 from a few days ago that I bookmarked for later. |
| 13:37 |
jeff |
(the "24" in my template fragment isn't significant, it's just the ID of a copy stat cat here) |
| 13:56 |
|
mantis1 left #evergreen |
| 14:24 |
berick |
eby: neat. |
| 14:24 |
berick |
also neat https://github.com/quickwit-oss/tantivy |
| 15:20 |
|
BDorsey_ joined #evergreen |
| 16:04 |
|
jihpringle joined #evergreen |
| 16:12 |
|
kmlussier joined #evergreen |
| 17:16 |
|
mmorgan left #evergreen |
| 20:45 |
|
Rogan joined #evergreen |
