| Time |
Nick |
Message |
| 00:01 |
|
jblyberg joined #evergreen |
| 07:24 |
|
kworstell-isl joined #evergreen |
| 07:56 |
|
Rogan joined #evergreen |
| 08:05 |
|
BDorsey joined #evergreen |
| 08:36 |
|
sandbergja joined #evergreen |
| 08:41 |
|
mmorgan joined #evergreen |
| 08:43 |
|
dguarrac joined #evergreen |
| 08:57 |
|
smayo joined #evergreen |
| 08:59 |
|
redavis joined #evergreen |
| 09:02 |
|
mmorgan1 joined #evergreen |
| 09:56 |
Bmagic |
@coffee [someone] |
| 09:56 |
* pinesol |
brews and pours a cup of Espresso Nuevo, and sends it sliding down the bar to bshum |
| 09:56 |
Bmagic |
@coffee [someone] |
| 09:56 |
* pinesol |
brews and pours a cup of Sumatra Danau Toba, and sends it sliding down the bar to degraafk |
| 09:56 |
Bmagic |
@coffee [someone] |
| 09:56 |
* pinesol |
brews and pours a cup of Kenya Peaberry Deep River Estate, and sends it sliding down the bar to Christineb |
| 09:56 |
Bmagic |
@coffee [someone] |
| 09:56 |
* pinesol |
brews and pours a cup of Colombia Cauca, Special Micro-Lot Selection Juvenal Penna, and sends it sliding down the bar to eeevil |
| 09:56 |
Bmagic |
@coffee [someone] |
| 09:56 |
* pinesol |
brews and pours a cup of Ethiopia Biloya Special, and sends it sliding down the bar to redavis |
| 09:57 |
Bmagic |
Good morning! (vietnam) |
| 09:57 |
redavis |
mmmmm, thank you |
| 09:59 |
Bmagic |
It's no sweat. I got up at 3am to start roasting it for you |
| 10:08 |
|
Dyrcona joined #evergreen |
| 10:26 |
|
mantis1 joined #evergreen |
| 10:41 |
redavis |
Thank you for your many hours of coffee roasting sacrifice. |
| 10:47 |
|
smayo joined #evergreen |
| 10:57 |
|
briank joined #evergreen |
| 11:00 |
Dyrcona |
berick: I don't know what the size of the file was at 143,000 records. The final file is 15GB with 2,251,864 bib records and 7,775,264 items. It took 18.3 hours to complete on my test system. |
| 11:00 |
Dyrcona |
berick++ |
| 11:04 |
Dyrcona |
Hmm. I guess that export running on production explains the "processor load too high on utility server emails." It keeps 1 CPU busy all the time, and there are only 8 cpus. Load was 11.5 when I just checked. |
| 11:04 |
Dyrcona |
Bmagic: I'm installing Rust on the CW MARS utility server today. |
| 11:07 |
Dyrcona |
We used to have 2 utility servers. |
| 11:19 |
Bmagic |
Dyrcona++ |
| 12:08 |
Christineb |
a cup of Kenya Peaberry Deep River Estate sounds delicious... thank you :D |
| 12:08 |
|
jihpringle joined #evergreen |
| 12:30 |
|
smayo joined #evergreen |
| 12:57 |
Dyrcona |
Oof... Making "simple" changes to a program in a new (to you) programming language are not always simple: https://www.sitepoint.com/rust-global-variables/ |
| 13:01 |
berick |
in short, don't use them, unless they are static scalar vars |
| 13:05 |
Dyrcona |
Yeah, so I want to add an option to modify ITEMS_QUERY, so I think it needs to be a RefCell and thread_local? Though, I love this quote: "it’s possible to build something that uses unsafe safely, if you know what you’re doing." |
| 13:06 |
berick |
Dyrcona: you can add a field to ExportOptions and a matching getopt entry -- if I follow what you're doing |
| 13:07 |
berick |
ExportOptions in this case is essentially our global vars collection |
| 13:07 |
Dyrcona |
Right. Then I want to modify ITEMS_QUERY to add to the WHERE clause. |
| 13:07 |
berick |
yeah, ITEMS_QUERY may need to be moved into a function that generates the SQL. |
| 13:08 |
berick |
so it can refer to the 'ops' values |
| 13:08 |
berick |
i started with the simplest possible approach for that bit |
| 13:08 |
Dyrcona |
Right. Starting simple makes sense. |
| 13:10 |
Dyrcona |
I think I'm also going to want to modify the ITEM_SUBFIELD_MAP, or at least figure out a way to skip some of the subfield b. Aspen apparently doesn't like more than 1, and I'd prefer to just send the copy circ_lib. |
| 13:12 |
Dyrcona |
For reference, I'm trying to implement Lp 2015484 and Lp 2004587. |
| 13:12 |
pinesol |
Launchpad bug 2015484 in Evergreen "marc_export: provide way to exclude OPAC-suppressed items" [Wishlist,Confirmed] https://launchpad.net/bugs/2015484 |
| 13:12 |
pinesol |
Launchpad bug 2004587 in Evergreen "multiple $bs in items from marc_export can confuse external systems" [Medium,Fix committed] https://launchpad.net/bugs/2004587 |
| 13:13 |
Dyrcona |
This is kind of cool. I'm jumping in, not at the deep end, but not exactly the shallow end either: mutexes, closures, and references.... I get it, just different syntax. :) |
| 13:19 |
sharpsie |
@blame add Stop trying to make $who happen. It's not going to happen! |
| 13:19 |
pinesol |
sharpsie: The operation succeeded. Blame #33 added. |
| 13:21 |
Dyrcona |
So, I think I'll by just hardcoding my desired behavior and seeing what happens. Start simple. |
| 13:25 |
jeff |
reminder to self: deleting a user's cat.copy.templates setting is no longer a valid way of getting their XUL-era templates to re-import. :-) |
| 13:26 |
Dyrcona |
jeff: The code is still there. I saw it the other day. Does it no longer fire? (By the "code" I mean the conversion function.) |
| 13:43 |
jeff |
Dyrcona: Angular vs AngularJS, I suspect. |
| 13:45 |
Dyrcona |
jeff: Yeah... That makes sense. |
| 13:57 |
sharpsie |
jeff: sounds like a West Wing double translator use case - import into AngJS so you can use in Angular |
| 13:58 |
sharpsie |
it's an older code, but it checks out - I was about to clear them |
| 13:58 |
|
kmlussier joined #evergreen |
| 13:58 |
jeff |
at least in 3.10, loading the Holdings Template Editor from local admin probably migrates them still, without requiring a dishwasher. |
| 13:59 |
sharpsie |
XUL... XUL... Now that's a name I've not heard in a long time... long time |
| 13:59 |
|
smayo joined #evergreen |
| 14:00 |
sharpsie |
what I'm saying y'all is we all need to do a Star Wars original trilogy rewatch |
| 14:03 |
JBoyer |
I haven't stayed on top of the marc_export discourse since I was out all last week, but I am also wondering if Dyrcona's system is missing one (or more) indexes because we do several exports that range from 85 rec/s to 240 rec/s with a ~.5M export taking only 40 mins. |
| 14:04 |
JBoyer |
Not that a new, faster exporter wouldn't be great, but I don't think that's the best effort / payoff ratio right now |
| 14:13 |
berick |
oh we're squarely in The Distraction Zone w/ the Rust stuff |
| 14:29 |
Dyrcona |
JBoyer: There's no way it is indexes. Did you read my latest email? It's sitting at 98% on a CPU usage. That's the Perl code, not the database. Database is on a different server in every case. |
| 14:30 |
Dyrcona |
berick: Speaking of Rust... I think you might have introduced a bug with a recent commit when you moved where OFFSET gets addded. I got the following when using --query-file: |
| 14:31 |
JBoyer |
Ah, still catching up here and there. Something is still bonkers with that extract though given what we're seeing here. |
| 14:32 |
Dyrcona |
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState(E42601), message: "syntax error at or near \"OFFSET\"", detail: None, hint: None, position: Some(Original(630)), where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("scan.l"), line: Some(1123), routi |
| 14:32 |
Dyrcona |
nner_yyerror") }) }', evergreen/src/bin/marc-export.rs:398:56 |
| 14:33 |
Dyrcona |
JBoyer: --items has always been slow for us, but it's worse than ever, and it really looks like it is Perl. |
| 14:34 |
berick |
Dyrcona: k. mind sharing your query file? |
| 14:35 |
Dyrcona |
berick: I'm modifying one of them. Let me try another run. |
| 14:35 |
berick |
Dyrcona: i think i see the issue.. |
| 14:37 |
Dyrcona |
One of the queries returns 3 columns: bre.id, bre.marc, and count(acp.id). It didn't loook like the 3rd column would be an issue. |
| 14:38 |
Dyrcona |
JBoyer: I would not be surprised if there is something wrong with the Perl versions I'm using or something, but I don't feel like I have time to deal with that. I'm under pressure to get them records last week. :) |
| 14:40 |
berick |
the chunked processing requires ordering/limiting/offseting which adds additional restraints to the format of the query file. for now, could just read query file as-is and avoid any paging. |
| 14:41 |
JBoyer |
True, finding the Right Fix when you're under a Right Now deadline is a lot like being technically correct but completely unhelpful. :) I just think that *after* you can get the initial export done and transported that replacing the exporter won't necessarily be the ideal fix. (Though, for later, I'm also curious what os the super slow exporter is running on) |
| 14:42 |
Dyrcona |
JBoyer: Both Ubuntu 22.04 and Ubuntu 20.04. |
| 14:43 |
JBoyer |
That is super weird. :-/ Well, I'll stop dragging things off course since I don't have a lot of helpful suggestions, but good luck and I hope this is easier to figure out once you're out from under your deadline. |
| 14:45 |
Dyrcona |
Meh. I am MORE than happy to replace Perl with Rust. |
| 14:46 |
Dyrcona |
This is not helpful, either: thread 'main' panicked at 'error retrieving column id: invalid column `id`', /home/opensrf/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-postgres-0.7.10/src/row.rs:151:25 |
| 14:51 |
|
jihpringle joined #evergreen |
| 14:52 |
berick |
Dyrcona: pushed a patch to avoid modification of the --query-file sql |
| 14:52 |
berick |
re: id, any chance you have multiple columns resolving to the name "id"? |
| 14:53 |
Dyrcona |
berick: Cool. I might.... I'm going to modify the query that I think is blowing up to use a CTE, and then grab bre.id and bre.marc. |
| 14:53 |
Dyrcona |
Currently, it's actually returning acn.record, bre.marc, and the count on acp.id. |
| 14:56 |
Dyrcona |
Dude..... I just noticed the file ends with two semicolons.....I'll bet that's it. |
| 14:56 |
Dyrcona |
Still, I think I'll do the CTE. |
| 14:57 |
berick |
my test file contain: SELECT id, marc FROM biblio.record_entry WHERE NOT deleted (no semicolons needed) |
| 14:58 |
berick |
afk for a bit |
| 14:59 |
Dyrcona |
Yeah, ; is a habit from writing stuff for psql. |
| 15:07 |
pinesol |
News from commits: LP2035287: Update selectors in e2e tests <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=f562b3ac30a3753d63d565c2d7be4d3a7121a2fb> |
| 15:11 |
Dyrcona |
Now, I'm cooking with gas. I got the "large" bibs file (85 records with 87,296 items) dumped to XML in under a minute. That took almost 2 hours with the Perl program the other day. |
| 15:18 |
Dyrcona |
Using query-file to feed the eg-marc-export, it is using a lot more RAM than before, about the same as the Perl export was using. It still uses less CPU. We'll see if that changes over time. |
| 15:28 |
jeff |
you have 87,296 items that are spread across only 85 bibs? |
| 15:29 |
Dyrcona |
Yes, we do. |
| 15:29 |
jeff |
color me intrigued. |
| 15:29 |
Dyrcona |
magazines, I assume. |
| 15:29 |
Dyrcona |
I think the one with the most has something like 4,000 items. |
| 15:30 |
Dyrcona |
I suppose I could run the query to count them to know for sure. |
| 15:32 |
Dyrcona |
Yeah, the "largest" has 4,127 items. |
| 15:32 |
berick |
Dyrcona: result of removing the paging support for --query-file |
| 15:32 |
berick |
can be added back, but i'd need to document the format of the query file a bit more |
| 15:33 |
Dyrcona |
berick: It stabilized around 11GB or 35.1% of RAM. Would it be better to throw ids at it? |
| 15:33 |
berick |
Dyrcona: yes, ids would be better w/ the current code |
| 15:34 |
JBoyer |
Yeah, serials get pretty wild if locations can't agree on a retention period. "Oh, you have 5 years of a weekly gossip rag? What's the circ look like on that one?" |
| 15:34 |
Dyrcona |
berick: I'll give that a try in production. |
| 15:37 |
|
adam_reid joined #evergreen |
| 15:41 |
adam_reid |
hello all! Does anyone have experience adjusting the formatting of "Series Title" facets? In our catalogue facets are displaying along with the individual volume number. Makes using the facets mostly pointless as it won't group items from the same series properly |
| 15:41 |
Dyrcona |
JBoyer: We have a winner! The largest one is "The New Yorker." |
| 15:43 |
JBoyer |
Yeah, I think at one point we cut people off from adding new copies to the People Magazine record and said "use this one instead, and stop trying to merge them!" |
| 15:43 |
JBoyer |
"we" - EGIN back in the Eg 1.2 or so days, that is. |
| 15:44 |
adam_reid |
I figured it must be managed under "Server Administration / MARC Search/Facet Fields" but changing the XPath value under Series Title doesn't seem to affect anything |
| 15:44 |
adam_reid |
(sorry if there is a better place to ask) |
| 15:44 |
Dyrcona |
adam_reid: Did you do an ingest after changing the XPath? |
| 15:45 |
Dyrcona |
I'm pretty sure you have to update the bib records or run one of the database function to regenerate the facet fields. |
| 15:46 |
adam_reid |
I did not! I was just searching for how to ingest / update the index. Is there a good place I can start to look for how to do that? |
| 15:47 |
adam_reid |
I did try saving the individual bib records but that didn't seem to trigger it |
| 15:48 |
Dyrcona |
There's a command line program that you can run on a utility server (recommended) or on any machine that has Perl and can communicate with your database: pingest.pl |
| 15:49 |
adam_reid |
excellent! (found the docs: https://docs.evergreen-ils.org/docs/3.3/_parallel_ingest_with_pingest_pl.html) |
| 15:49 |
adam_reid |
thanks! |
| 15:50 |
Dyrcona |
If you have any questions, feel free to ask. I wrote the initial implemenation. |
| 15:50 |
* Dyrcona |
grumbles: I wrote the current marc_export, too.....See what good being the author does me... ;) |
| 15:51 |
adam_reid |
lol, thanks Dyrcona I appreciate the help |
| 15:52 |
|
kmlussier1 joined #evergreen |
| 15:53 |
Dyrcona |
adam_reid: You're welcome! |
| 16:12 |
|
sleary joined #evergreen |
| 16:13 |
Dyrcona |
Hmm... I probably didn't have to install rust on the utility vm. As long as the Rust application is built with the same or older GLIBC, it should work..... |
| 16:15 |
Dyrcona |
I've run into that with compiled Lisp programs before.... |
| 16:15 |
Dyrcona |
BTW, I don't think Rust is a distraction. I think it could (should?) be our future. I think we should standardize on something, and Rust could be it. |
| 16:18 |
sleary |
We have a really big problem right now with the complexity of the project vs. new dev onboarding, and I think adding Rust to the mix would make it much, much worse. |
| 16:21 |
sleary |
We really need to weed the old interfaces before we add anything else. |
| 16:22 |
sleary |
Even a very large developer community would struggle to maintain the number of interfaces we are juggling right now, and we are not a large community. |
| 16:24 |
Dyrcona |
In the long run, I think it would make it easier: Just learn JavaScript and Rust. |
| 16:25 |
sleary |
Yes. Once we get rid of the other languages and obsolete frameworks. |
| 16:25 |
sleary |
We have to clear the lot before we can build the house. |
| 16:26 |
Dyrcona |
rm -rf Open-ILS/ # That's one lot cleared. :) |
| 16:26 |
Dyrcona |
I know it's not that simple. |
| 16:26 |
berick |
sleary: ageed the UI is much higher priority. for my part, i'm not aiming for any near-time large Rust code migration so much as getting a foot in the door. |
| 16:27 |
sleary |
I'm counting the scripts as an interface... just one with a very narrow audience :) |
| 16:28 |
Dyrcona |
What if the interfaces (i.e. cli options) don't change? |
| 16:28 |
sleary |
Developing for it still requires learning the underlying stuff that did change |
| 16:32 |
Dyrcona |
This would be a good topic for the hack-away next week. |
| 17:00 |
|
mantis1 left #evergreen |
| 17:06 |
|
mmorgan left #evergreen |
| 17:17 |
|
kmlussier joined #evergreen |
| 17:37 |
pinesol |
News from commits: LP#2036842: new reporting sources for invoice totals <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=4211fe0fb06ec94a29760d683a99813f9d9ee218> |
| 18:03 |
|
kmlussier left #evergreen |
| 18:24 |
|
jihpringle joined #evergreen |
| 22:33 |
|
sleary joined #evergreen |
