Evergreen ILS Website

IRC log for #evergreen, 2021-10-13

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
02:08 JBoyer joined #evergreen
03:25 Bmagic_ joined #evergreen
03:25 dluch_ joined #evergreen
03:28 pastebot0 joined #evergreen
03:31 jeff_ joined #evergreen
03:34 dbs joined #evergreen
06:00 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
07:26 rjackson_isl_hom joined #evergreen
08:35 mmorgan joined #evergreen
08:45 mantis joined #evergreen
09:02 Dyrcona joined #evergreen
09:58 collum joined #evergreen
10:05 jvwoolf joined #evergreen
10:29 Dyrcona There's something about declaring "package IO::Handle;" and then adding a new IO::Handle method in a Perl script that feels so wrong and so right at the same time.
10:30 gmcharlt heh
10:30 gmcharlt package Monkey::Patch...
10:37 Dyrcona All because I wanted to write $log->log_msg(...) and not log_msg($log, ...)
10:52 Bmagic joined #evergreen
11:00 csharp_ never seen this one before, but I feel seen today: https://xkcd.com/1205/
11:01 csharp_ (learning git submodules so integrating a third party thing locally will be "easier to manage")
11:02 csharp_ also learning about git archive, which I didn't previously know about
11:03 rhamby csharp_ the comic doesn't capture the angst of sitting there going "this is a newish thing ... am I going to spend 10 minutes writting this script and never need to do it again and wasting 8 minutes or am I being proactive?"
11:03 csharp_ exactly - something along the lines of Pascal's Wager maybe?
11:03 rhamby right
11:11 Dyrcona Well, I like automating things because automated mistakes usually lend themselves to automated fixes.
11:12 Dyrcona csharp_: I work with a project that includes a submodule. If you have questions, let me know. I might be able to help.
11:13 Dyrcona So, I'm looking at MARC export, and I think it's a bug that it exports holdings in 852. I think it's supposed to be 952.
11:31 Dyrcona Eh, maybe that isn't a bug. I should have looked at the full description again. ;)
11:33 Dyrcona Hmm.. The query that I'm working on is going to be more complicated than I first thought...
11:34 Dyrcona Or, maybe not. I probably don't have to include locations for deleted copies.
11:35 Dyrcona Thanks, rubber ducky!
11:44 mmorgan __(')<
12:02 Dyrcona Wonder if I messed up, or if there are really that many copies: Record length of 2216371 is larger than the MARC spec allows (99999 bytes). at /usr/share/perl5/MARC/File/USMARC.pm line 314.
12:03 Dyrcona mmorgan++
12:03 Dyrcona Guess I'll convert to XML and have a look after it finishes.
12:05 Dyrcona Heh. I messed up my query....
12:06 Dyrcona I'm dumping info for all of the libraries copies.
12:06 jihpringle joined #evergreen
12:11 Dyrcona Well, that should be "all of the library's copies." I forgot to join on asset.call_number where record = ?
12:13 Dyrcona Also, I wonder if we should add an OU setting for ISIL codes?
12:43 Dyrcona Well, the extract is a lot slower since I added the query to grab the copy locations per bib.
12:47 csharp_ Dyrcona: this solved my issue: https://pypi.org/project/git-archive-all/ - basically trying to change up how we deal with our customizations
12:47 csharp_ up to now, we pull down the stock tar.gz, uncompress it, and lay our changes on top and I'd rather do all the changes within git
12:49 Dyrcona OK. I just make a branch for my customizations and keep it up to date with master, but I guess adding the submodule complicates things?
12:50 csharp_ we'll see if it becomes a pain in the ass :-)
12:50 Dyrcona Always fun when you get the submodule out of sync with the main code. :)
12:52 Dyrcona Here's the project I'm talking about: https://github.com/Dyrcona/openfortigui
12:54 Dyrcona Ugh. Looks like this code might be too slow to be useful.
12:57 Dyrcona When it was just straight up dumping MARC, it took about 10 to 15 minutes to dump 48,000+ records. It has been running for about 45 minutes now and only dumped 1,296 records with holdings. I should probably modify the main query to return the marc and copy info.
13:05 Dyrcona Wonder if I can array_agg over an array_agg?
13:22 Bmagic Dyrcona: I've got some perl that dumps records in parallel, I've seen it hit 300 records/second
13:24 Bmagic IIRC, 8 threads. Mind you, it's not using the perl "threads" module because Encode.pm. Instead it launches a system command and monitors a mutual file on the fs
13:47 Dyrcona Bmagic: I'm not sure doing this in parallel is worth it/possible since I have to write to a single output file.
13:48 Dyrcona I don't bother with threads in Perl 5. I use fork.
13:49 Dyrcona Anyway, I think I've got a solution. Rather than run this time consuming query once for each record, I'll do it once for all records and make a hash table of the information per record id.
13:49 Dyrcona If I get the options right, I can probably have selectall_arrayref make the data structure for me.
13:53 Dyrcona My program basically works like this: Get a list of bre.id using one of 3 queries. After that loop through the array of ids and grab the marc for each one. If this is a batch of deletes, set the leader 05 to d and write to the binary output file. Otherwise, delete the 852 tags in the marc, look up the copy location and org unt name for each copy and add a 852 to the marc for each.
13:53 Dyrcona Then, write it to the output file.
13:54 Dyrcona It got really slow when I added the copy location/org_unit query.
14:02 Bmagic I had some of the same challenges
14:04 Dyrcona Yeah. Not the first time, not the last.
14:04 Bmagic I think I landed on: run query to get the bottom ID number. Start there and alter the query with Limit 100 OFFSET X. Feed that query into a thread, move the next thread and shift OFFSET. Measure the time, if it's faster than last time, then increase LIMIT value. If no results, then recalculate the bottom ID number using OFFSET to make sure I'm not starting at the same bottom
14:05 Bmagic worked out all the kinks and it runs pretty smooth now. Kinda neat. The master thread writes a progress file that you can cat during runtime to see how many threads are going, how many records per second each thread is producing, and how many total records per second for the whole operation
14:08 Bmagic each thread writes a mrc file into a temp folder and tells the master thread the location of the output. When all is done, the master thread reads all outputs and streams them into a single file, removing the temp files in the process. Then FTP's it to wherever or writes it to the configured hard drive folder, etc.
14:08 Dyrcona Yeah. I don't use threads in Perl5. I'd do something like that in Python or C++, though.
14:09 Dyrcona You synchronize writes to 1 file with a mutex lock. I just threads would overcomplicate this program.
14:09 Bmagic cool, sounds like you got it handled
14:10 Bmagic here's my think FWIW https://github.com/mcoia/mobius_e‚Äčvergreen/tree/master/bib_extract
14:10 Bmagic thing*
14:11 Dyrcona :)
14:11 Dyrcona Seems like we're constantly reinventing marc_export.pl. :)
14:11 Bmagic yeah, lol, in my defense, I started this in 2013
14:19 Dyrcona Well, I think this is going to be faster. I'll just assume that the entries in my two array_aggs line up.
14:41 Dyrcona Yeah. It's a lot faster, now.
14:48 jvwoolf joined #evergreen
14:49 Bmagic Dyrcona++
14:49 Bmagic excellent
14:50 Bmagic open question: anyone had issues with the new bootstrap OPAC (integrated into the staff client) when placing holds for patrons, Specifically copy level holds, not automatically filling in the patron barcode in the hold placement page?
14:51 Bmagic I couldn't find a bug report... but we all know how launchpad can be when it comes to search....
15:03 Dyrcona I seem to recall there being bugs about copy holds and the new catalog/staff client, but not sure of the specifics.
15:21 jvwoolf joined #evergreen
15:51 jvwoolf left #evergreen
16:01 jihpringle joined #evergreen
17:21 mmorgan left #evergreen
18:01 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
19:34 jihpringle joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat