Evergreen ILS Website

IRC log for #evergreen, 2023-10-10

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
02:33 Rogan joined #evergreen
02:59 Rogan joined #evergreen
05:47 jblyberg7 joined #evergreen
07:39 kworstell-isl joined #evergreen
07:40 redavis joined #evergreen
08:00 BDorsey joined #evergreen
08:46 mmorgan joined #evergreen
08:47 kmlussier joined #evergreen
08:51 mmorgan Bmagic: re: emailing bibs from a basket, see also bug 1955079
08:51 pinesol Launchpad bug 1955079 in Evergreen 3.8 "Emailing Bib Records from the catalog (staff and OPAC) reports success, but fails to send email" [High,Fix released] https://launchpad.net/bugs/1955079
08:53 dguarrac joined #evergreen
09:00 mantis1 joined #evergreen
09:03 redavis joined #evergreen
09:43 Bmagic mmorgan++
09:46 bott-grpl joined #evergreen
09:47 Dyrcona joined #evergreen
10:08 Dyrcona Wow! An export of all of our records with items is still running after 114 hours and 31 minutes. It has exported 1,315,113 records so far. The XML file is 7.7 GB.
10:11 dguarrac_ joined #evergreen
10:17 jeff That's what, 3.2 records per second? That seems exceptionally slow. Any ideas/suspects, or are you just noticing and looking into it?
10:22 jeff 5.9 kb average record size seems reasonable, especially with holdings data added. Our non-deleted bibs (excluding a bunch of deleted stubby ILL bibs) are about 4.5 kb on average.
10:23 jeff anyway, if you had something looping or some kind of unintentionally additive loop/array, i'd expect you'd have a lot more than 7.7 GB of output.
10:26 Dyrcona jeff: I'm not really looking into it. I'm testing some patches on a test system, and also doing a test run for a big Aspen export. I thought I'd just run marc_export like this and time it: /usr/bin/perl /openils/bin/marc_export --all --items --exclude-hidden --852b circ_lib --format XML
10:27 Dyrcona `ps -o etime 24586` said 4-19:00:01 just a few seconds ago.
10:28 Dyrcona I'm running it with time, but was curious how long it has been going so far.
10:29 Dyrcona Adding --items seems to really slow it down on this setup.
10:33 Dyrcona I should try it with a binary MARC file to see if that makes a difference. I wonder if writing the output locally is a problem, though I doubt it.
10:36 Dyrcona The db server does not appear to be under any strain.
10:37 Dyrcona Load is 0.08 and plenty of free RAM, which could mean its not cached, but with NVMe, who needs cache? ;)
10:44 sandbergja joined #evergreen
10:44 Dyrcona Makes me wonder if we're missing an index, or if adding a new index might help. It would be nice if there was an easy way to dump the SQL from Perl DBI... Maybe there is. I should check.
10:48 Dyrcona I suppose I could hack a copy of marc_export to dump the SQL instead of executing it.
10:50 briank joined #evergreen
10:50 Dyrcona I'd like to run it through explain. It's probably the queries to grab item information to add to the MARC, so I'll have to dump an example of that, too.
10:52 Dyrcona Guess I will be looking into it later.... *sigh*
10:52 jeff or tweak log_min_duration_statement just long enough to capture some sample queries. depends on how otherwise loaded your db server is, if this is prod.
10:56 Dyrcona This is a test system that hosts multiple databases, but this is the only instance currently doing anything.
10:56 jeff ah, that makes many things easier.
10:56 Dyrcona I wonder if the number of page faults in marc_export matters....
10:57 jeff if you're trying to hold that whole resultset in memory... I haven't looked at how marc_export does paging, if at all.
11:01 Dyrcona jeff: It uses DBI->execute and loops over the result set, then it runs additional queries if necessary. And, yeah, its RSS is about 8GB, or 25.7% of RAM according to ps.
11:03 Dyrcona It's using 98.2% CPU, which nothing because the VM is configured with 16. The hardware has 32 with HTT enabled.
11:03 Dyrcona It's not running on the same server as the DB either.
11:06 Dyrcona I'll have to do some investigation to see where the problem lies. Maybe I can get some improvements for everyone out of this.
11:20 Dyrcona jeff++ # I may just crank the logging up for a test run later. I suspect this one will finish sometime later today, but I also thought that it would have done by yesterday to start with.
11:24 Dyrcona FWIW, I'm dumping XML because it's "easier" to work with than binary MARC, but when a file is about 8GB in size, the format doesn't really matter any longer, does it? :)
11:25 collum joined #evergreen
11:33 kmlussier joined #evergreen
11:39 jihpringle joined #evergreen
13:00 Dyrcona Y'know. I think I'll also do a similar extract with the --items option and time that. I also want to check what sandbergja reported on Lp 2015484.
13:00 pinesol Launchpad bug 2015484 in Evergreen "marc_export: provide way to exclude OPAC-suppressed items" [Wishlist,Confirmed] https://launchpad.net/bugs/2015484 - Assigned to Jason Stephenson (jstephenson)
13:41 Dyrcona jeff: I think some of the patches that I am testing are responsible for the slow down, particularly the one for the above Lp bug.
13:45 Dyrcona I think I'll revert a couple of commits before I say much more.
14:21 Dyrcona Hmm... Looks like I have somewhere in the vicinity of 400,000 records left to export. I think I'll stop this one and try again with the suspected commits reverted.
14:25 Dyrcona Think I'll export to a binary MARC file this time. At least the file will be smaller.
14:43 mdriscoll joined #evergreen
14:50 shulabear joined #evergreen
14:50 Stompro joined #evergreen
14:51 terranm joined #evergreen
14:51 Stompro I survived our upgrade from 3.3 -> 3.11, I should really upgrade more often.
14:51 collum joined #evergreen
14:52 Dyrcona Stompro++
14:52 jeffdavis Stompro++ # that's a big leap, glad you survived!
14:52 Dyrcona We always skip a few versions, but never that many.
14:53 shulabear Stompro++
14:55 terranm Stompro++ yes, please, for your sanity, upgrade more often!
14:55 Stompro Trying to find all the instances of SMS Carrier needing to be set.
14:55 Dyrcona jeff: I reverted a couple of commits and the export seems to be gong faster. I'm inclined to blame a particular commit...
14:56 jeff what do you think was slowing you down? the accessors on the fm objects for status/location, or something else?
14:56 Dyrcona I also would not be surprised if writing binary is faster than writing XML.
14:56 jeffdavis Hey so I guess we've got a dev meeting starting in a few minutes?
14:57 jeff jeffdavis: indeed!
14:57 Dyrcona jeff: I strongly suspect this commit: https://git.evergreen-ils.org/?p=wor​king/Evergreen.git;a=commit;h=68b925​26949454ad08142e48656b911e0dc1c9a6
15:00 shulabear joined #evergreen
15:02 Bmagic #startmeeting 2023-10-10 - Developer Meeting
15:02 pinesol Meeting started Tue Oct 10 15:02:10 2023 US/Eastern.  The chair is Bmagic. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02 pinesol Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:02 pinesol The meeting name has been set to '2023_10_10___developer_meeting'
15:02 Bmagic #info Agenda at https://wiki.evergreen-ils.org/do​ku.php?id=dev:meetings:2023-10-10
15:02 Bmagic #topic Introductions
15:02 Bmagic #info Bmagic = Blake GH, MOBIUS
15:02 terranm #info terranm = Terran McCanna, PINES
15:02 collum #info collum = Garry Collum, KCPL
15:02 jeff #info jeff = Jeff Godin, Traverse Area District Library (TADL)
15:02 dluch #info dluch = Debbie Luchenbill, MOBIUS
15:03 mmorgan #info mmorgan = Michele Morgan, NOBLE
15:03 shulabear #info shulabear = Shula Link, GCHRL
15:03 Stompro #info Stompro = Josh Stompro, LARL and NWRL
15:03 berick #info berick Bill Erickson, KCLS
15:03 sleary joined #evergreen
15:03 abneiman #info abneiman = Andrea Buntz Neiman, Equinox
15:03 eeevil #info eeevil = Mike Rylander, EOLI
15:04 Dyrcona #info Dyrcona = Jason Stephenson, CW MARS
15:04 sleary #info sleary = Stephanie Leary, EOLI
15:04 Bmagic feel free to introduce yourselves as you arrive
15:04 Bmagic #topic Action Items from Last Meeting
15:04 Bmagic #info jeff will make tutorial: "Add missing field to print template"
15:04 Bmagic #info jeff will write tutorial "Retrieve a user's setting and do something based on its value"
15:04 jeff both tutorials remain on my todo list, and I expect that I'll be able to spend time on then by the next dev meeting.
15:05 Bmagic jeff++
15:05 Bmagic #info mmorgan will explore moving LP stats to community site and automating same
15:05 mmorgan Still exploring capabilities of the Launchpad api for this. That's all I have to report right now. I did add stats to the agenda for this meeting.
15:05 Bmagic mmorgan++
15:06 Bmagic #info sandbergja will write tutorial: "Do a database call (Galen’s cat counter)"
15:06 sandbergja Partially done!
15:07 sleary @sandbergja asked me to take a look at the draft, and I have not had a chance. It is on my agenda!
15:07 pinesol sleary: git diff origin/hamster
15:07 dluch jeff+ mmorgan++ sandbergja++
15:07 Bmagic sandbergja++
15:07 Bmagic sleary++
15:07 Bmagic #info berick will post something to the email list about merging OpenSRF into Evergreen
15:08 berick done
15:08 shulabear Jeff++ mmorgan++ sandbergja++
15:08 Bmagic berick++
15:08 dluch berick++
15:08 dluch sleary++
15:09 Bmagic carrying the action forward, incoming
15:09 Bmagic #action jeff will make tutorial: "Add missing field to print template"
15:09 Bmagic #action jeff will write tutorial "Retrieve a user's setting and do something based on its value"
15:09 Bmagic #action mmorgan will explore moving LP stats to community site and automating same
15:09 Bmagic #action sandbergja will write tutorial: "Do a database call (Galen’s cat counter)"
15:10 Bmagic #topic OpenSRF
15:10 Bmagic any last minute things on OpenSRF that didn't make it onto the agenda?
15:11 Bmagic #topic Evergreen -  3.12 Release Schedule & Roadmap
15:11 Bmagic #link https://wiki.evergreen-ils.org/doku.php?i​d=faqs:evergreen_roadmap#release_schedule
15:11 Bmagic #link https://wiki.evergreen-ils.org/doku​.php?id=faqs:evergreen_roadmap:3.12
15:12 abneiman the bugs-with-signoffs link is huge & ugly, sorry
15:12 abneiman anyway, those are the release things, any questions?
15:12 Bmagic #link https://bugs.launchpad.net/evergreen/+bugs?field.​searchtext=&orderby=-importance&field.sta​tus%3Alist=NEW&field.status%3Alist=CONFIRMED&​amp;field.status%3Alist=TRIAGED&field.status%​3Alist=INPROGRESS&field.status%3Alist=INCOMPL​ETE_WITH_RESPONSE&field.status%3Alist=INCOMPL​ETE_WITHOUT_RESPONSE&assignee_option=any&​field.assignee=&field.bug_reporter=&field​.bug_commenter=&field.subscriber=&field.s​tructural_subscriber=&field.tag=pullrequest+
15:12 Bmagic signedoff&field.tags_combinator=ALL&field.has_​cve.used=&field.omit_dupes.used=&field.omit_du​pes=on&field.affects_me.used=&field.has_patch.​used=&field.has_branches.used=&field.has_branc​hes=on&field.has_no_branches.used=&field.has_n​o_branches=on&field.has_blueprints.used=&field​.has_blueprints=on&field.has_no_blueprints.use​d=&field.has_no_blueprints=on&search=Search
15:13 Bmagic eeesh
15:13 abneiman cough
15:13 abneiman toldya :)
15:13 mmorgan gesundheit
15:13 Dyrcona Well, that's Launchpad for ya.
15:13 Bmagic hopefully that makes it through the parse
15:13 dluch Dang, lol
15:13 dluch https://wiki.evergreen-ils.org/do​ku.php?id=dev:meetings:2023-10-10
15:13 dluch oops, ignore that, sorry
15:13 Bmagic abneiman++
15:13 abneiman anyway, things with signoffs! any committers in the room, please feel free to review and commit!
15:14 shulabear Abneiman++
15:14 dluch abneiman++
15:14 sandbergja abneiman++
15:14 abneiman feature slush is the Monday after Hackaway, so coming up quick - get those features in for review :)
15:14 sandbergja I think those monday review meetings have been going well.  Please feel free to stop by if you haven't yet had the chance!  We've been able to move 5 pull requests forward, including getting 2 committed.
15:14 Bmagic #topic Evergreen - Jane is hosting weekly review meetings on Mondays at 2pm ET
15:14 abneiman sandbergja++ for the code review meetings, btw, which I have not yet been able to attend
15:14 Bmagic :)
15:14 berick sandbergja++ # indeed, great idea
15:14 dluch sandbergja++
15:14 mmorgan sandbergja++
15:14 Bmagic sandbergja++
15:14 shulabear Sandbergja++
15:15 collum sandbergja++
15:15 Bmagic #topic Documentation
15:16 Bmagic #info DIG Meeting this Thursday, October 12, at 11am/12pm/1pm/2pm
15:16 Bmagic #link https://wiki.evergreen-ils.org/doku.php?id=​evergreen-docs:dig_meetings:20231012-agenda
15:16 dluch Nothing to add to what I put on the agenda about Thursday's meeting
15:16 Bmagic dluch++
15:16 terranm sandbergja++
15:16 shulabear joined #evergreen
15:17 Bmagic we're going fast, feel free to type something and slow us down
15:18 Bmagic #topic Launchpad Status (as of noon Eastern)
15:18 Bmagic #info Open Bugs - 3072
15:18 Bmagic #info Pullrequests - 94
15:18 Bmagic #info Signedoff - 21
15:18 Bmagic #topic Launchpad Status since last meeting
15:18 Bmagic #info Bugs Added - 55
15:18 Bmagic #info Pullrequest tag Added - 19
15:18 Bmagic #info Signedoff tag Added - 8
15:18 Bmagic #info Fix Committed - 12
15:18 Bmagic #topic New Business - Should we interfile nightwatch tests into the typical eg2/src/app directory (bug 2036831)? - Jane
15:18 pinesol Launchpad bug 2036831 in Evergreen "Move nightwatch tests into the same directory as the code they test" [Wishlist,Confirmed] https://launchpad.net/bugs/2036831
15:18 mmorgan Lots to test, lots to commit :)
15:18 Bmagic #link https://bugs.launchpad.net/evergreen/+bug/2036831
15:19 sandbergja Ooh, This came up in the newdevs group!  Especially from sleary, so please jump in if I get something wrong :-)
15:19 Bmagic mmorgan++ # thanks for doing the stats! it's nice to see them like that
15:20 sandbergja the idea was to reduce some of the maintenance burden of the nightwatch tests by making it clear when a particular UI has some nightwatch tests that exercise it
15:20 sandbergja And one easy way to do that is to just house them in the same directories
15:20 sandbergja like we already do with the unit tests
15:20 sleary ah yes! I am 100% on board with doing more Nightwatch tests, but (as I am also rearranging our markup on a daily basis) I want to make sure it's easy to find the test that corresponds to the file you're looking at
15:20 sandbergja So what do you think?
15:21 Bmagic I've been confused by this before, but I figured that there was a reason for it that predates me
15:21 berick any concerns about moving them?
15:21 berick certainly makes sense
15:22 berick having the .html and .ts files side by side has been a huge help
15:22 berick just as an example
15:22 sleary sandbergja: it's relatively easy to change the command to run all the tests from a directory to a filename pattern, right?
15:22 terranm It will also be easier to see if there are things that should have Nightwatch tests that do not
15:22 Bmagic I'm all about file folder organization, and it makes sense to me to have related things together.
15:23 shulabear +1 to having tests in the same directory as the code being tested
15:23 sandbergja sleary: theoretically, yes, it should be just changing the config file.  I haven't actually tried it though
15:23 sleary sandbergja++
15:24 sandbergja Cool!  I'm not hearing any objections.  Anybody interested in taking this one on?
15:24 Bmagic do we vote maybe?
15:24 sandbergja ooh, vote sounds fun
15:25 Bmagic hah, lets see if I can pull that off
15:26 Bmagic #startvote Shall we move the tests to related places in the Evergreen codebase yes, or no
15:26 pinesol Unable to parse vote topic and options.
15:26 Bmagic lol
15:27 Bmagic #startvote Shall we move the tests to related places in the Evergreen codebase? yes, no
15:27 pinesol Begin voting on: Shall we move the tests to related places in the Evergreen codebase? Valid vote options are yes, no.
15:27 pinesol Vote using '#vote OPTION'. Only your last vote counts.
15:27 Bmagic #vote yes
15:27 shulabear #vote yes
15:27 Dyrcona #vote yes
15:27 sandbergja #vote yes
15:27 collum #vote yes
15:27 mmorgan #vote yes
15:27 Bmagic #showvote
15:27 pinesol yes (6): shulabear, sandbergja, mmorgan, Bmagic, Dyrcona, collum
15:27 Bmagic voting is still open
15:27 Bmagic #endvote
15:27 pinesol Voted on "Shall we move the tests to related places in the Evergreen codebase?" Results are
15:27 pinesol yes (6): shulabear, sandbergja, mmorgan, Bmagic, Dyrcona, collum
15:28 Bmagic that was fun
15:28 berick heh, "the tests" to "the places".   we're still talking about nightwatch yes?
15:28 terranm Sorry, "yes" - heh
15:28 dluch lol
15:28 berick #vote yes
15:28 Bmagic yeah, nightwatch, sorry I didn't specify
15:28 terranm #vote yes
15:29 Bmagic thanks all, moving on
15:29 sleary #vote yes
15:29 Bmagic #topic Announcements
15:29 Bmagic Anyone want to announce anything?
15:29 Dyrcona Bmagic, you have to #endvote
15:29 Bmagic Dyrcona: I did, at least it shows that for me
15:29 dluch yep
15:29 Dyrcona Yeah, I missed it, sorry.
15:30 Dyrcona Someone voted late.
15:30 Bmagic The bot didn't count the three late votes, but the results are clear: yes
15:32 Bmagic looping back to that topic actually, do we need volunteers to do the work?
15:32 mantis1 left #evergreen
15:33 Bmagic Crickets
15:33 abneiman lol
15:33 shulabear I'm not sure what all would be involved, but I'd be willing to learn
15:34 sandbergja shulabear++
15:34 shulabear If it's more than "move files and make a consistent naming scheme"
15:34 Bmagic sandbergja: maybe at your next meeting, we can go over it?
15:34 sandbergja Sure, if you're available then shulabear
15:35 kworstell_isl joined #evergreen
15:35 terranm sandbergja++ shulabear++
15:35 mmorgan shulabear++
15:35 sandbergja if not, I'm happy to be on hand with any questions you have shulabear
15:35 sandbergja ^on hand to answer
15:36 shulabear Assuming my girlfriend's surgical consult that morning doesn't run long, I should be free
15:36 Bmagic #action sandbergja will go over the Nightwatch test reorg with folks at the Monday at 2pm ET meeting or another time as available
15:36 shulabear Put it on my calendar
15:36 shulabear Sandbergja++
15:37 Bmagic sandbergja++
15:37 Bmagic We're technically in the Announcements topic so..
15:37 Bmagic #info Next Meeting is Tuesday, November 14th
15:38 Bmagic Anyone want to add anything before we close?
15:38 mmorgan I didn't put this on the agenda, bug just want to throw bug 2002693 out there for additional eyes.
15:38 pinesol Launchpad bug 2002693 in Evergreen "Logging out on a page with a pcrud call floods browser with errors" [High,Confirmed] https://launchpad.net/bugs/2002693
15:39 mmorgan I'll be adding comments soon and would appreciate anyone's attention.
15:39 dluch shulabear++ sandbergja++
15:39 sandbergja mmorgan++
15:39 shulabear Mmorgan++
15:39 dluch mmorgan++
15:39 Bmagic #info Hack-a-way is coming up!
15:39 Bmagic #link https://wiki.evergreen-ils.org/doku​.php?id=hack-a-way:hack-a-way-2023
15:39 sandbergja it occured to me that we should probably merge bug 2035287 before moving the nightwatch tests.  Make them work first before messing with them hahaha
15:39 pinesol Launchpad bug 2035287 in Evergreen "e2e tests are failing" [Undecided,New] https://launchpad.net/bugs/2035287
15:39 Bmagic mmorgan++
15:41 jihpringle joined #evergreen
15:41 Bmagic thanks sandbergja, that looks like a good one to take care of
15:42 Bmagic alrighty, see some of you in a couple weeks!
15:42 Bmagic #endmeeting
15:42 pinesol Meeting ended Tue Oct 10 15:42:21 2023 US/Eastern.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
15:42 pinesol Minutes:        http://evergreen-ils.org/meetings/evergr​een/2023/evergreen.2023-10-10-15.02.html
15:42 pinesol Minutes (text): http://evergreen-ils.org/meetings/evergr​een/2023/evergreen.2023-10-10-15.02.txt
15:42 pinesol Log:            http://evergreen-ils.org/meetings/evergree​n/2023/evergreen.2023-10-10-15.02.log.html
15:42 mmorgan Bmagic++
15:42 collum Bmagic++
15:42 abneiman Bmagic++
15:42 shulabear Bmagic++
15:43 Dyrcona Bmagic++
15:43 sandbergja Bmagic++
15:43 Bmagic Thank you, thank you. I'd like to thank my rubber ducky and my lovely parents!
15:43 dluch Bmagic++
15:43 terranm Bmagic++
16:13 mmorgan As promised at the dev meeting, comment #9 has been added to bug 2002693
16:13 pinesol Launchpad bug 2002693 in Evergreen "Logging out on a page with a pcrud call floods browser with errors" [High,Confirmed] https://launchpad.net/bugs/2002693
16:16 pinesol News from commits: LP#2004587: add option to marc_export to tune 852$b output <https://git.evergreen-ils.org/?p=E​vergreen.git;a=commitdiff;h=8e53c6​2c548e82d395ce8505173f4a7bd5cc1549>
16:20 kmlussier joined #evergreen
16:29 Bmagic mmorgan++
17:00 mmorgan left #evergreen
17:13 sleary_ joined #evergreen
17:27 sleary joined #evergreen
19:46 kworstell-isl joined #evergreen
20:30 kworstell-isl joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat