IRC log for #evergreen, 2022-03-14

All times shown according to the server's local time.

Time	Nick	Message
00:05		Keith-isl joined #evergreen
00:06		Keith_isl joined #evergreen
06:00	pinesol	News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
07:17		rjackson_isl_hom joined #evergreen
08:07		rfrasur joined #evergreen
08:14		mantis1 joined #evergreen
08:35		Dyrcona joined #evergreen
08:40		mmorgan joined #evergreen
09:11		terranm joined #evergreen
09:11	terranm	Good morning and happy Feedback Fest Week!
09:27	Bmagic	terranm++
09:27	Bmagic	@coffee [someone]
09:27	pinesol	An error has occurred and has been logged. Please contact this bot's administrator for more information.
09:27	Bmagic	@coffee
09:27	pinesol	Bmagic: An error has occurred and has been logged. Please contact this bot's administrator for more information.
09:27	Bmagic	awww darn
09:30	JBoyer	Well, I suppose a log entry is feedback for someone.
09:30	Bmagic	haha, yeah
09:35	mantis1	Feedback Fest starts on Pi Day!
09:37		collum joined #evergreen
09:42	Dyrcona	@tea
09:42	* pinesol	brews and pours a pot of Magic Summer Organic Tea, and sends it sliding down the bar to Dyrcona (http://ratetea.com/tea/organic-herbie/magic-summer-organic-tea/9533/)
09:42	Dyrcona	The coffee database is probably not accessible.
09:44		jvwoolf joined #evergreen
09:47		JBoyer joined #evergreen
10:32	jvwoolf	Good morning. Over the weekend, we tried running action_trigger.purge_events() on our production database for the first time. We ran SELECT action_trigger.purge_events() ; in pqsl rather than using the srfsh script. I tested it on test database with production data and it only took 15 - 20 minutes to run. This weekend, it ran for over 3 hours in production with all application servers disabled and cron jobs turned off. We had to st
10:32	jvwoolf	it because we didn't plan for that much down time. Any ideas why the discrapancy the test runs and the production run?
10:38	miker	jvwoolf: if the test db was a dump/reload of production, it'll likely be because of table bloat and/or bad stats. I'd recommend a vacuum analyze to start
10:39	jvwoolf	miker++
10:59	Dyrcona	jvwoolf: It should be OK to run the purge while Evergreen is running. We run it daily at 5:30 AM. We keep 6 months of events.
11:16	jvwoolf	Dyrcona: We actually did restart it after it failed, but it didn't seem to be doing anything.
11:18	jvwoolf	miker: I was reminded we do a daily vacuum analyze on the full production DB. Do you think doing a full vaccuum on the action trigger event and output tables would help?
11:42	Dyrcona	jvwoolf: A full vacuum on those two tables might help.
11:43	Dyrcona	jvwoolf: If you have a lot of events to purge, the first run will be tricky. After that, a daily or weekly purge should be routine.
11:44	jvwoolf	Dyrcona: We do. We have more events to purge than the bad call numbers. :P
11:44	jvwoolf	Dyrcona++
11:44		jihpringle joined #evergreen
11:45	mmorgan	IIRC, purging the events is quick, it's the associated output that takes time.
12:33	Bmagic	I've got a plain old marc file with a small number of records. I read it into perl like this: $file = MARC::File::USMARC->in($marcFile); then loop: while ( my $marc = $file->next() ) . And I call this: $marc->as_xml(); I'm met with a fat error "no mapping found for [0x39] at position 812"
12:34	Bmagic	So, do I need to pre-read that file and "clean" stuff first? then pass it into the MARC::Record object?
12:36	Dyrcona	Bmagic: Sounds like you have a batch of UTF-8 records that aren't labeled UTF-8. Would these records be from Overdrive, perchance?
12:38	Bmagic	I can't assume any character encoding from any given file. What's the perl to read UTF-8 encoded records from a MARC file?
12:40	Dyrcona	Bmagic: I asked about Overdrive because I've been consistently getting UTF-8 records from them that do not have the LDR 09 set correctly.
12:41	Bmagic	MARC::Record decides which encoding to use based on the leader (like you're saying). Documented on CPAN under "encoding()"
12:42	Dyrcona	Bmagic: Right, but sometimes the records are wrong.
12:42	Bmagic	and what you are saying is, I'm not doing anything wrong, the record is telling MARC::Record that it's USMARC when it's really UTF-8?
12:43	Dyrcona	Possibly. If you check the IRC logs from 2 weeks ago, you'll see my monologue about it. :)
12:43	Dyrcona	It could be an entirely different character set, too, but I got pretty much the same error as you.
12:43	Bmagic	right, we've all had this pain. I've posted here over the years about it. This project is slightly different, and my understanding of the issue has matured over the years
12:44	Bmagic	I've got the file open in MARCEdit. I'm seeing lots of &#xA0 and {acute}
12:45	Dyrcona	So, I'd recommend setting the leader to UTF-8 and seeing what happens. You can do that with $marc->encoding('UTF-8');
12:46	Bmagic	right, let's see
12:48	Bmagic	that did the trick. I would like to try it one way, catch an error, then handle it the other way. Can I "get the message" that MARC::Record bombed? I'm just seeing a screen dump, but my program goes on
12:50	Dyrcona	Bmagic: You're getting a warning, I think from MARC::Charset? You can do some stuff with signal in Perl to make the warnings fatal and then use an eval block to trap them.
12:52	Bmagic	ah! I'll look that up, thanks
12:53	Dyrcona	Here's an example using eval {...}; if ($@) { ... } to log errors: https://pastebin.com/g4RGDJLr
12:53	Bmagic	Dyrcona++
12:55	Dyrcona	Bmagic: To make warnings fatal, do this: $SIG{__WARN__} = sub { die @_; };
12:55	Dyrcona	If you do that in the eval block, it only affects warnings inside that block.
12:55	Bmagic	in that example, you're not converting warnings to fatal. Instead, assigning the warnings to a variable, which if exists, handle it as an error?
12:56	Dyrcona	Not exactly.
12:57	Dyrcona	In that example any fatal errors that happen inside the eval {} get handled by the code in the if ($@) {}.
12:57	Bmagic	Also: I see that you're reading the file raw with a separator "\x1E\x1D". I suppose that's the best way? You find that reading the file yourself (instead of MARC::Batch) is better?
12:57	Dyrcona	eval does two different things depending on how you use it.
12:58	Dyrcona	Bmagic: MARC::Batch usually works. I do tend to read the file manually because I've had issues in the past with records containing "smart" quotes. One of them looks like the the end of record character.
12:58	Bmagic	I'll see if this error (now that I've narrowed it down to a particular record) will cause "die" or not
12:59	Bmagic	your example forces a die when @warnings
12:59		jihpringle joined #evergreen
13:00	Dyrcona	Right. But not all warnings, just warnings from MARC::Record.
13:00	Dyrcona	I'm not sure the MARC::Charset warning shows up there, but you can try it.
13:00	Bmagic	This program I'm writing needs to be pretty hardy. Handling millions of records every year. So, I think I'll go with the manual reading of the files with raw, like what you've got there
13:02	Dyrcona	Suit yourself. MARC::Batch and MARC::File go to great lengths to read in otherwise garbage records, but they blow up sometimes on otherwise well-formed MARC.
13:05	Bmagic	haha, so, it's more* hardy to use MARC::File/Batch.... Maybe I go with MARC::File.... and if it breaks, catch it and go RAW manual. off to find a file that blows MARC::File
13:06	Dyrcona	The problem I'm fixing by reading the records the way that I do comes from copy/paste cataloging.
13:06	Bmagic	yeah, I bet I can copy/paste a smart quote into a record and produce this issue
13:07	Dyrcona	I think it's the closing smart quote contains \x1D. That truncates the current records and totally garbles the next one.
13:07	* Dyrcona	searches CPAN's RT.
13:07	Bmagic	microsoft--
13:09	Dyrcona	Bmagic: https://rt.cpan.org/Public/Bug/Display.html?id=70169
13:10	Bmagic	Dyrcona++
13:12	Dyrcona	I thought tsbere submitted a patch for that one, but it didn't work. Then again, I also think tsbere opened a separate bug. I don't know where that bug has gone.
13:13	Dyrcona	I spent a little time working on tsbere's patch, but kind of gave up.
13:15	Dyrcona	Oh, right. tsbere made a PR on github: https://github.com/perl4lib/marc-perl/pull/4
13:16	Dyrcona	Too many bug trackers....
13:18		JBoyer joined #evergreen
13:18	Bmagic	It would be nice if MARC::File would just handle it
13:19	Bmagic	Though, it sure seems rare. 100% of the records that I'm expecting are from vendors. Not an ILS with custom-pasted cataloged records... so maybe I don't need to worry too much :)
13:20	Bmagic	I'm going to go ahead and code for it though. Who knows what can happen. I expect this code to be running daily for years
13:21	Dyrcona	I've seen it in vendor records.
13:21	Bmagic	right on
13:23	csharp_	oh, almost missed that it's pi day
13:23	Bmagic	!!! csharp_++
13:23	Bmagic	me too
13:24	* csharp_	runs off to measure a buncha circles
13:25	csharp_	@coffee Bmagic
13:25	pinesol	csharp_: An error has occurred and has been logged. Check the logs for more information.
13:25	csharp_	shit
13:25	Bmagic	lol
13:25	Dyrcona	@tea csharp_
13:25	* pinesol	brews and pours a pot of Bangkok, and sends it sliding down the bar to csharp_ (http://ratetea.com/tea/harney/bangkok/2217/)
13:25	Bmagic	"a bunch of circkes" - HAHAHA
13:25	csharp_	trying to fix it - looks like python library changes
13:26	csharp_	in particular BeautifulSoup 3 -> 4
13:28	csharp_	@coffee
13:28	* pinesol	brews and pours a cup of Kenya AA Nyeri Tatu, and sends it sliding down the bar to csharp_
13:28	csharp_	aahhhhh
13:28	Dyrcona	Ugh... One drawback of multiplexed ssh is that agent forwarding doesn't work if you use scp or sftp before using ssh.
13:33	Dyrcona	Mmmmm...Pie!
13:51		terranm joined #evergreen
13:57	mantis1	-Homer drooling noise-
13:58		jvwoolf left #evergreen
14:09		bshum joined #evergreen
14:54	Bmagic	Dyrcona: what I came up with for the two functions https://pastebin.com/Nfzcsu16
15:01	csharp_	mantis1: https://frinkiac.com/video/S05E08/xmsdLylby7Vr6DjJ3eTOB-FpCaQ=.gif
15:01	Bmagic	Dyrcona: sorry, just realized I left out the Raw function, and syntax highlighting: https://pastebin.com/ZWfWbeqC
15:02	Bmagic	csharp_++ # Perrrrfect
15:04	Dyrcona	Bmagic: You could do it that way. I'd probably do the local SIG warn inside the eval and check for $@ rather than the return value from eval.
15:04	Dyrcona	What you've got looks like it would work.
15:05	Bmagic	oh, good idea to put the SIG inside the eval. I suppose that would prevent me from having to set it back?
15:08	Bmagic	https://pastebin.com/QAz6ESPN # with it inside the eval, and removed the second SIG
15:10	Dyrcona	yeah, it should be local to the eval block that way.
15:10	Bmagic	nice! Cleaner
16:52	Bmagic	Is it considered a bug on the grid interface for the holds pull list, where when you choose "25 rows", the server only provides 25 rows total (and doesn't paginate)? I think we are expecting it to give us the whole pull list regardless of how many rows at a time
16:55		rfrasur joined #evergreen
16:57	mmorgan	Bmagic: Which release of evergreen?
16:58	Bmagic	3.7.2
16:58	Bmagic	I just tried it on 3.7.1 and it wasn't an issue, but the two tests aren't exactly identical
17:00	mmorgan	Bmagic: I'm not seeing that issue on 3.7.2. The pull list respects my chosen number of rows and has pagination.
17:00	Bmagic	mmorgan++ # thanks for checking, must be a local issue
17:02	Bmagic	The behavior I'm seeing is: Let's say there are 100 holds. And you choose 10 rows. It will give you 5, no pagination. If you choose 25 rows, it will give you 20, no pagination. It's like it's croaking on a hold somewhere and giving you what it came up with so far
17:08	mmorgan	Could it be one of those holds that got broken? Like a part or metabib hold where it's target went away?
17:09	Bmagic	I was chasing that idea down, but the logs don't have that issue. It's like the DB is only giving the results for the JSON query, just like it's asking for "limit: 5"
17:12	Bmagic	But the DB is not even coming up with 5. It's like 2. But if the limit were higher, like 25, then the DB returns 20 (for example). And I know there are well over 100. If I set it to 100, I get 89 rows. I'm still running different tests. No worries. I'll reply if I get stuck again, maybe with more info
17:25	Bmagic	perhaps this is a problem? https://pastebin.com/PWWN5AbR
17:36		mmorgan left #evergreen
18:00	pinesol	News from qatests: Failed Installing Angular web client <http://testing.evergreen-ils.org/~live//archive/2022-03/2022-03-14_16:00:02/test.29.html>
18:17		jihpringle joined #evergreen
18:48		jihpringle joined #evergreen