IRC log for #evergreen, 2018-11-08

All times shown according to the server's local time.

Time	Nick	Message
00:16		sandbergja joined #evergreen
04:23		yar joined #evergreen
04:23		agoben joined #evergreen
07:01		annagoben joined #evergreen
07:07		rjackson_isl joined #evergreen
07:28		Dyrcona joined #evergreen
07:28	Dyrcona	hack-a-way++
07:32		bdljohn joined #evergreen
08:18		bos20k joined #evergreen
08:36		collum joined #evergreen
08:38		mmorgan joined #evergreen
08:47		jvwoolf joined #evergreen
09:02		bdljohn joined #evergreen
09:02		agoben joined #evergreen
09:03		gmcharlt joined #evergreen
09:03		cesardv joined #evergreen
09:09		kmlussier joined #evergreen
09:14		aabbee joined #evergreen
09:27		jvwoolf1 joined #evergreen
09:34		yboston joined #evergreen
09:48	berick	hackaway++
09:51	kmlussier	hackaway++ contributors++
10:18		plux joined #evergreen
10:34		khuckins joined #evergreen
11:15		sandbergja joined #evergreen
11:22		yboston joined #evergreen
11:27	Bmagic	with a database containing millions rows in metabib.keyword_field_entry - I would think the value for default_statistics_target would need to be largish? What's a good sane value, 200? Should I set it just for the table, or how big of a deal would it be to set it globally?
11:28	csharp	Bmagic: miker advised doing it per-table when we had a similar question a couple of years ago
11:29	csharp	most of the time the PG query planner does the Right Thing™
11:29	Bmagic	ah! sweet. haha "tm"
11:29	csharp	:-D
11:30	Bmagic	keyword searches are slow (as usual) - it seems that is the issue for everyone? At least that was the take-away I got from Kathy a couple of years ago
11:31	csharp	they can be slow here - if you can get the core query from the logs of a slow query and do explain analyze, that should help pinpoint the bottleneck
11:31	Bmagic	I was asking myself if it could be as simple as increasing the statistics target for the keyword_field_entry. I set it to 1000 and then vacuum analyzed, didn'tsee much improvement though
11:33	csharp	we've seen the same thing, where because of poor stats collection from the planner, it does something like a seq scan on a huge table
11:33	csharp	and it hasn't changed for us when doing the stats target
11:33	csharp	not sure how to force PG into doing it correctly
11:34	Bmagic	here is an example https://explain.depesz.com/s/klxN
11:34	Bmagic	I know it's like beating a dead horse
11:35	Bmagic	in that example, I think* the slowness was due to the sort
11:36	kmlussier	Bmagic: A lot of work has been on search since that time. The bib visibility changes that went into 3.0 should have helped address some speed issues. But I still find broad searches to be slow.
11:37	Bmagic	that's what I thought as well :)
11:37	* kmlussier	should revisit those search timings now that all of those sites are on 3.0 or later.
11:37	Bmagic	csharp: which version are you running in production?
11:37	kmlussier	In my spare time.
11:38	* jeff	laughs
11:39	csharp	Bmagic: 9.5
11:40	csharp	EG 3.0.2 + selected backports
11:40	csharp	(including visibility stuff)
11:40	Bmagic	ok, then we are comparing apples to apples. 3.0.9 on missourievergreen.org
11:40	csharp	yeah
11:40	Bmagic	your keyword search was fast and mine was slow
11:40	Bmagic	http://gapines.org/eg/opac/results?query=whale&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0&sort=
11:40	Bmagic	vs
11:40	Bmagic	http://missourievergreen.org/eg/opac/results?query=whale&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0&sort=
11:41	csharp	yours was fast for me (probably because of cache)
11:41	Bmagic	yeah, cache
11:41	Bmagic	change the word to monkey or another animal
11:42	* Bmagic	figures the animals in keywords are roughly equally represented lol
11:42	Dyrcona	Funny you should mention that.
11:43	csharp	getting the core_query from the logs...
11:43	Dyrcona	I have noticed what appears to be a slow down in our electronic resource URL mechanism since we upgraded to 3.0 back in May, though I've only done two "big" loads since then.
11:43	Dyrcona	My current load seems to take 18 hours to process approximately 1,000 MARC records.
11:44	Dyrcona	I don't recall it being that slow when we were on 2.12, but the Evergreen version may be a red herring and it is likely to be bad statistics.
11:44	Bmagic	the code I wrote is very slow. It's due to the fuzzy matching query it uses to attempt to match the incoming bib. It's not strictly matching fields
11:45	Dyrcona	I will share my code in a minute. There are two database queries used to search for matching records. It took me a long time to get the performance acceptable on 2.10/2.12.
11:46	Bmagic	It needed to branch out of the 035/001 matching before, but maybe now that the database is being maintained by a 3rd party...
11:47	Dyrcona	I'll share it with dummy place holders for the dbi connection info.
11:49	Dyrcona	My search queries begin at lines 216 and 250, respectively: https://pastebin.com/g4RGDJLr
11:49	Bmagic	with a terabyte of memory, I have max_connections = 600, shared_buffers = 8GB, default_statistics_target = 50, effective_cache_size = 520GB, work_mem = 224MB (at least this is where it's at now)
11:50	Dyrcona	Do you run anything else on that machine?
11:50	Bmagic	nope
11:50	Dyrcona	effective cache size could be much higher.
11:51	Dyrcona	I haven't tried explain analyze on samples of those queries lately, but I plan on doing so soonish.
11:51	Bmagic	I was reading about that... Recommend half the amount of memory on the machine. But maybe 75%... it's not clear
11:51	csharp	explain analyze on the "whale" query: https://pastebin.com/q8Ssbbs3
11:51	Dyrcona	Recommended amount is RAM on the server - O/S overhead, which is closer to 90% :)
11:52	Bmagic	csharp: I guess I need to fish out mine.. just a minute
11:52	Dyrcona	There's a lot of different recommendations, though, and different reasoning behind them. I often find one source contradicts another.
11:52	csharp	I'm happy to share our pg conf files too
11:52	Dyrcona	Yeah, I was just going to check what our effective cache size is.
11:53	csharp	ours is 352GB
11:53	csharp	(512GB RAM on box)
11:53	Bmagic	csharp: compared to how much system memory?
11:53	Dyrcona	With 768GB of RAM our effective_cache_size is 750GB.
11:54	Bmagic	I could stand to raise it up quite a bit then. I wonder how much of a difference it will really make
11:54	Bmagic	the database on disk is only 350GB
11:54	csharp	Bmagic: also, make sure the huge pages kernel param is off
11:54	Dyrcona	Our work_mem is 768MB.
11:54	Bmagic	csharp: yep, I got the memo on that from your email a couple of years ago!
11:55	csharp	good
11:55	Bmagic	:)
11:57	Dyrcona	y'know. I may not have set that in the sysctl config file. It could just be that I've rebooted the server recently. :(
11:57	Bmagic	csharp: wait - huge TRANSPARENT pages right?
11:57	Bmagic	there's another "huge" related thing in the kernel which I haven't touched
11:57	csharp	echo never > /sys/kernel/mm/transparent_hugepage/enable
11:57	csharp	and echo never > /sys/kernel/mm/transparent_hugepage/defrag
11:58	csharp	we have those in /etc/rc.local on those servers
11:58	Bmagic	well, defrag is set to always (default) but does that matter if the other settings is "never" ?
11:58	csharp	https://pastebin.com/Dhdq0T3L
11:59	Dyrcona	https://www.postgresql.org/docs/9.4/kernel-resources.html Says something slightly different about hugepages.
11:59		jihpringle joined #evergreen
11:59	csharp	https://community.microfocus.com/borland/managetrack/accurev/w/accurev_knowledge_base/27749/recommendation-to-disable-linux-kernel-transparent-hugepages-thp-setting-for-performance-improvement
12:02	Dyrcona	I'm often skeptical of advice that fails to include version numbers. Things have a tendency to change, but that is about transparent huge pages and the postgresql link is talking about configuring the number of huge pages.
12:03	Bmagic	ansible code for grub boot:
12:03	Bmagic	lineinfile: dest=/etc/default/grub state=present regexp='GRUB_CMDLINE_LINUX_DEFAULT' line='GRUB_CMDLINE_LINUX_DEFAULT="transparent_hugepage=never"'
12:03	csharp	that wasn't what we depended on, but it is consistent with the advice we were given a couple of years ago
12:06	Dyrcona	And, after checking I do have transparent_hugepage set to never, so that survived reboot.
12:06	Dyrcona	And, lunch!
12:06	csharp	lunch++
12:06	csharp	@dessert lunch
12:06	* pinesol	grabs some ngCookies for lunch
12:07	Bmagic	csharp: missourievergreen whale query https://explain.depesz.com/s/bNF
12:07	kmlussier	ngCookies?
12:07	kmlussier	I don't think they're very tasty.
12:07	csharp	@dessert kmlussier
12:07	* pinesol	grabs some Coconut Cream Cake for kmlussier
12:07	csharp	better?
12:07	Dyrcona	ngx-Cookie: missing dependency: @nguniversal/express-engine
12:07	kmlussier	Yes, much! Thanks csharp! :)
12:08	Bmagic	and here's yours using the same color coded view: https://explain.depesz.com/s/IvHi
12:10	kmlussier	heh. I don't even remember saying this - http://irc.evergreen-ils.org/evergreen/2017-03-08#i_294038
12:13	mmorgan	kmlussier: Maybe it wasn't you, maybe it was your clone
12:13	kmlussier	mmorgan: If I could clone myself, I would be getting a lot more work done right now.
13:34		tsadok joined #evergreen
13:57		mmorgan1 joined #evergreen
14:29		Dyrcona joined #evergreen
15:02		kmlussier joined #evergreen
15:15		jonadab joined #evergreen
15:42		BAMkubasa joined #evergreen
15:46	BAMkubasa	So, in the OPAC you can search for a title by ISBN with a URL like: /eg/opac/results?qtype=identifier%7Cisbn&query=0062374230
15:46	BAMkubasa	Is there a way to search by UPC?
15:51	jeff	this should work: /eg/opac/results?qtype=identifier%7Cupc&query=733961134339
15:54	BAMkubasa	That did it!
16:12		yboston joined #evergreen
16:19	csharp	Dyrcona: around?
16:20	Dyrcona	Yeah. I was just looking at IRC to see if anything new came up.
16:20	csharp	cool - looking at https://pastebin.com/feAsnERF and using it as a basis for my forgive payment script...
16:20	Dyrcona	OK. Questions?
16:21	csharp	I'm using DBI's fetchall_hashref to pull in my data, but when I hit the if (ref($xacts) eq 'ARRAY') condition, it pushes me into the "else" block - is there a reason you check to see if it's an ARRAY?
16:22	csharp	I guess the way you're getting data creates an array of hashes?
16:22	Dyrcona	Let me look, again.
16:22	csharp	k - thanks!
16:23	csharp	array/hash refs are where my perl-fu is pretty sketchy :-)
16:23	csharp	oh, I use $xacts where you used $bills, but potayto/potahto
16:23	Dyrcona	Yeah, open-ils.actor.user.transactions.history.have_balance returns an array ref of open transactions.
16:24	Dyrcona	Or possibly an event or nothing, I'd have to look at the back end code, but looking at my code that's what I'd expect.
16:26	Dyrcona	So, using DBI, you could skip that check and just stuff your xact.id and xact.balance_owed into the array of arrayrefs.
16:26	Dyrcona	A loop on handle->fetchrow_hashref would do it.
16:26	csharp	rock on - thanks for the guidance
16:27	csharp	Dyrcona++
16:27	Dyrcona	I don't usually mix DBI with OpenSRF, but no reason that you can't.
16:27	csharp	yeah, I'm under time pressure or I would learn to use cstore editor too :-)
16:28	Dyrcona	Yeah. Understood. You have criteria to search for the bills to forgive, whereas mine just takes a list of patron barcodes via standard input.
16:29	csharp	right - slightly different approaches
16:30	Dyrcona	I forget exactly why I did it that way. I probably was told, forgive bills for this list of patrons.
17:05		mmorgan left #evergreen
17:12	* csharp	goes insane: https://pastebin.com/jKPAwSpg
17:12	csharp	so I'm trying to get the xact_id and balance_owed fields into an array, but I'm getting this:
17:13	csharp	Can't locate object method "xact_id" via package "145183530" (perhaps you forgot to load "145183530"?) at ./batch_forgive_fines_csharp.pl line 74.
17:13	csharp	"145183530" is one of the bill ids
17:13	csharp	I just don't understand conceptually what's going on
17:39		Dyrcona joined #evergreen
19:25	jeffdavis	csharp: the line numbers don't match up but can you dump the contents of $xacts before the foreach loop?
19:30	jeffdavis	It looks like you are calling the xact_id function on each element in the $xacts hashref but it doesn't look like the elements of $xacts would be objects with a defined xact_id function. You may need something like `$xact->{xact_id}` rather than plain `$xact->xact_id`in your foreach loop but I'm not sure on the exact contents/data structure of $xacts
19:54	csharp	jeffdavis: thanks! I was able to avoid hash refs entirely and just use strings/scalars ($sth->fetch rather than $sth->fetchrow_hashref, etc.) and I have a working script
19:54	jeffdavis	excellent!
19:54	csharp	I'll learn the Right Way™ another night when I'm not under the gun :-)
20:13		lsach joined #evergreen