Evergreen ILS Website

IRC log for #evergreen, 2019-03-07

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
02:33 beanjammin joined #evergreen
05:00 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
07:06 agoben joined #evergreen
07:07 rjackson_isl joined #evergreen
07:35 bos20k joined #evergreen
07:47 bdljohn joined #evergreen
08:19 littlet joined #evergreen
08:38 mmorgan joined #evergreen
08:39 Dyrcona joined #evergreen
08:45 collum joined #evergreen
09:11 aabbee joined #evergreen
09:15 bdljohn1 joined #evergreen
09:16 bdljohn joined #evergreen
09:16 afterl joined #evergreen
09:21 terran joined #evergreen
09:26 derekz joined #evergreen
09:35 sandbergja joined #evergreen
09:46 yboston joined #evergreen
10:13 Christineb joined #evergreen
10:19 pinesol [evergreen|Jane Sandberg] Docs: Fixing broken link - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=7f7919e>
10:20 beanjammin joined #evergreen
10:42 jvwoolf joined #evergreen
10:43 terran mmorgan++ for finding the link to my patch when I forgot to include it :D
10:44 mmorgan Now if I could only find that thing I know is somewhere on my desk ... :)
10:49 terran :)
10:56 bdljohn1 joined #evergreen
11:14 sandbergja joined #evergreen
11:32 khuckins joined #evergreen
11:46 Bmagic I've noticed a search slowness after upgrading from 3.0.9 to 3.1.6. And no different with 3.1.10. Same data, same search, apples to apples. Anyone else noticed? I've a bigger database server just installed and synced and tuned with huge pages, etc. side by side
11:47 Bmagic and 3.0.9 wins the keyword search battle every time
11:48 Bmagic gapines's opac returns slower than 3.0.9 as well. Here is the apples to apples. (you will need to change the search critera to avoid cache)
11:48 Bmagic http://gapines.org/eg/opac/results?quer​y=dog&amp;qtype=keyword&amp;fi%3Asearch​_format=&amp;locg=1&amp;detail_record_v​iew=0&amp;_adv=1&amp;page=0&amp;sort=
11:48 Bmagic vs
11:48 Bmagic https://missourievergreen.org/eg/opac/results?qu​ery=dog&amp;qtype=keyword&amp;fi%3Asearch_format​=&amp;locg=1&amp;detail_record_view=1&amp;sort=
11:49 Bmagic 26 seconds for gapines, 14 seconds on missourievergreen
11:51 Bmagic It's not quite apples to apples of course, due to the db's not having the exact same data. The test machine with the exact same data is here http://104.155.152.239/eg/opac/home
11:53 csharp Bmagic: maybe a comparison of queries and explain data would help?
11:53 Bmagic yeah, good point
11:54 Bmagic I've been here before, and the two versions of evergreen are fundamentally different in the search strategy. No conclusion
11:54 Bmagic I'll get the queries, just a minute
11:58 beanjammin joined #evergreen
12:05 Bmagic 3.1.10 https://explain.depesz.com/s/24P5  3.0.9 https://explain.depesz.com/s/3Js9
12:05 Bmagic that was "cats"
12:06 pastebot "Bmagic" at 64.57.241.14 pasted "The two queries" (165 lines) at http://paste.evergreen-ils.org/10047
12:09 Bmagic one thing that stands out: the 3.1.10 analyze uses a merge join and the 3.0.9 does not
12:12 csharp Time: 13861.164 ms - 3.1 query directly on our server
12:13 csharp Time: 6673.575 ms - 3.0 query directly on our server
12:13 csharp so yeah, 2x slow
12:13 Bmagic Your results support my case
12:14 csharp we're on 3.2, but that may be the same as 3.1 bib-search-wise
12:16 Bmagic something is fishy here. I've been spending time on it off and on but I just don't seem to find the answer. I was chasing down postgres tuning for awhile. Messing with default_statistics_target globally and then on specific tables/columns
12:18 Bmagic We are upgrading to 3.1.10 tomorrow. Upgrading the DB server in the process (more ram). Slower searches are going to make people wonder why we upgraded. We can't stay on 3.0.x. Lots of great benefits in the staff client for sure
12:18 Dyrcona Bmagic: Search is more complicated in 3.1+ than it was in 3.0, and 3.0 was slightly more complicated than it was in 2.12. I noticed an increase in search times after upgrading from 2.12 to 3.0, and some SQL-based utilities I had written also seemed to take longer on 3.0 than on 2.12.
12:20 afterl joined #evergreen
12:21 jvwoolf joined #evergreen
12:21 miker larges contributor: more rows are filtered out more quickly via pushing down the source map join farther. could be join collapse limit difference (or other tuning) or just planner changes. (note: the 3.1 version of the inner-most index scan is 2x faster than the 3.0 version.) looks like the primary cause is virtual index defs. there are patches (security bug) that incidentally, but not directly, will likely change things such that plans will look more
12:21 miker like the pre-virt-defs ones
12:21 miker largest*
12:21 miker first sentence is "why 3.0 is faster"
12:24 Dyrcona Bmagic: Have you altered the join collapse limit in your PostgreSQL configuration?
12:24 Bmagic probably not since that doesn't ring a bell.... let me see if I can find what config your talking about
12:25 Dyrcona join_collapse_limit our is set to 9. Default is 8, IIRC.
12:25 Bmagic it's commented out
12:25 Bmagic I'll try 9 and see what happens
12:25 Dyrcona From what miker said, it might help.
12:26 miker Bmagic: probably not much, there are way more than 9 joins ;)
12:27 Bmagic so moar?
12:27 Bmagic like what if it's 100
12:28 Bmagic may not be related but, I've got huge pages set vm.nr_hugepages = 4285  (after doing some division)
12:29 Dyrcona It will take longer to do planning, the higher you set it.
12:29 Dyrcona I also thought that I read somewhere that the progression was not linear, but the actual documentation doesn't explain it very well, just that setting it to 1 causes the planner not to reorder explicit joins.
12:30 Dyrcona Bmagic: What you set hugepages to varies based on the amount of RAM on your server and what Pg needs.
12:30 Bmagic hey! collapsable joins set to 20 made an improvment. from 44seconds to 24
12:31 Bmagic by that rational, 40 would make it even better, lol
12:32 Bmagic still 24 seconds
12:35 Bmagic hahaha, commented it back out and it's 24 seconds.... so, red herring I guess. I am restarting postgres in between edits
12:38 Bmagic I was excited to get 44 seconds down to 24, but it's still not as good as 3.0.9 at 14 seconds
12:38 Dyrcona Results may be cached in RAM, even after restarting Postgres.
12:39 Bmagic really? wow, that is surprising
12:39 Dyrcona The kernel will cache recently read bits of files, so it's possibly that.
12:41 Bmagic ok, rebooting the machine between edits
12:42 csharp our join_collapse_limit is set to 12 fwiw
12:42 Bmagic interesting - a new setting was not thinking about
12:43 Dyrcona Bmagic csharp You might want to set from_collapse_limit to be the same as join_collapse_limit.
12:43 Bmagic csharp: any other non-defaults?
12:45 Dyrcona Bmagic: You've seen our production config for Pg 9.5 and Evergreen 3.0.
12:46 * Dyrcona is contemplating upgrading to Pg 10 soonish.....
12:48 csharp Bmagic: sent via email
12:48 Bmagic ty
12:48 csharp if it weren't for the move from QTS -> ITS, we would have already moved to Ubuntu 18.04/PG 10
12:49 csharp probably do that over Labor Day
12:50 Dyrcona Someone (me?) should start testing with Pg 11. We only started looking at Pg 10 when 11 was released. :)
12:55 jvwoolf joined #evergreen
13:06 Bmagic fresh reboot, same cats query 190 seconds!
13:06 Bmagic collapsable join was at 10
13:16 jeffdavis Bmagic: you'd need to warm up the cache after a reboot...
13:18 Bmagic yeah, but that was the point of rebooting
13:18 Bmagic restarting postgres wasn't enough
13:18 Bmagic what's the command to warm up the cache after reboot?
13:19 jeffdavis You don't want your test query in cache, but you want indexes etc to be cached so that performance is normal (or else test on two different cold-cache servers I guess)
13:21 jeffdavis doing some other searches or typical queries and/or vacuuming metabib index tables might help, I'm not actually sure what's best beyond letting the server get used a bit
13:23 jeffdavis you could ask in #postgres for advice on benchmarking and cache issues
13:25 Bmagic yeah, it's hard to get a good test after changing those config points
13:25 Bmagic to know whether or not the change made a difference
13:26 sandbergja_ joined #evergreen
13:35 Dyrcona Bmagic: Find your database's oid: select oid from pg_database where datname = 'evergreen'; -- Assuming your database is named evergreen and you run that query in the postgres database.
13:37 Dyrcona Then as root or the postgres user: cd /var/lib/postgresql/${pgversion}/main/base/${oid} # Replacing the ${pgversion} with your postgres version, ex. 9.5, and ${oid} with the value of the oid from the previous query
13:38 Dyrcona Once you're there you do: cat * > /dev/null
13:38 yboston joined #evergreen
13:39 Dyrcona If you're on 9.5 (maybe 9.4) or later, there is a SQL function you can run to preload tables, but you have to run it on each table you want preloaded. The above should load your whole database into RAM.
13:40 Dyrcona Note that within a day or two, most of you database will be evicted from RAM by the kernel for not having been used.
13:41 Bmagic Dyrcona++ # nice
13:43 csharp ymigh
13:44 csharp Bmagic: you might look into pg_prewarm
13:44 csharp https://www.postgresql.org/docs/9.5/pgprewarm.html
13:44 Bmagic Tons of gems. This conversation has bared some fruit
13:44 Dyrcona Bared or borne? ;)
13:44 Bmagic sure
13:45 Bmagic English is my 10th language
13:45 Dyrcona pg_prewarm is what I was referring to when I said "there is a SQL function...."
13:46 Dyrcona gmcharlt++ # For showing me how to do the steps that I shared.
13:46 littlet joined #evergreen
13:48 * Dyrcona suspects that English, itself, is 10 languages put together to make what looks like one language.
13:49 Bmagic yeah, totally!
13:49 Bmagic I'm working with my 1st grader. Learning to read English is tough. So many things do not have logic unless you learned Latin first, and french and spainsh
13:50 Bmagic "It's just the way it is, you'll have to just remember that's how it's spelled"
13:51 Bmagic lol spainish/spanish
13:58 Dyrcona Well, German helps, too. :)
13:58 Dyrcona BTW, I added this: https://wiki.evergreen-ils.org/doku​.php?id=scratchpad:random_magic_spe​lls#preload_your_database_into_ram
13:58 miker apropos of cache warming: https://www.postgresql.org/docs/9.6/pgprewarm.html
13:59 Dyrcona :)
14:01 sandbergja Are there any DIG folks around?  Anybody up for an informal DIG meeting?
14:01 abneiman sandbergja: I'm around
14:04 sandbergja abneiman: looks like a pretty small turnout, hahaha
14:04 * abneiman shrugs
14:04 abneiman just the cool kids right?! ;-)
14:04 sandbergja absolutely!
14:05 littlet I'm here, but I'm normally just a lurker :)
14:06 sandbergja littlet: hi!
14:06 sandbergja abneiman: littlet: I sent ohiojoe an email just now to check in about his DIG plans, so I'll keep you posted on what he has to say
14:07 abneiman ok cool
14:08 littlet Sounds good
14:08 sandbergja Launchpad says that there are 496 bugs with the webstaffclient tag
14:08 sandbergja Do we get to have a party when we hit 500?
14:08 derekz I'm here for the DIG meeting. First meeting for me
14:09 sandbergja derekz: welcome!  It looks like we might wait until next month to meet.
14:09 abneiman sandbergja: any excuse for an EG party!  In all seriousness though, I've spend BSW doing some tag cleanup and I found myself wondering at what point that tag has diminishing-usefulness-returns
14:09 littlet sandbergja Hah! :D  I've been going through the webstaffclient bugs with no additional tags and adding extra tags
14:10 abneiman derekz++ welcome!
14:10 littlet Yes, agree abneiman. Now that we're only web client going forward, seems less useful
14:11 abneiman littlet: OTOH, it does serve to distinguish them from older XUL-client bugs (though I know you and I and some others have been weilding the wontfix hammer pretty hard this week(
14:11 littlet It gives me a little thrill each time I mark a XUL-only bug as Won't Fix :D
14:11 terran abneiman++ littlet++ for clearing out those old bugs!
14:12 abneiman littlet++ # same kind of nerd as me hahahaha
14:12 littlet lol
14:12 abneiman terran++ # BSW wrangler
14:12 littlet Yes! terran++
14:13 sandbergja It seems like the webstaffclient tag can be helpful for folks who are comfortable contributing front-end code to the AngularJS client, and are looking for possible bugs to contribute to
14:13 sandbergja I wonder if we could create a more specific tag that would meet that purpose
14:13 terran webstaffclient does still help differentiate from the OPAC too
14:13 terran Speaking of opac - do we still need separate opac & tpac tags?
14:13 abneiman terran: I was asking myself that yesterday re opac & tpac
14:14 littlet That's true. I just assume only stuff tagged 'opac' deals specifically with OPAC. Yes! Opac/tpac -- pls advise
14:14 abneiman sandbergja: there is an angular tag that only has (as of when I pulled data yesterday) 11 bugs
14:14 abneiman tpac, of course, was historically used to differentiate the once-new Template Toolkit OPAC from its predecesor, the JSPAC
14:14 sandbergja abneiman: oh, I'm glad you brought that one up!  I was assuming that the goal of that one was to separate Angular vs AngularJS client
14:15 sandbergja And also there's an official staffclient tag
14:15 abneiman now the use of tpac seems to serve as a marker of just how long you've been hanging around this place ;-)
14:15 berick yes: re separate angular and angularjs
14:15 abneiman ah, thanks berick for the clarification
14:15 terran Since there aren't (or shouldn't be) any more JSPAC bugs can we agree to just use opac from now on? (If so, I'll volunteer to update the remaining tpac tags)
14:15 littlet Ah, that does help, thanks berick
14:16 abneiman +1 to using opac & converting tpac tags
14:16 littlet +1
14:16 sandbergja +1
14:16 terran Cool beans, I'll do that.
14:16 sandbergja terran++
14:17 littlet terran++
14:17 abneiman I also have been toying with the idea of more carefully documenting how certain tags should be used -- I know we do that for a few special ones like "needstest" "pullrequest" etc. but as berick's remark demonstrated, there's a need to do this for other tags too
14:17 abneiman terran++
14:17 littlet abneiman Yes, please do this! I'm also happy to do that, but that is a great idea
14:17 abneiman I'll see if I can incorporate that -- or some of it -- into my conference LP presentation
14:18 littlet I'd also love that for the different statuses on LP for bug wranglers. Like, when is Invalid vs. Won't Fix appropriate
14:18 littlet There's a little tooltip, but sometimes it's kinda squidgy
14:18 abneiman littlet: that exists on the wiki... somewhere... sec
14:18 terran In the same vein, like Jane said, there's a staffclient tag - should we keep that as an official tag, or go through those and change them to either webstaffclient or perhaps a new official xulstaffclient tag?
14:18 littlet Ah, excellent
14:18 terran abneiman++ for more usage documentation!
14:18 littlet +1 to xulstaffclient. That way it's explicit
14:19 abneiman https://wiki.evergreen-ils.org/d​oku.php?id=dev:bug_wrangler:faq
14:19 sandbergja abneiman++
14:19 littlet Awesome, thanks abneiman!
14:20 littlet abneiman++
14:20 pinesol [evergreen|Adam Bowling] LP #1705497 Replaces functionality in web client from legacy - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=f4a1955>
14:20 StomproJ joined #evergreen
14:20 terran Triaged always confuses me - acknowledged but not confirmed?
14:20 abneiman well, if it's truly a xul client-only issue it should just be marked as won't fix, IMO.  But +1 to reviewing existing staffclient-tagged bugs to see if webstaffclient can be applied
14:20 littlet I've got to scoot out the door, but will read back over whatever y'all talk about later :)
14:21 abneiman terran: yeah that one's a bit weird, it strikes me as something that used to be in more common practice but is now less used
14:23 terran abneiman: true, we don't really need the xulstaffclient tag since we should be marking those Won't Fix. I'll volunteer the ones tagged with staffclient to see if they should be retagged webstaffclient or marked Won't Fix
14:24 sandbergja terran++
14:24 sandbergja also, that will definitely get us to our 500 webstaffclient bug party
14:24 terran lol
14:24 abneiman sandbergja: terran: littlet: may as well share this with y'all https://docs.google.com/spreadsheets/d/1Ge4iCUjaZC​smWEMLF0_TKrLoLTJ7-2Sv0C_A-lKsu9E/edit?usp=sharing
14:25 abneiman I started it yesterday once I realized I was getting a little lost in the tags
14:25 sandbergja nice!
14:25 terran abneiman++ dang you're organized :)
14:25 abneiman terran: I'm a big fan of spreadsheets, LOL
14:26 terran There's not currently an angularjs tag - shall I create one?
14:27 sandbergja terran: I feel like there would be so many bugs that it could be applied to that it might be overwhelming
14:28 abneiman terran: I don't have strong feelings either way about angularjs tag -- berick, thoughts?
14:28 abneiman (or any other lurking devs)
14:28 terran BTW, I removed the memoryleak tag from the official list yesterday because it had 0 bugs
14:28 terran (And we always use memory leak as a punchline around my office)
14:29 abneiman also, for the record, my process has been to mostly leave unofficial tags on bugs (other than ones that are reassigned elsewhere) because I think they serve as a decent barometer of "crowdsourced" new tags
14:29 berick i don't see a need for an angularjs tag.
14:30 littlet_ joined #evergreen
14:31 terran Okay, thanks
14:32 terran I've been removing some of the unofficial tags if there is a better existing term already in place
14:32 abneiman another one I've been wondering about but haven't looked into -- test-writing-day-1
14:32 abneiman https://bugs.launchpad.net/evergreen​/+bugs?field.tag=test-writing-day-1
14:34 abneiman All file by Liam Whalen, 3 years ago, seemingly about things that need tests written.  We typically use needstest to flag new code that needs a unit/pgtap tests, so I'm not sure that applies here, but it seems like a bit of a one-off
14:35 abneiman oh, mmorgan does have one with that tag too, my mistake.  mmorgan please chime in if you have thoughts on this.
14:38 terran I'm unsure about that one as well
14:39 mmorgan abneiman: Looks like Liam added the tag to bug 1331174, I have no strong feelings about that tag.
14:39 terran Was it flagging things that needed end user test instructions added?
14:39 pinesol Launchpad bug 1331174 in Evergreen "Long Overdue processing needs org unit settings separate from Lost Processing" [Wishlist,Confirmed] https://launchpad.net/bugs/1331174
14:39 abneiman http://libmail.georgialibraries.org/piperm​ail/open-ils-dev/2015-November/009998.html
14:41 abneiman and here https://wiki.evergreen-ils.org/d​oku.php?id=dev:test_writing_day
14:41 abneiman I do not think this is something still ongoing but I've been pretty conservative generally about removing tags
14:42 abneiman mmorgan++ # thanks for weighing in
14:42 pinesol [evergreen|Jane Sandberg] LP1721109: Update item status when holdings are edited - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=2c16628>
14:43 * miker scrolls up... csharp++ (missed you'd pointed out pg_prewarm)
14:44 abneiman anyway, I have to dash in a couple minutes for another meeting -- but this was a very helpful conversation, so thanks sandbergja++ terran++ littlet++ berick++
14:51 terran abneiman++
14:51 terran Bug updaters have surpassed 500 updates! bugsquashers++
14:53 mmorgan Best Bug-Squashing Week Evah!
14:57 terran It really is! There have been a whopping 22 bug patches submitted this week! Amazing!
14:59 csharp @praise librarians
14:59 * pinesol You don't want to get mixed up with someone like librarians. librarians is a loner, Dottie. A rebel.
14:59 terran Heh
15:00 csharp @blame cache
15:00 pinesol csharp: cache was monkeying around too much on the prod servers!
15:10 jvwoolf I have noticed that when building a report in the web client, if I switch back to display fields from base filters or aggregate filters, the fields are in a different order than they were when I added them to the template. Has anybody else had this happen? I don't see a bug ticket.
15:13 csharp jvwoolf: bug 1751800 is similar if not the same thing
15:13 pinesol Launchpad bug 1751800 in Evergreen "web client: reports - column labels spontaneously re-sort when deleting template fields" [Medium,Confirmed] https://launchpad.net/bugs/1751800
15:13 csharp super annoying
15:13 * csharp makes a note to include that in his reports session at the conference
15:20 littlet That one and bug 1778521 make reports more difficult than they sometimes are, they're really irritating
15:20 pinesol Launchpad bug 1778521 in Evergreen "Reports: transform selection is sticky" [Medium,Confirmed] https://launchpad.net/bugs/1778521
15:21 csharp in my workflow, I add the field then the transform, so it doesn't hit me as hard (though I've run into it before)
15:23 littlet I also add the field and then the transform, but then if I move onto another field that I don't need to mess with the transform, I don't normally touch it or think about it. So I may end up getting stuck with the same transform for 5 fields and not notice it
15:27 mmorgan joined #evergreen
15:28 terran The transform is only a minor annoyance for me, but that resorting drives me bonkers.
15:29 littlet The resorting one is definitely the worst one, for sure
15:29 littlet I can (theoretically) pay more attention to the transforms and deal with it, but I can't make the resorting stop. So that's worse
15:30 beanjammin joined #evergreen
15:46 jvwoolf csharp: Thank you, I didn't see that one!
16:02 khuckins joined #evergreen
16:07 littlet If there are any core committers hanging out, there are 39 bugs with the pullrequest and signedoff tags: https://goo.gl/eMPRQ3
16:16 berick littlet++
16:16 berick will take some of that action tomorrow
16:16 Dyrcona littlet++
16:17 Dyrcona Maybe we should coordinate who looks at which bugs?
16:19 littlet Thanks y'all! berick++ Dyrcona++
16:22 yboston joined #evergreen
16:32 * csharp plans to work on committing more tomorrow too
16:56 afterl left #evergreen
17:01 pinesol News from qatests: Failed Running pgTAP tests <http://testing.evergreen-ils.org/~li​ve/test.42.html#2019-03-07T16:59:16,091764708-0500 -0>
17:07 mmorgan left #evergreen
17:13 jamesrf joined #evergreen
17:14 jvwoolf left #evergreen
18:56 Dyrcona Well, I suppose it is a good thing that I deleted a bunch of old bug mails recently.
19:06 beanjammin joined #evergreen
19:13 stephengwills joined #evergreen
20:07 jamesrf joined #evergreen
20:46 sandbergja joined #evergreen
23:36 yboston joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat