Evergreen ILS Website

IRC log for #evergreen, 2018-10-23

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
02:39 jonadab joined #evergreen
06:29 jonadab joined #evergreen
07:00 agoben joined #evergreen
07:08 rjackson_isl joined #evergreen
07:36 bdljohn joined #evergreen
08:19 bos20k joined #evergreen
08:22 Dyrcona joined #evergreen
08:36 collum joined #evergreen
08:40 bos20k joined #evergreen
08:40 mmorgan joined #evergreen
08:58 kmlussier joined #evergreen
09:24 * kmlussier makes a note to replace 'cart' with 'basket' in the 3.2 release notes.
09:27 Bmagic Notes to change notes
09:28 nfburton joined #evergreen
09:28 Bmagic My notes are so big, even my notes have notes!
09:28 Dyrcona heh.
09:32 Dyrcona jeff: Are your self-checks from 3M/Bibliotheca? I'm asking for the sake of honesty on my internal bug write up.
09:36 Dyrcona I suppose I can word it so it doesn't matter. :)
09:38 jvwoolf joined #evergreen
09:44 * Dyrcona may change his bug to Opinion or Invalid later.
09:48 plux joined #evergreen
09:49 pastebot "Bmagic" at 64.57.241.14 pasted "Long running search" (19 lines) at http://paste.evergreen-ils.org/14240
09:49 Bmagic We are seeing super long running search queries on production after upgrading to 3.1. Anyone see anything strange about that search query?
09:51 Bmagic Been running 22 hours. I am considering a cron job to kill long running queries like these
09:51 Dyrcona Bmagic: Is that the whole thing? It looks to me like your paste is truncated, but I'm not an expert on search queries.
09:53 Bmagic I believe it is.... unless I need to do something to psql to let more come out
09:55 Dyrcona It should be the whole query if you didn't truncate the field.
09:56 Bmagic I tried another way and got the same
09:56 Dyrcona It just looks funny to me, but then all search queries look funny to me. :)
09:57 * Dyrcona leaves it for miker or someone else more knowledgeable.
09:57 Bmagic Not all search queries have this issue, just a few and they end up accumulating over time brining the CPU up on the db server
09:58 Dyrcona We sometimes get long running searches, too, but not nearly as many in the past, nor running for as long, but we're on 3.0.
09:58 Bmagic There is no way the patron/staff is still waiting for the answer and I feel it to be safe to kill it, however, it seems plausable that the drone waiting for the answer is hanging on a brick and killing the query might result in drone death, then more bad things
09:58 Dyrcona Yeah, they're safe to kill after a couple of minutes, I would think.
09:59 Dyrcona csharp JBoyer and likely some others probably have cron jobs to do it automatically.
09:59 Bmagic in the early days of 3.0, I had to put in something to kill the money billing queries for the web client... but these queries are standard search (I think)
10:00 Dyrcona I think the drone gives up rather quickly. I don't think it waits forever for the response.
10:00 miker 1) that's not the whole query (pg_stat_activity defaults to, IIRC, 4k characters -- those queries are longer) 2) it's a bad plan because of bad statistics (or other tuning needs) -- step one, vacuum. step two, real tuning of db performance knobs (about which several books have been written, and it's data set dependent, even among evergreen instances of the same version)
10:02 * Dyrcona wondered about limits in pg_stat_activity....
10:03 miker but the top is enough to know that, if other similarly simple searches are working, it's a stats or tuning issue, not a cold-db issue at 22 hours
10:06 Bmagic miker: I do believe postgres tuning is my issue. I have tweaked a knob or two in an attempt to make it better recently as 3.1 seems to demand more connections. As you said, many books have been written about the subject. Over the years, I have read various in-depth articles and whatnot. Obviously it's insufficient for me at this point.
10:08 Dyrcona We could add that as a hack-away topic: Tuning PostgreSQL, but I'm afraid we'd get no hacking done.
10:10 Bmagic We have compared settings on occation and it seems that our settings are close if not exactly the same as other Evergreen instances. But that doesn't mean anything as the data sets are different (as you said). I understand that. Back to reading the internet. There has got to be a method for this in the context of an individual query
10:14 Dyrcona Bmagic: I'd start with vacuum analyze on the relevant tables.
10:16 Bmagic #vacuum analyze; has been done recently (all tables). And I still see autovacuum frequently. Probably due to the reingest and hold targeter changing lots of things everyday. Perhaps those search queries run long due to the same tables getting vacuumed at the same time.
10:20 mmorgan joined #evergreen
10:34 Dyrcona And, while that discussion was going on, I got an email that load on our db server was "warning" and 2 minutes later it was back to "ok." I missed it.
10:34 Bmagic ha!
10:35 Dyrcona Bmagic: What do you mean "reingest ... changing things every day?" Do you mean the normal reingest from bib updates or are doing something extra every day?
10:35 Bmagic there is a bib reingest running post upgrade
10:36 * Dyrcona thnks our load warning threshold is too low given the number of cores we have.
10:36 Dyrcona OK. I didn't realize that was still going on. I hear it takes a long time for some reason.
10:37 Dyrcona Guess I will find out this week, since I plan got upgrade our training server from 3.0 to 3.2.
10:37 Dyrcona got? really? s/got/to/
10:38 Bmagic it might be overkill, I used your pingest at first, but based upon some of the same issues in the database, I decided to do the low and slow reingest. update biblio.record_entry set marc=marc where id=x; in a loop for all bibs
10:41 khuckins__ joined #evergreen
10:41 Dyrcona I removed this the other day, but I'm adding it back:
10:41 Dyrcona @hate Launchpad Search
10:41 pinesol Dyrcona: The operation succeeded.  Dyrcona hates Launchpad Search.
10:42 Bmagic haha, yeah, I feel your pain
10:42 Dyrcona Tried to find bug 1768715 using just pingest as the search term. Didn't turn up until I added .pl.
10:43 pinesol Launchpad bug 1768715 in Evergreen "Add pingest.pl to evergreen" [Wishlist,Fix released] https://launchpad.net/bugs/1768715
10:43 Bmagic wow
10:43 Dyrcona I did know enough to use Advanded Search and add Fix Released.
10:43 Dyrcona I wasn't sure if it made it into 3.1 or 3.2.
10:43 Dyrcona I thought 3.2, but wanted to make sure.
10:44 Bmagic either way it made 3.2
10:46 Dyrcona True, but there's a difference: whether or not is in 3.1.
10:46 Dyrcona So, you've got that to look forward to. :P
10:46 Bmagic :)
10:56 * JBoyer appears
10:57 JBoyer Bmagic, Dyrcona is right, we cut off any search after 1:30 since Apache has likely given up already, even if OpenSRF hasn't.
10:57 Bmagic good to know. Do you happen to know the frequency of that occurance?
10:58 JBoyer It's really basic (I think a customized version of something we got from EOLI a long time ago) and it doesn't log anything, so short of scraping PG logs for cancellations I don't know how often it hits, no.
10:59 Bmagic no worries. Thanks for the info. Another behavior that I am noticing is: BANG 10 search queries with the exact same criteria all at once.
11:00 Bmagic I suppose the user is refreshing the browser over and over or clicking the search button like 10 times before it changes to a spinner
11:00 JBoyer Some kid leaning on the enter key, I would guess.
11:01 kmlussier @blame cats
11:01 pinesol kmlussier: cats WILL PERISH UNDER MAXIMUM DELETION! DELETE. DELETE. DELETE!
11:01 JBoyer No more JS than we use in the OPAC we can't prevent repeat submissions. (unless I'm misremembering)
11:02 * JBoyer has never understood the 2 second delay on the spinner either.
11:02 JBoyer I changed it to 2ms here, spinner means stop clicking!
11:04 Dyrcona Even with JS, you can't necessarily guarantee that duplicate searches don't get sent. On the plus, Eg will cache the results. On the down side, it usually happens before the results are cached.
11:04 Dyrcona Only thing you can do is setup a proxy and drop duplicate requests.
11:05 * Dyrcona has looked into the general problem of double form submissions.
11:05 Dyrcona I should say, the proxy is the only thing you can do that would be 100% effective.
11:07 JBoyer Sure, it can't really stop anyone determined, or multiple people actually submitting the same search, I don't mind that. But I just did a simple test where I held down enter until I it looked like the screen was about to change and got 108 identical searches.
11:08 Dyrcona JBoyer: Sounds about right.
11:10 JBoyer Interesting thing about doing that though, you can get a rough idea of how well your load balancer spreads things around. ;)
11:10 Dyrcona Or doesn't spread things around. :)
11:11 Dyrcona Ours seems to pick on 1 brick at times, but it probably has something to do with "same source" requests.
11:12 miker if the searches are truly identical, you shouldn't be seeing them at the db level unless they came in /super/ close to one another.  once we get to the middle layer and have the query we're going to send to the db, before getting to the storage app and running the query, we stick a placeholder in memcache to say "I've got this, check back soon" and the subsequent requests should wait for the first to succeed. the other way to get duplicates would be
11:12 miker for them to come in > 3mins (IIRC) apart, where we release the memcache placeholder because the results would be old anyway
11:13 miker if you don't tell all service nodes about the same set of memcache servers, that won't work, of course
11:15 JBoyer I wouldn't expect all of those to hit the db at once, no. But if they're coming in at essentially the keyboard refresh rate that's /quite/ close together, if perhaps not /super/ :)
11:17 JBoyer It has been a good long since we've had the odd "here's 100+ of the same search, time to get sad" issue. I should probably stomp around on our dev server to see if I can trip it up anymore
11:18 miker yeah, it's not perfect. there's also an opensrf patch that I think is still pending (after rework because of modern perl) to release the modperl instance once the client goes away (as in the case of cat-on-keyboard)
11:18 Dyrcona It has been a while since I've seen a problem caused by this issue, probably once since we went to the current db hardware.
11:19 miker all of these are just mitigation in the face of accidental duplicates. a determined attacker can take down any smallish site, not just an EG instance
11:21 Dyrcona Sure, but we're not concerned about a determined attacker, just someone who doesn't know how browsers work.
11:22 Dyrcona I.E. most people.
11:25 JBoyer At any rate, the osrf level stuff definitely helps, since I could hardly catch a single search query on my db while I stressed out the (single) app server so hard it started throwing 404s.
11:26 JBoyer Anyway, none of this really had to do with the 22hour old query, so that's enough sting pulling, lest a sweater get unraveled and it turns cold.
11:26 Dyrcona My sweater unraveled a bit ago, I'm afraid. :)
11:28 * JBoyer is definitely looking forward to the hackaway being somewhere warmer this year.
11:28 * Dyrcona feels a draft from the windows, should probably put the storm windows down and the screens up.
11:29 Dyrcona This will be first anything-Evergreen in the Atlanta area. Somehow, it was always tsbere's turn when I was at MVLC. :)
11:33 bshum ... really?
11:33 bshum Huh, lol
11:39 * kmlussier went to the Decatur conference and then went to Atlanta for some other library open source thing that wasn't specifically related to Evergreen.
11:39 kmlussier I was hoping to make it back there for the hack-a-way, but I guess it's not happening.
11:42 JBoyer kmlussier, well, there's at least one person that goes basically on their vacations... ;)
11:42 kmlussier JBoyer: Sorry, I'll miss all of you, but not enough to spend my vacation time with you. :D
11:42 jwoodard joined #evergreen
11:43 JBoyer Sounds fair. :D
11:44 kmlussier Of course, if you all plan to meet up in Hawaii some year, I could change my mind.
11:44 eady joined #evergreen
11:46 rhamby kmlussier: ok, I might as well cancel the karoake rental then....
11:46 kmlussier rhamby: There's no reason you can't do karoake! I'm sure somebody will post the video so that I can enjoy it.
11:50 Dyrcona :)
11:51 kmlussier If anyone is looking for a quick and easy patch to test, there's bug 1783602
11:51 pinesol Launchpad bug 1783602 in Evergreen "Copy counts should be removed from metarecord search results page" [Medium,New] https://launchpad.net/bugs/1783602
11:56 plux cqqqqq
11:58 mmorgan1 joined #evergreen
12:02 jwoodard so I have an odd question for the group
12:02 jwoodard is there a way to age protect items, in a consortium setup, without utilizing age protection?
12:04 jonadab Assign them to a "New Items" collection that is restricted from going out beyond the local library?  And then move them to another collection when they're not new anymore?
12:08 nfburton joined #evergreen
12:09 jwoodard but how would you restrict items to circulate locally only without age protection?
12:09 jwoodard i feel like the answer is staring me in my face
12:10 mmorgan jwoodard: You can set a hold policy with a transit range of Branch or System.
12:11 mmorgan jwoodard: Why do you not want to use age hold protection?
12:14 jwoodard even though if falls off after it set time it still shows in the catalog
12:16 jwoodard it drive our cataloger, and myself, crazy
12:17 jwoodard *drives
12:18 jwoodard also I am trying to find ways to protect our holiday books from all going out without having to batch edit or touch every item each holiday
12:19 mmorgan Yes, I agree the display of the field is misleading, see lp 1251761
12:19 pinesol Launchpad bug 1251761 in Evergreen "Age Hold Protection TPAC field doesn't change when expired" [Wishlist,Confirmed] https://launchpad.net/bugs/1251761
12:23 kmlussier Does anyone run a nightly script that removes age protection from copies that are no longer age protected? It's not a good permanent solution, but could serve as a workaround.
12:23 mmorgan jwoodard: We have some circ modifiers set up with a hold policy that restricts them to only be holdable at the owning library using the transit range.
12:26 jwoodard mmorgan: I was just looking over the hold policies to see what is possible
12:26 JBoyer We remove age protection the day it "expires" and also automatically roll some 'blah blah new' circ mods to regular old 'blah blah'
12:27 JBoyer An automated script is really the only way to do it. It could be made into a srfsh script, or something that uses the CronScript perl modules, but the end result is the same.
12:27 jwoodard We remove ours monthly from new items
12:27 mmorgan Also, I have not tested this, but I remember discussion about the ability to set up a hold policy that consults an age hold protection rule based on other selections in the hold policy.
12:29 jwoodard as we have grown we are encountering more special cases where age protection would be nice
12:29 jwoodard but we do not want to increase the workload
12:30 jwoodard how does the "item age<" field function?
12:30 jwoodard that was my first thought but I cannot find any documentation on it
12:30 JBoyer I should have been more specific. kmlussier is right, we do all of that stuff daily through a cron job for the entire consortium. There's no point in touching age protection once it's set.
12:32 sandbergja joined #evergreen
12:34 Bmagic we run that script (removing the age protection after it's expired) cron
12:40 jihpringle joined #evergreen
12:40 mmorgan jwoodard: I also want to mention that we use holds always go home, so if an item travels to another library, it will always come home if there's a hold at the owning library.
12:42 jvwoolf joined #evergreen
12:45 Dyrcona jwoodard: The item_age< field means that hold matrix matchpoint only applies to items whose age is less than the field value.
12:45 Dyrcona It could be used as a substitute for age hold protection if the matrix is  set up correctly.
12:57 khuckins joined #evergreen
13:19 jwoodard thanks everyone! you have given me some different idea on how to tackle our problems
13:28 mmorgan jwoodard: Good luck, BTW, I don't think your question was odd at all :)
13:38 sandbergja_ joined #evergreen
13:58 jeffdavis Are point releases expected tomorrow-ish? (The calendar has them scheduled for last week.)
13:59 berick jeffdavis: i believe so, yes.
14:01 jeffdavis cool, thanks
14:16 gmcharlt kmlussier: ping
14:16 kmlussier gmcharlt: Yes?
14:17 gmcharlt kmlussier: I'm wrapping up testing of the branch for bug 1746536
14:17 pinesol Launchpad bug 1746536 in Evergreen "web client: cannot edit vol/call number in item status" [Medium,Confirmed] https://launchpad.net/bugs/1746536 - Assigned to Galen Charlton (gmc)
14:17 gmcharlt I've verified that Janet's testing in commit 27 also works for me in the current branch
14:17 gmcharlt and I"m no longer seeing the issues reported in comments 23 and 24
14:18 gmcharlt before I merge, was there anything else you're aware of holding up that one?
14:18 kmlussier gmcharlt: I'm confident that the previous bug fix has addressed the issues I found. Those were the only two issues.
14:18 gmcharlt ok, great
14:18 gmcharlt merging now
14:18 kmlussier gmcharlt++
14:24 pinesol [evergreen|a. bellenir] LP1746536: cannot edit vol/call number in item status - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=317c2fd>
14:24 pinesol [evergreen|Cesar Velez] LP#1746536 - enable volcopy editor for multi-bib but only volumes - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=c7516dc>
14:24 pinesol [evergreen|Mike Rylander] LP#1746536: Allow batch-apply to volume fields in multi-record context - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=7674fd6>
14:24 pinesol [evergreen|Mike Rylander] LP#1746536: Restrict CN addition but allow CN edits... - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=6cd17f1>
14:24 pinesol [evergreen|Kathy Lussier] LP#1746536: Remove input-group-addon class from Add Call Number button - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=f6d7999>
14:25 khuckins joined #evergreen
14:25 berick if anyone's feeling adventurous, I've found the patch for bug #1797923 to be helpful in my webstaff testing
14:25 pinesol Launchpad bug 1797923 in Evergreen 3.1 "Browser client iframe (catalog, etc.) loading page" [Undecided,New] https://launchpad.net/bugs/1797923
14:25 berick before tomrrow's release cut, i mean
14:26 yboston joined #evergreen
14:26 Dyrcona gmcharlt++
14:38 JBoyer berick, I'll throw some time at that. I won't miss the iframe-ception once it's gone.
14:43 bwicksall joined #evergreen
14:55 plux jeff: Sorry I had to bail on the title/ config.metabib_field thing on the 19th - I replaced the table with the latest schema from 3.2.0 and on reingest it is working now
14:56 JBoyer Has anyone experimented with the pg settings for parallel workers? It's helpful enough for simple queries that it like it's worth the time to start an "is this function parallel-safe?" project
14:56 plux jeff: oddly enough though, I also had reset all the mods32 in that table to mods33 and that failed so I had to revert to the default schema which still was using mods 32
14:57 JBoyer (sadly, a ton of work will have to go into that or a larger refactoring to ||-ize search.)
14:57 plux that’s working fine but if I upgrade production I’ll hit the bug issue around the 1100 upgrade script
14:58 JBoyer plux, you don't have to change the mods version for the existing entries, you just have to make sure that the entries in that script are set to mods33 (usually by changing the default for that field before running that script)
14:59 gmcharlt folks - thoughts on backporting bug 1746536 to rel_3_1?
14:59 pinesol Launchpad bug 1746536 in Evergreen 3.1 "web client: cannot edit vol/call number in item status" [Medium,Confirmed] https://launchpad.net/bugs/1746536
15:00 plux JBoyer: thanks….a tad overzealous with my updating
15:00 Dyrcona gmcharlt: Does it backport cleanly?
15:00 jeffdavis gmcharlt: well obviously I'm in favor :)
15:01 jeffdavis (I added a 3.1 target for that bug, was about to ask if there's a better way to say "please backport this")
15:01 Dyrcona jeffdavis: That's the best way, followed by a comment. :)
15:04 kmlussier +1 to backporting
15:06 khuckins_ joined #evergreen
15:16 gmcharlt Dyrcona: yes, once  1739087 (and account for "volume" vs "call number")
15:17 Dyrcona Who is RM for 3.1? I guess it should be their decision, but if it backports without much conflict, then I say go for it. :)
15:32 khuckins joined #evergreen
15:49 khuckins joined #evergreen
15:51 dbwells gmcharlt: +1 from me to backport for both 1746536 and 1739087
15:55 khuckins left #evergreen
16:22 Dyrcona So, it looks like the 3.1.5-3.2.0-upgrade-db.sql was not forward ported to rel_3_2.
16:23 gmcharlt and backports complete
16:29 Dyrcona berick: do you have a problem with me forward porting the 3.1.5 to 3.2.0 upgrade db script?
16:30 berick Dyrcona: no problem, and much appreciated
16:30 Dyrcona OK. Will do.
16:31 dbwells Dyrcona: Not in master either, if you don't mind.
16:31 Dyrcona No problem I was going to check master, too, after doing 3_2.
16:33 Dyrcona And, done!
16:34 pinesol [evergreen|Jason Stephenson] Forward port 3.1.5-3.2.0-upgrade-db.sql. - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=09740b3>
16:49 khuckins joined #evergreen
17:07 mmorgan left #evergreen
17:30 berick dbwells: can you please clarify "loading records from a catalog search does not work for me" re: bug #1797923
17:30 pinesol Launchpad bug 1797923 in Evergreen 3.1 "Browser client iframe (catalog, etc.) loading page" [Medium,Confirmed] https://launchpad.net/bugs/1797923
17:30 dbwells berick: I mean if I do a catalog search, then click a result, the default tab setting is not honored, I always get the OPAC view.
17:31 berick dbwells: ah, and that's an existing bug?
17:31 khuckins joined #evergreen
17:35 dbwells berick: I think it is mentioned in the tangle of bugs listed on #1731272, but I might be mistaken.  It seems your branch fixes some things on that bug, so maybe it deserves to be separated out to a new bug for clarity anyway.
17:36 berick dbwells: ok, just making sure the patch didn't introduce a bug
17:37 dbwells berick: No, sorry if I implied that.  In my testing your branch made 'default view' go from majorly broken to considerably less broken.
17:38 dbwells in my experience
17:39 jihpringle default view is fixed in the patches on LP 1724348 and LP 1731272 but neither have made it into a release yet
17:39 pinesol Launchpad bug 1724348 in Evergreen 3.1 "Web client: set default view not sticky" [Undecided,Confirmed] https://launchpad.net/bugs/1724348
17:39 pinesol Launchpad bug 1731272 in Evergreen 3.1 "web client: "Set default view" breaks record page loading" [High,Confirmed] https://launchpad.net/bugs/1731272
17:41 berick dbwells: ah, thanks :)
17:43 dbwells berick: So, in further summary, I think your fix does fix bug #1731272 and is preferable to the fixes offered there.  Bug #1724348 was included in the branch for #1731272, but should now probably be considered separately.
17:43 pinesol Launchpad bug 1731272 in Evergreen 3.1 "web client: "Set default view" breaks record page loading" [High,Confirmed] https://launchpad.net/bugs/1731272
17:43 pinesol Launchpad bug 1724348 in Evergreen 3.1 "Web client: set default view not sticky" [Undecided,Confirmed] https://launchpad.net/bugs/1724348
17:44 dbwells I'll add some bug comments to that effect.
17:45 berick dbwells++

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat