Evergreen ILS Website

IRC log for #evergreen, 2015-01-27

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
00:42 abowling joined #evergreen
01:21 dcook__ joined #evergreen
02:46 remingtron joined #evergreen
05:32 dat_ joined #evergreen
07:26 graced joined #evergreen
07:38 sarabee joined #evergreen
07:46 rjackson-isl joined #evergreen
07:55 collum joined #evergreen
08:15 julialima_ joined #evergreen
08:33 abowling joined #evergreen
08:36 mrpeters joined #evergreen
08:54 jwoodard joined #evergreen
09:09 Shae joined #evergreen
09:43 csharp eeevil++ # https://bugs.launchpad.net/evergreen/+bug/1414112
09:43 pinesol_green Launchpad bug 1414112 in Evergreen 2.7 "Audience filter no longer searching blank spaces" (affected: 2, heat: 12) [Medium,Confirmed]
09:43 csharp https://bugs.launchpad.net/eve​rgreen/+bug/1414112/comments/2 is actually what I meant to link to ;-)
09:44 csharp in any case - reingest-free solutions are always welcome the week after an upgrade!
09:44 csharp gmcharlt++ # also ;-)
10:14 csharp ok, quick report after using PG 9.3 built-in replication for a week.  Everything works okay, except we're having to deal with some cancelled reports queries due to conflicts between the master and the reports replica.
10:14 csharp currently running with a max_standby_streaming_delay of 2 hours and that seems to have resolved most cancellations
10:20 jboyer-isl csharp, are you using the option that allows slaves to send feedback to the master to get around some of that? I forget what it's called, but feedback is in the name (at least in 9.1)
10:38 dbs csharp++
10:40 gmcharlt jboyer-isl: csharp: hot_standby_feedback
10:41 jboyer-isl Yeah, that sounds right. We're using that here and I haven't noticed many reports canceled because of it. (We've canceled a few by hand, but that's nothing new...)
10:42 gmcharlt jboyer-isl: csharp: also, tweaking max_standby_streaming_delay can be in order
10:42 * dbs looks forward to eventual 9.4 upgrade
10:50 csharp jboyer-isl: we have two replicas - one is meant to be the actual hot standby so isn't receiving reports queries (feedback on with a short streaming delay) - the other is meant to do reports, so is set with feedback off (to prevent bloating of the master -- from what I've read that's a real concern) and higher streaming delay
10:51 csharp the dying reports are querying asset.copy (weeding, etc.)
10:52 csharp I've asked them to try running after business hours to see if that lets them complete
10:59 Dyrcona joined #evergreen
11:04 graced joined #evergreen
11:04 jboyer-isl joined #evergreen
11:10 collum|2 joined #evergreen
11:13 vlewis joined #evergreen
11:17 77CAAG1BE joined #evergreen
11:18 vlewis joined #evergreen
11:26 abowling is there a way to identify the user agent in a TT2 under the current configuration without importing a new module or plugin?
11:28 abowling disregard. i found it. ResolverResolver
11:33 dbwells that sounds... not right.  Or at least I don't understand the answer.  ResolverResolver is for OpenURL holdings stuff.
11:34 * dbwells scratches his head
11:37 Dyrcona abowling: I think your module that calls the TT2 would have to provide the HTTP headers in the data variable to the templates.
11:37 Dyrcona As far as I know, EGWeb doesn't do that.
11:42 csharp huh - I have a request from our Acq person to add/remove fields from the "name" options available in the acq.provider_holding_subfield_map table, and they appear to be set up in Open-ILS/src/templates/conif​y/global/acq/provider.tt2...
11:43 csharp but I don't know where they go from there - are the "name" values used by the edi fetcher and pusher scripts from that point?
11:43 abowling Dyrcona: ResolverResolver seems to use LWP::UserAgent, which is where I was going with it, but I think there might be a simpler way to get there as you mention in obtaining the HTTP headers
11:45 csharp berick: do you remember much about http://git.evergreen-ils.org/?p=Ev​ergreen.git;a=commitdiff;h=76976c7​fd09e47ca71ebb8c2311b690f11dac594?
11:45 pinesol_green [evergreen|Bill Erickson] ACQ Provider holding subfield field name options - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=76976c7>
11:46 berick csharp: more or less, yeah
11:46 Dyrcona abowling, what are you trying to do?
11:46 Dyrcona Or, rather, why do you want to pass the user agent to a template?
11:46 csharp berick: so if I wanted to edit that list, do you have any hints about what else besides the tt2 file would need to be changed?
11:47 nhilton joined #evergreen
11:47 Stompro Hello, Upgrade question.  There is a note in the 2.7.1-2.7.2 db upgrade script that says you need to run 2.6.2-2.6.3 script to grab a missing update.  Does that mean that the upgrade path for 2.5 would be 2.5.3--2.6.0 -> 2.6.2--2.6.3 -> 2.6--2.7.0 -> 2.7.0-2.7.1.   So you run the 2.6.2-2.6.3 before the 2.6-2.7.0?  I'm just updating the ugprade docs for 2.7.3 and want that to be clear.
11:48 csharp Stompro: that's what I did and all is working okay
11:48 berick csharp: unless I missed a field, the backend Perl code only understands that exact list of fields.  to extract/import more fields from provider records, you'd also have to add support to the Perl code.
11:48 Stompro csharp, thanks.
11:49 berick as far as the UI goes, though, the TT2 is all you'd have to change.
11:49 bshum Stompro: Correct, that was a missed upgrade script in the path, and running it before or after shouldn't be a problem.
11:49 csharp berick: okay - is it Order.pm?
11:49 sandbergja joined #evergreen
11:50 Dyrcona Seriously? We have 200 cstores going across two servers and load is 0.75!
11:50 bshum o.O
11:51 csharp berick: found it (yes, it's Open-ILS/src/perlmods/lib/Open​ILS/Application/Acq/Order.pm)
11:51 csharp berick: thanks for the info
11:51 Dyrcona Everyone is closed, but still....
11:52 berick csharp: indeed, cool
11:53 berick Dyrcona: also note that, unlike Perl opensrf, which supports max/min spare children, C opensrf does not clean up idle drones until they have hit max requests
11:54 berick so it takes a lot longer for C service drone counts to come back down
11:55 berick ...and most of them will just be sitting there doing nothing
11:56 Dyrcona Yes, but I've got tons of these: "No request was received in 6 seconds, exiting stateful session"
11:56 Dyrcona Something in the OPAC must be keeping a cstore connected.
11:57 Dyrcona I think I'll take a deeper look, even though I'm "off." I can always get comp. time. :)
12:02 Dyrcona Maybe it is something in our of our customizations, but I didn't think we customized the OPAC code so much, just the templates.
12:02 Dyrcona Think I'll start by diffing our production branch with master.
12:03 Dyrcona Hmm.. Wonder if I can limit it to certain directories to skip all of our templates and stuff?
12:05 Dyrcona To answer my question for the logs: git diff works if the path is a directory. I'd only ever used it on individual files before.
12:06 goood boo... the bottom of c3c8b74d5169 breaks authority validation (STABLE is Right(tm) for the functions, but the planner makes a bad choice). we should run the function first and then use the value in the query...
12:06 pinesol_green [evergreen|Dan Wells] Remove unwanted index recreation - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=c3c8b74>
12:07 goood make that 4ffa62207
12:07 pinesol_green [evergreen|Ben Shum] LP#1253163: stamping upgrade for authority.in-line-headings - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=4ffa622>
12:09 bshum Honestly, I don't remember what I was testing that day when I merged that.  Wasn't that related to the PG 9.3 weirdness and function breakage?
12:11 jihpringle joined #evergreen
12:11 bshum Wait, I'm confused by those two commits you linked now.
12:11 bshum Are you saying the index should be there or shouldn't be there?
12:11 pastebot "Dyrcona" at 64.57.241.14 pasted "MVLC WWW diff with master" (40 lines) at http://paste.evergreen-ils.org/27
12:12 bshum Dyrcona: That doesn't look too unusual to me.
12:18 bshum Or to put it another way, what did I break now?  10+ months later :(
12:21 * bshum dangles off the edge of sanity
12:32 dreuther_ joined #evergreen
12:33 goood berick: is http://git.evergreen-ils.org/?p=working/random.g​it;a=shortlog;h=refs/heads/collab/berick/hatch2 still the newest hatch branch?
12:36 berick goood: yes
12:41 goood berick: thanks. are there any instructions for installing on windows or mac?
12:45 berick goood: no instructions.  same concepts for both, though, just different commands
13:03 bmills joined #evergreen
13:05 csharp goood: I'm attempting to apply the fix for bug 1414112 on a test server and getting this error: http://pastie.org/9865951 ... it looks like it should work, though :-/
13:05 pinesol_green Launchpad bug 1414112 in Evergreen 2.7 "Audience filter no longer searching blank spaces" (affected: 2, heat: 12) [Medium,Confirmed] https://launchpad.net/bugs/1414112
13:09 Dyrcona berick: After comparing notes wth bshum, I really think it is something under EGCatloader, probably doing xact_begin followed by xact_commit or xact_rollback, but no disconnect.
13:09 Dyrcona He sees the messages in his logs, also, just not as many.
13:10 Dyrcona They also only come from his two main app servers that also run Apache, not from the utility server or other servers.
13:10 Dyrcona That is, the no request received messages.
13:11 Dyrcona Anyway, I'm going to see what I find and patch it for our server. If that works, I'll make a branch in a few days.
13:13 csharp Dyrcona: we aren't seeing performance problems (yet), but I'm seeing many many such messages in our logs last hour
13:13 Dyrcona csharp: Thanks for the corroboration.
13:13 csharp No request was received in 6 seconds, exiting stateful session
13:14 Dyrcona bshum's timeout is 8 seconds, but other than that he got about 1300 in the last hour.
13:14 Dyrcona I get over 17,000!
13:14 Dyrcona I only have about 1/3 more entries in Apache access.log for that hour than bshum has, so the problem could be exponential rather than linear.
13:15 Dyrcona He also runs drones on 3x as many servers as I do.... Bingo!
13:15 jeff heh
13:15 Dyrcona I think we have a winner, folks.
13:15 bshum Heh
13:17 Dyrcona I'll make myself another tea, and then I'll see if I can find the culprit(s).
13:19 csharp Dyrcona++
13:23 dbwells Hey all.  Just a quick double-check: is PG 9.1 still the minimum version for EG 2.7?
13:23 csharp dbwells: from my understanding, 9.1 has not been deprecated yet for any EG version
13:24 bshum dbwells: PG 9.1 is the minimum version, but personally I only tested with PG 9.3 and recommend that to everyone who goes up to EG 2.7 :)
13:24 dbwells csharp++ Thanks.  I thought that to be true, but didn't want to find out mid-upgrade that I was wrong :)
13:25 csharp dbwells: I'll mention that we're on 9.3, so I can speak from experience upgrading to 2.7 on 9.1 ;-)
13:25 dbwells bshum: That's probably good advice, but my upgrade window is a little tight this time around.  We'll get there eventually.
13:26 Dyrcona We're still running Pg 9.1 with 2.7
13:27 yboston joined #evergreen
13:28 dbwells Thanks again, all.
13:29 berick Dyrcona: good find, re: util vs. app server
13:29 berick opac could definitely be it
13:36 csharp goood: FYI - I found the problem and submitted a signed-off branch to correct it: https://bugs.launchpad.net/eve​rgreen/+bug/1414112/comments/4
13:36 pinesol_green Launchpad bug 1414112 in Evergreen 2.7 "Audience filter no longer searching blank spaces" (affected: 2, heat: 12) [Medium,Confirmed]
13:49 Dyrcona Well, I found one problem in fetch_user_circs from EGCatLoader/Account.pm.
13:49 Dyrcona It does xact_rollback instead of rollback, so that would leave a connection open.
13:50 berick indeed
13:52 Dyrcona That's all I can find under EGCatLoader that looks like a problem.
13:52 Dyrcona Some others do at first glance, but there is always a commit or rollback before a return.
13:52 Dyrcona Think I'll just look under WWW for completeness-sake.
13:55 berick ugh, I did that?  lame!
13:57 Dyrcona KPACLoader, has a similar situation on line 225.
13:57 csharp @who is to blame?
13:57 pinesol_green graced is to blame.
13:58 Dyrcona I'm going to skim the other files my grep turns up, but they only show up because the explicitly disconnect.
13:58 Dyrcona I did this to find things that look troublesome: grep -n -R -P '(?:disconnect|xact_(?:begin|commit|rollback))' ./
14:01 Dyrcona Nifty. BadDebt.pm actually calls open-ils.cstore.transaction.begin. I don't see that very often in the code.
14:02 Dyrcona Ditto TemplateBatchBibUpdate.pm
14:03 Dyrcona ->gather(1) disconnects doesn't it? Does it not work if the connect was done explicitly?
14:04 berick no, gather only calls recv()
14:04 csharp Dyrcona: BadDebt.pm is basically cruft from a PINES wishlist feature from ages back
14:06 yboston_ joined #evergreen
14:07 Dyrcona berick: Yeah, I just looked through OpenSRF::AppSession again, and was about to say it doesn't disconnect.
14:07 Dyrcona That's OK. The lines that prompted the question had a gather(1) followed by a disconnect.
14:08 dreuther joined #evergreen
14:09 Dyrcona And, AutoSuggest.pm apparently disconnects when it doesn't have to.
14:09 Dyrcona Which should be harmless.
14:09 Dyrcona So, I wonder if fetch_user_circs is causing us all this trouble.
14:10 Dyrcona We don't use KPAC.
14:10 berick it's called every time some one views their circ list and a few other places
14:11 berick renewals, circ history
14:11 berick in any event, it's adding to the problem
14:11 berick and def. needs to be cleaned up
14:15 Dyrcona Yep.
14:15 Dyrcona I just realized that I'm using my personal laptop and don't have a copy of my development branch on it, and can't get it without messing with my git repos.....
14:16 Dyrcona Or firing up my work laptop.
14:16 Dyrcona I should put this in my existing CStoreEditor branch anyway.
14:24 Dyrcona Wow.
14:24 Dyrcona We've been consistently hovering around 200 cstore drones over two servers, with the load on the main Evergreen server currently at 0.51.
14:24 Dyrcona This has been going on for 3 hours.
14:25 Dyrcona It has to be fetch_user_circs....
14:25 Dyrcona Only the OPAC should be in use today, everyone is closed.
14:31 Dyrcona Ah ha!
14:32 Dyrcona fetch_user_circs runs any time a patron tries to renew.
14:32 berick yeah, or even view their items out
14:32 Dyrcona Guess what I saw in the logs during one of episodes that occurred at night on the weekends....
14:32 Dyrcona Library Elf renewing stuff for a number of patrons.
14:33 berick Library Elf renews through the tpac?
14:33 Dyrcona Oop.. never mind. they go through xml-rpc...
14:33 goood csharp: doh! thanks
14:33 Dyrcona I bet a lot of people are renewing today.
14:33 goood (he said an hour later)
14:34 Dyrcona Since the libraries are closed.
14:34 Dyrcona goood++
14:34 Dyrcona Just 'cause eeevil gets all the karma. ;)
14:41 dreuther_ joined #evergreen
14:52 dreuther joined #evergreen
14:58 Dyrcona So, I load my checkouts in the OPAC.
14:58 Dyrcona I renew a couple.
14:58 Dyrcona I reload the list of checkouts.
14:59 Dyrcona I just potentially talked to 3 drones, and each one has a connection dangling if I did all of that in < {timeout} seconds.
15:07 Stompro I just tried to get a better Qualys SSL score and managed to break the staff clients ability to connect.  Anyone remember what the staff client needs in reguards to ssl versions/cyphers that doesn't work with a modern locked down ssl setup?
15:09 csharp hmm - can you get to Admin -> For developers... -> about:config when you can't connect?
15:10 csharp doesn't look like it
15:12 Stompro Wireshark says it is trying to use tls1.0, which is enabled, so maybe the problem is with the ciphers enabled.  Apache logs show nothing so it is't getting past the ssl negotiation phase.
15:12 dbs Stompro: umm. works pretty great for me
15:12 dbs https://www.ssllabs.com/ssltest/analyze​.html?d=laurentian.concat.ca&amp;latest anyways
15:13 Stompro dbs: would you mind sharing your SSLCipherSuite line?  I also just loaded a non self signed cert.. I should probably not change too many things at once before testing :)
15:13 * dbs using https://wiki.mozilla.org/Security/Server_Side_TLS (intermediate support)
15:14 Shae joined #evergreen
15:14 dbs SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GC​M-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-​AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-D​SS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128​-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES12​8-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA​384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SH​A:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DH​E-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-R
15:14 jeff Stompro: i'd be interested in your config and in your failure mode / errors, but purely for selfish reasons -- i'd like to know how it can break. doesn't help you fix it right now.
15:14 Stompro Firefox can connect just fine to the catalog... just not the staff client.
15:14 * dbs also redirecting everything from HTTP to HTTPS
15:19 Stompro jeff: The error mode is that the server status check fails with a red "There was an error testing this hostname".  The other issue that could be related is that the cert is so new that the OCSP info hasn't been updated, so maybe the staff client is trying to validate the cert against the OCSP list and failing because of that?
15:19 Stompro And it has nothing to do with the cipher suites.
15:19 jeff What OCSP issue do you think you're running into?
15:22 Stompro I think I was supposed to wait an hour or two before using the new cert for the OCSP info to get added.  I think a negative reply is cached now on the OCSP server so I have to wait for 24 hours for it to time out.  ssllabs does give me an "OCSP ERROR" so they are seeing it also.
15:27 Stompro That was it.  I disabled OCSP checking in the staff client and now I can connect.  Added user_pref("security.OCSP.enabled", 0); to the prefs.js.
15:45 gmcharlt because, apparently, we collectively don't have enough to do: https://security-tracker.debi​an.org/tracker/CVE-2015-0235
15:45 pinesol_green ** RESERVED ** This candidate has been reserved by an organization or individual that will use it when announcing a new security problem.  When the candidate has been publicized, the details for this candidate will be provided. (http://cve.mitre.org/cgi-bin/c​vename.cgi?name=CVE-2015-0235)
15:45 bmills joined #evergreen
15:46 * bshum grumbles
15:50 Dyrcona Does pinesol pickup commits from the working repo?
15:50 jeff commit a65e1ad
15:50 jeff apparently not.
15:50 bshum No it does not
15:50 Dyrcona OK, I'll paste the URL from gitweb.
15:51 Dyrcona http://git.evergreen-ils.org/?p=worki​ng/Evergreen.git;a=commitdiff;h=ad5a2​9d2dd887e29dabbf733813294f58ce66752
15:51 bshum We could track the working repo, but it would only be tracking the main branch and spitting dupes to the channel I think.
15:51 bshum Since master there is the same as master in the other place.
15:51 Dyrcona I applied that in production (after checking that it didn't break the OPAC in development).
15:51 bshum But I wonder, hmm
15:51 Dyrcona Since I applied that, the number of cstore drones in use on our server has been steadily going downward, if slowly.
15:52 Dyrcona It was going down a little and then back up to 200.
15:52 Dyrcona We're currently down to about 55 on one and 82 on the other.
15:52 Dyrcona I think I'm gonna go shovel some snow.
15:54 goood Dyrcona: hrm... doesn't that break the contract with callers to not destroy their editor?
15:55 goood (obviously, if there are only a few callers, and they're not using the editor after, they should be disconnecting it, but ...)
15:56 Dyrcona goood: If you look at the rest of the code, those rollbacks come after explicit xact_begin calls.
15:57 Dyrcona Everywhere else in EGCatLoader and family that happens, an explicit commit or rollback is done.
15:57 goood Dyrcona: right, but xact_commit doesn't make the editor unusable, where rollback does, unless I'm misremembering (which is totally possible
15:58 csharp for Ubuntu users, the same alert as gmcharlt posted for Debian: http://www.ubuntu.com/usn/usn-2485-1/
15:59 goood the xact_begin/xact_commit pairs leave the editor intact, I mean, where rollback doesn't
15:59 goood so a theoretical caller doesn't have an editor now, after fetching circs
15:59 berick rollback causes a disconnect, but it does not destroy the editor
15:59 goood but, if that's the last thing that any caller does,
16:00 goood berick: will it magically reconnect?
16:00 goood sorry, I shouldn't have used "destroy"
16:00 berick goood: if you call xact_begin, yes
16:00 berick but it won't just reconnect with any new request
16:00 Dyrcona And, it looks like xact_begin is used everywhere.
16:01 Dyrcona in the OPAC anyway.
16:01 berick agreed if the caller connected the editor before passing it down to the user-circs function, the user-circs function should not disconnect it
16:01 berick but I think in this case it was just a mistake
16:01 berick the caller never connected it
16:01 berick user-circs did, by virtue of calling xact_begin, then left it connected
16:01 berick and since the caller never connected, it never disconnected
16:02 goood ack says only EGCatLoader/Account.pm calls that sub, so there are few callers to check
16:02 goood which it sounds like Dyrcona had done ;)
16:03 goood so, I'll lower my red flag to a yellow for "hypothetical future callers" :)
16:03 berick goood++
16:03 berick all valid points
16:04 goood well
16:04 goood handle_circ_renew looks like it uses the editor ... ah! but only to pull values off it, not to make requests
16:05 goood so, every use of fetch_user_circs is the last (remote) use of the editor in each code path
16:05 dbwells Dyrcona++ # 'cause I'm feeling hopeful you've cracked it
16:06 berick at minimum, a comment in the function warning the caller
16:07 * csharp schedules maintenance window to apply the glibc updates for PINES :-/
16:07 goood Dyrcona++
16:22 nhilton joined #evergreen
16:33 Dyrcona While I shoveled snow, I figured we should add a connection_count to CStoreEditor like a reference count.
16:33 Dyrcona It gets incremented every time connect is called and decremented each time disconnect is called.
16:34 Dyrcona When the count reaches 0, disconnect actually disconnects.
16:34 Dyrcona reset would set count to 0 before calling disconnect.
16:34 Dyrcona That's the OO way to fix it, anyway.
16:34 Dyrcona Fortunately, my neighbor plows my driveway.
16:35 Dyrcona We're down to 14 cstore drones, 7 on each server!
16:36 Dyrcona I'll do that in a branch tomorrow.
16:36 vlewis_ joined #evergreen
16:37 bshum Huzzah
16:40 Dyrcona I'd Huzzah, too, but I'm tired from shoveling my walk and shoveling out my car.
16:40 Dyrcona We got two feet plus, all right.
16:52 Dyrcona Well, I'm signing off.
16:53 Dyrcona TTYL, #evergreen!
17:17 pinesol_green Incoming from qatests: Test Success - http://testing.evergreen-ils.org/~live/test.html <http://testing.evergreen-ils.org/~live/test.html>
17:21 gmcharlt goood: bug 1415234
17:21 pinesol_green Launchpad bug 1415234 in Evergreen "uncontrolled attribute values that consistent only of spaces are normalized away" (affected: 1, heat: 6) [Undecided,New] https://launchpad.net/bugs/1415234
21:26 csharp joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat