Evergreen ILS Website

IRC log for #evergreen, 2016-07-29

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary

All times shown according to the server's local time.

Time Nick Message
03:09 gsams joined #evergreen
03:40 gsams joined #evergreen
03:41 dbwells_ joined #evergreen
03:59 Ewan_ joined #evergreen
04:06 Ewan_ Hello, I have a question: is my /openils/var/web supposed to have 3 Makefiles in it when evergreen is installed?
04:09 dcook joined #evergreen
06:52 gsams_ joined #evergreen
07:15 rjackson_isl joined #evergreen
07:25 graced @later tell Ewan_ that most Evergreeners are in channel between 8:00am and 7:00pm EST
07:25 pinesol_green graced: The operation succeeded.
07:26 druthb @tea
07:26 * pinesol_green brews and pours a pot of Huang Shan Mao Feng Green Tea, and sends it sliding down the bar to druthb (http://ratetea.com/tea/teavivre/hu​ang-shan-mao-feng-green-tea/6041/)
07:27 druthb that'll do.
08:02 dcook__ joined #evergreen
08:04 mrpeters joined #evergreen
08:09 ericar joined #evergreen
08:14 Dyrcona joined #evergreen
08:20 bshum Happy sys admin day out there! :) hope you all get some cake (and/or other treats)
08:25 * csharp settles the usual patting self on the back
08:26 csharp settles *for*
08:36 JBoyer Ho, ho, ho! Merry... wait, no, that's something else.
08:36 JBoyer Feliz Sysadmin... also no.
08:37 JBoyer Congratulations on sysadmin-ing for another year! Here's to more of that.
08:37 JBoyer Close enough.
08:38 mmorgan joined #evergreen
08:38 csharp JBoyer++
08:39 Dyrcona heh
08:40 mmorgan @coffee sysadmins
08:40 * pinesol_green brews and pours a cup of Espresso Nuevo, and sends it sliding down the bar to sysadmins
08:41 * Dyrcona wonders if being "DevOps" counts. :)
08:47 * csharp starts flamewar about whether DevOps counts
08:50 * Dyrcona launches a volley of Greek Fire at csharp over the bow.
08:55 kmlussier joined #evergreen
08:59 ericar_ joined #evergreen
09:06 maryj joined #evergreen
09:07 kmlussier @dessert 41 bshum
09:07 * pinesol_green grabs some birthday cake with lots of candles for bshum
09:14 Dyrcona I imagine Bmagic and JBoyer may not be around yet because timezones.
09:14 Dyrcona Oh, yeah... Happy Birthday, bshum!
09:14 * Dyrcona waits to bug them about timeouts in ipvs.
09:15 * Dyrcona wants to reopen a conversation from June 3.
09:15 Dyrcona Oh wait. Duh. JBoyer is here already....
09:15 JBoyer Not at all, while Indiana has lost it's way RE: DST, most of us are still within the comforting confines of the Eastern Timezone.
09:15 * Dyrcona has a stupid.....
09:16 Dyrcona Bmagic can hopefully pick up when he shows up, then.
09:17 jvwoolf joined #evergreen
09:17 Dyrcona Back on June 3, http://irc.evergreen-ils.org/​evergreen/2016-06-03#i_252050, the three of us were talking about ipvsadm and Apache settings.
09:17 Dyrcona I was mostly talking about Apache.
09:18 Dyrcona Anyway, Bmagic shared his changes to ipvsadm timeout values that improved performance and JBoyer suggested slight different values.
09:18 Dyrcona We're currently using the defaults for the three timeouts and I'd like to know if anyone has suggestions for better values to use.
09:19 Dyrcona It seems like JBoyer recommended 60 to 120 seconds, and Bmagic dropped some to 30 seconds.
09:19 JBoyer I suggested trying that, but ours are also still on the defaults
09:20 JBoyer Because testing is an enormous PITA since you never know when things will happen. :(
09:21 Dyrcona Bmagic mentions issues with sip that might be related to the 30 second values.
09:22 Dyrcona Before I do something drastic with connection limiting via a firewall, I'd like to try load balancer changes.
09:22 JBoyer That's a good plan.
09:23 Dyrcona My concern is that I'll probably end up with firewall changes anyway since the problem happens in environments without a load balancer.
09:23 JBoyer I wonder if our coordinator would like me playing with values semi-randomly during a 2-library migration.
09:23 Dyrcona Ha ha! I'm sure that would be no problem, JBoyer. ;)
09:24 JBoyer Dyrcona, that does sound like you're going to end up having to go that route. The book on the keyboard problem?
09:24 Dyrcona Yes.
09:24 JBoyer Yeah, I can't imagine an LB actually helping with that. Maybe one of those web application firewalls or something that can look at the url before it hits apache, but $$$
09:24 Dyrcona Another question, is anyone using something other than round robin for the scheduler?
09:25 Dyrcona A "web application firewall" is a fancy name for a proxy server. :)
09:25 Dyrcona I can build one of those for nothing with Apache or nginx.
09:26 JBoyer I'm using weighted least connections since I've got unevenly spec'd app servers.
09:26 Dyrcona I've got evenly spec'd app servers, but I was still thinking of using that one.
09:26 JBoyer Dyrcona, that's true. The $$$ kind has been on my mind because that's what our central IT agency is putting everyone behind.
09:26 Dyrcona It might help with 1 and 2 being superloaded and the other three doing next to nothing, but not sure.
09:27 Dyrcona Ah, yes, central IT often doesn't get Free, so $$$ still win.
09:28 Dyrcona Gov't: where budgets are power.
09:28 JBoyer I could complain for days, but it wouldn't stop apache from falling over, so...
09:32 ericar_ joined #evergreen
09:32 Dyrcona Yep.
09:33 Dyrcona This has been very helpful. I have a better grasp on what is going on with LVS and load balancing and what my options are.
09:34 yboston joined #evergreen
09:34 JBoyer Good. It has a lot of interesting options (I really like maintenancedir, since SIP servers don't have the equiv of a ping file really) but you're right, it hasn't been touched much in ages.
09:35 JBoyer Neither have the docs or howtos...
09:35 Dyrcona I guess if it ain't broke.... ;)
09:36 JBoyer True enough for the code, but I'd feel better if there were more docs written in this decade. ;)
09:36 * Dyrcona wishes he had time to experiment with different load balancing options, i.e. relayd and the others.
09:37 Dyrcona Yeah, kinda scary when you do the google search and one of the top 3 results is from 1998. :)
09:37 jeff were this another channel, @band add relayd and the others
09:37 * JBoyer wishes he had time to experiment. :-/
09:37 * jeff experiments with time
09:38 Dyrcona jeff: Watch out! Those experiments can get you into some trouble. ;)
09:38 jeff anyone have interest in logging changes to SIPServer? i'm thinking of adding a connection id and logging username/workstation (after that patch is in), etc.
09:39 jeff actually, thinking of adding a new log line for each request.
09:39 jeff not just adjusting the ones that are there.
09:39 JBoyer Oh, that reminds me, I think I've got pullrequest tags on both of those...
09:40 Dyrcona jeff: Sounds useful. I'd certainly be willing to look at it.
09:40 JBoyer Ah, yes.
09:41 JBoyer jeff: what kind of conn id do you mean? something that better ties src and dest together, or something else?
09:42 jeff JBoyer: just a made up number that you can use to distinguish two messages that may otherwise share the same institution/username/workstation
09:43 jeff JBoyer: also possibly a distinct id for backends in a prefork environment.
09:43 JBoyer Oh, ok. So you don't have to say "Is this message being logged twice, or sent twice?" That kind of thing?
09:43 jeff that specific use case hadn't occurred to me.
09:44 JBoyer There are some creative clients.
09:46 jeff in the example you gave, i'd start with "sent twice". nice thing there is that i could look at the code to pretty much rule out "logged twice"
09:46 JBoyer Sure.
09:47 jeff though that's not to say that there couldn't be a subtle bug lurking in Net::Server::Prefork that could cause double-logging.
09:47 jeff but i'd expect that kind of bug to also involve double-handling. :-)
09:47 JBoyer or rsyslog when sending it to a central logging server, etc. ;)
09:47 * jeff nods
09:48 jeff so yes, in an example where you are questioning your logs a connection id could be a useful additional bit of distinguishing information in addition to the timestamp.
09:48 jeff (or timestamps, rather)
09:50 Dyrcona Previous message repeated X times.
09:50 Dyrcona ;)
09:51 JBoyer I loosened the throttle on that a fair bit. Don't remember the reason off hand though.
09:54 Dyrcona I thought of another question regarding ipvs/ldirector and Apache.
09:54 Dyrcona How does Apache's keep alive come into play?
09:55 Dyrcona We apparently are not persisting http and https through ipvs, so do established connections bypass the load balancer?
09:55 Dyrcona Oh, they must....
09:55 Dyrcona It's tcp.
09:55 Dyrcona Never mind, a little thought goes a long way. :)
09:57 Dyrcona Hmm... That may complicate things if multiple requests come over the same tcp connection.....
09:58 jeff You could consider MaxKeepAliveRequests as a useful limiter there.
09:59 jeff But if you're adding mod_perl logic to classify requests you can probably set Connection: close before Apache normally would...
09:59 jeff I'm not actually sure how Apache would handle that.
09:59 jeff It might be on the client to honor it, and it might not.
09:59 jeff I've not had reason to test that.
10:00 Dyrcona I'm thinking of the book on the keyboard, and the constant stream of requests.
10:00 jeff right.
10:01 Dyrcona If they're all coming over the same tcp connection, then a firewall rule isn't gonna help. :(
10:03 * Dyrcona is also reading https://tools.ietf.org/html/rfc7230#section-6.3
10:04 Dyrcona jeff: I think I would have to send Connection: close after each response as you suggest, or just disable persistent connection in Apache.
10:05 Dyrcona And, I think either would likely cause a performance hit in general use that outweighs the benefit of fixing the occasional problem from multiple requests from the same client.
10:05 jeff Which would long-term have greater detriment on your performance than occasional book-on-keyboard incidents.
10:05 * jeff nods
10:05 Dyrcona I think so.
10:06 jeff I can think of a few options that don't exist yet...
10:07 Dyrcona I could run wireshark to see what is really happening with connections.
10:07 miker Dyrcona: if the book-on-keyboard requests were coming in through one connections, you wouldn't have the apache-spawn-bomb
10:07 jeff miker++
10:07 miker so, keepalive isn't an issue for that case
10:07 Dyrcona miker: I was just thinking that before you typed it.
10:08 Dyrcona miker++
10:08 Dyrcona Or, rather, as you typed it.
10:09 Dyrcona Firewall rules will work.
10:09 miker what would be really great is if we could cut a pending request off at the knees if the browser disconnects, which, in book-on-keyboard, it should
10:09 miker and kill the apache backend in that case
10:09 miker s/pending/in-process
10:10 jeff related to the "naming things" problem: "finding the thing in a list of other things after you've named the thing"
10:21 Dyrcona miker: It would be nice if there were a HTTP response for a duplicate request....
10:22 miker HTTP 420: go home, client, you're high
10:23 Dyrcona heh.
10:23 Dyrcona 406 or 409 might do... but a duplicate is not normally an error.
10:29 JBoyer HTTP:409 Clean up your mess
10:30 Dyrcona 409 Conflict
10:30 Dyrcona 406 Not Acceptable
10:31 Dyrcona "You're behavior is not acceptable, young web client...."
10:31 Dyrcona Gah. Wrong your....
10:31 Dyrcona Have I mentioned I hate English, lately?
10:31 JBoyer I was thinking more https://www.google.com/searc​h?q=formula+409&tbm=isch but cleaning the junk out of Google's URL made me too slow. :D
10:32 Dyrcona Ha! That's good.
10:32 Dyrcona We should use 409 and one of those graphics on the response page. :)
10:32 JBoyer joeks++
10:49 maryj joined #evergreen
10:55 Dyrcona Guess I am not going to find the Google searches that I did approximately two months ago on a different computer.
10:55 JBoyer Well, if you're logged in and have search history turned on...
10:56 JBoyer (That skeeved me out the instant it was announced, the first thing it logged me searching was likely how to turn it off)
10:56 JBoyer Oh, Web History, that's what I'm thinking of. Bing calls it Search History. Something.
10:58 csharp JBoyer: re: allowing more "repeated" log messages, as I recall, you recommended that because messages were getting plain old dropped with that setting set too strictly (i.e., the default setting)
10:58 JBoyer These keyword searches for -1' make me think that a very basic, fairly loose JS query quality checker would make a lot of slow/poor searches go away without any fancy config. :-/
10:59 JBoyer csharp, Oh, right. And I couldn't tell what was happening for a while because the message about things being dropped was logged locally on the machine, while only the logs that made it through went to the central server.
11:02 jihpringle joined #evergreen
11:05 Dyrcona Yeah, but I'm not seeing it. The new "My Activity" interface is kind of bad.
11:06 Dyrcona and, yeah, I saw something like that with dropped messages on rsyslogd before.
11:09 Bmagic Anyone have any call from their libraries to capture who and where patrons are registered?
11:13 mmorgan Bmagic: We have the occasional "who" question, the "where" question is important to us for statistical purposes. We currently use Home Library to determine the "where", though it's imperfect.
11:14 Bmagic mmorgan: yeah, we have a library system with 9 branches. The home library doesn't satisfy their need. And who registered the patron is just a gaping hole right?
11:16 mmorgan Bmagic: Yes, if there's a clear record of who edited a patron, there oughta be a record of who created it.
11:20 Bmagic bug 1607827
11:20 pinesol_green Launchpad bug 1607827 in Evergreen "Need Evergreen to capture who and where patrons are registered" [Undecided,New] https://launchpad.net/bugs/1607827
11:34 bmills joined #evergreen
11:35 Dyrcona More on the "book on keyboard" issue:
11:36 Dyrcona I'm consideing mod_qos or similar to limit a client to 1 connection per second to search results or something like that.
11:36 Dyrcona Options.
11:38 eady joined #evergreen
11:41 miker Dyrcona: using a non-mod_perl proxy instance?
11:41 Dyrcona miker: Not sure. I'm just kind of thinking out loud.
11:43 Dyrcona Guess it would have to be through a proxy to be effective.
11:45 brahmina joined #evergreen
12:04 maryj_ joined #evergreen
12:42 jvwoolf joined #evergreen
13:02 gsams joined #evergreen
13:17 maryj joined #evergreen
13:18 miker Dyrcona: lunch time thought experiment: http://nox.esilibrary.com/~miker/sta​sh-apache-requestrec.evergreen.patch + http://nox.esilibrary.com/~miker/te​st-apache-requestrec.opensrf.patch
13:22 Dyrcona miker: Looks interesting.
13:22 JBoyer I love a parallel bib load on migration day. 67K records, < 45 minutes. *sunglasses-emoji*
13:24 Dyrcona heh
14:08 jeff found the basis for our new server naming scheme: http://mewo2.com/notes/naming-language/
14:13 ssss joined #evergreen
14:18 bmills left #evergreen
14:30 csharp jeff++
14:31 * csharp logs into pumkulnukkulkuv01
14:32 jeff pretty limiting. i think you're going to want pumkulnukkulkuv0001 to leave room for growth.
14:32 bshum o.O
14:33 Dyrcona :)
14:34 Dyrcona I wrote a character name generator based on Jack Vance's The Dying Earth series.
14:36 Dyrcona I was going to use it in an online Neverwinter Nights module that never completely panned out.
15:08 jeff fatal: [eg-siptest01.REDACTED]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'eg_memcache_ip' is undefined"}
15:09 jeff "hey, i see what you're trying to do and all, but i'm going to need just a BIT more to work with here."
15:12 jeff that said, I really appreciate the "mandatory" filter and the fact that the behavior defaults to on to begin with.
15:13 JBoyer Yeah, I'm not a fan of the "Huh, haven't seen that variable name before, it must be equal to ''" pattern that has infected so many languages/frameworks.
15:27 hbrennan joined #evergreen
15:39 jeff I'm pretty sure that this product's SIP2 support is purely based on pattern matching.
15:41 jeff That is, pattern matching against the raw SIP2 messages, no other attempt made to parse.
15:43 Dyrcona jeff: Now, you've got two problems. :)
15:53 Dyrcona So this is fun. I think I've found the parts of the log where a problem occurred on Wednesday, 'cause I've got searches returning 500 after about two minutes in the logs.
15:54 Dyrcona However, I don't see any signs, yet, of "book on the keyboard" type situations.
15:57 JBoyer jeff: re sip pattern matching, It didn't see a CO in an address and decide that was a new field, did it? I saw that once and was very, very amused. (except that we're still paying for that product...)
15:57 jeff hah
15:58 jeff i was thinking of a few cases where you could confuse that kind of code, and that specific example had not occurred to me.
15:58 jeff you may recall the situation where a barcode with AY in it caused us some fun times, though.
15:59 JBoyer Well, given that 90% of libraries still ENTER PATRON INFORMATION LIKE THIS and SIP field codes are just 2 uppercase characters...
16:02 Dyrcona So, now, you've got three problems. :)
16:02 Dyrcona I think lines like this might have something to do with my lack of evidence: rsyslogd-2177: imuxsock lost 195 messages from pid 4910 due to rate-limiting
16:03 JBoyer Wow. Something was feeling pretty chatty and told to just zip it.
16:06 Dyrcona Probably an Apache process trying to log all the requests it was receiving for the same URL, but I'm guessing.
16:06 Dyrcona I'll look into the syslog rate limit settings on Monday. Bit late in the day to mess with that.
16:07 Dyrcona I see similar messages throughout the day in the logs on the brick heads.
16:07 JBoyer I can send you what we're doing just as a starting point, but yeah, I wouldn't do anything with it until Monday, heh.
16:07 Dyrcona Yeah, that might be useful.
16:08 Dyrcona About two minutes later, the logs say it lost 527 messages from the same PID.
16:11 pastebot "JBoyer" at 64.57.241.14 pasted "Log Limiting such and such" (13 lines) at http://paste.evergreen-ils.org/28
16:18 Dyrcona JBoyer++
16:21 JBoyer To be fair you only need $SystemLogRateLimitInterval 0 to turn off limiting but the rest of that sets up what to do when the messages start coming in a bit too fast.
16:28 Dyrcona Thanks. I'll read up some more on rsyslog on Monday.
16:28 Dyrcona Think I'll call it a day for now since I started working a little early this morning.
16:29 Dyrcona Have a great weekend, everyone!
16:44 maryj_ joined #evergreen
16:52 kmlussier Wow! Work days almost over. Let's see if I can merge one branch before I go on vacation next week.
16:59 kmlussier Actually, that might be somewhat optimistic.
17:01 bshum I believe in you kmlussier!
17:03 kmlussier No, don't believe in me.  I have to feed the children, and there are upgrade scripts to deal with. Otherwise, it would be merged already.
17:03 bshum I suppose children are important.
17:07 mmorgan left #evergreen
17:14 jvwoolf left #evergreen
17:36 kmlussier OK, children are fed. Calling 0983 and 0984
17:38 Bmagic kmlussier++

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary