Evergreen ILS Website

IRC log for #evergreen, 2016-07-28

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
00:01 genpaku joined #evergreen
04:15 dbwells_ joined #evergreen
07:17 rjackson_isl joined #evergreen
07:56 ericar joined #evergreen
08:08 mrpeters joined #evergreen
08:10 agoben joined #evergreen
08:10 collum joined #evergreen
08:26 gsams_ joined #evergreen
08:41 mmorgan joined #evergreen
08:45 ericar joined #evergreen
08:52 bos20k joined #evergreen
09:04 jwoodard joined #evergreen
09:11 jvwoolf joined #evergreen
09:11 maryj joined #evergreen
09:18 kmlussier joined #evergreen
09:25 yboston joined #evergreen
09:26 mce joined #evergreen
09:42 remingtron joined #evergreen
10:05 Dyrcona joined #evergreen
10:35 Dyrcona So, after a "book on the keyboard" incident yesterday, I am looking into ldirectord and LVS, and this stuff smells of bit rot.
10:36 Dyrcona I also wonder why (though I have ideas) that two out of 5 bricks can go bonkers on running Apache, and the other three sit there looking like they're not even used.
10:39 jeff what's currently in front of the five bricks?
10:39 Dyrcona LVS + ldirectord apparently.
10:40 jeff and what's the current strategy for load balancing requests? are you being bitten by most of your requests coming from a small handful of source IPs and being persisted based on those source IPs?
10:41 Dyrcona The strategy, I don't know, yet... But the second answer is yes, which is one of my thoughts about why I saw what I saw.
10:41 * Dyrcona is just learning about this [ancient] stuff.
10:43 jeff there's more money in selling Application Delivery Controllers than there is in updating documentation for load balancing software. ;-)
10:45 Dyrcona Well, from what I can see, IPVS is not available past kernel 2.6.x.
10:46 Dyrcona Maybe I'm missing something, but this project smells funky to me.
10:48 Dyrcona My goal is to solve the "book on the keyboard" issue spawning so many apache children that the load skyrockets on the brick head, even though individual processes seem to be doing very little......
10:48 Dyrcona And I just thought of something.... Number of ejabberd processes....
10:49 * Dyrcona tries to check the logs for ejabberd.
10:49 Dyrcona I managed to tweak MVLC's server so it could still function with over 150 apache drones going, though it looks like one server hit that limit yesterday.
10:50 Dyrcona That could be part of it, too, configure more apaches to run on the brick heads.
10:54 Christineb joined #evergreen
10:58 Dyrcona exception exit: {process_limit,{max_queue,1695}} in erlang.log
10:58 Dyrcona Ah, but for the wrong day.... Never mind that.
10:58 Dyrcona Yesterday had a lot of timeouts.
10:59 Dyrcona That herring is definitely red.
11:01 Bmagic Dyrcona: We recently had to switch to centos for our load balancer, because my research lead me to the same issue. We used this to help get it setup correctly
11:01 Bmagic https://siva2009.wordpress.com/2013​/12/03/setup-load-balancing-cluster​-with-lvs-and-piranha-on-centos-6/
11:03 Bmagic Then again recently, (I believe we had a brief discussion here in channel) we had to change some settings on LVS in order to get those connections closed sooner. App servers were spiraling out of control over a magic number of connections
11:03 Dyrcona Yep, I recall that conversation.
11:04 Dyrcona Bmagic: Any particular reason to use Centos over Debian?
11:04 Bmagic The article had it tried and true for centos
11:04 Bmagic And I happen to like Redhat stuff
11:04 Dyrcona OK. Just wondered if anything else came up in your research.
11:05 Bmagic It comes with a handy UI for editing the config as well
11:05 miker Dyrcona: ipvsadm and ldirector are certainly in modern distros and work on modern kernels: http://manpages.ubuntu.com/manpag​es/xenial/man8/ldirectord.8.html http://manpages.ubuntu.com/manp​ages/xenial/man8/ipvsadm.8.html
11:07 Dyrcona miker: Thanks. I saw some reference in the Ubuntu man pages, but most of the stuff turned up by Google is old.
11:07 Dyrcona My problem isn't load balancing anyway.
11:08 * miker attributes that to ldirectord+ipvsadm being a "solved problem"
11:08 Dyrcona It's too many apache processes spawning.
11:08 Dyrcona I was mainly looking to find a way to limit that in ldirectord if possible.
11:09 miker Dyrcona: re, the book-on-keyboard issue, we actively stop duplicate searches inside apache, but yes, they still eat memory. they won't kill the db anymore, though :)
11:09 Dyrcona Load balancing is a solved problem, and there are other options.
11:09 Dyrcona miker: Right, the database isn't the issue. It's Apache spinning out of control and two brick heads becoming useless.
11:09 miker that'll likely be an iptables or ipvsadm thing
11:10 Dyrcona I dealt this with this to some extent at MVLC by tweaking Apache configurations. Now I'm in a different environment.
11:10 miker as in, one could say "no more than X concurrent connections from the same IP within Y time"
11:10 miker but that will then lock out the library with the book on a keyboard
11:11 miker so... it's not a simple problem. which is why it's yet to be solved
11:11 Dyrcona Right. Ideally, I'd like an Apache solution to basically cutoff repeated requests for the same url, including query, etc.
11:11 miker that's there already (for searches, at least, which is the worst-offender for book-on-keyboard, IME)
11:12 miker but by the time apache has spun up a backend far enough to detect that, mod_perl et al are loaded and the memory is eaten
11:12 Dyrcona Um, I don't see that at all. Looking through the apache logs the exact same url is repeated many times in a row, and it search results.
11:12 Dyrcona Oh. Right... ;)
11:13 Dyrcona I could probably do it with an apache proxy, maybe with a combination of mod_proxy_balancer.....
11:13 miker that's a thought (well, nginx) we've had, too.
11:14 Dyrcona In my particular case, I see the non-routeable IPs for the computers at libraries because of a VPN, so I might be able to do something ipvsadm to limit connections per second.
11:15 miker that adds several moving parts, with attendant issues ... and is itself pretty nontrival, and in reality, site specific
11:15 miker Dyrcona: indeed!
11:15 miker you could even make it a static rule for internal IPs known to be public opac stations
11:16 Dyrcona I'll read up on how ipvsadm works.
11:16 Dyrcona Thanks!
11:16 Dyrcona miker++ Bmagic++ # for suggestions
11:17 miker Dyrcona: it'll probably be iptables rather than ipvsadm, btw, for the conntrack-based limiting
11:17 JBoyer I'm certainly not volunteering to figure it out at the moment, but what would be required to cut the per-process mem use so this isn't a problem anymore?
11:17 Dyrcona I should experiment with the proxy solution some day, too. I've always wanted to play with nginx but never made the time.
11:17 Dyrcona JBoyer: Serve static pages. :)
11:18 miker just about, yeah :)
11:18 JBoyer That's one end of the spectrum, yes. ;)
11:19 miker shard all our mod_perl and gateway/translator stuff into a tonne of apache instances, and mod_proxy them inside ... like we do with websockets now (sans mod_proxy)
11:20 miker there will be some lower bound on memory, though. mod_perl just ain't cheap, in a retail sense
11:20 Dyrcona About iptables: One of the systems staff here is going to see if the current firewall has any QoS features.
11:21 Dyrcona Well, it's tricky to control memory use in Perl anyway.
11:21 Dyrcona mod_perl and the persistence of Apache processes makes it more challenging.
11:21 miker Dyrcona: even if it doesn't, the LB can mark packets based on IP, and later rules can do QoS
11:22 Dyrcona Right. I should be able to do it with iptables.
11:23 JBoyer That's about what I thought re: mem. Shame it's not easier. (though I suppose we'd have already done it if so...)
11:32 brahmina joined #evergreen
11:33 bmills joined #evergreen
11:35 Dyrcona JBoyer: There are some things that we can do, probably, but it won't be easy to identify them or to fix them once identified.
11:57 dbwells Heads up, y'all.  Yesterday was the cutoff for the 2.11 alpha, so I am in the process of rolling the release.  I am not thinking this affects much, unless you are reviewing something you really want in the alpha.
12:08 jihpringle joined #evergreen
12:20 Dyrcona dbwells++
12:22 gmcharlt kmlussier: I've pushed a fix for that remaining issue for LP#154505
12:22 gmcharlt er, 1549505
12:22 kmlussier bug 1549505
12:22 pinesol_green Launchpad bug 1549505 in Evergreen "Statistically generated record ratings" [Wishlist,New] https://launchpad.net/bugs/1549505
12:22 kmlussier Ah, ok. Thanks!
12:22 kmlussier gmcharlt++
12:22 * kmlussier now ponders how she will test it.
12:52 jeff anyone ever have the desire to mark patron stat cats as read-only?
12:52 jeff and "show only if already set"?
12:53 jeff mostly thinking of migrated/legacy stat cats.
12:54 Dyrcona jeff: More or less.
12:54 Dyrcona Same reasons, too, most likely.
12:55 Dyrcona We migrated some things from the legacy ILS as stat cats so we could identify certain groups of patrons if need be.
12:55 * jeff nods
12:55 Dyrcona They do nothing at all otherwise, and some staff add them to new patrons.
12:56 jeff hrm. new patron editor doesn't let you unset a non-freeform stat cat.
12:56 * jeff checks old patron editor
12:56 Dyrcona Err, "we" should probably be MVLC. :)
12:56 jeff heh
12:56 hbrennan joined #evergreen
12:56 * Dyrcona is still in a mental transition state from MVLC to C/W MARS.
13:07 maryj joined #evergreen
13:27 Dyrcona @karma irc_logs
13:27 pinesol_green Dyrcona: irc_logs has neutral karma.
13:28 Dyrcona @karma irclogs
13:28 pinesol_green Dyrcona: irclogs has neutral karma.
13:28 Dyrcona @karma logs
13:28 pinesol_green Dyrcona: logs has neutral karma.
13:28 Dyrcona irc_logs++ Bmagic++
13:29 * Dyrcona saves URLs for 3 conversations on Apache load and LVS from May and June.
13:34 jeff hrm. DoB months are off-by-one in the angular patron editor on -- oh, nevermind. new datepicker should fix.
13:35 Dyrcona That reminds me...I had a thought last week to open a Launchpad bug recommending using four digits for the year  in the date picker.
13:36 Dyrcona It's behavior seems to be non-spec, adding 2000 and not 1900 to two-digit years, and DoB is therefore in the future for many patrons.
13:36 Dyrcona I think we should force the entry of four digit years to avoid confusion.
13:37 jeff as someone who was identified as the parent of a 105 year old patient at the hospital a while back, yes please.
13:37 * Dyrcona returns to sifting through the remains of LVS documentation.
13:47 druthb gmcharlt++
14:00 jvwoolf1 joined #evergreen
14:02 Dyrcona @tea [someone]
14:02 * pinesol_green brews and pours a pot of Dragon Pearl Jasmine, and sends it sliding down the bar to sarabee (http://ratetea.com/tea/harney​/dragon-pearl-jasmine/1320/)
14:10 JBoyer @whisky
14:10 pinesol_green JBoyer: http://scientopia.org/blogs/scicu​rious/files/2013/03/cousin-it.png
14:10 JBoyer @whiskey
14:10 pinesol_green JBoyer: Yeah, well, you know, that's just like uh, your opinion, man.
14:10 JBoyer Fail.
14:23 csharp @bartender
14:23 * pinesol_green fills a pint glass with New Belgium 2° Below Winter Ale, and sends it sliding down the bar to csharp (http://beeradvocate.com/beer/profile/192/26542/)
14:23 * csharp slides it further down to JBoyer
14:33 JBoyer Pro-tip: Maybe don't play with config.metabib_field_index_norm_map in the middle of the day. I've heard that can cause trouble.
14:34 csharp heh
14:35 Dyrcona :)
14:36 JBoyer Actually, what I want to do probably requires new entries only for facets anyway. Hmm.
14:37 JBoyer Either that, or get every cataloger in the system to do everything consistently and correct all of the existing records...
14:38 Dyrcona delete cascade from biblio.record_entry; That should do it! ;)
14:39 JBoyer "There! Everything is perfectly consistent and 100% correct." "But we can't do anything!" "Do? But it's right, right now!"
14:41 Dyrcona heh
14:50 Dyrcona Zen and the Art of Database Management. :)
14:52 Stompro Hello everyone, are there any tables like metabib.real_full_rec that have the raw subfield data?  I want to grab data where punctuation is important, I need to see colons, commas and slashes, which get stripped out of metabib.real_full_rec.  Thanks
14:53 Dyrcona Stompro: You could use xpath on the marc from biblio.record_entry.
14:53 Dyrcona That is the most common way to do it with SQL.
14:54 Dyrcona But, to answer your question as stated: No.
14:54 jvwoolf joined #evergreen
14:55 Stompro Dyrcona, thanks, I had forgetten about the xpath method, that should work.
15:12 jwoodard @weather
15:12 pinesol_green jwoodard: Aubrey, TX :: Clear :: 92F/33C | Heat Index: 98F/37C | Thursday: Sunshine and clouds mixed. A stray shower or thunderstorm is possible. High 93F. Winds SSE at 10 to 15 mph. Thursday Night: Partly cloudy skies early followed by scattered thunderstorms overnight. Low around 75F. Winds SSE at 5 to 10 mph. Chance of rain 50%.
15:20 berick @weather
15:20 pinesol_green berick: Error: I did not find a preset location for you. Set via setweather <location>
15:20 berick heh
15:20 berick @weather 27712
15:20 pinesol_green berick: Durham, NC :: Mostly Cloudy :: 94F/34C | Heat Index: 103F/40C | Thursday: Partly cloudy. Lows overnight in the mid 70s. Thursday Night: Some clouds this evening will give way to mainly clear skies overnight. Low around 75F. Winds SW at 5 to 10 mph. | Updated: 24m ago
15:21 berick @eightball should I go outside?
15:21 pinesol_green berick: What are you asking me for?
15:22 Dyrcona @weather
15:22 pinesol_green Dyrcona: Methuen, MA :: Clear :: 95F/35C | Heat Index: 95F/35C | Thursday: A mix of clouds and sun. High 93F. Winds WNW at 5 to 10 mph. Thursday Night: Cloudy. A stray shower or thunderstorm is possible. Low 71F. Winds NNE at 5 to 10 mph.
15:23 berick Dyrcona: dang, did not know it got that hot there
15:23 Dyrcona Me, neither. :)
15:24 Dyrcona Actually on the day we unloaded the truck in July of 2002, it was a record high temperature of over 100 degrees F.
15:24 Bmagic @weather 65203
15:24 pinesol_green Bmagic: Columbia, MO :: Mostly Cloudy :: 86F/30C | Heat Index: 94F/34C | Thursday: Sunshine and clouds mixed. High 87F. Winds NNW at 5 to 10 mph. Thursday Night: Some clouds. Low 67F. Winds light and variable. | Updated: 30m ago
15:48 jeffdavis The SuperCat SRU search retrieves holdings via a call to open-ils.cstore.direct.bib​lio.record_entry.retrieve, fleshing out the record to include copies. This seems to be pretty inefficient. Would it be crazy to rework it to use the unapi functions instead?
15:49 Dyrcona Not crazy in my opinion.
15:49 Dyrcona You should do some profiling to make sure it really does improve performance.
16:33 kmlussier @quote random
16:33 pinesol_green kmlussier: Quote #92: "< bshum> Maybe after I hang up my RM cape, I can put back on my bug wrangler gloves." (added by csharp at 05:34 PM, September 18, 2014)
16:34 bshum Haha
17:05 kmlussier Dyrcona: bug 1501781 is ready to go, right? I want to make sure it makes it for 2.11
17:05 pinesol_green Launchpad bug 1501781 in Evergreen "Patron name search should be diacritic-insensitive" [Wishlist,Confirmed] https://launchpad.net/bugs/1501781
17:07 Dyrcona Um, yes.
17:07 kmlussier OK, thanks! I might look at it tonight. I didn't realize how soon the beta deadline is coming up.
17:09 mmorgan left #evergreen
17:20 jvwoolf left #evergreen
17:23 bmills joined #evergreen
17:38 mrpeters1 joined #evergreen
17:38 mrpeters1 left #evergreen
18:30 kmlussier joined #evergreen
18:32 * kmlussier is adding 2.11 dates to the dev calendar.
18:32 kmlussier I really wish Google calendar wouldn't change 2.11 to 2:11 p.m.
18:33 kmlussier The good news is, by the time we get to the 25.x release series, Google calendar will stop interpreting it as a time.
18:36 dbwells joined #evergreen
19:35 bmills joined #evergreen
22:05 gsams joined #evergreen
23:43 gsams_ joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat