| Time |
Nick |
Message |
| 00:01 |
|
genpaku joined #evergreen |
| 04:15 |
|
dbwells_ joined #evergreen |
| 07:17 |
|
rjackson_isl joined #evergreen |
| 07:56 |
|
ericar joined #evergreen |
| 08:08 |
|
mrpeters joined #evergreen |
| 08:10 |
|
agoben joined #evergreen |
| 08:10 |
|
collum joined #evergreen |
| 08:26 |
|
gsams_ joined #evergreen |
| 08:41 |
|
mmorgan joined #evergreen |
| 08:45 |
|
ericar joined #evergreen |
| 08:52 |
|
bos20k joined #evergreen |
| 09:04 |
|
jwoodard joined #evergreen |
| 09:11 |
|
jvwoolf joined #evergreen |
| 09:11 |
|
maryj joined #evergreen |
| 09:18 |
|
kmlussier joined #evergreen |
| 09:25 |
|
yboston joined #evergreen |
| 09:26 |
|
mce joined #evergreen |
| 09:42 |
|
remingtron joined #evergreen |
| 10:05 |
|
Dyrcona joined #evergreen |
| 10:35 |
Dyrcona |
So, after a "book on the keyboard" incident yesterday, I am looking into ldirectord and LVS, and this stuff smells of bit rot. |
| 10:36 |
Dyrcona |
I also wonder why (though I have ideas) that two out of 5 bricks can go bonkers on running Apache, and the other three sit there looking like they're not even used. |
| 10:39 |
jeff |
what's currently in front of the five bricks? |
| 10:39 |
Dyrcona |
LVS + ldirectord apparently. |
| 10:40 |
jeff |
and what's the current strategy for load balancing requests? are you being bitten by most of your requests coming from a small handful of source IPs and being persisted based on those source IPs? |
| 10:41 |
Dyrcona |
The strategy, I don't know, yet... But the second answer is yes, which is one of my thoughts about why I saw what I saw. |
| 10:41 |
* Dyrcona |
is just learning about this [ancient] stuff. |
| 10:43 |
jeff |
there's more money in selling Application Delivery Controllers than there is in updating documentation for load balancing software. ;-) |
| 10:45 |
Dyrcona |
Well, from what I can see, IPVS is not available past kernel 2.6.x. |
| 10:46 |
Dyrcona |
Maybe I'm missing something, but this project smells funky to me. |
| 10:48 |
Dyrcona |
My goal is to solve the "book on the keyboard" issue spawning so many apache children that the load skyrockets on the brick head, even though individual processes seem to be doing very little...... |
| 10:48 |
Dyrcona |
And I just thought of something.... Number of ejabberd processes.... |
| 10:49 |
* Dyrcona |
tries to check the logs for ejabberd. |
| 10:49 |
Dyrcona |
I managed to tweak MVLC's server so it could still function with over 150 apache drones going, though it looks like one server hit that limit yesterday. |
| 10:50 |
Dyrcona |
That could be part of it, too, configure more apaches to run on the brick heads. |
| 10:54 |
|
Christineb joined #evergreen |
| 10:58 |
Dyrcona |
exception exit: {process_limit,{max_queue,1695}} in erlang.log |
| 10:58 |
Dyrcona |
Ah, but for the wrong day.... Never mind that. |
| 10:58 |
Dyrcona |
Yesterday had a lot of timeouts. |
| 10:59 |
Dyrcona |
That herring is definitely red. |
| 11:01 |
Bmagic |
Dyrcona: We recently had to switch to centos for our load balancer, because my research lead me to the same issue. We used this to help get it setup correctly |
| 11:01 |
Bmagic |
https://siva2009.wordpress.com/2013/12/03/setup-load-balancing-cluster-with-lvs-and-piranha-on-centos-6/ |
| 11:03 |
Bmagic |
Then again recently, (I believe we had a brief discussion here in channel) we had to change some settings on LVS in order to get those connections closed sooner. App servers were spiraling out of control over a magic number of connections |
| 11:03 |
Dyrcona |
Yep, I recall that conversation. |
| 11:04 |
Dyrcona |
Bmagic: Any particular reason to use Centos over Debian? |
| 11:04 |
Bmagic |
The article had it tried and true for centos |
| 11:04 |
Bmagic |
And I happen to like Redhat stuff |
| 11:04 |
Dyrcona |
OK. Just wondered if anything else came up in your research. |
| 11:05 |
Bmagic |
It comes with a handy UI for editing the config as well |
| 11:05 |
miker |
Dyrcona: ipvsadm and ldirector are certainly in modern distros and work on modern kernels: http://manpages.ubuntu.com/manpages/xenial/man8/ldirectord.8.html http://manpages.ubuntu.com/manpages/xenial/man8/ipvsadm.8.html |
| 11:07 |
Dyrcona |
miker: Thanks. I saw some reference in the Ubuntu man pages, but most of the stuff turned up by Google is old. |
| 11:07 |
Dyrcona |
My problem isn't load balancing anyway. |
| 11:08 |
* miker |
attributes that to ldirectord+ipvsadm being a "solved problem" |
| 11:08 |
Dyrcona |
It's too many apache processes spawning. |
| 11:08 |
Dyrcona |
I was mainly looking to find a way to limit that in ldirectord if possible. |
| 11:09 |
miker |
Dyrcona: re, the book-on-keyboard issue, we actively stop duplicate searches inside apache, but yes, they still eat memory. they won't kill the db anymore, though :) |
| 11:09 |
Dyrcona |
Load balancing is a solved problem, and there are other options. |
| 11:09 |
Dyrcona |
miker: Right, the database isn't the issue. It's Apache spinning out of control and two brick heads becoming useless. |
| 11:09 |
miker |
that'll likely be an iptables or ipvsadm thing |
| 11:10 |
Dyrcona |
I dealt this with this to some extent at MVLC by tweaking Apache configurations. Now I'm in a different environment. |
| 11:10 |
miker |
as in, one could say "no more than X concurrent connections from the same IP within Y time" |
| 11:10 |
miker |
but that will then lock out the library with the book on a keyboard |
| 11:11 |
miker |
so... it's not a simple problem. which is why it's yet to be solved |
| 11:11 |
Dyrcona |
Right. Ideally, I'd like an Apache solution to basically cutoff repeated requests for the same url, including query, etc. |
| 11:11 |
miker |
that's there already (for searches, at least, which is the worst-offender for book-on-keyboard, IME) |
| 11:12 |
miker |
but by the time apache has spun up a backend far enough to detect that, mod_perl et al are loaded and the memory is eaten |
| 11:12 |
Dyrcona |
Um, I don't see that at all. Looking through the apache logs the exact same url is repeated many times in a row, and it search results. |
| 11:12 |
Dyrcona |
Oh. Right... ;) |
| 11:13 |
Dyrcona |
I could probably do it with an apache proxy, maybe with a combination of mod_proxy_balancer..... |
| 11:13 |
miker |
that's a thought (well, nginx) we've had, too. |
| 11:14 |
Dyrcona |
In my particular case, I see the non-routeable IPs for the computers at libraries because of a VPN, so I might be able to do something ipvsadm to limit connections per second. |
| 11:15 |
miker |
that adds several moving parts, with attendant issues ... and is itself pretty nontrival, and in reality, site specific |
| 11:15 |
miker |
Dyrcona: indeed! |
| 11:15 |
miker |
you could even make it a static rule for internal IPs known to be public opac stations |
| 11:16 |
Dyrcona |
I'll read up on how ipvsadm works. |
| 11:16 |
Dyrcona |
Thanks! |
| 11:16 |
Dyrcona |
miker++ Bmagic++ # for suggestions |
| 11:17 |
miker |
Dyrcona: it'll probably be iptables rather than ipvsadm, btw, for the conntrack-based limiting |
| 11:17 |
JBoyer |
I'm certainly not volunteering to figure it out at the moment, but what would be required to cut the per-process mem use so this isn't a problem anymore? |
| 11:17 |
Dyrcona |
I should experiment with the proxy solution some day, too. I've always wanted to play with nginx but never made the time. |
| 11:17 |
Dyrcona |
JBoyer: Serve static pages. :) |
| 11:18 |
miker |
just about, yeah :) |
| 11:18 |
JBoyer |
That's one end of the spectrum, yes. ;) |
| 11:19 |
miker |
shard all our mod_perl and gateway/translator stuff into a tonne of apache instances, and mod_proxy them inside ... like we do with websockets now (sans mod_proxy) |
| 11:20 |
miker |
there will be some lower bound on memory, though. mod_perl just ain't cheap, in a retail sense |
| 11:20 |
Dyrcona |
About iptables: One of the systems staff here is going to see if the current firewall has any QoS features. |
| 11:21 |
Dyrcona |
Well, it's tricky to control memory use in Perl anyway. |
| 11:21 |
Dyrcona |
mod_perl and the persistence of Apache processes makes it more challenging. |
| 11:21 |
miker |
Dyrcona: even if it doesn't, the LB can mark packets based on IP, and later rules can do QoS |
| 11:22 |
Dyrcona |
Right. I should be able to do it with iptables. |
| 11:23 |
JBoyer |
That's about what I thought re: mem. Shame it's not easier. (though I suppose we'd have already done it if so...) |
| 11:32 |
|
brahmina joined #evergreen |
| 11:33 |
|
bmills joined #evergreen |
| 11:35 |
Dyrcona |
JBoyer: There are some things that we can do, probably, but it won't be easy to identify them or to fix them once identified. |
| 11:57 |
dbwells |
Heads up, y'all. Yesterday was the cutoff for the 2.11 alpha, so I am in the process of rolling the release. I am not thinking this affects much, unless you are reviewing something you really want in the alpha. |
| 12:08 |
|
jihpringle joined #evergreen |
| 12:20 |
Dyrcona |
dbwells++ |
| 12:22 |
gmcharlt |
kmlussier: I've pushed a fix for that remaining issue for LP#154505 |
| 12:22 |
gmcharlt |
er, 1549505 |
| 12:22 |
kmlussier |
bug 1549505 |
| 12:22 |
pinesol_green |
Launchpad bug 1549505 in Evergreen "Statistically generated record ratings" [Wishlist,New] https://launchpad.net/bugs/1549505 |
| 12:22 |
kmlussier |
Ah, ok. Thanks! |
| 12:22 |
kmlussier |
gmcharlt++ |
| 12:22 |
* kmlussier |
now ponders how she will test it. |
| 12:52 |
jeff |
anyone ever have the desire to mark patron stat cats as read-only? |
| 12:52 |
jeff |
and "show only if already set"? |
| 12:53 |
jeff |
mostly thinking of migrated/legacy stat cats. |
| 12:54 |
Dyrcona |
jeff: More or less. |
| 12:54 |
Dyrcona |
Same reasons, too, most likely. |
| 12:55 |
Dyrcona |
We migrated some things from the legacy ILS as stat cats so we could identify certain groups of patrons if need be. |
| 12:55 |
* jeff |
nods |
| 12:55 |
Dyrcona |
They do nothing at all otherwise, and some staff add them to new patrons. |
| 12:56 |
jeff |
hrm. new patron editor doesn't let you unset a non-freeform stat cat. |
| 12:56 |
* jeff |
checks old patron editor |
| 12:56 |
Dyrcona |
Err, "we" should probably be MVLC. :) |
| 12:56 |
jeff |
heh |
| 12:56 |
|
hbrennan joined #evergreen |
| 12:56 |
* Dyrcona |
is still in a mental transition state from MVLC to C/W MARS. |
| 13:07 |
|
maryj joined #evergreen |
| 13:27 |
Dyrcona |
@karma irc_logs |
| 13:27 |
pinesol_green |
Dyrcona: irc_logs has neutral karma. |
| 13:28 |
Dyrcona |
@karma irclogs |
| 13:28 |
pinesol_green |
Dyrcona: irclogs has neutral karma. |
| 13:28 |
Dyrcona |
@karma logs |
| 13:28 |
pinesol_green |
Dyrcona: logs has neutral karma. |
| 13:28 |
Dyrcona |
irc_logs++ Bmagic++ |
| 13:29 |
* Dyrcona |
saves URLs for 3 conversations on Apache load and LVS from May and June. |
| 13:34 |
jeff |
hrm. DoB months are off-by-one in the angular patron editor on -- oh, nevermind. new datepicker should fix. |
| 13:35 |
Dyrcona |
That reminds me...I had a thought last week to open a Launchpad bug recommending using four digits for the year in the date picker. |
| 13:36 |
Dyrcona |
It's behavior seems to be non-spec, adding 2000 and not 1900 to two-digit years, and DoB is therefore in the future for many patrons. |
| 13:36 |
Dyrcona |
I think we should force the entry of four digit years to avoid confusion. |
| 13:37 |
jeff |
as someone who was identified as the parent of a 105 year old patient at the hospital a while back, yes please. |
| 13:37 |
* Dyrcona |
returns to sifting through the remains of LVS documentation. |
| 13:47 |
druthb |
gmcharlt++ |
| 14:00 |
|
jvwoolf1 joined #evergreen |
| 14:02 |
Dyrcona |
@tea [someone] |
| 14:02 |
* pinesol_green |
brews and pours a pot of Dragon Pearl Jasmine, and sends it sliding down the bar to sarabee (http://ratetea.com/tea/harney/dragon-pearl-jasmine/1320/) |
| 14:10 |
JBoyer |
@whisky |
| 14:10 |
pinesol_green |
JBoyer: http://scientopia.org/blogs/scicurious/files/2013/03/cousin-it.png |
| 14:10 |
JBoyer |
@whiskey |
| 14:10 |
pinesol_green |
JBoyer: Yeah, well, you know, that's just like uh, your opinion, man. |
| 14:10 |
JBoyer |
Fail. |
| 14:23 |
csharp |
@bartender |
| 14:23 |
* pinesol_green |
fills a pint glass with New Belgium 2° Below Winter Ale, and sends it sliding down the bar to csharp (http://beeradvocate.com/beer/profile/192/26542/) |
| 14:23 |
* csharp |
slides it further down to JBoyer |
| 14:33 |
JBoyer |
Pro-tip: Maybe don't play with config.metabib_field_index_norm_map in the middle of the day. I've heard that can cause trouble. |
| 14:34 |
csharp |
heh |
| 14:35 |
Dyrcona |
:) |
| 14:36 |
JBoyer |
Actually, what I want to do probably requires new entries only for facets anyway. Hmm. |
| 14:37 |
JBoyer |
Either that, or get every cataloger in the system to do everything consistently and correct all of the existing records... |
| 14:38 |
Dyrcona |
delete cascade from biblio.record_entry; That should do it! ;) |
| 14:39 |
JBoyer |
"There! Everything is perfectly consistent and 100% correct." "But we can't do anything!" "Do? But it's right, right now!" |
| 14:41 |
Dyrcona |
heh |
| 14:50 |
Dyrcona |
Zen and the Art of Database Management. :) |
| 14:52 |
Stompro |
Hello everyone, are there any tables like metabib.real_full_rec that have the raw subfield data? I want to grab data where punctuation is important, I need to see colons, commas and slashes, which get stripped out of metabib.real_full_rec. Thanks |
| 14:53 |
Dyrcona |
Stompro: You could use xpath on the marc from biblio.record_entry. |
| 14:53 |
Dyrcona |
That is the most common way to do it with SQL. |
| 14:54 |
Dyrcona |
But, to answer your question as stated: No. |
| 14:54 |
|
jvwoolf joined #evergreen |
| 14:55 |
Stompro |
Dyrcona, thanks, I had forgetten about the xpath method, that should work. |
| 15:12 |
jwoodard |
@weather |
| 15:12 |
pinesol_green |
jwoodard: Aubrey, TX :: Clear :: 92F/33C | Heat Index: 98F/37C | Thursday: Sunshine and clouds mixed. A stray shower or thunderstorm is possible. High 93F. Winds SSE at 10 to 15 mph. Thursday Night: Partly cloudy skies early followed by scattered thunderstorms overnight. Low around 75F. Winds SSE at 5 to 10 mph. Chance of rain 50%. |
| 15:20 |
berick |
@weather |
| 15:20 |
pinesol_green |
berick: Error: I did not find a preset location for you. Set via setweather <location> |
| 15:20 |
berick |
heh |
| 15:20 |
berick |
@weather 27712 |
| 15:20 |
pinesol_green |
berick: Durham, NC :: Mostly Cloudy :: 94F/34C | Heat Index: 103F/40C | Thursday: Partly cloudy. Lows overnight in the mid 70s. Thursday Night: Some clouds this evening will give way to mainly clear skies overnight. Low around 75F. Winds SW at 5 to 10 mph. | Updated: 24m ago |
| 15:21 |
berick |
@eightball should I go outside? |
| 15:21 |
pinesol_green |
berick: What are you asking me for? |
| 15:22 |
Dyrcona |
@weather |
| 15:22 |
pinesol_green |
Dyrcona: Methuen, MA :: Clear :: 95F/35C | Heat Index: 95F/35C | Thursday: A mix of clouds and sun. High 93F. Winds WNW at 5 to 10 mph. Thursday Night: Cloudy. A stray shower or thunderstorm is possible. Low 71F. Winds NNE at 5 to 10 mph. |
| 15:23 |
berick |
Dyrcona: dang, did not know it got that hot there |
| 15:23 |
Dyrcona |
Me, neither. :) |
| 15:24 |
Dyrcona |
Actually on the day we unloaded the truck in July of 2002, it was a record high temperature of over 100 degrees F. |
| 15:24 |
Bmagic |
@weather 65203 |
| 15:24 |
pinesol_green |
Bmagic: Columbia, MO :: Mostly Cloudy :: 86F/30C | Heat Index: 94F/34C | Thursday: Sunshine and clouds mixed. High 87F. Winds NNW at 5 to 10 mph. Thursday Night: Some clouds. Low 67F. Winds light and variable. | Updated: 30m ago |
| 15:48 |
jeffdavis |
The SuperCat SRU search retrieves holdings via a call to open-ils.cstore.direct.biblio.record_entry.retrieve, fleshing out the record to include copies. This seems to be pretty inefficient. Would it be crazy to rework it to use the unapi functions instead? |
| 15:49 |
Dyrcona |
Not crazy in my opinion. |
| 15:49 |
Dyrcona |
You should do some profiling to make sure it really does improve performance. |
| 16:33 |
kmlussier |
@quote random |
| 16:33 |
pinesol_green |
kmlussier: Quote #92: "< bshum> Maybe after I hang up my RM cape, I can put back on my bug wrangler gloves." (added by csharp at 05:34 PM, September 18, 2014) |
| 16:34 |
bshum |
Haha |
| 17:05 |
kmlussier |
Dyrcona: bug 1501781 is ready to go, right? I want to make sure it makes it for 2.11 |
| 17:05 |
pinesol_green |
Launchpad bug 1501781 in Evergreen "Patron name search should be diacritic-insensitive" [Wishlist,Confirmed] https://launchpad.net/bugs/1501781 |
| 17:07 |
Dyrcona |
Um, yes. |
| 17:07 |
kmlussier |
OK, thanks! I might look at it tonight. I didn't realize how soon the beta deadline is coming up. |
| 17:09 |
|
mmorgan left #evergreen |
| 17:20 |
|
jvwoolf left #evergreen |
| 17:23 |
|
bmills joined #evergreen |
| 17:38 |
|
mrpeters1 joined #evergreen |
| 17:38 |
|
mrpeters1 left #evergreen |
| 18:30 |
|
kmlussier joined #evergreen |
| 18:32 |
* kmlussier |
is adding 2.11 dates to the dev calendar. |
| 18:32 |
kmlussier |
I really wish Google calendar wouldn't change 2.11 to 2:11 p.m. |
| 18:33 |
kmlussier |
The good news is, by the time we get to the 25.x release series, Google calendar will stop interpreting it as a time. |
| 18:36 |
|
dbwells joined #evergreen |
| 19:35 |
|
bmills joined #evergreen |
| 22:05 |
|
gsams joined #evergreen |
| 23:43 |
|
gsams_ joined #evergreen |
