| Time |
Nick |
Message |
| 03:09 |
|
gsams joined #evergreen |
| 03:40 |
|
gsams joined #evergreen |
| 03:41 |
|
dbwells_ joined #evergreen |
| 03:59 |
|
Ewan_ joined #evergreen |
| 04:06 |
Ewan_ |
Hello, I have a question: is my /openils/var/web supposed to have 3 Makefiles in it when evergreen is installed? |
| 04:09 |
|
dcook joined #evergreen |
| 06:52 |
|
gsams_ joined #evergreen |
| 07:15 |
|
rjackson_isl joined #evergreen |
| 07:25 |
graced |
@later tell Ewan_ that most Evergreeners are in channel between 8:00am and 7:00pm EST |
| 07:25 |
pinesol_green |
graced: The operation succeeded. |
| 07:26 |
druthb |
@tea |
| 07:26 |
* pinesol_green |
brews and pours a pot of Huang Shan Mao Feng Green Tea, and sends it sliding down the bar to druthb (http://ratetea.com/tea/teavivre/huang-shan-mao-feng-green-tea/6041/) |
| 07:27 |
druthb |
that'll do. |
| 08:02 |
|
dcook__ joined #evergreen |
| 08:04 |
|
mrpeters joined #evergreen |
| 08:09 |
|
ericar joined #evergreen |
| 08:14 |
|
Dyrcona joined #evergreen |
| 08:20 |
bshum |
Happy sys admin day out there! :) hope you all get some cake (and/or other treats) |
| 08:25 |
* csharp |
settles the usual patting self on the back |
| 08:26 |
csharp |
settles *for* |
| 08:36 |
JBoyer |
Ho, ho, ho! Merry... wait, no, that's something else. |
| 08:36 |
JBoyer |
Feliz Sysadmin... also no. |
| 08:37 |
JBoyer |
Congratulations on sysadmin-ing for another year! Here's to more of that. |
| 08:37 |
JBoyer |
Close enough. |
| 08:38 |
|
mmorgan joined #evergreen |
| 08:38 |
csharp |
JBoyer++ |
| 08:39 |
Dyrcona |
heh |
| 08:40 |
mmorgan |
@coffee sysadmins |
| 08:40 |
* pinesol_green |
brews and pours a cup of Espresso Nuevo, and sends it sliding down the bar to sysadmins |
| 08:41 |
* Dyrcona |
wonders if being "DevOps" counts. :) |
| 08:47 |
* csharp |
starts flamewar about whether DevOps counts |
| 08:50 |
* Dyrcona |
launches a volley of Greek Fire at csharp over the bow. |
| 08:55 |
|
kmlussier joined #evergreen |
| 08:59 |
|
ericar_ joined #evergreen |
| 09:06 |
|
maryj joined #evergreen |
| 09:07 |
kmlussier |
@dessert 41 bshum |
| 09:07 |
* pinesol_green |
grabs some birthday cake with lots of candles for bshum |
| 09:14 |
Dyrcona |
I imagine Bmagic and JBoyer may not be around yet because timezones. |
| 09:14 |
Dyrcona |
Oh, yeah... Happy Birthday, bshum! |
| 09:14 |
* Dyrcona |
waits to bug them about timeouts in ipvs. |
| 09:15 |
* Dyrcona |
wants to reopen a conversation from June 3. |
| 09:15 |
Dyrcona |
Oh wait. Duh. JBoyer is here already.... |
| 09:15 |
JBoyer |
Not at all, while Indiana has lost it's way RE: DST, most of us are still within the comforting confines of the Eastern Timezone. |
| 09:15 |
* Dyrcona |
has a stupid..... |
| 09:16 |
Dyrcona |
Bmagic can hopefully pick up when he shows up, then. |
| 09:17 |
|
jvwoolf joined #evergreen |
| 09:17 |
Dyrcona |
Back on June 3, http://irc.evergreen-ils.org/evergreen/2016-06-03#i_252050, the three of us were talking about ipvsadm and Apache settings. |
| 09:17 |
Dyrcona |
I was mostly talking about Apache. |
| 09:18 |
Dyrcona |
Anyway, Bmagic shared his changes to ipvsadm timeout values that improved performance and JBoyer suggested slight different values. |
| 09:18 |
Dyrcona |
We're currently using the defaults for the three timeouts and I'd like to know if anyone has suggestions for better values to use. |
| 09:19 |
Dyrcona |
It seems like JBoyer recommended 60 to 120 seconds, and Bmagic dropped some to 30 seconds. |
| 09:19 |
JBoyer |
I suggested trying that, but ours are also still on the defaults |
| 09:20 |
JBoyer |
Because testing is an enormous PITA since you never know when things will happen. :( |
| 09:21 |
Dyrcona |
Bmagic mentions issues with sip that might be related to the 30 second values. |
| 09:22 |
Dyrcona |
Before I do something drastic with connection limiting via a firewall, I'd like to try load balancer changes. |
| 09:22 |
JBoyer |
That's a good plan. |
| 09:23 |
Dyrcona |
My concern is that I'll probably end up with firewall changes anyway since the problem happens in environments without a load balancer. |
| 09:23 |
JBoyer |
I wonder if our coordinator would like me playing with values semi-randomly during a 2-library migration. |
| 09:23 |
Dyrcona |
Ha ha! I'm sure that would be no problem, JBoyer. ;) |
| 09:24 |
JBoyer |
Dyrcona, that does sound like you're going to end up having to go that route. The book on the keyboard problem? |
| 09:24 |
Dyrcona |
Yes. |
| 09:24 |
JBoyer |
Yeah, I can't imagine an LB actually helping with that. Maybe one of those web application firewalls or something that can look at the url before it hits apache, but $$$ |
| 09:24 |
Dyrcona |
Another question, is anyone using something other than round robin for the scheduler? |
| 09:25 |
Dyrcona |
A "web application firewall" is a fancy name for a proxy server. :) |
| 09:25 |
Dyrcona |
I can build one of those for nothing with Apache or nginx. |
| 09:26 |
JBoyer |
I'm using weighted least connections since I've got unevenly spec'd app servers. |
| 09:26 |
Dyrcona |
I've got evenly spec'd app servers, but I was still thinking of using that one. |
| 09:26 |
JBoyer |
Dyrcona, that's true. The $$$ kind has been on my mind because that's what our central IT agency is putting everyone behind. |
| 09:26 |
Dyrcona |
It might help with 1 and 2 being superloaded and the other three doing next to nothing, but not sure. |
| 09:27 |
Dyrcona |
Ah, yes, central IT often doesn't get Free, so $$$ still win. |
| 09:28 |
Dyrcona |
Gov't: where budgets are power. |
| 09:28 |
JBoyer |
I could complain for days, but it wouldn't stop apache from falling over, so... |
| 09:32 |
|
ericar_ joined #evergreen |
| 09:32 |
Dyrcona |
Yep. |
| 09:33 |
Dyrcona |
This has been very helpful. I have a better grasp on what is going on with LVS and load balancing and what my options are. |
| 09:34 |
|
yboston joined #evergreen |
| 09:34 |
JBoyer |
Good. It has a lot of interesting options (I really like maintenancedir, since SIP servers don't have the equiv of a ping file really) but you're right, it hasn't been touched much in ages. |
| 09:35 |
JBoyer |
Neither have the docs or howtos... |
| 09:35 |
Dyrcona |
I guess if it ain't broke.... ;) |
| 09:36 |
JBoyer |
True enough for the code, but I'd feel better if there were more docs written in this decade. ;) |
| 09:36 |
* Dyrcona |
wishes he had time to experiment with different load balancing options, i.e. relayd and the others. |
| 09:37 |
Dyrcona |
Yeah, kinda scary when you do the google search and one of the top 3 results is from 1998. :) |
| 09:37 |
jeff |
were this another channel, @band add relayd and the others |
| 09:37 |
* JBoyer |
wishes he had time to experiment. :-/ |
| 09:37 |
* jeff |
experiments with time |
| 09:38 |
Dyrcona |
jeff: Watch out! Those experiments can get you into some trouble. ;) |
| 09:38 |
jeff |
anyone have interest in logging changes to SIPServer? i'm thinking of adding a connection id and logging username/workstation (after that patch is in), etc. |
| 09:39 |
jeff |
actually, thinking of adding a new log line for each request. |
| 09:39 |
jeff |
not just adjusting the ones that are there. |
| 09:39 |
JBoyer |
Oh, that reminds me, I think I've got pullrequest tags on both of those... |
| 09:40 |
Dyrcona |
jeff: Sounds useful. I'd certainly be willing to look at it. |
| 09:40 |
JBoyer |
Ah, yes. |
| 09:41 |
JBoyer |
jeff: what kind of conn id do you mean? something that better ties src and dest together, or something else? |
| 09:42 |
jeff |
JBoyer: just a made up number that you can use to distinguish two messages that may otherwise share the same institution/username/workstation |
| 09:43 |
jeff |
JBoyer: also possibly a distinct id for backends in a prefork environment. |
| 09:43 |
JBoyer |
Oh, ok. So you don't have to say "Is this message being logged twice, or sent twice?" That kind of thing? |
| 09:43 |
jeff |
that specific use case hadn't occurred to me. |
| 09:44 |
JBoyer |
There are some creative clients. |
| 09:46 |
jeff |
in the example you gave, i'd start with "sent twice". nice thing there is that i could look at the code to pretty much rule out "logged twice" |
| 09:46 |
JBoyer |
Sure. |
| 09:47 |
jeff |
though that's not to say that there couldn't be a subtle bug lurking in Net::Server::Prefork that could cause double-logging. |
| 09:47 |
jeff |
but i'd expect that kind of bug to also involve double-handling. :-) |
| 09:47 |
JBoyer |
or rsyslog when sending it to a central logging server, etc. ;) |
| 09:47 |
* jeff |
nods |
| 09:48 |
jeff |
so yes, in an example where you are questioning your logs a connection id could be a useful additional bit of distinguishing information in addition to the timestamp. |
| 09:48 |
jeff |
(or timestamps, rather) |
| 09:50 |
Dyrcona |
Previous message repeated X times. |
| 09:50 |
Dyrcona |
;) |
| 09:51 |
JBoyer |
I loosened the throttle on that a fair bit. Don't remember the reason off hand though. |
| 09:54 |
Dyrcona |
I thought of another question regarding ipvs/ldirector and Apache. |
| 09:54 |
Dyrcona |
How does Apache's keep alive come into play? |
| 09:55 |
Dyrcona |
We apparently are not persisting http and https through ipvs, so do established connections bypass the load balancer? |
| 09:55 |
Dyrcona |
Oh, they must.... |
| 09:55 |
Dyrcona |
It's tcp. |
| 09:55 |
Dyrcona |
Never mind, a little thought goes a long way. :) |
| 09:57 |
Dyrcona |
Hmm... That may complicate things if multiple requests come over the same tcp connection..... |
| 09:58 |
jeff |
You could consider MaxKeepAliveRequests as a useful limiter there. |
| 09:59 |
jeff |
But if you're adding mod_perl logic to classify requests you can probably set Connection: close before Apache normally would... |
| 09:59 |
jeff |
I'm not actually sure how Apache would handle that. |
| 09:59 |
jeff |
It might be on the client to honor it, and it might not. |
| 09:59 |
jeff |
I've not had reason to test that. |
| 10:00 |
Dyrcona |
I'm thinking of the book on the keyboard, and the constant stream of requests. |
| 10:00 |
jeff |
right. |
| 10:01 |
Dyrcona |
If they're all coming over the same tcp connection, then a firewall rule isn't gonna help. :( |
| 10:03 |
* Dyrcona |
is also reading https://tools.ietf.org/html/rfc7230#section-6.3 |
| 10:04 |
Dyrcona |
jeff: I think I would have to send Connection: close after each response as you suggest, or just disable persistent connection in Apache. |
| 10:05 |
Dyrcona |
And, I think either would likely cause a performance hit in general use that outweighs the benefit of fixing the occasional problem from multiple requests from the same client. |
| 10:05 |
jeff |
Which would long-term have greater detriment on your performance than occasional book-on-keyboard incidents. |
| 10:05 |
* jeff |
nods |
| 10:05 |
Dyrcona |
I think so. |
| 10:06 |
jeff |
I can think of a few options that don't exist yet... |
| 10:07 |
Dyrcona |
I could run wireshark to see what is really happening with connections. |
| 10:07 |
miker |
Dyrcona: if the book-on-keyboard requests were coming in through one connections, you wouldn't have the apache-spawn-bomb |
| 10:07 |
jeff |
miker++ |
| 10:07 |
miker |
so, keepalive isn't an issue for that case |
| 10:07 |
Dyrcona |
miker: I was just thinking that before you typed it. |
| 10:08 |
Dyrcona |
miker++ |
| 10:08 |
Dyrcona |
Or, rather, as you typed it. |
| 10:09 |
Dyrcona |
Firewall rules will work. |
| 10:09 |
miker |
what would be really great is if we could cut a pending request off at the knees if the browser disconnects, which, in book-on-keyboard, it should |
| 10:09 |
miker |
and kill the apache backend in that case |
| 10:09 |
miker |
s/pending/in-process |
| 10:10 |
jeff |
related to the "naming things" problem: "finding the thing in a list of other things after you've named the thing" |
| 10:21 |
Dyrcona |
miker: It would be nice if there were a HTTP response for a duplicate request.... |
| 10:22 |
miker |
HTTP 420: go home, client, you're high |
| 10:23 |
Dyrcona |
heh. |
| 10:23 |
Dyrcona |
406 or 409 might do... but a duplicate is not normally an error. |
| 10:29 |
JBoyer |
HTTP:409 Clean up your mess |
| 10:30 |
Dyrcona |
409 Conflict |
| 10:30 |
Dyrcona |
406 Not Acceptable |
| 10:31 |
Dyrcona |
"You're behavior is not acceptable, young web client...." |
| 10:31 |
Dyrcona |
Gah. Wrong your.... |
| 10:31 |
Dyrcona |
Have I mentioned I hate English, lately? |
| 10:31 |
JBoyer |
I was thinking more https://www.google.com/search?q=formula+409&tbm=isch but cleaning the junk out of Google's URL made me too slow. :D |
| 10:32 |
Dyrcona |
Ha! That's good. |
| 10:32 |
Dyrcona |
We should use 409 and one of those graphics on the response page. :) |
| 10:32 |
JBoyer |
joeks++ |
| 10:49 |
|
maryj joined #evergreen |
| 10:55 |
Dyrcona |
Guess I am not going to find the Google searches that I did approximately two months ago on a different computer. |
| 10:55 |
JBoyer |
Well, if you're logged in and have search history turned on... |
| 10:56 |
JBoyer |
(That skeeved me out the instant it was announced, the first thing it logged me searching was likely how to turn it off) |
| 10:56 |
JBoyer |
Oh, Web History, that's what I'm thinking of. Bing calls it Search History. Something. |
| 10:58 |
csharp |
JBoyer: re: allowing more "repeated" log messages, as I recall, you recommended that because messages were getting plain old dropped with that setting set too strictly (i.e., the default setting) |
| 10:58 |
JBoyer |
These keyword searches for -1' make me think that a very basic, fairly loose JS query quality checker would make a lot of slow/poor searches go away without any fancy config. :-/ |
| 10:59 |
JBoyer |
csharp, Oh, right. And I couldn't tell what was happening for a while because the message about things being dropped was logged locally on the machine, while only the logs that made it through went to the central server. |
| 11:02 |
|
jihpringle joined #evergreen |
| 11:05 |
Dyrcona |
Yeah, but I'm not seeing it. The new "My Activity" interface is kind of bad. |
| 11:06 |
Dyrcona |
and, yeah, I saw something like that with dropped messages on rsyslogd before. |
| 11:09 |
Bmagic |
Anyone have any call from their libraries to capture who and where patrons are registered? |
| 11:13 |
mmorgan |
Bmagic: We have the occasional "who" question, the "where" question is important to us for statistical purposes. We currently use Home Library to determine the "where", though it's imperfect. |
| 11:14 |
Bmagic |
mmorgan: yeah, we have a library system with 9 branches. The home library doesn't satisfy their need. And who registered the patron is just a gaping hole right? |
| 11:16 |
mmorgan |
Bmagic: Yes, if there's a clear record of who edited a patron, there oughta be a record of who created it. |
| 11:20 |
Bmagic |
bug 1607827 |
| 11:20 |
pinesol_green |
Launchpad bug 1607827 in Evergreen "Need Evergreen to capture who and where patrons are registered" [Undecided,New] https://launchpad.net/bugs/1607827 |
| 11:34 |
|
bmills joined #evergreen |
| 11:35 |
Dyrcona |
More on the "book on keyboard" issue: |
| 11:36 |
Dyrcona |
I'm consideing mod_qos or similar to limit a client to 1 connection per second to search results or something like that. |
| 11:36 |
Dyrcona |
Options. |
| 11:38 |
|
eady joined #evergreen |
| 11:41 |
miker |
Dyrcona: using a non-mod_perl proxy instance? |
| 11:41 |
Dyrcona |
miker: Not sure. I'm just kind of thinking out loud. |
| 11:43 |
Dyrcona |
Guess it would have to be through a proxy to be effective. |
| 11:45 |
|
brahmina joined #evergreen |
| 12:04 |
|
maryj_ joined #evergreen |
| 12:42 |
|
jvwoolf joined #evergreen |
| 13:02 |
|
gsams joined #evergreen |
| 13:17 |
|
maryj joined #evergreen |
| 13:18 |
miker |
Dyrcona: lunch time thought experiment: http://nox.esilibrary.com/~miker/stash-apache-requestrec.evergreen.patch + http://nox.esilibrary.com/~miker/test-apache-requestrec.opensrf.patch |
| 13:22 |
Dyrcona |
miker: Looks interesting. |
| 13:22 |
JBoyer |
I love a parallel bib load on migration day. 67K records, < 45 minutes. *sunglasses-emoji* |
| 13:24 |
Dyrcona |
heh |
| 14:08 |
jeff |
found the basis for our new server naming scheme: http://mewo2.com/notes/naming-language/ |
| 14:13 |
|
ssss joined #evergreen |
| 14:18 |
|
bmills left #evergreen |
| 14:30 |
csharp |
jeff++ |
| 14:31 |
* csharp |
logs into pumkulnukkulkuv01 |
| 14:32 |
jeff |
pretty limiting. i think you're going to want pumkulnukkulkuv0001 to leave room for growth. |
| 14:32 |
bshum |
o.O |
| 14:33 |
Dyrcona |
:) |
| 14:34 |
Dyrcona |
I wrote a character name generator based on Jack Vance's The Dying Earth series. |
| 14:36 |
Dyrcona |
I was going to use it in an online Neverwinter Nights module that never completely panned out. |
| 15:08 |
jeff |
fatal: [eg-siptest01.REDACTED]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'eg_memcache_ip' is undefined"} |
| 15:09 |
jeff |
"hey, i see what you're trying to do and all, but i'm going to need just a BIT more to work with here." |
| 15:12 |
jeff |
that said, I really appreciate the "mandatory" filter and the fact that the behavior defaults to on to begin with. |
| 15:13 |
JBoyer |
Yeah, I'm not a fan of the "Huh, haven't seen that variable name before, it must be equal to ''" pattern that has infected so many languages/frameworks. |
| 15:27 |
|
hbrennan joined #evergreen |
| 15:39 |
jeff |
I'm pretty sure that this product's SIP2 support is purely based on pattern matching. |
| 15:41 |
jeff |
That is, pattern matching against the raw SIP2 messages, no other attempt made to parse. |
| 15:43 |
Dyrcona |
jeff: Now, you've got two problems. :) |
| 15:53 |
Dyrcona |
So this is fun. I think I've found the parts of the log where a problem occurred on Wednesday, 'cause I've got searches returning 500 after about two minutes in the logs. |
| 15:54 |
Dyrcona |
However, I don't see any signs, yet, of "book on the keyboard" type situations. |
| 15:57 |
JBoyer |
jeff: re sip pattern matching, It didn't see a CO in an address and decide that was a new field, did it? I saw that once and was very, very amused. (except that we're still paying for that product...) |
| 15:57 |
jeff |
hah |
| 15:58 |
jeff |
i was thinking of a few cases where you could confuse that kind of code, and that specific example had not occurred to me. |
| 15:58 |
jeff |
you may recall the situation where a barcode with AY in it caused us some fun times, though. |
| 15:59 |
JBoyer |
Well, given that 90% of libraries still ENTER PATRON INFORMATION LIKE THIS and SIP field codes are just 2 uppercase characters... |
| 16:02 |
Dyrcona |
So, now, you've got three problems. :) |
| 16:02 |
Dyrcona |
I think lines like this might have something to do with my lack of evidence: rsyslogd-2177: imuxsock lost 195 messages from pid 4910 due to rate-limiting |
| 16:03 |
JBoyer |
Wow. Something was feeling pretty chatty and told to just zip it. |
| 16:06 |
Dyrcona |
Probably an Apache process trying to log all the requests it was receiving for the same URL, but I'm guessing. |
| 16:06 |
Dyrcona |
I'll look into the syslog rate limit settings on Monday. Bit late in the day to mess with that. |
| 16:07 |
Dyrcona |
I see similar messages throughout the day in the logs on the brick heads. |
| 16:07 |
JBoyer |
I can send you what we're doing just as a starting point, but yeah, I wouldn't do anything with it until Monday, heh. |
| 16:07 |
Dyrcona |
Yeah, that might be useful. |
| 16:08 |
Dyrcona |
About two minutes later, the logs say it lost 527 messages from the same PID. |
| 16:11 |
pastebot |
"JBoyer" at 64.57.241.14 pasted "Log Limiting such and such" (13 lines) at http://paste.evergreen-ils.org/28 |
| 16:18 |
Dyrcona |
JBoyer++ |
| 16:21 |
JBoyer |
To be fair you only need $SystemLogRateLimitInterval 0 to turn off limiting but the rest of that sets up what to do when the messages start coming in a bit too fast. |
| 16:28 |
Dyrcona |
Thanks. I'll read up some more on rsyslog on Monday. |
| 16:28 |
Dyrcona |
Think I'll call it a day for now since I started working a little early this morning. |
| 16:29 |
Dyrcona |
Have a great weekend, everyone! |
| 16:44 |
|
maryj_ joined #evergreen |
| 16:52 |
kmlussier |
Wow! Work days almost over. Let's see if I can merge one branch before I go on vacation next week. |
| 16:59 |
kmlussier |
Actually, that might be somewhat optimistic. |
| 17:01 |
bshum |
I believe in you kmlussier! |
| 17:03 |
kmlussier |
No, don't believe in me. I have to feed the children, and there are upgrade scripts to deal with. Otherwise, it would be merged already. |
| 17:03 |
bshum |
I suppose children are important. |
| 17:07 |
|
mmorgan left #evergreen |
| 17:14 |
|
jvwoolf left #evergreen |
| 17:36 |
kmlussier |
OK, children are fed. Calling 0983 and 0984 |
| 17:38 |
Bmagic |
kmlussier++ |
| 17:54 |
pinesol_green |
Showing latest 5 of 16 commits to Evergreen... |
| 17:54 |
pinesol_green |
[evergreen|Mike Rylander] LP#1549505: Query literal interpolation casts incorrectly - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=ae3bb32> |
| 17:54 |
pinesol_green |
[evergreen|Kathy Lussier] LP#1549505: Remove stray semicolon from PgTap test - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=cf61449> |
| 17:54 |
pinesol_green |
[evergreen|Kathy Lussier] LP#1549505: Decrease value of Max popularity importance multiplier - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=9402707> |
| 17:54 |
pinesol_green |
[evergreen|Galen Charlton] LP#1549505: update $modal to $uibModal - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=ac561d3> |
| 17:54 |
pinesol_green |
[evergreen|Kathy Lussier] LP#1549505: Stamping upgrade scripts 0983-84 for stat pop ratings - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=d5ce592> |
| 17:54 |
kmlussier |
miker++ gmcharlt++ |
| 17:56 |
kmlussier |
Have a nice weekend everyone! |
| 22:02 |
hbrennan |
Happy weekend everyone! |
