| Time |
Nick |
Message |
| 01:55 |
|
StomproJ joined #evergreen |
| 03:34 |
|
stephengwills joined #evergreen |
| 06:30 |
|
agoben joined #evergreen |
| 06:58 |
|
stephengwills joined #evergreen |
| 07:00 |
|
collum joined #evergreen |
| 07:12 |
|
rjackson_isl joined #evergreen |
| 07:27 |
|
Dyrcona joined #evergreen |
| 07:29 |
Dyrcona |
Hm... Restarted the load balancers and with "nothing" going on, ldirectord memory use is trending upwards again.... |
| 07:44 |
Dyrcona |
Yeah, this is pathological. I wonder what's so special about the nginx proxy on the brick heads that it causes this behavior? I'm assuming that's the cause because we didn't have trouble with crashes/memory until after setting up the proxy. |
| 07:52 |
Dyrcona |
"Nothing" going on, i.e. < 200 total connections, and ldirectord memory use is climbing by 20K per minute or so. |
| 07:52 |
Dyrcona |
Nope. Much faster than that. |
| 07:56 |
|
rfrasur joined #evergreen |
| 08:04 |
JBoyer |
what kind of timeouts do you have in your nginx config? Maybe it's holding things open for too long? |
| 08:09 |
|
bos20k joined #evergreen |
| 08:11 |
JBoyer |
(seems unlikely since I know you've been looking into this a bit, but it's the thing that makes the most sense.) |
| 08:19 |
|
_bott_ joined #evergreen |
| 08:23 |
csharp |
Dyrcona: in case it helps, this is our nginx osrf config: https://pastebin.com/ggkhSMFj |
| 08:23 |
csharp |
I do remember needing to adjust the timeout thresholds |
| 08:23 |
Dyrcona |
Right now, I'm dealing with ldirectord working for a little bit, then showing no connections. |
| 08:23 |
Dyrcona |
Thanks, I'll have a look when people can actually get in. |
| 08:24 |
Dyrcona |
I think I'm switching to haproxy on the load balancer and getting rid of ldirectord and nginx in the not too distant future. |
| 08:25 |
Dyrcona |
Since I can't seem to get any answers in #clusterlabs, should lidrectord only be running on 1 load balancer? The problems always seem to start a few minutes after I kill one. |
| 08:27 |
Dyrcona |
Memory usages is still ballistic.... |
| 08:28 |
csharp |
<REDACTED> |
| 08:28 |
csharp |
*sigh* |
| 08:29 |
Dyrcona |
:) |
| 08:30 |
Dyrcona |
Memory usage has gone up 4MB since I typed my "ballistic" line, and probably gone up more while typing this. |
| 08:30 |
Dyrcona |
So, roughly 2MB/minute. |
| 08:31 |
Dyrcona |
csharp: I'm going to try your proxy timeouts. It can't hurt any more than everything else I've been doing.... :) |
| 08:31 |
csharp |
yeah, I don't remember details atm but I know that solved a serious problem we were having at the time |
| 08:35 |
Dyrcona |
Well, I've made the changes and reloaded nginx. We'll see how things go. |
| 08:35 |
|
mmorgan joined #evergreen |
| 08:36 |
Dyrcona |
websocket--- # Seriously: re-implementing TCP over HTTP, which is itself implemented on TCP, is a sign you're doing something very wrong. |
| 08:36 |
Dyrcona |
And, that's not leveled at Evergreen specifically, that's a general commentary on the state of web development. |
| 08:41 |
Dyrcona |
RSZ of ldirectord continues to creep upward on both load balancers. Still don't know if only 1 should be running, but Evergreen is accessible, now, so I'm not touching it. |
| 09:00 |
JBoyer |
Dyrcona, when you say "both," do you mean you have two ldirector load balancers up and running simultaneously for the same backend servers? |
| 09:01 |
JBoyer |
I was under the impression you should only have 1 running and 1 ready to take over if there's an issue. |
| 09:01 |
Dyrcona |
Yes, and it seems to be working. |
| 09:01 |
JBoyer |
(i.e. up but not actually doing anything) |
| 09:01 |
Dyrcona |
So was I, but when I rebooted them, they both started up. I killed the one with the least connections, and after a minute or two, no one could connect to Evergreen, and the one showed a handful of connections. |
| 09:02 |
Dyrcona |
I have disabled ldirectord in systemd on both load balancers. |
| 09:02 |
Dyrcona |
I let ha/heartbeat manage it. |
| 09:03 |
Dyrcona |
Since it is working, now, and stopping one seems to cause us problems today, I'm leaving them alone for now. |
| 09:03 |
JBoyer |
That's perfectly reasonable. |
| 09:03 |
Dyrcona |
Now, I'm trying to find out more about this memory consumption. DDG is not proving very helpful as I keep find things from 2006 or so. |
| 09:04 |
JBoyer |
I've never looked into using ha or heartbeat; I've only ever had a single ldirector running on its own, which while not entirely of the spirit of things, does work well. |
| 09:06 |
Dyrcona |
Well, I think it is supposed to work that way: ldirectord runs on 1 load balancer and heart beat, or pacemaker or whatever they call it this minute, will start the other one if the first one dies. |
| 09:06 |
Dyrcona |
How this is working with IP addressing I have no clue, but it seems to be working, and I'll settle for "wrong" if correct blows up in my face. |
| 09:07 |
|
tlittle joined #evergreen |
| 09:10 |
|
nfBurton joined #evergreen |
| 09:28 |
Dyrcona |
Fun thing, too, because ipvs goes through the kernel, the problems could be kernel bugs and not just issues with ldirectord itself. |
| 09:32 |
Dyrcona |
Memory use may have slowed. After 10 minutes it has gone up only 14MB. |
| 09:40 |
|
mdriscoll joined #evergreen |
| 09:42 |
|
yboston joined #evergreen |
| 09:44 |
|
aabbee joined #evergreen |
| 09:46 |
|
rfrasur joined #evergreen |
| 10:52 |
Dyrcona |
csharp++ # For sharing the proxy config. |
| 11:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 11:18 |
csharp |
@praise Testing Success |
| 11:18 |
* pinesol |
Testing Success LOVES the RESISTANCE! |
| 11:20 |
|
jamesrf joined #evergreen |
| 11:21 |
Dyrcona |
heh |
| 11:21 |
Dyrcona |
@who loves the Resistance? |
| 11:21 |
pinesol |
ericar loves the Resistance. |
| 11:22 |
ericar |
wut |
| 11:22 |
Dyrcona |
@who is a Spy? |
| 11:22 |
pinesol |
dbs is a Spy. |
| 11:22 |
Dyrcona |
I knew it! |
| 11:24 |
dbs |
So true |
| 11:24 |
dbs |
I've been unmasked |
| 11:25 |
* Dyrcona |
is waiting on an action trigger run to finish to see if adding filters helps with something not related to ldirectord. :) |
| 11:25 |
phasefx |
Resistance will always have to be an EG conference thing for me, my local gaming friends don't like it :D |
| 11:27 |
Dyrcona |
That's too bad, but at least you have local gaming friends. I'm limited to mostly Scrabble, Yahtzee, and Dutch Blitz with the family. |
| 11:28 |
phasefx |
kind of local.. 40 minute drive once or twice a month for that group. 1.5 hour drive once a week for D&D |
| 11:28 |
Dyrcona |
D&D is worth it! |
| 11:28 |
phasefx |
times for going in one direction |
| 11:28 |
phasefx |
oh yes |
| 11:28 |
* Dyrcona |
should find a group. |
| 11:28 |
* JBoyer |
feels like a wet blanket re: games. So many bad experiences like Monopoly and so on that I tend to stick to the electronic games. |
| 11:28 |
Dyrcona |
Monopoly is meant to be a bad experience. |
| 11:29 |
dbs |
my daughter just joined a D&D group at her high school. i'm so proud |
| 11:29 |
phasefx |
find a local adventurer's league (which sucks now, rules-wise), find a group there, and then poach them from AL into your own non-AL game :) |
| 11:29 |
JBoyer |
It is, but when the game takes 4+ hours because of house rules, I hate it for reasons other than the intended purpose. |
| 11:29 |
Dyrcona |
dbs: Cool. |
| 11:30 |
Dyrcona |
@band add House Rules Monopoly |
| 11:30 |
pinesol |
Dyrcona: Band 'House Rules Monopoly' added to list |
| 11:31 |
Dyrcona |
The trouble with making configuration changes on low activity days is you can't tell if the changes really made an improvement. |
| 11:32 |
Dyrcona |
Or if the numbers are just low because Thursday is usually less busy than Tuesday and Wednesday. |
| 11:33 |
|
Christineb joined #evergreen |
| 11:37 |
|
rfrasur joined #evergreen |
| 11:40 |
|
nfBurton joined #evergreen |
| 11:46 |
csharp |
my 16-year-old son is also in a D&D group |
| 11:46 |
csharp |
@dnd |
| 11:46 |
pinesol |
csharp: strength:11 dexterity:11 constitution:17 intelligence:13 wisdom:15 charisma:10 |
| 11:46 |
csharp |
@librarian |
| 11:46 |
pinesol |
csharp: Management:16, Cataloging:10, Acquisitions:14, Reference:10, Circulation:17, Systems:8, Research:9, Custodial:14 |
| 11:51 |
* phasefx |
offers to run a game during hackaway |
| 11:51 |
|
yboston joined #evergreen |
| 11:52 |
berick |
phasefx: https://i.imgur.com/CJNovec.jpg |
| 11:53 |
phasefx |
it's true! |
| 11:54 |
JBoyer |
berick++ |
| 11:55 |
Dyrcona |
:) |
| 12:07 |
|
jihpringle joined #evergreen |
| 12:12 |
Dyrcona |
Well, that's nice: Can't locate object method "content" via package "OpenSRF::DomainObject::oilsMethodException" at /openils/bin/action_trigger_runner.pl line 174. |
| 12:34 |
|
collum_ joined #evergreen |
| 12:40 |
|
collum joined #evergreen |
| 12:49 |
|
rfrasur joined #evergreen |
| 13:00 |
|
khuckins joined #evergreen |
| 13:07 |
|
cschroth joined #evergreen |
| 13:08 |
|
HomerPublic joined #evergreen |
| 13:14 |
|
yboston joined #evergreen |
| 13:29 |
|
rfrasur joined #evergreen |
| 13:33 |
JBoyer |
I know I've seen this before but I've forgotten everything about it apparently. I'm getting a JS error when loading the reporter, complaining that it can't insert the list of OUs into oils_rpt_top_folder_lib_picker. |
| 13:34 |
JBoyer |
And in the logs there's an xml parser error looking at oils_rpt_editor.xhtml, preventing that page from being included in any other. |
| 13:48 |
|
yboston joined #evergreen |
| 14:04 |
|
yboston joined #evergreen |
| 14:21 |
dbs |
JBoyer: are the two copies of the fm_IDL.xml (in /openils/conf and /openils/var/web/reports) in sync (assuming you're unilingual)? |
| 14:33 |
JBoyer |
dbs, they are indeed. |
| 14:39 |
dbs |
Well, that, and autoconf, is about the extent of my troubleshooting suggestions :/ |
| 14:50 |
JBoyer |
dbs++ |
| 15:05 |
jeffdavis |
gmcharlt: for bug 1817645 you had a couple of errata that should be fixed - should I open a separate bug for those now that the original is Fix Released? |
| 15:05 |
pinesol |
Launchpad bug 1817645 in Evergreen "Configurable patron auth and retrieval" [Wishlist,Fix released] https://launchpad.net/bugs/1817645 |
| 15:06 |
gmcharlt |
jeffdavis: yes pelase |
| 15:06 |
jeffdavis |
ok, will do |
| 15:14 |
|
yboston joined #evergreen |
| 15:21 |
|
frank joined #evergreen |
| 15:23 |
frank |
Hi all, I have a question, I already had closed dates in EG by staff client, but I am trying to see them in web client and the window is empty, I mean, it does not display the closed date registerede previously by staff client, Do I have to save it againt by web client? |
| 15:24 |
mmorgan |
frank: By default, the web client only shows closed dates in the future. You can change the date filter in the upper right corner of the screen and refresh |
| 15:25 |
|
mmorgan1 joined #evergreen |
| 15:26 |
|
khuckins joined #evergreen |
| 15:56 |
csharp |
JBoyer: assuming it's behind a proxy, is the originating IP being passed through? (bouncing back the suggestion you had for me back in April with a similar issue :-) ) |
| 16:08 |
JBoyer |
csharp, looks that way, yep. (RemoteIP* settings in eg_vhost.conf are un-commented.) |
| 16:09 |
csharp |
JBoyer++ |
| 16:09 |
JBoyer |
The worst part is that I've dealt with this in the last 3-4 months and forget what I did (or if the answer was "blow it all away and let /dev/null sort it out") |
| 16:09 |
csharp |
@praise /dev/null |
| 16:09 |
* pinesol |
You don't want to get mixed up with someone like /dev/null. /dev/null is a loner, Dottie. A rebel. |
| 16:09 |
|
mmorgan joined #evergreen |
| 16:14 |
|
frank joined #evergreen |
| 16:15 |
frank |
mmorgan: tks for responding, Do you know where does the closed dates are saved? |
| 16:16 |
mmorgan |
frank: They are saved in the database table actor.org_unit_closed |
| 16:54 |
frank |
mmorgan: thanks |
| 17:00 |
|
mdriscoll left #evergreen |
| 17:09 |
|
mmorgan left #evergreen |
| 18:07 |
|
yboston joined #evergreen |
| 18:29 |
|
khuckins joined #evergreen |
| 19:48 |
|
cmalm joined #evergreen |
| 20:08 |
|
ejk__ joined #evergreen |
| 21:12 |
|
stephengwills joined #evergreen |
| 23:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
