| Time |
Nick |
Message |
| 00:54 |
|
StomproJ joined #evergreen |
| 03:04 |
|
book` joined #evergreen |
| 03:17 |
|
Stompro joined #evergreen |
| 05:01 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 06:17 |
csharp |
dbs++ # closed date fixes |
| 07:00 |
|
agoben joined #evergreen |
| 07:13 |
|
rjackson_isl joined #evergreen |
| 07:22 |
|
JBoyer joined #evergreen |
| 08:08 |
|
rlefaive joined #evergreen |
| 08:38 |
|
mmorgan joined #evergreen |
| 08:40 |
|
collum joined #evergreen |
| 08:54 |
|
bos20k joined #evergreen |
| 08:54 |
|
Dyrcona joined #evergreen |
| 09:04 |
|
krvmga joined #evergreen |
| 09:08 |
|
kmlussier joined #evergreen |
| 09:17 |
|
yboston joined #evergreen |
| 09:24 |
|
mdriscoll joined #evergreen |
| 09:28 |
dbs |
csharp++ # signing off! I'll push those in a bit |
| 09:33 |
pinesol_green |
[evergreen|Dan Scott] LP#1432753 Restore "All day" verbiage to Closed Dates editor - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=8bfa94f> |
| 09:33 |
pinesol_green |
[evergreen|Dan Scott] LP#1594937 Fix off-by-one display of closed dates - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=233e61f> |
| 09:35 |
csharp |
hooray! |
| 09:35 |
mmorgan |
dbs++ |
| 09:36 |
mmorgan |
csharp++ |
| 09:42 |
kmlussier |
dbs++ csharp++ |
| 09:57 |
|
Christineb joined #evergreen |
| 10:06 |
Bmagic |
dbs++ |
| 10:06 |
Bmagic |
csharp++ |
| 10:41 |
|
Jason_ joined #evergreen |
| 10:51 |
Bmagic |
I keep having this memcached issue and I can't figure it out. memcached doesn't log anything. Our libraries have to continue to login every step of the way through the staff client. It's like memcached is evicting the key, but memcstat shows 0 |
| 10:52 |
Bmagic |
I found these in osrfsys.log open-ils.circ [ERR :21628:Cache.pm:126:1481641510398122] Unable to store oils_permit_key_740b499e8fdcd95de95ce0a8d48c0cc0 => [1 bytes] in memcached server |
| 10:52 |
Bmagic |
then I restart memcached, and everything is fine for a few days |
| 11:01 |
Dyrcona |
Bmagic: Never seen that before but sounds like a memcached bug to me. |
| 11:01 |
Stompro |
Bmagic, how many connections does memcached have open when this happens? Any chance new connections are being denied? With our openvz VM's, we sometimes hit the artificial resource limits on tcp buffers and connections, running openvz? |
| 11:01 |
pastebot |
"Bmagic" at 64.57.241.14 pasted "Failed state of memcached" (105 lines) at http://paste.evergreen-ils.org/36 |
| 11:04 |
Bmagic |
Stompro: not running openvz |
| 11:05 |
Bmagic |
if it was TCP, then restarting memcached wouldn't help right? |
| 11:05 |
Stompro |
Bmagic, have you captured any traffic between the memcache server and your app servers that shows the issue? |
| 11:06 |
Bmagic |
no, that would require us to be in a failed state while everyone waited for me to set that up. I want to get it up and running asap. but maybe I could have it capturing all the time.... |
| 11:06 |
berick |
you can hit resource limits regardless of openvz, of course |
| 11:07 |
berick |
memcache has a connection limit setting (defaults to 1024) |
| 11:07 |
Stompro |
Bmagic, if the memcached process was holding onto connections, not fully closing them, then it might report X, but linux might show more. |
| 11:07 |
berick |
the OS will have its own open connection limits |
| 11:07 |
Bmagic |
I thought it was resource limits at first, so I upped the ram from 4gb to 8gb (as you can see from the pasted config) - also changed servers from no swap to swap |
| 11:09 |
Bmagic |
not sure about the tcp connections though. the OS will block new tcp connections after the limit is reached? Where is that config? |
| 11:09 |
Bmagic |
oh sysctl |
| 11:11 |
Stompro |
Bmagic, there was a bug with added content a while back where the connection wasn't being fully closed, so apache was eating up connections... let me find it, the troubleshooting steps might help look for a problem. |
| 11:11 |
Dyrcona |
net.core.somaxconn |
| 11:11 |
Dyrcona |
That's a per socket max connections. |
| 11:12 |
Dyrcona |
I believe it's per socket. Looks like the default is 128. You had 123 in your output. |
| 11:12 |
Bmagic |
hmmm... that could be something |
| 11:12 |
Dyrcona |
probably another way to set it other than sysctl. |
| 11:13 |
Dyrcona |
There are probably additional admin type connections to the socket that aren't counted by memcached. |
| 11:13 |
Bmagic |
hmmm |
| 11:14 |
Bmagic |
how many connections do you have in production for example? |
| 11:14 |
Dyrcona |
We run two memcached servers. I'll check in a minute. |
| 11:14 |
Stompro |
Bmagic, Found it, Bug 1478123. |
| 11:14 |
pinesol_green |
Launchpad bug 1478123 in Evergreen 2.8 "Missing close() after shutdown() in EGCatloader/Record.pm FD Leak" [Medium,Fix released] https://launchpad.net/bugs/1478123 |
| 11:16 |
Dyrcona |
Stompro: That shouldn't affect memcached, unless memcached is on the save server and you run out of descriptors to open new sockets with. :) |
| 11:16 |
Bmagic |
Stompro: my memcached server is on a separate machine from all of the apache app servers |
| 11:16 |
Dyrcona |
s/save/same/ |
| 11:16 |
Bmagic |
it is interesting though |
| 11:17 |
Stompro |
Bmagic, I just meant that looking for incorrectly closed connections might be a troubleshooting step. Not that this specific bug was a problem. |
| 11:17 |
Bmagic |
roger that |
| 11:18 |
Dyrcona |
Bmagic: What's the command you ran to get that output? It has been a while since I've looked at memcached statistics. |
| 11:18 |
* Stompro |
was just googling the same thing. |
| 11:18 |
Bmagic |
memcstat --servers (ip address) |
| 11:19 |
Bmagic |
I wish I knew what my connection numbers were on our old setup. I do recall all of the app servers running their own memcached instances (not realizing) which turned out to be in use from eg_vhost.conf. Now, I have edited that config to point to the main server |
| 11:20 |
Bmagic |
perhaps with the splintered memcached servers before, I never hit the 128 limit, but now that I have all of the connections talking to the same server..... |
| 11:21 |
Bmagic |
in any event, 128 is low, or there is a bug with getting those closed |
| 11:22 |
Bmagic |
well, I am looking again, and now I have 133 curr_connections - everything is working fine right now, so, 128 must not be the limit |
| 11:23 |
Dyrcona |
Bmagic: We have 529 and 728 current connections on the two servers. |
| 11:23 |
Bmagic |
and what is your net.core.somaxconn setting? |
| 11:23 |
Dyrcona |
Hm.. It's 128 on the first one. |
| 11:24 |
Bmagic |
darn, I was hoping it was that |
| 11:26 |
Dyrcona |
That's the core value. There's a way to override it. |
| 11:29 |
Dyrcona |
Great... Running into this on jessie: /usr/bin/ld: cannot find -ldbdpgsql |
| 11:31 |
Dyrcona |
Crazy, but I'm not finding it. Mabye I'm thinking of BSD systems. |
| 11:33 |
Dyrcona |
Bmagic: not sure if raising net.core.somaxconn will help you or not. |
| 11:34 |
Dyrcona |
And, funny thing about the link error, I don't really need cstore compiled on this machine, 'cause I don't plan to run services. I'm just installing for some of the libs. |
| 11:39 |
Dyrcona |
Weird. Is this just a problem with 32-bit systems? 'Cause I didn't have this problem on the 64-bit vm. |
| 11:40 |
|
brahmina joined #evergreen |
| 11:40 |
bshum |
It wouldn't surprise me |
| 11:40 |
bshum |
We haven't tested 32-bit installs with Evergreen in a long time... |
| 11:40 |
bshum |
At least I haven't |
| 11:40 |
bshum |
And there were problems the last times we had anyone try it |
| 11:42 |
Dyrcona |
It's right there: /usr/lib/i386-linux-gnu/dbd/libdbdpgsql.so |
| 11:42 |
Dyrcona |
Thing is, I don't need this. I just wanted some of the Perl libs that I use in some scripts and they don't depend on that. |
| 11:42 |
Dyrcona |
However, we have no flags to disable building the services. |
| 11:44 |
Bmagic |
I think I need more information to figure out my issue. I have -vv switch in memcached config but I still get zero log entries in /var/log/memcached.log |
| 11:45 |
Bmagic |
maybe that file wasn't writable? I have created it manually, and gave memcace user permissions. But memcached was already started, maybe restart tonight and we'll see |
| 11:46 |
bshum |
Dyrcona: https://bugs.launchpad.net/evergreen/+bug/1445182 maybe? |
| 11:46 |
pinesol_green |
Launchpad bug 1445182 in Evergreen 2.7 "Debian Jessie 2.8.0 libdbi-drivers cannot find libpq.so" [Undecided,Fix released] |
| 11:46 |
bshum |
Ghost of, I mean |
| 11:47 |
Dyrcona |
Yeah, no. You're not supposed to install it by hand on Jessie, 'cause there's a package. |
| 11:48 |
Dyrcona |
It should just work. |
| 11:48 |
Dyrcona |
And, it did on 64-bit. |
| 11:49 |
bshum |
Ah |
| 11:49 |
bshum |
Right then :) |
| 11:49 |
Dyrcona |
So, I'll mangle the makefiles and add the full path to the lib where it should go. That should get me past this. |
| 11:52 |
Dyrcona |
Bingo! We have a winner! |
| 11:55 |
Bmagic |
What do we have for him Jonny? It's a NEW CAR! |
| 11:56 |
Dyrcona |
heh! Now, I hear the music in my head. |
| 12:01 |
Jason_ |
Question: I'm installing evergreen on a test server (ubuntu-trusty) and I've encountered an issue with opensrf configuration > Websockets installation. The git clone does not contain the example configuration file "examples/apache_24/websockets/apache2.conf" listed in the instructions. Any ideas? Or should I skip it for now as websockets are optional? |
| 12:01 |
bshum |
Jason_: The example config file is in the opensrf source files |
| 12:01 |
bshum |
Not in the git clone for websockets |
| 12:02 |
bshum |
That instruction is getting updated in a future release to be more clear I think. |
| 12:03 |
bshum |
Websockets are "optional" if you're not planning to use the web-based staff client (which is still under development). So yes, you could technically skip the step. |
| 12:03 |
Jason_ |
Got it, and found the file! Thanks :) |
| 12:04 |
kmlussier |
Jason_: Dyrcona's blog post on the web client installation is helpful. It identifies a couple of things like that issue that aren't clear in the instructions. http://blog.mvlcstaff.org/2015/06/setting-up-evergreen-browser-staff_2.html |
| 12:05 |
Jason_ |
I'll check that out, thanks |
| 12:45 |
|
_adb joined #evergreen |
| 12:54 |
|
bmills joined #evergreen |
| 13:03 |
sandbergja |
I'm running into an issue with authority_control_fields.pl: every time I run it with the -a or --all flag, it gives me the error message "Can't use an undefined value as an ARRAY reference at /openils/bin/authority_control_fields.pl line 77." |
| 13:04 |
sandbergja |
Running it with the --start and --end works just fine, but I want to set it up to run regularly, and don't want to have to calculate the --end value each time |
| 13:10 |
Dyrcona |
sandbergja: That would seem to indicate that it is either timing out or failing to retrieve all of the non-deleted bibs in your database. |
| 13:12 |
Dyrcona |
Or the list of ids returned is too big for ejabberd to handle. |
| 13:12 |
sandbergja |
That's helpful, thanks |
| 13:14 |
sandbergja |
If it is a timing out issue, would that be in opensrf or postgres or somewhere else? |
| 13:15 |
Dyrcona |
OpenSRF. Postgres doesn't time out. |
| 13:15 |
sandbergja |
Got it. |
| 13:17 |
Dyrcona |
If you specify the days_back option, it connects to the database to get the list of ids. |
| 13:17 |
Dyrcona |
It should probably do that for the all option as well. |
| 13:20 |
sandbergja |
Oooh, let me try that out! |
| 13:21 |
|
bmills joined #evergreen |
| 13:21 |
Dyrcona |
By "it should," I mean it doesn't now, but should be changed to do so. Are you going to take a crack at changing the code? |
| 13:23 |
sandbergja |
I think that'd be a little advanced for me, but I think that the --days_back flag is actually what we need, rather than --all |
| 13:24 |
Dyrcona |
OK. I ususally use days_back, like 90 or so for authority updates. |
| 13:24 |
sandbergja |
I might throw together some documentation, because it doesn't look like the --days_back flag is mentioned in the official docs or the help screen on authority_control_fields.pl -h |
| 13:26 |
sandbergja |
Thanks so much for your help! |
| 13:26 |
sandbergja |
Dyrcona++ |
| 13:26 |
|
mmorgan joined #evergreen |
| 13:29 |
Dyrcona |
Looks like the days_back option was added in 2012 and the documentation for the help was added in 2011. |
| 13:29 |
Dyrcona |
Two potential Lp bugs, there. :) |
| 13:30 |
sandbergja |
Fun fun! |
| 13:31 |
sandbergja |
I can report those and work on the documentation aspects. |
| 13:33 |
Dyrcona |
Cool. I'll see about fixing the all option. |
| 13:35 |
sandbergja |
Thanks! It seems like that would be very helpful for us to run with the --all option eventually (e.g. 2 years from now, when we get an authority record that needs to be linked to a bib record we haven't touched in forever) |
| 15:04 |
|
mmorgan1 joined #evergreen |
| 15:08 |
|
Dyrcona joined #evergreen |
| 15:17 |
JBoyer |
Bmagic, I ran into the same problem you're seeing when we increased the number of app servers and the number of spare children at the same time. memcached has a connection limit specified in it's config file and the default is 1024. Here I'm using 8192 or thereabouts. It's the "# -c 1024" line in the memcached.conf file you pasted, uncomment it and bump it up and you should stop seeing that happen. |
| 15:18 |
Bmagic |
JBoyer: I can't believe we are at 1024, the output of the running config of memcached only showed curr_connections: 123 |
| 15:19 |
JBoyer |
Oops, missed that part of the paste. Carry on! |
| 15:19 |
Bmagic |
perhaps the number of connections with regaurds to that setting is not output from memcstat? |
| 15:20 |
JBoyer |
I would trust memcstat over netstat, since memcached will likely not "count" connections in CLOSE_WAIT or other non-connected statues. |
| 15:29 |
Dyrcona |
Bmagic: I don't think the defaults are output, and the config file could actually be different from the options set when memcached was built. |
| 15:30 |
Dyrcona |
I'd try just uncommenting that line to see and restarting memcached to see what happens. |
| 15:30 |
Bmagic |
I sure will |
| 15:30 |
Dyrcona |
oops. bad on the fly editing. |
| 15:30 |
Bmagic |
1024 seems fine |
| 15:30 |
Dyrcona |
I'll check out configuration. |
| 15:30 |
Bmagic |
of course, I have to wait till 4am |
| 15:32 |
Dyrcona |
Ours is still commented, but we're on a different distro from what I think you're running. |
| 15:37 |
Dyrcona |
Bmagic: You're running this in a docker appliance, yeah? |
| 15:37 |
Bmagic |
memcached is running on a plain ol vm |
| 15:38 |
Bmagic |
the docker stuff is app bricks only |
| 15:38 |
Dyrcona |
OK. Thanks. Thought maybe some docker setting could apply here, but I don't know docker, so... |
| 15:38 |
Bmagic |
I don't think that is in the equation |
| 15:39 |
Dyrcona |
Have you had another look with memcstat and has anything changed much? |
| 15:40 |
Bmagic |
curr_connections: 129 total_connections: 5180 connection_structures: 155 |
| 15:40 |
Bmagic |
not really |
| 15:41 |
bshum |
That's a bit higher total |
| 15:42 |
bshum |
But probably not insane |
| 15:42 |
* bshum |
keeps musing to himself |
| 15:43 |
Bmagic |
yeah, I agree, and because the system is running fine right now.... |
| 16:39 |
|
mmorgan joined #evergreen |
| 17:01 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 17:07 |
|
mmorgan left #evergreen |
| 18:35 |
|
bmills joined #evergreen |
| 18:56 |
|
bmills joined #evergreen |
| 19:44 |
|
StomproJ joined #evergreen |
| 21:36 |
|
bmills joined #evergreen |
| 22:42 |
|
littleDude joined #evergreen |
