| Time |
Nick |
Message |
| 00:17 |
|
sandbergja joined #evergreen |
| 00:22 |
|
jvwoolf joined #evergreen |
| 00:27 |
|
jvwoolf joined #evergreen |
| 05:53 |
|
cmalm joined #evergreen |
| 06:57 |
|
agoben joined #evergreen |
| 07:15 |
|
rjackson_isl joined #evergreen |
| 08:09 |
|
bos20k joined #evergreen |
| 08:43 |
|
mmorgan joined #evergreen |
| 08:47 |
|
Dyrcona joined #evergreen |
| 09:01 |
|
aabbee joined #evergreen |
| 09:09 |
csharp |
I haven't yet been able to resolve my Item Tags adding UI problem - I rolled back the commit and synced the fm_IDL.xml's |
| 09:10 |
csharp |
as long as the problem wasn't caused by my push to master, I'm not worried about it though |
| 09:10 |
|
yboston joined #evergreen |
| 09:15 |
jeff |
i really wish we didn't have to deal with spaces in barcodes, but we do, and we're more likely to be scanning more of these directly into evergreen interfaces. based on some opinions in a handful of bugs i'm looking at this morning, i suspect we're going to need to do some extra work as a result. |
| 09:17 |
csharp |
coming from basic unfamiliarity with the issue here - can't we just strip out all spaces as the barcode is scanned? |
| 09:18 |
csharp |
or whitespace of any kind, I guess |
| 09:19 |
|
jvwoolf joined #evergreen |
| 09:20 |
|
tlittle joined #evergreen |
| 09:21 |
jeff |
then you would have multiple input values that "normalize" to the same stored value. |
| 09:21 |
jeff |
and your stored barcode in your system wouldn't match the actual barcode on the item, or match the stored value in other systems. |
| 09:22 |
csharp |
ah - makes sense |
| 09:25 |
jeff |
though removing the whitespace has been proposed and partially implemented in at least some interfaces, i think. |
| 09:25 |
jeff |
(thankfully leading/trailing whitespace isn't something we see in the wild often, if ever) |
| 09:26 |
jeff |
but another issue is having space be treated the same as escape for dismissing dialogs, etc. |
| 09:27 |
jeff |
and stripping the space isn't something that could help there. the input device is going to pass the space, and if there's a dialog that you don't want dismissed with Enter, your space is going to dismiss it if you've set things up so that space does that. |
| 09:28 |
jeff |
I'll think it over a little and comment on a few bugs and likely have other things to deal with this morning. :-) |
| 09:29 |
mmorgan |
Do we need to support barcodes stored in the database with spaces? |
| 09:29 |
|
tlittle joined #evergreen |
| 09:30 |
jeff |
we support this at present, though there have been a few things threatening to sneak in that would break that. |
| 09:30 |
jeff |
i think i've commented on at least one before. |
| 09:30 |
jeff |
(in the bug itself) |
| 09:31 |
mmorgan |
If all whitespace was stripped out of the barcode prior to storing, the stored barcode would match input stripped of whitespace |
| 09:31 |
jeff |
(in all of the above I'm talking about item barcodes, to be clear) |
| 09:31 |
jeff |
mmorgan: yes, csharp suggested that above. see my response there. |
| 09:33 |
jeff |
ah, I was thinking of my comment on bug 1818692 in my (i think i've commented on at least one before) |
| 09:33 |
pinesol |
Launchpad bug 1818692 in Evergreen "Docs: We should document Evergreen's barcode requirements" [Undecided,New] https://launchpad.net/bugs/1818692 |
| 09:33 |
jeff |
which was a tangent from bug 1798187 |
| 09:34 |
pinesol |
Launchpad bug 1798187 in Evergreen 3.2 "Item status -- Barcodes with initial or trailing spaces fail silently when in txt file" [Medium,Fix released] https://launchpad.net/bugs/1798187 |
| 09:35 |
jeff |
looks like the code to strip *all* whitespace may have made it in anyway. |
| 09:35 |
mmorgan |
jeff: So above you're suggesting you could have a barcode '3 1234' and a barcode '3 12 24' and those could be two distinct barcodes? |
| 09:35 |
jeff |
right. |
| 09:36 |
|
yboston joined #evergreen |
| 09:36 |
Dyrcona |
GIGO |
| 09:38 |
jeff |
mmorgan: and you might communicate with an external system regarding item with barcode "T12345" and the external system knows the item by its actual barcode value, which is "T 12345", and either it has no idea what item you're talking about, or (worse!) it finds another, different item... all because you're referring to it by the wrong barcode. |
| 09:39 |
jeff |
mostly I blame one ILS popular in education for these barcodes. |
| 09:40 |
mmorgan |
My sensibilities tell me that those two distinct barcodes should be the same. In my prior ILS experiences spaces were stripped out of barcodes |
| 09:41 |
jeff |
if I had to order the two concerns, I'd put "your stored barcode in your system wouldn't match the actual barcode on the item, or match the stored value in other systems" first. |
| 09:43 |
jeff |
my other favorite is case sensitive barcodes. |
| 09:43 |
jeff |
s/my other/another/ |
| 09:45 |
Dyrcona |
@blame Character Set B |
| 09:45 |
pinesol |
Dyrcona: Character Set B musta been an Apple employee. |
| 09:46 |
* mmorgan |
thinks things would be a lot less problematic if barcode data were normalized before storing, no whitespace, no case sensitivity |
| 09:47 |
mmorgan |
IMHO :) |
| 09:47 |
Dyrcona |
Aren't most barcodes for library books done in Codabar? |
| 09:47 |
jeff |
and since you have to deal with barcodes with spaces and without spaces, you can't just "un normalize" before transmission to external systems, because if you strip the whitespace there's no way to tell which of the following barcodes needs whitespace added where: |
| 09:48 |
jeff |
T 81047 |
| 09:48 |
mmorgan |
Dyrcona: Yes, in my experience |
| 09:48 |
jeff |
T0031356 |
| 09:48 |
jeff |
T333000019726K |
| 09:48 |
jeff |
etc. |
| 09:48 |
Dyrcona |
Then, how do you get lowercase, letters greater than D, or spaces in them? |
| 09:48 |
jeff |
I suppose I should have stripped the space from that first for purposes of my example: |
| 09:48 |
jeff |
T81047 |
| 09:51 |
|
sandbergja joined #evergreen |
| 09:52 |
mmorgan |
jeff: I follow what you're saying. Just wondering, though, where does the barcode data come from for the external system? Entered independently, and you just have to work with it? |
| 09:52 |
mmorgan |
I mean how does it get stored in the external system that evergreen talks to. |
| 09:52 |
jeff |
right. |
| 09:52 |
jeff |
statewide resource sharing system. |
| 09:53 |
mmorgan |
:-( |
| 09:53 |
Dyrcona |
Well, what part of Evergreen is currently stripping spaces from inside barcodes? |
| 09:54 |
jeff |
Dyrcona: recent versions of the "upload a file" item status interface, by my quick read of the patch in bug 1798187 |
| 09:54 |
pinesol |
Launchpad bug 1798187 in Evergreen 3.2 "Item status -- Barcodes with initial or trailing spaces fail silently when in txt file" [Medium,Fix released] https://launchpad.net/bugs/1798187 |
| 09:54 |
mmorgan |
I know we're talking about items, but spaces are currently stripped from patron barcodes in the checkout interface. |
| 09:55 |
jeff |
but there were similar "barcodes don't have spaces" or "barcodes shouldn't have spaces" opinions in a few other bugs like bug 1826584 |
| 09:55 |
pinesol |
Launchpad bug 1826584 in Evergreen "continue past alert and confirm dialogs with keyboard space or enter" [Undecided,Confirmed] https://launchpad.net/bugs/1826584 |
| 09:55 |
Dyrcona |
jeff: Well, if they're Codabar, they can't have spaces..... |
| 09:55 |
Dyrcona |
Code 128 is another story. |
| 09:56 |
mmorgan |
Dyrcona: Scanners can be programmed to insert spaces when codabar barcodes are scanned. Who remembers CLSI Editing? |
| 09:57 |
Dyrcona |
Sure, but Codabar has no way to represent spaces itself and the above sounds like an extraordinarily dumb idea. |
| 09:57 |
Dyrcona |
Eh, not extraordinarily, just plain dumb.... :) |
| 09:58 |
mmorgan |
Unless spaces are stripped out before storing :) |
| 10:00 |
Dyrcona |
Let's switch to Aztec and use images of the cover instead of barcodes. |
| 10:00 |
jeff |
i have no reason to believe that they are codabar. |
| 10:00 |
* mmorgan |
forms strong opinions about barcodes when looking at precat reports |
| 10:00 |
jeff |
last one i inspected was interleaved 2 of 5. |
| 10:01 |
Dyrcona |
jeff: I don't either, so I think Evergreen should accept whatever it's given. |
| 10:01 |
Dyrcona |
I'm just "arguing" from the side of "barcodes don't have spaces because...." |
| 10:02 |
jeff |
got it. just trying to clarify that mystery. :-) |
| 10:02 |
Dyrcona |
I also disagree with continuing past dialogs with keyboard space or enter. I think the user should pay attention, otherwise why have a human do the work at all? |
| 10:02 |
jeff |
lest someone think that these libraries have found a clever way of encoding unsupported characters in codabar. :-) |
| 10:03 |
Dyrcona |
Right. I was suspecting Code 128, myself. |
| 10:03 |
Dyrcona |
Hence, the blame of Character Set B. :) |
| 10:03 |
mmorgan |
Dyrcona: For accessibility reasons, there should be a way to continue past dialogs using keyboard. |
| 10:04 |
Dyrcona |
mmorgan: OK. |
| 10:18 |
Bmagic |
grrr spaces! What kills me is when I see someone login to their account somewhere (not Evergreen) and their username is a whole name with spaces. It's relatively new in computer land to see usernames with spaces. :) |
| 10:19 |
Dyrcona |
Spaces by themselves are not a problem. The problem is not taking spaces into account. |
| 10:19 |
Bmagic |
"It's just another character" |
| 10:20 |
mmorgan |
But it's a character you can't always see. Spaces at the end or beginning, or two spaces instead of one are all problematic. |
| 10:21 |
Bmagic |
that's the point I was thinking of making mmorgan++ |
| 10:21 |
jeff |
making usernames case insensitive remains an outstanding todo. |
| 10:21 |
jeff |
and our count of duplicate-but-for-case users is at 208. |
| 10:21 |
Bmagic |
eeek |
| 10:21 |
jeff |
impressively, none with more than 2. |
| 10:21 |
mmorgan |
ouch. |
| 10:22 |
* csharp |
decides to begin all his usernames with "<space>0" from now on |
| 10:22 |
Bmagic |
MS / Linux = case sensitive |
| 10:24 |
* Dyrcona |
enabled case sensitivity on a Mac OS X file system, once. A couple of applications broke. |
| 10:25 |
Bmagic |
:) |
| 10:26 |
mmorgan |
jeff: Why do you want to make your usernames case insensitive? |
| 10:26 |
JBoyer |
mmorgan, to match expectation with every other online system in use since 1985, are my personal feelings on it. :) |
| 10:27 |
Bmagic |
what JBoyer said |
| 10:27 |
JBoyer |
(I also shake my fists at clouds when spaces in barcodes come up.) |
| 10:27 |
mmorgan |
So why should barcodes be any different? |
| 10:27 |
* mmorgan |
ducks |
| 10:28 |
Dyrcona |
:) |
| 10:28 |
* JBoyer |
sees all case sensitivity as a dev / user battle that users have lost. |
| 10:29 |
Dyrcona |
A battle? |
| 10:30 |
JBoyer |
In UI, I should say. If I fed a line like ' INT mAIn(char**argv,CHAR argc)' at a compiler and it didn't complain I'd have serious concerns re: using it. |
| 10:30 |
Bmagic |
let's BATTLE: https://www.youtube.com/watch?v=rCwn1NTK-50 |
| 10:30 |
Dyrcona |
Well, that's different from what we've been talking about barcodes and usernames. |
| 10:30 |
tlittle |
Bmagic++ |
| 10:31 |
Dyrcona |
SQL is usually case insensitive, for example. |
| 10:31 |
JBoyer |
Dyrcona, maybe more a tug-of-war between dev convenience and user convenience. |
| 10:31 |
|
Christineb joined #evergreen |
| 10:31 |
Dyrcona |
Usernames should be case insensitive because JBoyer and jboyer should not be different users, IMHO. |
| 10:32 |
JBoyer |
agreed. |
| 10:32 |
Dyrcona |
Barcodes are tricky when you accept them from other systems. |
| 10:32 |
mmorgan |
We're going to run out of usernames ;-) |
| 10:33 |
Dyrcona |
Particularly, when they both have using D[0-9]{8} for items. |
| 10:33 |
JBoyer |
b00kwerm1987 gives this book two thumbs up! |
| 10:33 |
Dyrcona |
mmorgan: I doubt it, since most usrnames [sic] in our system are barcodes. :) |
| 10:35 |
jeff |
including the five barcodes-as-usernames in this db which case-collide. *cringe* |
| 10:35 |
Dyrcona |
Naming things is hard. :) |
| 10:49 |
rhamby_ |
one could argue that the creation of a distinction between upper and lowercase was just a bad idea ..... |
| 10:50 |
Bmagic |
In writing as well..... |
| 10:55 |
|
khuckins joined #evergreen |
| 11:02 |
pinesol |
News from qatests: Failed Running Evergreen tests <http://testing.evergreen-ils.org/~live/test.31.html#2019-09-04T11:00:52,122990657-0400 -0> |
| 11:05 |
|
sandbergja joined #evergreen |
| 11:16 |
sandbergja |
Hmmm... looks like I forgot to stamp the upgrade script for carousels when I pushed it yesterday |
| 11:18 |
Dyrcona |
sandbergja: Easy enough to fix. |
| 11:18 |
|
yboston joined #evergreen |
| 11:19 |
sandbergja |
I can plan to do that later today |
| 11:27 |
sandbergja |
And also look into the failed qa test |
| 11:30 |
csharp |
@who failed the QA test? |
| 11:30 |
pinesol |
jeff_ failed the QA test. |
| 11:31 |
mmorgan |
If our usernames were normalized, we could blame jeff ;-) |
| 11:41 |
csharp |
@blame the usernames |
| 11:41 |
pinesol |
csharp: the usernames must eat cottage cheese! |
| 11:54 |
jeff |
usernames may not contain dairy products. |
| 11:54 |
jeff |
usernames must be divisible by themselves. |
| 11:55 |
mmorgan |
:) |
| 11:55 |
Dyrcona |
So, my username can't be 0? |
| 12:11 |
|
jihpringle joined #evergreen |
| 12:20 |
Dyrcona |
Has anyone had to increase resources, i.e. add a brick, since implementing the web staff client? |
| 12:26 |
miker |
Dyrcona: pcrud, actor, auth, and maybe cstore use does go up some with webclient, but I'm not sure enough to add whole servers. maybe just increasing max kids for those |
| 12:27 |
Dyrcona |
miker: I've increased max kids on those, twice. I'm seeing us get very close to max Apache2 on several of our bricks as I sit here and watch things. |
| 12:27 |
miker |
and, with websocketd instead of Apache ws, front end load goes /way/ down... |
| 12:29 |
csharp |
I had to up max children on pcrud, auth, and actor, and have occasionally approached some DB connection limits, but at this point not looking at adding a brick |
| 12:29 |
csharp |
we're using websocketd very happily |
| 12:29 |
Dyrcona |
We're using websocketd with nginx proxy. |
| 12:29 |
csharp |
same here |
| 12:30 |
csharp |
I have to do a rolling opensrf restart every couple of weeks or so, depending on the situation |
| 12:30 |
Dyrcona |
I don't do that. I let things run until they crash, which historically has not been very often. |
| 12:31 |
|
zbanks joined #evergreen |
| 12:31 |
Dyrcona |
ldirectord doesn't seem to like the proxy, but it could just be that we've had an increase in traffic since we switched to using the proxy. It's hard to tell without analyzing the logs, which is rather difficult given how rsyslog mangles them. |
| 12:34 |
Dyrcona |
Looks like I have run out of open-ils.acq children today, but no one complained about acq, AFAIK. |
| 12:39 |
Dyrcona |
Looks like I have recently run out of actor drones on at least 1 brick, so guess I need to bump that number some more. |
| 12:51 |
Dyrcona |
The web staff client uses open-ils.actor way too much, IMHO. |
| 12:57 |
jeff |
what do you mean by "way too much"? what do you think it should be doing instead? |
| 13:02 |
Dyrcona |
Well, I have 120 actor drones allowed to run per brick and I'm still running out of actor drones from time to time (i.e. every couple of days) on 1 brick or sometimes 2. |
| 13:04 |
Dyrcona |
The XUL client didn't use actor this much. It also didn't use pcrud much at all. I've looked at web staff client code using pcrud that basically mimics the backend calls that XUL made. Wouldn't it be more efficient to use the existing back end API? |
| 13:09 |
rjackson_isl |
Here at ISL we are set up to reboot our app servers at 0,7,11,16 and 19 hours daily to avoid lack of resources |
| 13:09 |
rjackson_isl |
seems to keep things cleaned up enough to where we don't hit limits |
| 13:11 |
rjackson_isl |
JBoyer++ for smooth daily running of all things Evergreen! |
| 13:13 |
jeff |
rjackson_isl: can you elaborate at all on what lack of resources led you to implement that solution? |
| 13:14 |
rjackson_isl |
jeff: no statistics but we were hitting issues on a regular basis (almost daily) on at least one of the app servers requiring manual restarts |
| 13:14 |
rjackson_isl |
now we hit all 11 app servers at those times one at a time thru cron to relieve the problem |
| 13:18 |
Dyrcona |
I don't think I could do that with the way that our bricks are set up, and I really don't think reboots should be necessary. |
| 13:19 |
Dyrcona |
I mean, what is this? Windows? |
| 13:20 |
|
zbanks joined #evergreen |
| 13:27 |
Dyrcona |
udp-- # syslog dropping messages |
| 13:34 |
jeffdavis |
11 app servers! |
| 13:36 |
jeffdavis |
We have 3. Restarts are occasionally required but not regularly or in the normal course of things. |
| 13:38 |
Dyrcona |
Well, to make things even more interesting, our memcached servers are also close to full, both currently using close to 3.2GB. |
| 13:38 |
Dyrcona |
So, I guess a third memcached server is in order. |
| 13:39 |
csharp |
when we see memcached saturation it's usually caused by too many DB connections |
| 13:40 |
|
yboston joined #evergreen |
| 13:44 |
Dyrcona |
"too many DB connections" please elaborate. |
| 13:47 |
csharp |
often when we see higher-than-usual memcached connections it's because the DB server has hit its connection limit and EG just spawns more and more attempts to connect aggressively |
| 13:47 |
Dyrcona |
I count 529 "postgres" proceses on my database server, including postmaster and the other housekeeping processes. |
| 13:48 |
Dyrcona |
Well, I'm not talking about memcached connections, I'm talking about RAM/bytes used. |
| 13:48 |
csharp |
oh |
| 13:49 |
Dyrcona |
max connections is 2000 on Pg. |
| 13:49 |
csharp |
we have a 6GB cap on our memcached |
| 13:50 |
Dyrcona |
Our is 4GB. |
| 13:50 |
csharp |
our PG max connections is 1000 |
| 13:50 |
Dyrcona |
4GB per memcached instance. |
| 13:51 |
Dyrcona |
I do have over 1,700 "curr_connections" in memcached stats. |
| 13:51 |
Dyrcona |
Thought that number seems to fluctuate down and back up by several hundred every time I check. |
| 13:52 |
Dyrcona |
And, it seems to be orthogonal to number of database processes running. |
| 13:53 |
Dyrcona |
csharp: How many memcached instances do you run? |
| 13:53 |
|
sandbergja joined #evergreen |
| 13:53 |
csharp |
we run two |
| 13:54 |
csharp |
curr_connections on each are 369 and 415 |
| 13:55 |
Dyrcona |
Do your run separate memcached on your utility servers? |
| 13:56 |
csharp |
no, they all use those two |
| 13:56 |
Dyrcona |
hmm. That's how I think we're set up.... |
| 13:56 |
csharp |
6 app (EG/apache) servers, 3 utility servers, 2 sip servers |
| 13:57 |
csharp |
(one of the utility servers is not super active - used for high-RAM/processing cron jobs) |
| 13:57 |
Dyrcona |
5 bricks (1 head/2 drones each), 3 utility serviers, 4 SIP servers. |
| 14:00 |
Dyrcona |
According to ipvsadm -l, we have 501 active https connections. |
| 14:01 |
sandbergja |
csharp: it looks like the upgrades for bug 1721036 didn't get stamped either. Okay if I stamp them while I'm on a stamping mission? |
| 14:01 |
pinesol |
Launchpad bug 1721036 in Evergreen "Add the ability for copy tags to be links" [Wishlist,Fix committed] https://launchpad.net/bugs/1721036 |
| 14:03 |
Dyrcona |
And, memcached is not accepting connections. We must have hit 1,000 per.... |
| 14:03 |
csharp |
sandbergja: oh - oops! |
| 14:03 |
csharp |
sandbergja: yes, please! |
| 14:03 |
Dyrcona |
sandbergja: Fine with me... Thanks for catching it! |
| 14:04 |
csharp |
sandbergja++ |
| 14:04 |
sandbergja |
Sure! Let's ignore the fact that I only caught it because I did the exact same thing. :-) |
| 14:04 |
csharp |
<-- still a n00b |
| 14:05 |
sandbergja |
Grabbing 1174 and 1175 |
| 14:06 |
csharp |
jeff: heh - looks like we just hit a "spaces in barcodes" issue here today too |
| 14:07 |
csharp |
310 330 096 495 12 is apparently unable to be retrieved with a normal scan |
| 14:08 |
csharp |
dummy title: hOME fREE dummy author: fERN mICHAELS dummy who input the record: UNKNOWN |
| 14:12 |
Dyrcona |
sandbergja++ |
| 14:13 |
Dyrcona |
Well, I've made sure that our memcached is not visible on the Internet. :) |
| 14:13 |
Dyrcona |
For a moment, I was thinking that might be why we have so many connections. |
| 14:14 |
JBoyer |
Because I've only just caught up, I don't think restarting the app servers was a good fix, but it was the fast one. :-/ |
| 14:14 |
Dyrcona |
Well, rebooting is never a good fix, unless you're updating a kernel. :) |
| 14:15 |
Dyrcona |
But, yes, it can be a fast fix. |
| 14:15 |
JBoyer |
I think the issue was apache (possibly apache-websockets given the timeframe) pigging out on memory and since it only freaked out under very heavy load a proper experiment was hard to set up. |
| 14:15 |
Dyrcona |
Yes, Apache without websockets can be a pig. We've been hitting upwards of 120 Apache processes on some bricks today. |
| 14:15 |
pinesol |
[evergreen|Jane Sandberg] LP1832897 (follow-up): Fixing oils_i18n_gettext key - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=d3b93e4> |
| 14:15 |
pinesol |
[evergreen|Jane Sandberg] LP1832897 and LP1721036: stamping upgrade scripts - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=40b5853> |
| 14:16 |
JBoyer |
Dyrcona, and yes, I had the same reaction "what is this!?" when my previous ILS vendor required weekly restarts of their software on a Windows server. ;) |
| 14:17 |
Dyrcona |
:) |
| 14:17 |
Dyrcona |
You sure that wasn't for Windows update? |
| 14:18 |
JBoyer |
It wasn't a reboot. :p |
| 14:18 |
|
cmalm joined #evergreen |
| 14:18 |
Dyrcona |
:) |
| 14:18 |
JBoyer |
Proper Windows server software should be expected to run for a month, minimum. (Because Patch Tuesday, heh.) |
| 14:18 |
Dyrcona |
So, I'm assuming that memcached running but not responding to a stats request means it is not accepting connections because it is saturated. |
| 14:19 |
Dyrcona |
True... |
| 14:20 |
Dyrcona |
And, now, there are 562 postgres processes, though I'm not sure if that number has any correlation to memcached. |
| 14:21 |
Dyrcona |
I could have just checked the log: Sep 4 14:17:21 mem1 systemd-memcached-wrapper[958]: accept4(): Too many open files |
| 14:32 |
Dyrcona |
Interesting that no one is calling about Evergreen problems..... |
| 14:33 |
JBoyer |
I don't think PG should have any effect on memcached unless you've got the pg-memcache plugin installed, and I don't know if anything uses that in Eg. |
| 14:34 |
JBoyer |
Are you hitting the memcache connection limit (can change in /etc/memcached.conf, I think), or the real per-process open file limit? (yikes) |
| 14:40 |
dbs |
ulimit? |
| 14:43 |
JBoyer |
I just checked since I have a fresh VM and the default limit is 1024 unless you change memcached.conf, so it's probably that; I've run into it before. As for *why*, :emojishrug: |
| 14:57 |
gmcharlt |
well, there is Open-ILS/src/sql/Pg/pgmemcache-ou_tree_traversal_cache.sql |
| 14:58 |
gmcharlt |
so it is in the realm of possibility that there are a couple Evergreen instances invoking memcached directly from Pg backends |
| 14:58 |
JBoyer |
Is that installed by default? |
| 14:59 |
gmcharlt |
no |
| 14:59 |
|
yboston joined #evergreen |
| 15:04 |
Dyrcona |
Sorry, I was away from my desk seeing if we should restart memcached or not. |
| 15:05 |
Dyrcona |
So, to answer dbs and JBoyer: Yes. :) ulimint -n and memached.conf are set at 1024. Though I'm not sure if my user's ulimit applies to daemons. |
| 15:06 |
JBoyer |
Your user's shouldn't, but memcached will manage that limit on its own even if the ulimit is 10x that. |
| 15:06 |
JBoyer |
I assume if you get the limits the other-way-round you get a nice core dump though, heh. |
| 15:07 |
Dyrcona |
Maybe. I've hit ulimit before in my own programs and didn't dump core, but I had code to handle failure to open a file/socket. |
| 15:08 |
JBoyer |
It's that last bit that proves tricky for some folk. ;) (Edge cases? Is that like a sharp corner on a coffee table?) |
| 15:15 |
|
khuckins joined #evergreen |
| 15:19 |
Dyrcona |
Well.... curr_connections: mem1: 114 mem2: 997 |
| 15:19 |
Dyrcona |
That's after about 15 minutes after a restart of memcached. |
| 15:21 |
Dyrcona |
So, yeah, we need another memcached server, or we got a buggy memcached server/client somewhere. |
| 15:22 |
Dyrcona |
And, all of our bricks are heading for 151 Apache processes, one brick is already there. |
| 15:25 |
|
zbanks joined #evergreen |
| 15:26 |
JBoyer |
Dyrcona, well, unless it looks like there's a significant need for more memory you could just up that limit a bit, say 2048 or so. (if *that* limit gets hit without any other changes locally you might have a bigger mystery) |
| 15:29 |
Dyrcona |
True, and we're now using less than 50 Apaches per brick..... |
| 15:33 |
|
yboston joined #evergreen |
| 15:35 |
JBoyer |
Speaking of problems, anyone having any fun EINTEGRITY errors with npm today? (OpenILS/src/eg2) They're my favorite. |
| 15:45 |
JBoyer |
Never mind! rm package-lock.json ; npm install ; success. |
| 15:45 |
JBoyer |
"npm notice created a lockfile as package-lock.json. You should commit this file." -- Something should be committed, alright. |
| 15:47 |
jeff |
*sigh* |
| 15:49 |
JBoyer |
Something *MUST* be done wrong at some stage here, if npm were no more reliable than this it would (hopefully) have died off years ago. Finding out for certain is difficult given that documentation is sparse beyone "you should commit this file..." |
| 15:49 |
JBoyer |
beyond that , even. |
| 15:58 |
Dyrcona |
I've not found npm to ever be reliable. |
| 15:58 |
Dyrcona |
I don't think matters to its main audience. |
| 16:04 |
JBoyer |
They must also be keeping node_modules around or something that we're not doing, because this is silly, even for JS development. |
| 16:05 |
Dyrcona |
:) |
| 16:06 |
Dyrcona |
Well, memcached is full, again.... |
| 16:06 |
Dyrcona |
Or, not. It was and now it isn't, though both are close... |
| 16:09 |
gmcharlt |
I'm only half-joking when I suggest that CPAN should become the distribution mechanism for Node modules |
| 16:11 |
JBoyer |
If CPAN would stop asking and Just Config Already I could get behind that too. |
| 16:13 |
sandbergja |
I should not have to run `rm -r node_modules` nearly as much as I do, but I find it oddly satisfying |
| 16:16 |
Dyrcona |
JBoyer: In CPAN's defense, you *might* know the mirrors that you want to use. For instance, I have worked at two places that used to mirror CPAN, so I put our mirror first in the list. |
| 16:17 |
|
jvwoolf left #evergreen |
| 16:18 |
JBoyer |
And there should be a reasonable way to do that, but asking 100% of the users a question that only .05% will answer one way or the other is doing questions wrong. |
| 16:22 |
Dyrcona |
Well, it's only 1 question now, instead of 2. :) |
| 16:23 |
JBoyer |
They're halfway there! ;) |
| 16:23 |
Dyrcona |
But, yeah....just configure automatically, and let the experts run cpan configure. |
| 16:26 |
Dyrcona |
There's plenty of "you're doing it wrong" to go around. :) |
| 16:31 |
Stompro |
Hello Load Balancer users, what do you use for your health check URL? |
| 16:42 |
JBoyer |
Stompro, I think example.org/ldirectorping.txt is frequently used, unless you mean literally health like checking the apache status plugin? |
| 16:49 |
Stompro |
JBoyer, thanks, I'm using pound proxy, and right now just have it set to check the default catalog page, but that can load fine even if the rest of the system is borked. I wonder if using the OSRF http gateway to make a request like the system version would work better for telling if the system is actually up. |
| 16:50 |
JBoyer |
Depends on what you'd like to check I suppose (and if you have an alternate method for manually removing servers, which is why the .txt method is handy) Even a version check will succeed if the db is dead, for instance. |
| 16:51 |
Stompro |
JBoyer, ah, I was assuming that the version was stored in the server. |
| 16:51 |
Stompro |
Er, database. |
| 16:52 |
JBoyer |
I think that value is hardcoded somewhere, the release scripts set it to a friendly value when tarballs are built. A simple query against something like ccs would be both fast and effective. |
| 16:53 |
JBoyer |
(And should be available without auth) |
| 16:55 |
Stompro |
JBoyer, nice, thanks for the suggestion. |
| 17:06 |
|
mmorgan left #evergreen |
| 17:26 |
|
sandbergja joined #evergreen |
| 17:40 |
jeffdavis |
hm, turns out we have a lot of item barcodes with spaces in them |
| 17:58 |
|
zbanks joined #evergreen |
| 18:02 |
|
zbanks joined #evergreen |
| 18:16 |
|
jihpringle joined #evergreen |
| 19:23 |
|
sandbergja joined #evergreen |
| 20:58 |
|
sandbergja joined #evergreen |
| 23:02 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 23:54 |
|
sandbergja joined #evergreen |
