| Time |
Nick |
Message |
| 06:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 06:07 |
|
csharp joined #evergreen |
| 07:15 |
|
rjackson_isl_hom joined #evergreen |
| 08:05 |
|
collum joined #evergreen |
| 08:34 |
|
mmorgan joined #evergreen |
| 09:00 |
|
Dyrcona joined #evergreen |
| 09:00 |
|
rfrasur joined #evergreen |
| 09:07 |
|
mantis1 joined #evergreen |
| 09:36 |
ejk |
I mostly lurk here, but I'm disconnecting now. This thread from my IRC client of choice was very interesting: https://twitter.com/IRCCloud/status/1397475744448663564 |
| 09:37 |
ejk |
Hope to see y'all again soon |
| 09:38 |
|
collum joined #evergreen |
| 09:41 |
|
jvwoolf joined #evergreen |
| 09:47 |
|
Keith-isl joined #evergreen |
| 10:11 |
jvwoolf |
Anybody have any thoughts about what would cause an error like this: |
| 10:11 |
jvwoolf |
osrfwarn.log:2021-05-25 15:55:51 *servername* open-ils.cstore: [ERR :40209:osrf_prefork.c:1305:162197250840024184] Drone terminating: unable to notify listener of availability: Broken pipe |
| 10:16 |
Dyrcona |
jvwoolf: Lots of things. |
| 10:16 |
csharp |
jvwoolf: what are the problems that led you to that error |
| 10:16 |
csharp |
? |
| 10:17 |
jvwoolf |
Dyrcona: csharp: Wondering if it's a symptom or a cause. We ran out of cstore drones yesterday and changed the max. We've been having some brick troubles since then. |
| 10:18 |
csharp |
jvwoolf: I would be checking "osrf_control --diagnostic" on the bricks and would be looking at pg_top on the DB |
| 10:18 |
Dyrcona |
Look in kern.log or syslog for oom killer messages. |
| 10:18 |
csharp |
that too |
| 10:19 |
csharp |
jvwoolf: if the DB is having trouble, you might grep for ERROR in the logs when the issues were happening |
| 10:19 |
Dyrcona |
The database could be out of connections.... |
| 10:20 |
csharp |
show max_connections; will let you see what the current setting is from psql |
| 10:20 |
csharp |
jvwoolf: if it were me, I would rule out database connection saturation first |
| 10:21 |
jvwoolf |
chsarp: Thanks, will take a look at that |
| 10:21 |
jvwoolf |
csharp++ Dyrcona++ |
| 10:22 |
Dyrcona |
Yeah, I agree with csharp on checking the database first. |
| 10:22 |
jvwoolf |
Aha |
| 10:22 |
jvwoolf |
That's look like what it is. We increased the max_children to higher than our number of DB connections. |
| 10:23 |
csharp |
pinesol: what about THE CHILDREN? |
| 10:23 |
pinesol |
csharp: Yeah, well, you know, that's just like uh, your opinion, man. |
| 10:24 |
* csharp |
's own max_children setting is 2 |
| 10:24 |
Dyrcona |
Heh. |
| 10:24 |
mmorgan |
csharp: But is that higher than your number of db connections? ;-) |
| 10:25 |
csharp |
mmorgan: certainly |
| 10:25 |
Dyrcona |
if (newc > INT_MAX - currentc) { errno = EOVERFLOW; return -1;} |
| 10:28 |
jvwoolf |
So new question then: Any ideas about what might cause an overabundance of cstore drones? |
| 10:28 |
jvwoolf |
We've seen a lot of actor drones before, but this one is new to us |
| 10:32 |
Dyrcona |
Cstore is only accessible from the back end, unless you've changed your opensrf_core.xml. |
| 10:32 |
Dyrcona |
So, it would be some spamming of back end calls that in turn spam cstore requests. |
| 10:33 |
Dyrcona |
Most things end up going through cstore, pcrud, or storage eventually. |
| 10:36 |
jvwoolf |
Dyrcona: So basically, it could be anything. |
| 10:37 |
Dyrcona |
Unfortunately, yes. |
| 10:38 |
Dyrcona |
You could try some tricks with grep to get counts of cstore requests at given times. This could give you some where to start looking. |
| 10:38 |
jvwoolf |
Dyrcona: Thanks. I'll see if the apache logs yield anything. |
| 10:39 |
csharp |
jvwoolf: when we've seen that kind of thing over and over, it's usually something causing DB slowness (lack of RAM, slow disk, etc.) |
| 10:39 |
Dyrcona |
I'm not sure how useful that would actually be. |
| 10:39 |
Dyrcona |
It's likely the connection limit. |
| 10:40 |
jvwoolf |
We just did a hardware upgrade, so it doesn't seem like it would be that. |
| 10:40 |
jvwoolf |
Slowness, that is |
| 10:41 |
Dyrcona |
Did you tune Pg for the new hardware? |
| 10:41 |
Dyrcona |
Our connection limit is 2000, fwiw. |
| 10:41 |
jvwoolf |
Dyrcona: Thanks, that was going to be my next question |
| 10:42 |
jvwoolf |
I would imagine John did that, but I will ask him. |
| 10:43 |
csharp |
jvwoolf: do you know the specs of the new hardware? |
| 10:43 |
jvwoolf |
csharp: Not offhand |
| 10:43 |
Dyrcona |
We currently have 533 connections. |
| 10:44 |
csharp |
we have 492 |
| 10:44 |
Dyrcona |
I'm going to share our pg configuration along with hardware specs. |
| 10:44 |
jvwoolf |
Our max is 512. So it looks like we may need to increase that. |
| 10:44 |
jvwoolf |
How do I see current connections? |
| 10:45 |
Bmagic |
we have 500 DB connections to > 700 during mid day |
| 10:45 |
Bmagic |
SELECT sum(numbackends) FROM pg_stat_database; |
| 10:47 |
jvwoolf |
Bmagic++ |
| 10:47 |
jvwoolf |
We have 346 right now, I can imagine that going up mid day |
| 10:47 |
|
nfBurton joined #evergreen |
| 10:47 |
Bmagic |
jvwoolf++ # get your DB on! |
| 10:48 |
jeff |
there was some discussion about IRC in the chat interface for the conference this morning, mostly focused around moving the project channels from Freenode to either Libera or OFTC. I won't copy and paste the entire conversation, but I'll echo my part here: |
| 10:48 |
jeff |
> While we will need to have a conversation at some point about any changes we want to make to the project's IRC channels, this is probably not the place to have those conversations, and I don't think that those conversations should happen now and detract from the conference-in-progress. |
| 10:48 |
jeff |
> Active IRC users might want to claim their nicknames on OFTC and/or Libera if they haven't already, and we can discuss and decide if/where to relocate the channels once the important work of the conference is no longer at risk of being interrupted/derailed. :-) |
| 10:51 |
Dyrcona |
jvwoolf: https://docs.google.com/document/d/1atUcq_rDelpiB_6SrvF8eX_sRaQ-ZpzQVDixlkJVdNE/edit#heading=h.br8toojvg3y |
| 10:52 |
jvwoolf |
Dyrcona++ #thank you! |
| 10:55 |
Bmagic |
jeff: I hope you can get "jeff" - it'll be weird trying to re-map my brain to each of our changed Nicks |
| 10:56 |
Dyrcona |
I would be in favor of OFTC.net, since a number of us already have our nicks reserved over there. |
| 10:56 |
Bmagic |
Are there features that we enjoy with IRC that we would lose if we changed to a completely different thing? Like Slack? We'd have to rewrite pinsol I guess |
| 10:57 |
jeff |
Bmagic: I indeed have "jeff" on both of those networks. Worry not! |
| 10:57 |
* Dyrcona |
is going to have a look at Libera. |
| 10:58 |
Dyrcona |
Bmagic: I would be opposed Slack as it is proprietary software. |
| 10:58 |
Bmagic |
fair enough, I only mention Slack to make my point understood. Fill in the blank with whatever |
| 11:01 |
Dyrcona |
Sure. I suggested ejabberd, but we'd likely end up with a single point of failure if someone in the community ran the server. |
| 11:02 |
Bmagic |
AIM, heck yeah |
| 11:04 |
Dyrcona |
:) |
| 11:12 |
rhamby |
I have no objection to OFTC though I admit I keep forgetting about it even though Koha is there. It's setup in my quassel client so I mentally forget it's not on Freenode. :) |
| 11:24 |
|
jihpringle joined #evergreen |
| 11:33 |
gmcharlt |
also re Slack: its moderation tools are almost nil, as Slack essentially assumes that moderation ultimately gets performed by people's bosses and HR departments |
| 11:36 |
gmcharlt |
more broadly - a discussion about alternatives to IRC needs to happen, but given the narrowness of the current question about which IRC network to use, I suggest that the one not be allowed to become a blocker for making a decision on the other |
| 11:37 |
|
stompro_ joined #evergreen |
| 12:08 |
|
collum joined #evergreen |
| 12:14 |
stompro_ |
Has anyone ever looked at which permissions actually get used by staff based on logs? |
| 12:14 |
|
collum joined #evergreen |
| 12:38 |
Dyrcona |
Hmm. websockets seems to not work over a SOCKS proxy.... |
| 12:38 |
Dyrcona |
At least going to 1 brick head... |
| 12:40 |
Dyrcona |
Is there a way to logout of the staff client if you get a blank worstation registration screen? |
| 12:45 |
Dyrcona |
OK. So it isn't the proxy. It may have had something to do with trying to use a private Firefox window. |
| 12:50 |
Dyrcona |
Hmm. Starting the SOCKS proxy broke my other SSH tunnels, and vice versa. I wasn't aware that would happen. |
| 12:56 |
|
sandbergja joined #evergreen |
| 13:02 |
|
jihpringle joined #evergreen |
| 13:04 |
csharp |
stompro_: not all perm checks make it to the logs - I think it's useful to compare the list to the perl/JS though |
| 13:04 |
csharp |
also the fieldmapper |
| 13:12 |
JBoyer |
Dyrcona, I think FF Private Browsing just turns off certain localstorage features rather than throwing them away when the session ends, and we expect some things to always be there. |
| 13:12 |
* JBoyer |
is hazy on the details, but may have affected Safari also? |
| 13:16 |
Dyrcona |
JBoyer: That sounds about right and jibes with my experience. |
| 13:20 |
Dyrcona |
I checked the console. It looks like certain local storage features are unavailable. |
| 13:21 |
JBoyer |
That might be fixed in later versions; I have ios devices and Safari working on a local 3.7 install, but I don't recall if I had to add a patch or if that's part of the release. |
| 13:22 |
|
mmorgan1 joined #evergreen |
| 13:25 |
|
Stompro joined #evergreen |
| 13:41 |
|
StomproJ joined #evergreen |
| 13:45 |
|
jihpringle joined #evergreen |
| 13:52 |
Dyrcona |
My bucket mystery from yesterday remains mysterious. I was using the SOCKS proxy to try the same steps on the brick that the logs indicate the staff person was connected to yesterday. Of course, it works for me. |
| 14:26 |
|
StomproJosh joined #evergreen |
| 14:54 |
pinesol |
[evergreen|Garry Collum] LP1920042 Bootstrap Opac: Simple Selector in Advanced Search - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=83b3ea1> |
| 14:58 |
pinesol |
[evergreen|Garry Collum] LP1910288 Bootstrap opac: E-items lines always display in account summary. - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=37ee16e> |
| 15:10 |
pinesol |
[evergreen|Garry Collum] LP1916936-Bootstrap opac: item tags display as separate copies on small screen. - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=1d83a42> |
| 15:10 |
Dyrcona |
Ha! So the problem that I earlier thought was related to the SOCKS proxy killing my ssh tunnels was actually caused by my VPN client disconnecting. |
| 15:12 |
Bmagic |
Dycrona++ # figuring everything out |
| 15:12 |
Bmagic |
Dyrcona++ # dang it |
| 15:12 |
Bmagic |
curse my metal body |
| 15:18 |
|
sandbergja joined #evergreen |
| 15:20 |
|
jihpringle joined #evergreen |
| 15:26 |
pinesol |
[evergreen|Jason Boyer] LP1901710: Properly nest HTML tags and TT2 directives - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=b72f90c> |
| 15:36 |
pinesol |
[evergreen|Garry Collum] lp1902302 Opac. SMS and Carriers appear in Edit Holds Regardless of OU Setting. - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=33c341f> |
| 15:37 |
gmcharlt |
sandbergja: re 1898114, any reason not to slap an limitPerms onto the new org selector? |
| 15:43 |
gmcharlt |
sandbergja: *quack* cries the duck! to answer my own question, we can't because that interface is still AngularJS |
| 15:47 |
|
Christineb joined #evergreen |
| 15:47 |
|
JamiG joined #evergreen |
| 15:56 |
|
tlittle joined #evergreen |
| 15:57 |
|
jvwoolf left #evergreen |
| 16:04 |
tlittle |
berick Your new WIP branch for the Angular acq interface doesn't touch Load MARC Order Records, correct? It doesn't look like it, but just double-checking I didn't miss something |
| 16:06 |
pinesol |
[evergreen|Jane Sandberg] LP1898114: Add org selector to holds pull list - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=9b7a29a> |
| 16:06 |
pinesol |
[evergreen|Galen Charlton] LP#1898114: (follow-up) adjust the style of the org selector - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=2d94406> |
| 16:07 |
|
Stompro joined #evergreen |
| 16:09 |
|
jvwoolf joined #evergreen |
| 16:21 |
* gmcharlt |
claims 1263 |
| 16:24 |
jvwoolf |
Well, I think we found our "Typhoid Mary" that lead to the overabundance of cstore drones yesterday |
| 16:24 |
jvwoolf |
Evidently a library added 78 ILL patron accounts to a single patron group |
| 16:26 |
jvwoolf |
Looking to see if there is a Launchpad bug on this... |
| 16:26 |
gmcharlt |
jvwoolf: as in a family gruop? |
| 16:26 |
jvwoolf |
gmcharlt: Yes |
| 16:26 |
gmcharlt |
thanks |
| 16:27 |
mmorgan |
jvwoolf: Was it intentional, or did they just clone a record 77 times? |
| 16:27 |
jvwoolf |
mmorgan: It was intentional |
| 16:27 |
jvwoolf |
They wanted an easy way to get to the other accounts. We discussed other options. |
| 16:27 |
mmorgan |
Interesting. Very creative! |
| 16:28 |
jvwoolf |
Indeed! Too bad it crashed the system. LOL |
| 16:28 |
pinesol |
[evergreen|Bill Erickson] LP1913338 Catalog search form visibility stickiness - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=2955ee6> |
| 16:28 |
pinesol |
[evergreen|Chris Sharp] LP#1913338: Typo correction - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=11dc6e2> |
| 16:28 |
mmorgan |
Sounds like a good use for a patron bucket. |
| 16:28 |
pinesol |
[evergreen|Galen Charlton] LP#1913338: (follow-up) aria-hide the icon names - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=f274668> |
| 16:28 |
pinesol |
[evergreen|Galen Charlton] LP#1913338: stamp schema update - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=6c43a08> |
| 16:29 |
mmorgan |
gmcharlt++ # committing! |
| 16:31 |
jvwoolf |
mmorgan: That was one of the ideas I gave them |
| 16:34 |
jvwoolf |
Not coming up with anything in Launchpad. Perhaps I'll try to recreate the problem on a test system during the hackfest on Friday so I can do a complete bug report. |
| 16:34 |
pinesol |
[evergreen|Jason Stephenson] LP#1856868: Remove deprecated open-ils.circ.renew.auto API - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=4210059> |
| 16:55 |
|
jihpringle joined #evergreen |
| 16:56 |
|
kip joined #evergreen |
| 16:56 |
|
rhamby joined #evergreen |
| 16:56 |
|
tlittle joined #evergreen |
| 17:00 |
|
yar_ joined #evergreen |
| 17:01 |
|
StomproJosh joined #evergreen |
| 17:03 |
|
jwoodard joined #evergreen |
| 17:04 |
|
StomproJ joined #evergreen |
| 17:06 |
|
mantis2 joined #evergreen |
| 17:08 |
|
Stompro joined #evergreen |
| 17:09 |
|
jwoodard left #evergreen |
| 17:29 |
|
jwoodard joined #evergreen |
| 17:45 |
|
Christineb joined #evergreen |
| 17:58 |
|
mmorgan left #evergreen |
| 18:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:03 |
|
StomproJosh joined #evergreen |
| 18:38 |
|
mantis2 left #evergreen |
| 19:13 |
|
jvwoolf1 joined #evergreen |
| 19:22 |
|
jvwoolf1 left #evergreen |
| 19:49 |
|
nfBurton joined #evergreen |
| 20:59 |
|
StomproJosh joined #evergreen |
| 21:20 |
|
sandbergja joined #evergreen |
| 22:12 |
|
sandbergja joined #evergreen |
| 22:50 |
|
dickreckard joined #evergreen |
| 23:42 |
|
nfBurton joined #evergreen |
