Evergreen ILS Website

IRC log for #evergreen, 2021-05-26

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
06:00 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
06:07 csharp joined #evergreen
07:15 rjackson_isl_hom joined #evergreen
08:05 collum joined #evergreen
08:34 mmorgan joined #evergreen
09:00 Dyrcona joined #evergreen
09:00 rfrasur joined #evergreen
09:07 mantis1 joined #evergreen
09:36 ejk I mostly lurk here, but I'm disconnecting now. This thread from my IRC client of choice was very interesting: https://twitter.com/IRCCloud​/status/1397475744448663564
09:37 ejk Hope to see y'all again soon
09:38 collum joined #evergreen
09:41 jvwoolf joined #evergreen
09:47 Keith-isl joined #evergreen
10:11 jvwoolf Anybody have any thoughts about what would cause an error like this:
10:11 jvwoolf osrfwarn.log:2021-05-25 15:55:51 *servername* open-ils.cstore: [ERR :40209:osrf_prefork.c:1305:162197250840024184] Drone terminating: unable to notify listener of availability: Broken pipe
10:16 Dyrcona jvwoolf: Lots of things.
10:16 csharp jvwoolf: what are the problems that led you to that error
10:16 csharp ?
10:17 jvwoolf Dyrcona:  csharp: Wondering if it's a symptom or a cause. We ran out of cstore drones yesterday and changed the max. We've been having some brick troubles since then.
10:18 csharp jvwoolf: I would be checking "osrf_control --diagnostic" on the bricks and would be looking at pg_top on the DB
10:18 Dyrcona Look in kern.log or syslog for oom killer messages.
10:18 csharp that too
10:19 csharp jvwoolf: if the DB is having trouble, you might grep for ERROR in the logs when the issues were happening
10:19 Dyrcona The database could be out of connections....
10:20 csharp show max_connections; will let you see what the current setting is from psql
10:20 csharp jvwoolf: if it were me, I would rule out database connection saturation first
10:21 jvwoolf chsarp: Thanks, will take a look at that
10:21 jvwoolf csharp++ Dyrcona++
10:22 Dyrcona Yeah, I agree with csharp on checking the database first.
10:22 jvwoolf Aha
10:22 jvwoolf That's look like what it is. We increased the max_children to higher than our number of DB connections.
10:23 csharp pinesol: what about THE CHILDREN?
10:23 pinesol csharp: Yeah, well, you know, that's just like uh, your opinion, man.
10:24 * csharp 's own max_children setting is 2
10:24 Dyrcona Heh.
10:24 mmorgan csharp: But is that higher than your number of db connections? ;-)
10:25 csharp mmorgan: certainly
10:25 Dyrcona if (newc > INT_MAX - currentc) { errno = EOVERFLOW; return -1;}
10:28 jvwoolf So new question then: Any ideas about what might cause an overabundance of cstore drones?
10:28 jvwoolf We've seen a lot of actor drones before, but this one is new to us
10:32 Dyrcona Cstore is only accessible from the back end, unless you've changed your opensrf_core.xml.
10:32 Dyrcona So, it would be some spamming of back end calls that in turn spam cstore requests.
10:33 Dyrcona Most things end up going through cstore, pcrud, or storage eventually.
10:36 jvwoolf Dyrcona: So basically, it could be anything.
10:37 Dyrcona Unfortunately, yes.
10:38 Dyrcona You could try some tricks with grep to get counts of cstore requests at given times. This could give you some where to start looking.
10:38 jvwoolf Dyrcona: Thanks. I'll see if the apache logs yield anything.
10:39 csharp jvwoolf: when we've seen that kind of thing over and over, it's usually something causing DB slowness (lack of RAM, slow disk, etc.)
10:39 Dyrcona I'm not sure how useful that would actually be.
10:39 Dyrcona It's likely the connection limit.
10:40 jvwoolf We just did a hardware upgrade, so it doesn't seem like it would be that.
10:40 jvwoolf Slowness, that is
10:41 Dyrcona Did you tune Pg for the new hardware?
10:41 Dyrcona Our connection limit is 2000, fwiw.
10:41 jvwoolf Dyrcona: Thanks, that was going to be my next question
10:42 jvwoolf I would imagine John did that, but I will ask him.
10:43 csharp jvwoolf: do you know the specs of the new hardware?
10:43 jvwoolf csharp: Not offhand
10:43 Dyrcona We currently have 533 connections.
10:44 csharp we have 492
10:44 Dyrcona I'm going to share our pg configuration along with hardware specs.
10:44 jvwoolf Our max is 512. So it looks like we may need to increase that.
10:44 jvwoolf How do I see current connections?
10:45 Bmagic we have 500 DB connections to > 700 during mid day
10:45 Bmagic SELECT sum(numbackends) FROM pg_stat_database;
10:47 jvwoolf Bmagic++
10:47 jvwoolf We have 346 right now, I can imagine that going up mid day
10:47 nfBurton joined #evergreen
10:47 Bmagic jvwoolf++ # get your DB on!
10:48 jeff there was some discussion about IRC in the chat interface for the conference this morning, mostly focused around moving the project channels from Freenode to either Libera or OFTC. I won't copy and paste the entire conversation, but I'll echo my part here:
10:48 jeff > While we will need to have a conversation at some point about any changes we want to make to the project's IRC channels, this is probably not the place to have those conversations, and I don't think that those conversations should happen now and detract from the conference-in-progress.
10:48 jeff > Active IRC users might want to claim their nicknames on OFTC and/or Libera if they haven't already, and we can discuss and decide if/where to relocate the channels once the important work of the conference is no longer at risk of being interrupted/derailed. :-)
10:51 Dyrcona jvwoolf: https://docs.google.com/document/d/1atUcq_rDelp​iB_6SrvF8eX_sRaQ-ZpzQVDixlkJVdNE/edit#heading=h.br8toojvg3y
10:52 jvwoolf Dyrcona++ #thank you!
10:55 Bmagic jeff: I hope you can get "jeff" - it'll be weird trying to re-map my brain to each of our changed Nicks
10:56 Dyrcona I would be in favor of OFTC.net, since a number of us already have our nicks reserved over there.
10:56 Bmagic Are there features that we enjoy with IRC that we would lose if we changed to a completely different thing? Like Slack? We'd have to rewrite pinsol I guess
10:57 jeff Bmagic: I indeed have "jeff" on both of those networks. Worry not!
10:57 * Dyrcona is going to have a look at Libera.
10:58 Dyrcona Bmagic: I would be opposed Slack as it is proprietary software.
10:58 Bmagic fair enough, I only mention Slack to make my point understood. Fill in the blank with whatever
11:01 Dyrcona Sure. I suggested ejabberd, but we'd likely end up with a single point of failure if someone in the community ran the server.
11:02 Bmagic AIM, heck yeah
11:04 Dyrcona :)
11:12 rhamby I have no objection to OFTC though I admit I keep forgetting about it even though Koha is there.  It's setup in my quassel client so I mentally forget it's not on Freenode.  :)
11:24 jihpringle joined #evergreen
11:33 gmcharlt also re Slack: its moderation tools are almost nil, as Slack essentially assumes that moderation ultimately gets performed by people's bosses and HR departments
11:36 gmcharlt more broadly - a discussion about alternatives to IRC needs to happen, but given the narrowness of the current question about which IRC network to use, I suggest that the one not be allowed to become a blocker for making a decision on the other
11:37 stompro_ joined #evergreen
12:08 collum joined #evergreen
12:14 stompro_ Has anyone ever looked at which permissions actually get used by staff based on logs?
12:14 collum joined #evergreen
12:38 Dyrcona Hmm. websockets seems to not work over a SOCKS proxy....
12:38 Dyrcona At least going to 1 brick head...
12:40 Dyrcona Is there a way to logout of the staff client if you get a blank worstation registration screen?
12:45 Dyrcona OK. So it isn't the proxy. It may have had something to do with trying to use a private Firefox window.
12:50 Dyrcona Hmm. Starting the SOCKS proxy broke my other SSH tunnels, and vice versa. I wasn't aware that would happen.
12:56 sandbergja joined #evergreen
13:02 jihpringle joined #evergreen
13:04 csharp stompro_: not all perm checks make it to the logs - I think it's useful to compare the list to the perl/JS though
13:04 csharp also the fieldmapper
13:12 JBoyer Dyrcona, I think FF Private Browsing just turns off certain localstorage features rather than throwing them away when the session ends, and we expect some things to always be there.
13:12 * JBoyer is hazy on the details, but may have affected Safari also?
13:16 Dyrcona JBoyer: That sounds about right and jibes with my experience.
13:20 Dyrcona I checked the console. It looks like certain local storage features are unavailable.
13:21 JBoyer That might be fixed in later versions; I have ios devices and Safari working on a local 3.7 install, but I don't recall if I had to add a patch or if that's part of the release.
13:22 mmorgan1 joined #evergreen
13:25 Stompro joined #evergreen
13:41 StomproJ joined #evergreen
13:45 jihpringle joined #evergreen
13:52 Dyrcona My bucket mystery from yesterday remains mysterious. I was using the SOCKS proxy to try the same steps on the brick that the logs indicate the staff person was connected to yesterday. Of course, it works for me.
14:26 StomproJosh joined #evergreen
14:54 pinesol [evergreen|Garry Collum] LP1920042 Bootstrap Opac: Simple Selector in Advanced Search - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=83b3ea1>
14:58 pinesol [evergreen|Garry Collum] LP1910288 Bootstrap opac: E-items lines always display in account summary. - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=37ee16e>
15:10 pinesol [evergreen|Garry Collum] LP1916936-Bootstrap opac: item tags display as separate copies on small screen. - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=1d83a42>
15:10 Dyrcona Ha! So the problem that I earlier thought was related to the SOCKS proxy killing my ssh tunnels was actually caused by my VPN client disconnecting.
15:12 Bmagic Dycrona++ # figuring everything out
15:12 Bmagic Dyrcona++ # dang it
15:12 Bmagic curse my metal body
15:18 sandbergja joined #evergreen
15:20 jihpringle joined #evergreen
15:26 pinesol [evergreen|Jason Boyer] LP1901710: Properly nest HTML tags and TT2 directives - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=b72f90c>
15:36 pinesol [evergreen|Garry Collum] lp1902302 Opac. SMS and Carriers appear in Edit Holds Regardless of OU Setting. - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=33c341f>
15:37 gmcharlt sandbergja: re 1898114, any reason not to slap an limitPerms onto the new org selector?
15:43 gmcharlt sandbergja: *quack* cries the duck! to answer my own question, we can't because that interface is still AngularJS
15:47 Christineb joined #evergreen
15:47 JamiG joined #evergreen
15:56 tlittle joined #evergreen
15:57 jvwoolf left #evergreen
16:04 tlittle berick Your new WIP branch for the Angular acq interface doesn't touch Load MARC Order Records, correct? It doesn't look like it, but just double-checking I didn't miss something
16:06 pinesol [evergreen|Jane Sandberg] LP1898114: Add org selector to holds pull list - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=9b7a29a>
16:06 pinesol [evergreen|Galen Charlton] LP#1898114: (follow-up) adjust the style of the org selector - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=2d94406>
16:07 Stompro joined #evergreen
16:09 jvwoolf joined #evergreen
16:21 * gmcharlt claims 1263
16:24 jvwoolf Well, I think we found our "Typhoid Mary" that lead to the overabundance of cstore drones yesterday
16:24 jvwoolf Evidently a library added 78 ILL patron accounts to a single patron group
16:26 jvwoolf Looking to see if there is a Launchpad bug on this...
16:26 gmcharlt jvwoolf: as in a family gruop?
16:26 jvwoolf gmcharlt: Yes
16:26 gmcharlt thanks
16:27 mmorgan jvwoolf: Was it intentional, or did they just clone a record 77 times?
16:27 jvwoolf mmorgan: It was intentional
16:27 jvwoolf They wanted an easy way to get to the other accounts. We discussed other options.
16:27 mmorgan Interesting. Very creative!
16:28 jvwoolf Indeed! Too bad it crashed the system. LOL
16:28 pinesol [evergreen|Bill Erickson] LP1913338 Catalog search form visibility stickiness - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=2955ee6>
16:28 pinesol [evergreen|Chris Sharp] LP#1913338: Typo correction - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=11dc6e2>
16:28 mmorgan Sounds like a good use for a patron bucket.
16:28 pinesol [evergreen|Galen Charlton] LP#1913338: (follow-up) aria-hide the icon names - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=f274668>
16:28 pinesol [evergreen|Galen Charlton] LP#1913338: stamp schema update - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=6c43a08>
16:29 mmorgan gmcharlt++ # committing!
16:31 jvwoolf mmorgan: That was one of the ideas I gave them
16:34 jvwoolf Not coming up with anything in Launchpad. Perhaps I'll try to recreate the problem on a test system during the hackfest on Friday so I can do a complete bug report.
16:34 pinesol [evergreen|Jason Stephenson] LP#1856868: Remove deprecated open-ils.circ.renew.auto API - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=4210059>
16:55 jihpringle joined #evergreen
16:56 kip joined #evergreen
16:56 rhamby joined #evergreen
16:56 tlittle joined #evergreen
17:00 yar_ joined #evergreen
17:01 StomproJosh joined #evergreen
17:03 jwoodard joined #evergreen
17:04 StomproJ joined #evergreen
17:06 mantis2 joined #evergreen
17:08 Stompro joined #evergreen
17:09 jwoodard left #evergreen
17:29 jwoodard joined #evergreen
17:45 Christineb joined #evergreen
17:58 mmorgan left #evergreen
18:00 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
18:03 StomproJosh joined #evergreen
18:38 mantis2 left #evergreen
19:13 jvwoolf1 joined #evergreen
19:22 jvwoolf1 left #evergreen
19:49 nfBurton joined #evergreen
20:59 StomproJosh joined #evergreen
21:20 sandbergja joined #evergreen
22:12 sandbergja joined #evergreen
22:50 dickreckard joined #evergreen
23:42 nfBurton joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat