Evergreen ILS Website

IRC log for #evergreen, 2022-01-18

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
06:02 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
06:36 rjackson_isl_hom joined #evergreen
07:56 collum joined #evergreen
08:31 mantis1 joined #evergreen
08:36 mmorgan joined #evergreen
09:02 rfrasur joined #evergreen
09:03 terranm joined #evergreen
09:27 Dyrcona joined #evergreen
10:44 Dyrcona Chrome's tab UI suddently became frustrating. I swear I could move tabs around just by clicking and dragging. Today, when I drag a tab it makes a new window.
10:46 Dyrcona So, to fix the issue I closed all tabs to the right of where I wanted this tab to be, then reopened the others. I wouldn't have this problem if it wasn't so easy to accidentally drag a tab.
10:46 Dyrcona Maybe I should try tab groups, but honestly, it's "just a browser" the UI shouldn't be complicated.
10:47 mmorgan Dyrcona: I move tabs around in Chrome all the time, and am not seeing any changes in behavior.
10:48 * mmorgan always has too many tabs open...
10:48 Dyrcona I'm using Version 97.0.4692.71 (Official Build) (64-bit) on Ubuntu, and I only noticed this today, but I don't think I've needed to move tabs much lately.
10:49 Dyrcona Anyway, I have a meeting at 11, and I've wasted too much time on this.
10:50 mmorgan rabbit-holes--
10:51 Dyrcona OMG! The Agenda is HUGE!
10:53 Dyrcona Oh.... It has stuff from past meetings in the document.... :/
11:14 nfBurton joined #evergreen
11:14 nfBurton2 joined #evergreen
11:33 jihpringle joined #evergreen
11:55 jihpringle joined #evergreen
12:20 nfBurton Is there an easy way to do date manipulation in the action/trigger email templates?
12:21 mmorgan nfBurton: Do you mean to control how the date appears on notifications?
12:22 nfBurton Well, we keep holds for 5 days and they'd like the hold expiry to represented. So I'm looking to use today's date and add 5 days
12:23 nfBurton But now I think of it, maybe there is a hold expiry variable?
12:24 mmorgan shelf_expire_time is the field in the database
12:25 nfBurton Making a mountain out of a molehill I guess. I tried a few different ways to date manipulate and kept getting errors
12:28 mmorgan nfBurton: It's all perspective. You don't know whether it's a mountain or a molehill till you get closer to it :)
12:29 Dyrcona nfBurton: There's a helpers.date_format function available in the a/t templates.
12:30 nfBurton yeah. I can format time but seems to be no way to manipulate it
12:30 Dyrcona Oh, nm. I should everything more carefully before answering.
12:30 nfBurton Guess I am stuck in the prese nt
12:31 nfBurton The variable worked. I just want to manipulate time so bad lol
12:31 Dyrcona :)
12:33 * mmorgan has done something like this to add text that will automatcally expire to a notice:
12:33 mmorgan [%- IF date.format(format => '%Y%m%d') < 20211227 %]
12:40 nfBurton Sometimes you just need to say something to think of the answer lol. Thanks :D
12:42 csharp_ we're dying on 3.8
12:42 csharp_ open-ils.actor listener keeps falling offline and actor drones are pushing up to 100% kind of constantly
12:42 csharp_ Ubuntu 18.04
12:43 csharp_ I'm thinking it has to be related to Angular cataloging UIs, but I don't know
12:51 miker csharp_: did you update opensrf?
12:52 collum joined #evergreen
12:52 csharp_ miker: we're running 3.2.2
12:57 terranm joined #evergreen
13:01 miker csharp_: a couple log error lines to look for: "server: died with error Use of freed value in iteration" and "server: died with error Can't kill a non-numeric process ID" ... if you're seeing either of those, I recommend looking at Galen's three branches at the top of the working repo: https://git.evergreen-ils.org/?​p=working/OpenSRF.git;a=summary
13:16 miker csharp_: fwiw, it looks like 18.04 has perl 5.26, and at least the problem described in https://git.evergreen-ils.org/?p=wor​king/OpenSRF.git;a=commitdiff;h=6fad​f10c8dee0fe64e5273fef3595e82f1ac9baa can happen on perl 5.24 or later
13:17 csharp_ yes, I had applied those to OpenSRF 3.2.1, but forgot to add them to 3.2.2 :-/
13:17 csharp_ thanks for the tip - we'll see how it goes
13:27 csharp_ miker: that seems to have calmed things down
13:27 csharp_ miker++
13:30 terranm miker++ csharp_++
13:30 JBoyer csharp_, this can certainly wait until you've shaken loose any other post-upgrade blues, but as that makes multiple large production systems running those patches a signoff would help get those into 3.2.3 (or 3.3.0 or whatever) :)
13:30 JBoyer and yes, csharp_ ++ miker ++
13:31 csharp_ yeah, I'd be happy to
13:37 csharp_ spoke too soon though - something's still killing off open-ils.actor
13:38 csharp_ I think it has to be something an individual is doing and I suspect either cataloging or acq
13:40 Dyrcona csharp_: Is it a lot of settings requests?
13:40 csharp_ I know there *are* a lot of settings requests, but I haven't definitely connected those to the cause
13:43 csharp_ 2263 open-ils.actor.settings.retrieve.atomic, 2003 open-ils.actor.ou_setting.ancestor_default.batch,  1440 open-ils.actor.get_barcodes, 1454 open-ils.actor.user.has_work_perm_at.batch
13:43 csharp_ those are the highest on the particular server that just died off
13:44 Dyrcona Well, on the plus side, the batch ones could be getting more than 1 setting at a time. :)
13:50 Dyrcona Doesn't the client cache the settings?
13:52 jeffdavis I'd expect to see numbers like that over the course of an hour
13:53 csharp_ yeah
13:54 csharp_ I'm not sure if this is a quantity problem or a data problem (or a settings problem in ejabberd)
13:54 csharp_ the listener dying off doesn't happen consistently with high drone counts
13:54 jeff when you say "killing off open-ils.actor", what are your symptoms?
13:55 jeff classic "NOT CONNECTED TO THE NETWORK", or something else?
13:57 csharp_ jeff: http://irc.evergreen-ils.org/​evergreen/2021-12-14#i_497042 - also the logs are WS received XMPP error message in response to thread=0.72683392379099751642531798868 and recipient=router@public.brick04-head.gapines.org/open-ils.actor.  Likely the recipient is not accessible/available.
13:58 csharp_ and2022-01-18 13:36:53 brick03-head open-ils.actor: [ERR :92115:XMPPReader.pm:171:] Disconnected from Jabber server, exiting immediately
13:59 csharp_ it's like ejabberd drops the client then opensrf discovers that later, then removes the service
14:00 csharp_ @eightball will OpenSRF ever just respawn the listener?
14:00 pinesol csharp_: Maybe...
14:06 jeffdavis anything of interest in dmesg output?
14:21 jeff Ubuntu 18.04, so ejabberd... 18?
14:22 jeff What does the process appear to be doing / handling when it's disconnected from the Jabber server? Any clues there?
14:22 jeff Any reason for the disconnection in the ejabberd logs?
14:23 jeff Have you already shared your ejabberd config / changes?
14:24 jeff the log you linked to was talking about death with a very specific template being fetched, I think. Is this that again, or was that specific example resolved?
14:31 Dyrcona jeff: From my experience, the ejabberd logs are useless when things like this happen. You have to crank the verbosity up to where there's way more noise than signal.
14:31 Dyrcona IIRC on the ejabberd side it looks like the client went away.
14:32 Dyrcona While on the client side, it looks like ejabberd told the client to go away.
14:35 * jeff nods
14:42 jeff packet capture might help isolate the 503 if logs don't (or rule out a 503). if you're not exceeding max connections, I wonder if you're sending invalid xml, either due to a weird value or due to a message being split in a bad place. I thought that there was at least one of those and it was fixed a while ago.
14:50 jihpringle80 joined #evergreen
15:00 csharp_ jeffdavis: huh - your question about dmesg sent me down a path that has led me to a couple of possibilities
15:00 jeffdavis yay for possible progress!
15:00 csharp_ one is apparmor, which is showing denials for ejabberd (that's an obvious possibility) and another: TCP: request_sock_TCP: Possible SYN flooding on port 5222. Sending cookies.  Check SNMP counters.
15:03 JBoyer csharp_, just to rule this out since I've seen it do some really bizarre stuff before, the max_user_sessions value is 10000 and not just 10, right?
15:03 csharp_ yeah, it's at 10000
15:03 Dyrcona Yay for apparmor keeping us "safe" from ourselves......
15:04 Dyrcona Seems to be a lot of that going around these days.....
15:05 Dyrcona Although, I don't mind if the server sends me cookies, preferably chocolate chip.
15:06 csharp_ @quote add  <Dyrcona> Although, I don't mind if the server sends me cookies, preferably chocolate chip.
15:06 pinesol csharp_: Error: You must be registered to use this command. If you are already registered, you must either identify (using the identify command) or add a hostmask matching your current hostmask (using the "hostmask add" command).
15:06 Dyrcona Speaking off..... *gets up and walks off to get a cookie from the kithcen*
15:07 csharp_ @quote add  <Dyrcona> Although, I don't mind if the server sends me cookies, preferably chocolate chip.
15:07 pinesol csharp_: The operation succeeded.  Quote #220 added.
15:09 jihpringle joined #evergreen
15:15 csharp_ ok, I've removed ejabberd from apparmor and increased the net.core.somaxconn setting from 128 to 2048
15:16 csharp_ another possible setting to tweak in sysctl is net.ipv4.tcp_max_syn_backlog (currently 1024)
15:16 jeff don't forget to bump the ejabberd listen backlog also.
15:19 * csharp_ looks for that in the config
15:21 jeff it's an optional argument to the "listen" blocks
15:21 jeff https://docs.ejabberd.im/admin/con​figuration/listen-options/#backlog
15:21 jeff by default i think the mqtt listener has one specified
15:25 jeff "ss -nlt" should help you see if the value has changed once you reconfigure.
15:25 jeff look for the port matching the listener you changed, like 5222... listen backlog is the value in the SendQ column.
15:35 jeff (Send-Q, sorry)
15:40 csharp_ jeff: thanks - I upped it to 10 on each server - not sure if that's enough
15:53 csharp_ now for some reason, the login window isn't loading and no obvious JS console output :-)
15:53 csharp_ :-(
16:04 csharp_ ...which of course was self-inflicted when I didn't build opensrf with --with-websockets-port=443
16:25 csharp_ drone counts are a bit better but the open-ils.actor listener is dying randomly
16:25 csharp_ it's enough that I can't walk away for 10 minutes
16:34 terranm joined #evergreen
16:40 JBoyer csharp_, anything useful in the logs?
16:40 JBoyer Or just more of the same as above? (Disconnected from Jabber...)
16:52 jihpringle joined #evergreen
16:54 miker csharp_: if you haven't already, perhaps check the the stderr log at /openils/var/log/open-ils.actor_stderr.log (by default) to see if anything is getting into there when the listener has issues. and catch-all syslog logs
17:09 mmorgan left #evergreen
17:49 jeffdavis might also be worth adding some more logging to Server.pm to try and narrow down the exact point where the listener is dying
18:01 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat