| Time |
Nick |
Message |
| 06:02 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 06:36 |
|
rjackson_isl_hom joined #evergreen |
| 07:56 |
|
collum joined #evergreen |
| 08:31 |
|
mantis1 joined #evergreen |
| 08:36 |
|
mmorgan joined #evergreen |
| 09:02 |
|
rfrasur joined #evergreen |
| 09:03 |
|
terranm joined #evergreen |
| 09:27 |
|
Dyrcona joined #evergreen |
| 10:44 |
Dyrcona |
Chrome's tab UI suddently became frustrating. I swear I could move tabs around just by clicking and dragging. Today, when I drag a tab it makes a new window. |
| 10:46 |
Dyrcona |
So, to fix the issue I closed all tabs to the right of where I wanted this tab to be, then reopened the others. I wouldn't have this problem if it wasn't so easy to accidentally drag a tab. |
| 10:46 |
Dyrcona |
Maybe I should try tab groups, but honestly, it's "just a browser" the UI shouldn't be complicated. |
| 10:47 |
mmorgan |
Dyrcona: I move tabs around in Chrome all the time, and am not seeing any changes in behavior. |
| 10:48 |
* mmorgan |
always has too many tabs open... |
| 10:48 |
Dyrcona |
I'm using Version 97.0.4692.71 (Official Build) (64-bit) on Ubuntu, and I only noticed this today, but I don't think I've needed to move tabs much lately. |
| 10:49 |
Dyrcona |
Anyway, I have a meeting at 11, and I've wasted too much time on this. |
| 10:50 |
mmorgan |
rabbit-holes-- |
| 10:51 |
Dyrcona |
OMG! The Agenda is HUGE! |
| 10:53 |
Dyrcona |
Oh.... It has stuff from past meetings in the document.... :/ |
| 11:14 |
|
nfBurton joined #evergreen |
| 11:14 |
|
nfBurton2 joined #evergreen |
| 11:33 |
|
jihpringle joined #evergreen |
| 11:55 |
|
jihpringle joined #evergreen |
| 12:20 |
nfBurton |
Is there an easy way to do date manipulation in the action/trigger email templates? |
| 12:21 |
mmorgan |
nfBurton: Do you mean to control how the date appears on notifications? |
| 12:22 |
nfBurton |
Well, we keep holds for 5 days and they'd like the hold expiry to represented. So I'm looking to use today's date and add 5 days |
| 12:23 |
nfBurton |
But now I think of it, maybe there is a hold expiry variable? |
| 12:24 |
mmorgan |
shelf_expire_time is the field in the database |
| 12:25 |
nfBurton |
Making a mountain out of a molehill I guess. I tried a few different ways to date manipulate and kept getting errors |
| 12:28 |
mmorgan |
nfBurton: It's all perspective. You don't know whether it's a mountain or a molehill till you get closer to it :) |
| 12:29 |
Dyrcona |
nfBurton: There's a helpers.date_format function available in the a/t templates. |
| 12:30 |
nfBurton |
yeah. I can format time but seems to be no way to manipulate it |
| 12:30 |
Dyrcona |
Oh, nm. I should everything more carefully before answering. |
| 12:30 |
nfBurton |
Guess I am stuck in the prese nt |
| 12:31 |
nfBurton |
The variable worked. I just want to manipulate time so bad lol |
| 12:31 |
Dyrcona |
:) |
| 12:33 |
* mmorgan |
has done something like this to add text that will automatcally expire to a notice: |
| 12:33 |
mmorgan |
[%- IF date.format(format => '%Y%m%d') < 20211227 %] |
| 12:40 |
nfBurton |
Sometimes you just need to say something to think of the answer lol. Thanks :D |
| 12:42 |
csharp_ |
we're dying on 3.8 |
| 12:42 |
csharp_ |
open-ils.actor listener keeps falling offline and actor drones are pushing up to 100% kind of constantly |
| 12:42 |
csharp_ |
Ubuntu 18.04 |
| 12:43 |
csharp_ |
I'm thinking it has to be related to Angular cataloging UIs, but I don't know |
| 12:51 |
miker |
csharp_: did you update opensrf? |
| 12:52 |
|
collum joined #evergreen |
| 12:52 |
csharp_ |
miker: we're running 3.2.2 |
| 12:57 |
|
terranm joined #evergreen |
| 13:01 |
miker |
csharp_: a couple log error lines to look for: "server: died with error Use of freed value in iteration" and "server: died with error Can't kill a non-numeric process ID" ... if you're seeing either of those, I recommend looking at Galen's three branches at the top of the working repo: https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=summary |
| 13:16 |
miker |
csharp_: fwiw, it looks like 18.04 has perl 5.26, and at least the problem described in https://git.evergreen-ils.org/?p=working/OpenSRF.git;a=commitdiff;h=6fadf10c8dee0fe64e5273fef3595e82f1ac9baa can happen on perl 5.24 or later |
| 13:17 |
csharp_ |
yes, I had applied those to OpenSRF 3.2.1, but forgot to add them to 3.2.2 :-/ |
| 13:17 |
csharp_ |
thanks for the tip - we'll see how it goes |
| 13:27 |
csharp_ |
miker: that seems to have calmed things down |
| 13:27 |
csharp_ |
miker++ |
| 13:30 |
terranm |
miker++ csharp_++ |
| 13:30 |
JBoyer |
csharp_, this can certainly wait until you've shaken loose any other post-upgrade blues, but as that makes multiple large production systems running those patches a signoff would help get those into 3.2.3 (or 3.3.0 or whatever) :) |
| 13:30 |
JBoyer |
and yes, csharp_ ++ miker ++ |
| 13:31 |
csharp_ |
yeah, I'd be happy to |
| 13:37 |
csharp_ |
spoke too soon though - something's still killing off open-ils.actor |
| 13:38 |
csharp_ |
I think it has to be something an individual is doing and I suspect either cataloging or acq |
| 13:40 |
Dyrcona |
csharp_: Is it a lot of settings requests? |
| 13:40 |
csharp_ |
I know there *are* a lot of settings requests, but I haven't definitely connected those to the cause |
| 13:43 |
csharp_ |
2263 open-ils.actor.settings.retrieve.atomic, 2003 open-ils.actor.ou_setting.ancestor_default.batch, 1440 open-ils.actor.get_barcodes, 1454 open-ils.actor.user.has_work_perm_at.batch |
| 13:43 |
csharp_ |
those are the highest on the particular server that just died off |
| 13:44 |
Dyrcona |
Well, on the plus side, the batch ones could be getting more than 1 setting at a time. :) |
| 13:50 |
Dyrcona |
Doesn't the client cache the settings? |
| 13:52 |
jeffdavis |
I'd expect to see numbers like that over the course of an hour |
| 13:53 |
csharp_ |
yeah |
| 13:54 |
csharp_ |
I'm not sure if this is a quantity problem or a data problem (or a settings problem in ejabberd) |
| 13:54 |
csharp_ |
the listener dying off doesn't happen consistently with high drone counts |
| 13:54 |
jeff |
when you say "killing off open-ils.actor", what are your symptoms? |
| 13:55 |
jeff |
classic "NOT CONNECTED TO THE NETWORK", or something else? |
| 13:57 |
csharp_ |
jeff: http://irc.evergreen-ils.org/evergreen/2021-12-14#i_497042 - also the logs are WS received XMPP error message in response to thread=0.72683392379099751642531798868 and recipient=router public.brick04-head.gapines.org/open-ils.actor. Likely the recipient is not accessible/available. |
| 13:58 |
csharp_ |
and2022-01-18 13:36:53 brick03-head open-ils.actor: [ERR :92115:XMPPReader.pm:171:] Disconnected from Jabber server, exiting immediately |
| 13:59 |
csharp_ |
it's like ejabberd drops the client then opensrf discovers that later, then removes the service |
| 14:00 |
csharp_ |
@eightball will OpenSRF ever just respawn the listener? |
| 14:00 |
pinesol |
csharp_: Maybe... |
| 14:06 |
jeffdavis |
anything of interest in dmesg output? |
| 14:21 |
jeff |
Ubuntu 18.04, so ejabberd... 18? |
| 14:22 |
jeff |
What does the process appear to be doing / handling when it's disconnected from the Jabber server? Any clues there? |
| 14:22 |
jeff |
Any reason for the disconnection in the ejabberd logs? |
| 14:23 |
jeff |
Have you already shared your ejabberd config / changes? |
| 14:24 |
jeff |
the log you linked to was talking about death with a very specific template being fetched, I think. Is this that again, or was that specific example resolved? |
| 14:31 |
Dyrcona |
jeff: From my experience, the ejabberd logs are useless when things like this happen. You have to crank the verbosity up to where there's way more noise than signal. |
| 14:31 |
Dyrcona |
IIRC on the ejabberd side it looks like the client went away. |
| 14:32 |
Dyrcona |
While on the client side, it looks like ejabberd told the client to go away. |
| 14:35 |
* jeff |
nods |
| 14:42 |
jeff |
packet capture might help isolate the 503 if logs don't (or rule out a 503). if you're not exceeding max connections, I wonder if you're sending invalid xml, either due to a weird value or due to a message being split in a bad place. I thought that there was at least one of those and it was fixed a while ago. |
| 14:50 |
|
jihpringle80 joined #evergreen |
| 15:00 |
csharp_ |
jeffdavis: huh - your question about dmesg sent me down a path that has led me to a couple of possibilities |
| 15:00 |
jeffdavis |
yay for possible progress! |
| 15:00 |
csharp_ |
one is apparmor, which is showing denials for ejabberd (that's an obvious possibility) and another: TCP: request_sock_TCP: Possible SYN flooding on port 5222. Sending cookies. Check SNMP counters. |
| 15:03 |
JBoyer |
csharp_, just to rule this out since I've seen it do some really bizarre stuff before, the max_user_sessions value is 10000 and not just 10, right? |
| 15:03 |
csharp_ |
yeah, it's at 10000 |
| 15:03 |
Dyrcona |
Yay for apparmor keeping us "safe" from ourselves...... |
| 15:04 |
Dyrcona |
Seems to be a lot of that going around these days..... |
| 15:05 |
Dyrcona |
Although, I don't mind if the server sends me cookies, preferably chocolate chip. |
| 15:06 |
csharp_ |
@quote add <Dyrcona> Although, I don't mind if the server sends me cookies, preferably chocolate chip. |
| 15:06 |
pinesol |
csharp_: Error: You must be registered to use this command. If you are already registered, you must either identify (using the identify command) or add a hostmask matching your current hostmask (using the "hostmask add" command). |
| 15:06 |
Dyrcona |
Speaking off..... *gets up and walks off to get a cookie from the kithcen* |
| 15:07 |
csharp_ |
@quote add <Dyrcona> Although, I don't mind if the server sends me cookies, preferably chocolate chip. |
| 15:07 |
pinesol |
csharp_: The operation succeeded. Quote #220 added. |
| 15:09 |
|
jihpringle joined #evergreen |
| 15:15 |
csharp_ |
ok, I've removed ejabberd from apparmor and increased the net.core.somaxconn setting from 128 to 2048 |
| 15:16 |
csharp_ |
another possible setting to tweak in sysctl is net.ipv4.tcp_max_syn_backlog (currently 1024) |
| 15:16 |
jeff |
don't forget to bump the ejabberd listen backlog also. |
| 15:19 |
* csharp_ |
looks for that in the config |
| 15:21 |
jeff |
it's an optional argument to the "listen" blocks |
| 15:21 |
jeff |
https://docs.ejabberd.im/admin/configuration/listen-options/#backlog |
| 15:21 |
jeff |
by default i think the mqtt listener has one specified |
| 15:25 |
jeff |
"ss -nlt" should help you see if the value has changed once you reconfigure. |
| 15:25 |
jeff |
look for the port matching the listener you changed, like 5222... listen backlog is the value in the SendQ column. |
| 15:35 |
jeff |
(Send-Q, sorry) |
| 15:40 |
csharp_ |
jeff: thanks - I upped it to 10 on each server - not sure if that's enough |
| 15:53 |
csharp_ |
now for some reason, the login window isn't loading and no obvious JS console output :-) |
| 15:53 |
csharp_ |
:-( |
| 16:04 |
csharp_ |
...which of course was self-inflicted when I didn't build opensrf with --with-websockets-port=443 |
| 16:25 |
csharp_ |
drone counts are a bit better but the open-ils.actor listener is dying randomly |
| 16:25 |
csharp_ |
it's enough that I can't walk away for 10 minutes |
| 16:34 |
|
terranm joined #evergreen |
| 16:40 |
JBoyer |
csharp_, anything useful in the logs? |
| 16:40 |
JBoyer |
Or just more of the same as above? (Disconnected from Jabber...) |
| 16:52 |
|
jihpringle joined #evergreen |
| 16:54 |
miker |
csharp_: if you haven't already, perhaps check the the stderr log at /openils/var/log/open-ils.actor_stderr.log (by default) to see if anything is getting into there when the listener has issues. and catch-all syslog logs |
| 17:09 |
|
mmorgan left #evergreen |
| 17:49 |
jeffdavis |
might also be worth adding some more logging to Server.pm to try and narrow down the exact point where the listener is dying |
| 18:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
