| Time |
Nick |
Message |
| 08:49 |
|
mmorgan joined #evergreen |
| 09:25 |
|
Dyrcona joined #evergreen |
| 09:26 |
|
mmorgan joined #evergreen |
| 10:06 |
Dyrcona |
Great. It looks like the opensrf.math service (or other C services) are hanging while connecting to redis on the vm upgraded from Ubuntu 24.04 to Ubuntu 26.04. |
| 10:07 |
Dyrcona |
I'll log my debugging session and add the backtrace to the bug. |
| 10:07 |
Dyrcona |
I'm not sure this is anything we're doing, and it might just be busted configuration on an upgraded system, but /etc/redis/redis.conf is the same between the two systems. |
| 10:39 |
|
mmorgan1 joined #evergreen |
| 10:53 |
Dyrcona |
We're not making it to osrfAppRegisterApplication in the child process it seems. |
| 11:04 |
Dyrcona |
OK. I have confirmed it stalls trying to communicate with opensrf.settings. |
| 11:09 |
Bmagic |
is AppArmor messing with things? |
| 11:16 |
Dyrcona |
berick | goood: https://bugs.launchpad.net/evergreen/+bug/2155417/comments/7 |
| 11:16 |
Dyrcona |
Bmagic: I hadn't though of apparmor, particularly since it works on a fresh installation. |
| 11:17 |
Bmagic |
I've had that bite me before |
| 11:17 |
Bmagic |
ulimit and apparmor |
| 11:19 |
Dyrcona |
I think I'll disable apparmor across the board and see what happens. |
| 11:22 |
Dyrcona |
Doesn't seem to have made a difference, but I'll disable it in grub.cfg and reboot if I ever get my prompt back. |
| 11:22 |
Bmagic |
:) |
| 11:25 |
Dyrcona |
I'm thinking it is not apparmor, particularly given that it hangs in recv() from the C library. |
| 11:28 |
Bmagic |
yeah, probably not, but it came to mind.. so now it's in your mind |
| 11:31 |
Dyrcona |
Well, I disabled it in the grub config as well as with systemctl, rebooted, and opensrf_control still hangs when it tries to start opensrf.math. It looks like it times out after a few minutes. |
| 11:32 |
Bmagic |
I've observed opensrf.settings hanging for a long time during restarts. Which is something new as of 22.04. I've learned to ignore it, but I wonder if the symtoms started years ago |
| 11:35 |
Dyrcona |
Well, the hang happens when starting opensrf.math regardless of how long opensrf.settings has been running. I should try a perl program to look up settings. It may be that redis is just busted after an upgrade. |
| 11:41 |
Dyrcona |
opensrf.persist and opensrf.validator are able to read their configuration, so it is something peculiar to C. |
| 11:44 |
|
jihpringle joined #evergreen |
| 12:51 |
|
Dyrcona joined #evergreen |
| 14:07 |
|
jihpringle joined #evergreen |
| 15:00 |
Dyrcona |
So much wasted time.... I previously kept /openils/conf, but deleted everything else on the upgraded Ubuntu vm. After doing rm -r /openils/* and reinstalling. OpenSRF appears to work. |
| 15:05 |
Dyrcona |
Some lessons are expensive, but at least this one only cost me a day or so. |
| 15:17 |
Dyrcona |
My advice: Don't upgrade vms, just build a new one. |
| 15:22 |
eby |
https://cloudipdb.io/providers/ |
| 15:22 |
goood |
indeed. "servers" are disposable these days. deploy config into a clean env every time, where possible. (ideally without human hands touching the config, instead generating it from a constrained input system). sorry, man... |
| 15:35 |
Dyrcona |
https://xkcd.com/303/ |
| 15:47 |
Dyrcona |
On the plus side, I got to experiment with gdb. I don't get to do that very often. |
| 16:01 |
Dyrcona |
I guess that xkcd could also applying to restoring a database dump or running tests. |
| 16:01 |
Dyrcona |
'also applying?' meh. |
| 16:11 |
Dyrcona |
If we add Ubuntu 26.04 for 3.18 beta, we'll need to have a discussion about an ejabberd branch of OpenSRF perhaps 3.4, and that will require a whole new round of testing, so maybe we won't support Ubuntu 26.04 for 3.18.... I should update the bug. |
| 16:14 |
goood |
IMO, since the upgrade path from 24.04 /just now/ openned with 26.04.1, I don't think it's critical for 3.18. but(!) I don't use ubuntu, nor would I stay on the bleeding edge if I did, so do take that with a grain of salt... |
| 16:29 |
Dyrcona |
I'm ok with waiting for Ubuntu 26.04. I just mailed the list with the question as I see it. If no one else steps up, then it likely won't happen. |
| 16:30 |
Dyrcona |
Yeah, I forced the upgrade a few days early. There were holding it back because if issues with Rust coreutils. |
| 17:15 |
|
mmorgan1 left #evergreen |
| 19:46 |
|
beardicus15 joined #evergreen |
| 21:33 |
|
beardicus15 joined #evergreen |
| 23:03 |
|
beardicus157 joined #evergreen |
