| Time |
Nick |
Message |
| 06:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:09 |
|
rjackson_isl joined #evergreen |
| 07:56 |
|
mantis joined #evergreen |
| 08:37 |
|
mmorgan joined #evergreen |
| 08:56 |
|
Dyrcona joined #evergreen |
| 09:21 |
Dyrcona |
So, storage service "crashed" last night while the fine generator was running. Interestingly, I got an email from nagios that no storage processes were running, then a bit later that it was OK. |
| 09:22 |
Dyrcona |
I'm going to check the logs. Does anyone know the telltale signs of one of the drones "replacing" the listener? |
| 09:23 |
|
rfrasur joined #evergreen |
| 09:25 |
Dyrcona |
In the logs, it looks another process that uses storage ran just fine at 3:00 AM. The email about no storage processes was sent at 00:09, and the fine generator hung up about that time and was still "running" at 9:15 AM when I logged in to check it. |
| 09:26 |
Dyrcona |
We also get emails when a process tries to start, but can't because of the lock file, so I have a bunch of those from the fine generator, too, which is what caused me to log in and check the fine generator. |
| 09:31 |
Dyrcona |
I've seen this movie before: Sep 1 00:02:18 cron open-ils.storage: [INFO:1616:System.pm:125:] server: restarting after fatal crash... |
| 09:43 |
Dyrcona |
Yeah, definitely: Sep 1 00:02:18 cron open-ils.storage: [ERR :1616:System.pm:123:] server: died with error Use of freed value in iteration at /usr/lib/x86_64-linux-gnu/perl/5.26/IO/Select.pm line 71. |
| 09:43 |
Dyrcona |
Seems like a bug in Perl or IO::Select? Maybe how we use it? |
| 10:11 |
|
jvwoolf joined #evergreen |
| 11:47 |
|
JBoyer joined #evergreen |
| 12:17 |
jeff |
I think there's some additional nuance about how that adjustment -> billing mapping works, but I'll have to double check. |
| 12:18 |
jeff |
(rewinding the conversation to yesterday's mention of it in relation to mmppbt) |
| 12:20 |
* Dyrcona |
thinks we should have an IDL class of ackthppt or maybe acqthppt. :) |
| 12:26 |
berick |
heh |
| 12:26 |
|
jihpringle joined #evergreen |
| 12:33 |
jeff |
i have issued the (valid in my environment at the time) command "ack mmpbbt" before, mostly for the sake of amusement. :-) |
| 12:34 |
Dyrcona |
:) |
| 13:04 |
jeffdavis |
I was hoping to raise pcrud max_children to 300 but I have discovered that there's a hard-coded upper limit of 256 on max_children, at least for C services (ABS_MAX_CHILDREN in osrf_prefork.c) |
| 13:09 |
berick |
"no one will ever need that many, it's unthinkable" |
| 13:09 |
Dyrcona |
jeffdavis: I had not noticed that, but I've gone above 200. You could always change the code and recompile. |
| 13:09 |
Dyrcona |
"I've gone above 200," should be "I've not gone above 200." |
| 13:10 |
Dyrcona |
But, now I'm curious what you're doing that requires so many pcrud drones. |
| 13:11 |
Dyrcona |
640K should be enough for anybody..... |
| 14:02 |
jeffdavis |
There are some parallel-requests bugs in the client that can exhaust pcrud drones - bug 1930617 for example |
| 14:02 |
pinesol |
Launchpad bug 1930617 in Evergreen "Too many requests when loading Holdings Editor for multiple items" [Undecided,New] https://launchpad.net/bugs/1930617 |
| 14:03 |
jeffdavis |
I had hoped to increase pcrud max_children as a workaround for that, since it's at 225 and still maxing out |
| 14:04 |
jeffdavis |
and I haven't been able to come up with a fix for that bug |
| 14:05 |
Dyrcona |
I thought it would end up being work around for something. |
| 14:13 |
jeffdavis |
https://bugs.launchpad.net/opensrf/+bug/1942359 |
| 14:13 |
pinesol |
Launchpad bug 1942359 in OpenSRF "Raise or remove ceiling on max_children" [Undecided,New] |
| 14:15 |
Dyrcona |
I think fixing the web client(s) would be a better option. You could end up fork bombing the server after a while. |
| 14:17 |
jeffdavis |
I don't think there should be a hard cap built into OpenSRF to prevent that, though? Particularly a fairly low one. Increasing the limit or making it configurable would be OK to me. (I agree the web client does also need to be fixed.) |
| 14:27 |
|
jvwoolf joined #evergreen |
| 14:29 |
|
jvwoolf left #evergreen |
| 14:39 |
|
jihpringle joined #evergreen |
| 14:46 |
|
collum joined #evergreen |
| 15:04 |
JBoyer |
jeffdavis, rather than letting 300+ pcrud processes party on your ram is there a reason you're not adding more vms? They could be set up to run nothing but pcrud to leave a healthy buffer even with only 50-100 on each. |
| 15:09 |
Dyrcona |
JBoyer: I've found that when "out of control" requests are made by a client, they end up hitting the same brick/vm. |
| 15:09 |
jeffdavis |
JBoyer: In our environment, the hundreds of parallel requests generated by workflows like the ones in 1930617 all go to the same server. If they were being load-balanced across all available servers then yeah, expanding the VM roster would be a good way to go. |
| 15:09 |
jeffdavis |
Jinx! |
| 15:09 |
Dyrcona |
:) |
| 15:11 |
JBoyer |
Ah, that does make it more annoying then. |
| 15:16 |
JBoyer |
So here's a fun thing that literally just happened to me. Has anyone seen issues with the offline-db-worker.js background worker just spinning and chewing on your CPU? I had 3 master-ish tabs open in Chrome 92 and those tabs are totally unresponsive. (and they will be until I kill that background task in the Chrome task manager, which is an obnoxious thing to type out loud.) |
| 15:19 |
JBoyer |
And now my fans finally slow back down. |
| 15:21 |
jeffdavis |
I haven't run into that before, but yuck. |
| 15:46 |
|
eby joined #evergreen |
| 15:59 |
|
jeffdavis joined #evergreen |
| 16:13 |
|
Stompro joined #evergreen |
| 16:28 |
Stompro |
Git question, within one branch, can I copy a file to a different location, but still have it track any changes made to the original file.. |
| 16:28 |
Stompro |
Thinking about how to store custom templates directories. |
| 16:33 |
Dyrcona |
Stompro: I don't think so, but there are some ways to compare directories and files in git. |
| 16:36 |
Dyrcona |
And, maybe I am misremembering.... I've used emacs' emerge feature to maintain custom trees in the past, but I swear I used to have some git magic for that, too.... |
| 16:37 |
Dyrcona |
I don't bother with custom template directories these days. I keep our modifications in the main directories in our own branch. If you have more than one set of templates, then you do need custom directories. |
| 16:43 |
Stompro |
Dyrcona, thanks, We have a custom template dir for each of our 2 systems and then one shared one for customizations that both use, trying to leave the /templates folder itself alone. |
| 16:44 |
Stompro |
But if I use git more I can probably get rid of the shared one. |
| 16:46 |
Dyrcona |
Sure, that makes a lot of sense. We used to have a similar situation, with 2 custom templates directories, but the second used the first. They will be searched in order specified in the Apache config. |
| 16:46 |
Dyrcona |
When we no longer needed the second, I switched to not having a custom template directory at all. This coincided with our upgrade to 3.5. |
| 16:47 |
Dyrcona |
Closest thing that I've found in git would be: git diff -- templates/ custom_templates/ |
| 16:47 |
Dyrcona |
Which is pretty much the same as using command line diff with the -R option. |
| 16:55 |
Dyrcona |
I'm not finding anything in my scripts of "stupid shell/git tricks." |
| 16:57 |
Stompro |
I was just hopeful there was a magical way to keep those custom template files up to date with the original + whatever our local changes are.... but that probably doesn't make sense.. how would git know which changes to make going forward. |
| 16:58 |
Stompro |
Thank you for looking. |
| 17:12 |
|
mmorgan left #evergreen |
| 17:45 |
|
sandbergja joined #evergreen |
| 18:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:13 |
|
jihpringle joined #evergreen |
| 21:10 |
|
Stompro joined #evergreen |
