| Time |
Nick |
Message |
| 06:17 |
|
troy joined #evergreen |
| 07:17 |
|
redavis joined #evergreen |
| 07:39 |
csharp_ |
wellll, it looks like pingest.pl for big reingests and search.symspell_dictionary_updates are not friends :-( |
| 07:39 |
redavis |
Ugh |
| 07:40 |
csharp_ |
search.symspell_dictionary_updates grew to 1.1TB and filled the DB server's disk of 2.5TB |
| 07:40 |
csharp_ |
which cascaded into the local backup server's disk filling up too |
| 07:41 |
csharp_ |
there's a bug in there somewhere - I may just report what I see to be the problem - I'm not fully grokking the symspell process in the midst of panicked troubleshooting |
| 07:42 |
csharp_ |
that is to say, it might be a "don't do that"-type configuration problem rather than a bug |
| 07:50 |
csharp_ |
I guess setting enabling the internal_flag "ingest.disable_symspell_reification" would be one solution |
| 07:50 |
csharp_ |
I guess if I better understood the consequences of that setting I'd be more comfortable |
| 07:52 |
csharp_ |
er... *intended* consequences I mean |
| 07:52 |
redavis |
filed under "this won't help you now" but I think it would be a great thing to have a conference session or even pre-conference on how to fully optimize DYM. On large datasets, as we know, it gets...unwieldy quickly |
| 07:54 |
csharp_ |
DYM to fill your disk? (yes)/(no)? |
| 07:57 |
redavis |
lol, exactly |
| 08:43 |
|
mmorgan joined #evergreen |
| 09:14 |
|
collum joined #evergreen |
| 09:20 |
|
dguarrac joined #evergreen |
| 09:34 |
|
mantis joined #evergreen |
| 09:37 |
mantis |
Do patron baskets get saved somewhere in the db? |
| 09:41 |
redavis |
baskets or lists? |
| 09:42 |
redavis |
I don't think baskets do (that's saved "prior" to the db). But if a patron "saves basket to list" or creates a list some other way, that does. |
| 09:44 |
mantis |
Just baskets |
| 09:45 |
mantis |
redavis: I also thought the same but just wanted to make sure |
| 09:46 |
redavis |
csharp_ is kinda wandering around here, and might know the exact place. |
| 09:48 |
berick |
csharp_: same happened here, result of a fudged code merge on my part. |
| 09:48 |
berick |
mantis: i believe baskets from the catalog are stored in memcache |
| 09:49 |
mantis |
berick++ |
| 09:49 |
redavis |
berick++ |
| 10:32 |
csharp_ |
berick: oh, interesting |
| 10:33 |
csharp_ |
auth/mfa has increased resource needs here apparently |
| 10:38 |
csharp_ |
also changes to HoldTargeter.pm are spamming our warn logs |
| 10:38 |
csharp_ |
2025-02-18 10:38:02 utility01 open-ils.hold-targeter: [WARN:416416:Application.pm:624:17398926014345034] open-ils.hold-targeter.target: Use of uninitialized value in numeric eq (==) at /usr/local/share/perl/5.34.0/OpenILS/Application/HoldTargeter.pm line 114. |
| 10:38 |
csharp_ |
Perl devs: please check that variables have values before doing stuff with them |
| 10:46 |
mmorgan |
csharp_++ |
| 10:59 |
|
Christineb joined #evergreen |
| 11:14 |
|
Dyrcona joined #evergreen |
| 11:18 |
csharp_ |
system load is very high for us this morning - we're still on ejabberd - currently wondering if we need to tweak apache2 settings |
| 11:19 |
berick |
much deja vu today. we got hit pretty good last week. aggressively lowered our apache max workers |
| 11:19 |
mantis |
bots/probes? |
| 11:20 |
mantis |
if so, us, too |
| 11:20 |
berick |
yeah |
| 11:20 |
mantis |
it was super bad - we're also on ejabberd still and the activity was crazy |
| 11:23 |
csharp_ |
berick: our MaxClients/MaxRequestWorkers is set at 150 |
| 11:25 |
csharp_ |
current mpm config: https://pastebin.com/0Hm7An8s |
| 11:25 |
mantis |
Is there a working LP on this specifically? Just wondering how to report it if it hasn't already. Our issue was an overload of basket activity someone else mentioned before. |
| 11:25 |
csharp_ |
we are blocking most foreign countries who were giving us trouble |
| 11:25 |
berick |
csharp_: if you're tracking how many are typically active, that should give some sense how much you can lower it |
| 11:26 |
csharp_ |
via iptables/geoip |
| 11:27 |
berick |
i lowered ours to 50, but we don't host a patron catalog, so your mileage will vary on the exact number |
| 11:28 |
csharp_ |
makes sense |
| 11:36 |
berick |
csharp_: mantis: see my dev email as well, which should help reduce apache load |
| 11:40 |
mantis |
berick++ |
| 11:50 |
csharp_ |
berick: trying now with nginx update |
| 11:57 |
csharp_ |
berick: it didn't work immediately - had to revert and will test later - but I love the approach |
| 11:59 |
berick |
huh, k. copied from our prod setup. may be some variables i'm not considering |
| 11:59 |
berick |
or the miasma of caching |
| 11:59 |
eby |
i was debating opening a discussion bug about an updated nginx example that is more evg centric vs opensrf. the opensrf example doesn't really take into account some of the growing size of files like fm_IDL which would be buffered to file with default proxy settings. |
| 12:03 |
berick |
eby: would be a good step toward bug #2032835 (but, more importatnly, useful in the short term). |
| 12:03 |
pinesol |
Launchpad bug 2032835 in OpenSRF "Discussion: Merge OpenSRF Into Evergreen?" [Wishlist,New] https://launchpad.net/bugs/2032835 |
| 12:04 |
jeff |
conversation bug, or conversation on the dev list, either way... sounds worth having. |
| 12:22 |
csharp_ |
aaaayyy |
| 12:22 |
csharp_ |
I think it was badly configured ejabberd |
| 12:23 |
csharp_ |
Ubuntu 22.04 adds "burst_size" to the shaper config, which has spotty docs on ejabberd's site and no mention on Open |
| 12:23 |
csharp_ |
...SRF install instructions |
| 12:24 |
csharp_ |
it can be safely removed so it just says "normal: 50000" |
| 12:26 |
eby |
since the goal is example code in the evg tree i went ahead and did a conversation bug #2098783 |
| 12:26 |
pinesol |
Launchpad bug 2098783 in Evergreen "Discussion: Evergreen Specific Proxy Configuration Examples" [Undecided,New] https://launchpad.net/bugs/2098783 |
| 12:29 |
csharp_ |
never upgrade anything ever, for any reason, no matter what |
| 12:31 |
Dyrcona |
csharp_: Contrarywise, I have less trouble with Arch where I upgrade everything almost weekly. |
| 12:32 |
mmorgan |
csharp_: +1 |
| 12:32 |
Dyrcona |
Sometimes more than once per week. |
| 12:32 |
* mmorgan |
would still be on Windows XP given the choice. |
| 12:42 |
Dyrcona |
Windows 2000 Pro was the best Windows ever. |
| 12:52 |
JBoyer |
csharp_, I did notice the 10000 request limit in your mpm config, that seems extremely high. If process churn when that's down around 3-500 is too high to tolerate you're pretty much already hosed. |
| 12:53 |
JBoyer |
And extremely high request counts give each process a chance to grow their memory use. |
| 12:53 |
JBoyer |
Speaking of, how many apache front-ends do you have for PINES? |
| 12:54 |
JBoyer |
er, "a chance to grow..." should be "more of a chance to grow" because otherwise tautology is tautological. |
| 13:13 |
Dyrcona |
We should top using mod_perl. |
| 13:13 |
Dyrcona |
stop using it, that is. |
| 13:15 |
Dyrcona |
Ah, well... Too little to do, too much time. ... Wait. Strike that. Reverse it. |
| 13:15 |
csharp_ |
JBoyer: 6 frontends |
| 13:16 |
csharp_ |
mmorgan: maybe you should come down to Georgia where apparently multiple library systems are still running Windows 7 that can't run modern Chrome and is not playing well with dark-mode-ish changes |
| 13:16 |
csharp_ |
we learned that yesterday and today |
| 13:17 |
mantis |
csharp: oh god |
| 13:17 |
* csharp_ |
sends USBs with Ubuntu installers on them to all the libraries |
| 13:17 |
redavis |
Whoa |
| 13:17 |
redavis |
And a bonus CD Rom for the Encyclopedia Brittanica |
| 13:18 |
JBoyer |
"Ahh, Chrome 54, a fine vintage." |
| 13:18 |
redavis |
lol |
| 13:18 |
berick |
back when hatch extension v2 worked.. good times |
| 13:19 |
csharp_ |
argh - still seeing crazy load after ejabberd fix |
| 13:52 |
berick |
eby++ # proxy bug |
| 14:04 |
JBoyer |
For a minute I thought scrapers were starting to use Hemlock's user agent to get around some blocking here and there, but on closer inspection, while the IP is registerd at RIPE, it's used by a Texas company for their VPN exit. I guess someone really wanted to renew their items while Bake Off was on. |
| 14:05 |
csharp_ |
switched one of our bricks to rediSRF - doing ok/better so far |
| 14:40 |
csharp_ |
I'm guessing this is still bad ejabberd config, but I'm seeing redis working and system load falling to normal levels on the servers I've updated with it |
| 14:41 |
csharp_ |
stupid ejabberd project and their different config file formats between versions |
| 14:41 |
csharp_ |
makes diffing very dicey |
| 14:58 |
Dyrcona |
@band add Clockwork Elves |
| 14:58 |
pinesol |
Dyrcona: Band 'Clockwork Elves' added to list |
| 14:59 |
Dyrcona |
@praise [band] for their album, Dicey Diffing |
| 14:59 |
* pinesol |
The Rural Jurors is the very model of a modern major hacker for their album, Dicey Diffing |
| 15:02 |
berick |
@blame [band] for letting [band] open for them |
| 15:02 |
pinesol |
berick: Go With Action wants the TRUTH?! Go With Action CAN'T HANDLE THE TRUTH!! for letting Seflcheck open for them |
| 15:24 |
csharp_ |
berick++ # Redis |
| 15:24 |
csharp_ |
that apparently has calmed everything here |
| 15:24 |
csharp_ |
and it's dead simple |
| 15:24 |
csharp_ |
as an admin, Redis is ideal |
| 15:37 |
pinesol |
News from commits: Docs: 3.13.9 release notes <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=35d59fa5fbc9d5d1ba7db7dcecdeefb8fff4cd96> |
| 15:55 |
* Dyrcona |
disagrees. |
| 16:00 |
|
mantis left #evergreen |
| 16:04 |
Rogan |
jboyer don't underestimate the need to watch Bake Off before someone spoils it |
| 16:35 |
csharp_ |
@band add Stoned Cat |
| 16:35 |
pinesol |
csharp_: Band 'Stoned Cat' added to list |
| 16:38 |
pinesol |
News from commits: Docs: 3.14.4 release notes <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=3ed8f71196a3ba12a28a5e397f7e952a2838570d> |
| 16:39 |
redavis |
abneiman++ |
| 16:40 |
redavis |
also, csharp_++ |
| 17:10 |
|
mmorgan left #evergreen |
