| Time |
Nick |
Message |
| 07:31 |
|
collum joined #evergreen |
| 08:11 |
|
Stompro joined #evergreen |
| 08:35 |
|
mantis1 joined #evergreen |
| 08:39 |
|
mmorgan joined #evergreen |
| 08:57 |
|
dguarrac joined #evergreen |
| 09:17 |
|
Dyrcona joined #evergreen |
| 09:21 |
|
jvwoolf joined #evergreen |
| 09:51 |
|
mantis1 joined #evergreen |
| 10:18 |
csharp_ |
@dunno add SysTEmContROL ReeStarTT Opan |
| 10:18 |
pinesol |
csharp_: The operation succeeded. Dunno #75 added. |
| 10:18 |
csharp_ |
dammit - foiled by premature enter key |
| 10:19 |
Dyrcona |
csharp_: Regarding your Pg 14 slow search query from yesterday, have you set the join_collapse_limit option in PostgreSQL? |
| 10:19 |
csharp_ |
Dyrcona: yeah - set it to 12 (for both 10 and 14) |
| 10:19 |
Dyrcona |
OK. Mine is at 9. |
| 10:19 |
csharp_ |
but on one of the servers it was set to 8 (default) and it was working fine |
| 10:19 |
Dyrcona |
So, that's probably NOT an issue. |
| 10:20 |
csharp_ |
it was actually my first experiment - we had an issue that was solved by that setting a number of years ago |
| 10:20 |
Dyrcona |
Yeah, we set it to 9 a few years for some large query. Don't remember if it was search or what. |
| 10:21 |
Dyrcona |
Every increase doubles the number of joins that can be collapsed or whatever that setting does. |
| 10:22 |
Dyrcona |
I might remove it from my Pg 14 config to see what happens at 8. |
| 10:23 |
Dyrcona |
I haven't started performance testing, yet. I'm still mucking about with getting a database upgraded and dumped so I can load it on Pg 14. |
| 11:42 |
jvwoolf |
We saw some high open-ils.storage drones on our OPAC servers and high load on our database this morning. I'm looking at the logs but not finding a cause that I can directly point to. Any ideas on what to look for? |
| 11:53 |
jeffdavis |
csharp_: fyi we have the fix for bug 1990306 in production now - too soon to say whether it resolves the problem but at least it doesn't seem to break anything |
| 11:53 |
pinesol |
Launchpad bug 1990306 in Evergreen "Flood of open-ils.actor.user.has_work_perm_at.batch requests for VIEW_USER with null authtoken" [Undecided,New] https://launchpad.net/bugs/1990306 |
| 12:02 |
|
jihpringle joined #evergreen |
| 12:02 |
|
Christineb joined #evergreen |
| 12:09 |
jeffdavis |
jvwoolf: did you see high open-ils.search drones at the same time? I usually only see high storage drones in conjunction with that |
| 12:11 |
Dyrcona |
I |
| 12:11 |
Dyrcona |
We've seen high storage drones without high search drones. |
| 12:17 |
Dyrcona |
I can't find it on Lp, but I thought I filed a bug about it. The best I can find is Lp 1819798 that I opened about the fine generator killing the storage listener. |
| 12:17 |
pinesol |
Launchpad bug 1819798 in Evergreen "Storage Listener Dies During Fine Generation" [Undecided,New] https://launchpad.net/bugs/1819798 |
| 12:20 |
jeffdavis |
Just in general, when looking at the logs for high drone counts I usually start by doing something like `grep 'app1.*CALL: open-ils.storage' osrfsys.log | less` and just looking for what was going on when the drone spike started (we monitor drone counts so I usually have a rough idea of the timing). Parallel requests are usually pretty visually obvious. |
| 12:21 |
jeffdavis |
(replace "app1" with the hostname of your affected server or whatever) |
| 12:21 |
jeffdavis |
If you spot parallel requests for a specific API call you can look in the code for where that call is used. |
| 12:21 |
Dyrcona |
Yeah. I have seen them coincide with searches, but not always. I haven't seen any in a while. |
| 12:25 |
jeffdavis |
If it's not parallel requests (many near-simultaneous near-identical requests), another good one to look for is whether any "Message processing duration" times are unusually long for the affected service (open-ils.storage in this case) when the spike happened. |
| 12:26 |
jeffdavis |
jvwoolf: ^ |
| 12:46 |
|
collum joined #evergreen |
| 12:58 |
jvwoolf |
jeffdavis++ Dyrcona++ |
| 13:11 |
Bmagic |
I'm looking at LP, and there are some bugs having to do with misspellings. If I fix those, do I need to update all of the .po files as well? Or are those generated? |
| 13:27 |
Bmagic |
cd build/i18n && make newpot reads the source code and generates the po files right? |
| 14:11 |
Dyrcona |
Bmagic: I believe it does, yes. |
| 14:44 |
Bmagic |
Dyrcona++ # ty |
| 16:58 |
|
mmorgan left #evergreen |
| 17:08 |
|
jvwoolf joined #evergreen |
| 18:53 |
|
jihpringle joined #evergreen |
| 19:13 |
|
jihpringle joined #evergreen |
| 20:29 |
|
JBoyer joined #evergreen |
