| Time |
Nick |
Message |
| 04:46 |
|
pastebot joined #evergreen |
| 06:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:15 |
|
rjackson_isl_hom joined #evergreen |
| 08:16 |
|
mantis1 joined #evergreen |
| 08:20 |
|
Dyrcona joined #evergreen |
| 08:34 |
|
jvwoolf joined #evergreen |
| 08:43 |
|
mmorgan joined #evergreen |
| 08:46 |
|
rfrasur joined #evergreen |
| 08:54 |
|
collum joined #evergreen |
| 09:04 |
|
terranm joined #evergreen |
| 09:42 |
|
mantis1 joined #evergreen |
| 10:08 |
|
jvwoolf1 joined #evergreen |
| 11:41 |
|
sandbergja joined #evergreen |
| 11:50 |
|
malexander joined #evergreen |
| 11:57 |
|
jihpringle joined #evergreen |
| 12:05 |
|
khuckins joined #evergreen |
| 12:26 |
|
malexander_ joined #evergreen |
| 12:29 |
|
sandbergja joined #evergreen |
| 12:34 |
csharp |
OMG we're getting *slayed* by open-ils.actor right now |
| 12:34 |
csharp |
trying to find out what's causing it |
| 12:37 |
csharp |
23968 calls of open-ils.actor.settings.apply.user_or_ws since noon on one of the affected servers |
| 12:40 |
|
jvwoolf joined #evergreen |
| 12:40 |
Dyrcona |
All from the same client, I'll bet. |
| 12:44 |
csharp |
I'm watching it happen in real time via osrf_control --diagnostic - open-ils.actor gets flooded up to 100%, then cstore, then cstore goes offline |
| 12:44 |
csharp |
all in a matter of seconds |
| 12:45 |
csharp |
maybe someone trying over and over to do something and it's just hitting server after server? |
| 12:46 |
csharp |
it's happening so fast that I'm unable to take the time to log dive |
| 12:47 |
csharp |
might have to set up a cron to restart opensrf every minute unless this stops |
| 12:50 |
jeffdavis |
Are the calls all for the same authtoken, and if so, can you track down an IP for that user and block it? |
| 12:50 |
jeffdavis |
as an emergency stopgap |
| 12:51 |
csharp |
jeffdavis: good thought - I'll take a look |
| 12:52 |
|
mrisher joined #evergreen |
| 12:53 |
|
sandbergja joined #evergreen |
| 12:54 |
csharp |
jeffdavis++ # your suggestion has led me to find the offending user :-) |
| 12:55 |
csharp |
that single user has 33479 calls to open-ils.actor.settings.apply.user_or_ws since noon |
| 12:56 |
csharp |
oof - actually more like 125K |
| 12:56 |
csharp |
I was only looking at one of the affected servers |
| 12:57 |
csharp |
a huge wall of {"circ.checkin.strict_barcode":null} and {"circ.checkin.do_inventory_update":null} |
| 12:57 |
jeffdavis |
Defective Hatch install maybe? Or local storage disabled? |
| 12:57 |
csharp |
maybe uploading a file for checkin? |
| 12:58 |
csharp |
oh hmm |
| 12:59 |
csharp |
looks like hundreds of calls of that per second for over an hour from that single user |
| 12:59 |
csharp |
actually, still happening |
| 13:01 |
csharp |
and... looks like we don't have IPs being forwarded to the logs :-( |
| 13:03 |
jeffdavis |
If you can grep back for the first occurrence of the authtoken, I think you can find the username and workstation? |
| 13:03 |
csharp |
I just killed the session via memcache |
| 13:04 |
csharp |
yeah, I've got the user/workstation |
| 13:12 |
csharp |
everything is calm after knocking that session offline |
| 13:13 |
csharp |
I AM ROOT |
| 13:13 |
Dyrcona |
csharp: There's a branch to document fixing the Apache logging: bug 1908540 |
| 13:13 |
pinesol |
Launchpad bug 1908540 in Evergreen "Add Apache proxy logging example" [Wishlist,New] https://launchpad.net/bugs/1908540 |
| 13:13 |
csharp |
pinesol: ahem |
| 13:13 |
pinesol |
csharp: http://www.firstpersontetris.com/ |
| 13:14 |
csharp |
Dyrcona++ # thanks |
| 13:20 |
Dyrcona |
After figuring that out for the third time, I decided that we should document it. :) |
| 13:54 |
|
jvwoolf joined #evergreen |
| 13:58 |
|
sandbergja joined #evergreen |
| 14:07 |
Dyrcona |
rebase-- fixup-- 'Cause now, I've to edit a commit to undo changes to two files that fixed up in a branch.... Or, I suppose I could add a commit to replace the modified files with the originals and then fixup that commit in place..... |
| 14:10 |
jeff |
nah, too much work. just do a fresh clone of the repo, then copy your working copy over the top of it and git add . && git commit -m "some changes" |
| 14:14 |
Dyrcona |
Wouldn't have to clone the repo. There are faster ways to do that, and it wold be a decent use of a detached HEAD. |
| 14:15 |
Dyrcona |
I don't want to do that, because I want to keep my custom changes grouped in 6 logical commits. I should have waited before adding this stuff and squashing it in my "main" branch. |
| 14:17 |
Dyrcona |
toomanybranches-- |
| 14:18 |
Dyrcona |
If I had left these as individual commits, I could just drop them. |
| 14:21 |
Dyrcona |
I'll just add the original images back in there own commit and bother with the rebase later. |
| 14:22 |
Dyrcona |
"Ah, but I was so much older then. I'm younger than that, now." |
| 14:35 |
csharp |
TOO MANY SECRETS |
| 14:35 |
csharp |
(for jeff's benefit) |
| 14:39 |
Dyrcona |
"Don't put on any airs when you're down on Rue Morgue Aven-ooooo." |
| 15:32 |
|
mantis1 left #evergreen |
| 17:01 |
|
sandbergja joined #evergreen |
| 17:23 |
jeff |
Dyrcona: I was attempting a joke/sarcasm above. I fear that I may not have expressed that clearly enough. :-) |
| 17:24 |
* jeff |
. o ("a sarcasm? eesh.") |
| 17:25 |
|
mmorgan left #evergreen |
| 17:37 |
jeffdavis |
terranm++ # Launchpad maintenance |
| 17:38 |
jeffdavis |
There are lots of folks contributing to the feedback fest, but wow, that's a lot of bugs getting cleaned up! |
| 17:39 |
terranm |
:) |
| 17:39 |
terranm |
Also X) |
| 18:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 19:21 |
jeffdavis |
We really need to make it easier to test AuthProxy. |
| 19:40 |
|
sandbergja joined #evergreen |
| 19:44 |
|
stephengwills joined #evergreen |
| 20:58 |
|
sandbergja joined #evergreen |
| 21:02 |
|
devted joined #evergreen |
| 21:03 |
|
dluch joined #evergreen |
| 23:50 |
|
khuckins joined #evergreen |
