| Time |
Nick |
Message |
| 06:57 |
|
jboyer-isl joined #evergreen |
| 07:48 |
|
rjackson_isl joined #evergreen |
| 07:55 |
|
ericar joined #evergreen |
| 07:57 |
|
mrpeters joined #evergreen |
| 08:01 |
|
rjackson_isl_ joined #evergreen |
| 08:11 |
|
collum joined #evergreen |
| 08:54 |
|
ericar joined #evergreen |
| 09:10 |
|
Dyrcona joined #evergreen |
| 09:25 |
dbs |
Y'all might have noticed a failure of the planet (or not) over the past two days; I had upgraded my server to jessie but failed to make the required apache 2.4 config changes for the planet. Sorry! |
| 09:26 |
dbs |
Should be good now though. |
| 09:28 |
Dyrcona |
dbs++ |
| 09:29 |
bshum |
dbs++ |
| 09:44 |
|
yboston joined #evergreen |
| 09:46 |
|
maryj joined #evergreen |
| 09:49 |
|
pmurray_away joined #evergreen |
| 09:49 |
|
pmurray joined #evergreen |
| 09:49 |
|
pmurray left #evergreen |
| 10:28 |
|
collum_ joined #evergreen |
| 10:29 |
|
collum joined #evergreen |
| 10:30 |
|
collum joined #evergreen |
| 10:51 |
csharp |
hmm - isn't age protection defined in a database table somewhere? It exists on asset.copy as an integer... |
| 10:51 |
csharp |
I don't see it in config.* or asset.* |
| 10:52 |
bshum |
csharp: Age protection for... like holds? |
| 10:52 |
csharp |
right - as it exists on a copy |
| 10:52 |
bshum |
Or wait, I see |
| 10:52 |
bshum |
There's no reference to the table it goes to? |
| 10:52 |
bshum |
Usually there's a reference in the table definition |
| 10:52 |
csharp |
nope, not that I see :-/ |
| 10:53 |
berick |
config.rule_age_hold_protect |
| 10:53 |
bshum |
berick beat me to it |
| 10:53 |
csharp |
ah |
| 10:53 |
bshum |
berick++ |
| 10:53 |
csharp |
berick++ # thanks |
| 10:56 |
|
Christineb joined #evergreen |
| 11:00 |
|
sandbergja joined #evergreen |
| 11:05 |
|
RoganH joined #evergreen |
| 11:10 |
Dyrcona |
dbs: Follow up from Friday: I didn't have to add anything to MANIFEST to get the new perlmods under OpenILS/WWW to install. |
| 11:11 |
Dyrcona |
I just did a clean install on a new vm and Build.PL found them and installed them automagically. |
| 11:23 |
mceraso |
Over the weekend, I got a mail delivery failure from mailer-daemon evergreen-ils.org when trying to send an email to gitadminevergreen-ils.org |
| 11:24 |
Dyrcona |
mceraso: What was the failure reason? |
| 11:24 |
mceraso |
Dyrcona: <gitadminevergreen-ils.org>: delivery temporarily suspended: connect to |
| 11:24 |
mceraso |
127.0.0.1[127.0.0.1]:10024: Connection refused |
| 11:25 |
Dyrcona |
Hmm. Sounds like a problem on the server or possibly a grey lister. So, you're server bounced the email back to you? It didn't keep trying? |
| 11:25 |
Dyrcona |
Guess it depends more on the code, though.... |
| 11:26 |
mceraso |
I received the bouceback almost 5 days after sending the first email |
| 11:28 |
Dyrcona |
OK, so your sever kept trying....Hopefully, someone with the power: csharp, gmcharlt ? will see this and can do something about it. |
| 11:30 |
Dyrcona |
mceraso: If you're submitting a public ssh key, you could email it directly to gmcharlt and/or tsbere and they could add it in the meantime. |
| 11:30 |
mceraso |
Dyrcona: Excellent! Thanks! |
| 11:30 |
* tsbere |
had been wondering why that email had never shown up after being told it should be on the way to gitadmin |
| 11:30 |
csharp |
Dyrcona: https://www.youtube.com/watch?v=-dJolYw8tnk |
| 11:31 |
Dyrcona |
:) |
| 11:34 |
mceraso |
Dyrcona: Thanks! :) |
| 11:35 |
Dyrcona |
mceraso: YW! |
| 11:36 |
jeff |
and /cl |
| 11:36 |
jeff |
er |
| 11:47 |
dbs |
csharp: test received (on gitadmin) |
| 11:56 |
berick |
dbs: thought of one of your SEO talks today as google news showed a headline of 'Heartburn pills linked to increased risk of kidney disease' next to a picture of David Bowie. |
| 12:04 |
dbs |
berick: ouch |
| 12:09 |
csharp |
dbs: thanks! |
| 12:10 |
csharp |
for some reason the amavis (spam filter) daemon and clamav (virus scanner) were not running on lupin, so aliased emails like feedback and gitadmin weren't coming through |
| 12:29 |
|
jihpringle joined #evergreen |
| 12:33 |
bshum |
csharp: That's happened before a couple times now |
| 12:33 |
bshum |
csharp: I think it's a memory killer issue |
| 12:33 |
jeff |
anyone generating logs should be forced to parse those logs before committing. |
| 12:33 |
jeff |
unparsable log entries should be a test failure. |
| 12:34 |
jeff |
(mostly unrelated to evergreen, just a general "if i ruled the world" whim of this morning) |
| 12:35 |
jeff |
i'm parsing logs generated by software which i think i wrote in 2008. |
| 12:43 |
tsbere |
jeff: At least once I wrote something that logged perfectly parseable logs, then the logging system changed and they became useless. So maybe not past-you's fault? ;) |
| 12:45 |
csharp |
yay! now I'm seeing feedbackevergreen-ils.org spam! |
| 12:45 |
csharp |
(meaning that all is back to normal :-)) |
| 13:04 |
* Dyrcona |
forgets how to left join where the outer table is null, 'cause what he's doing isn't working. |
| 13:06 |
Dyrcona |
Either that or I have no deleted precat copies from this library that don't have a permanent copy. |
| 13:07 |
tsbere |
Dyrcona: left join table on condition, then "table is null" included in the where clause? |
| 13:08 |
Dyrcona |
tsbere: yes, but missing extra criterion for the join.... acp2.id <> acp.id.... |
| 13:08 |
Dyrcona |
Or, acp2.callnumber <> -1 |
| 13:08 |
Dyrcona |
It was joining the precat with itself.... |
| 13:08 |
tsbere |
heh, oops. |
| 13:09 |
Dyrcona |
'cause I only specified deleted on the inner table. |
| 13:10 |
Dyrcona |
I'm searching for example for the restore deleted copies interface demo tomorrow. |
| 13:28 |
Dyrcona |
Oof... Typos in the messages means doing make newpot again. |
| 13:29 |
Dyrcona |
Or not... it did nothing. |
| 13:41 |
Dyrcona |
Ah. I would need to add a new target for Open-ILS/src/templates/utils/ |
| 13:41 |
Dyrcona |
Guess I'll do that if (when) I decided to really share it with the rest of the community. |
| 13:42 |
Dyrcona |
I doubt many would be interested, because this would be better implemented in the web staff client at this point. |
| 13:48 |
|
mrpeters left #evergreen |
| 14:00 |
|
mmorgan joined #evergreen |
| 14:02 |
csharp |
I'm seeing that we're going to need to up the timeout in action_trigger_runner.pl to accommodate the large number of reminder notices - do others see that problem as well? |
| 14:02 |
|
mrpeters joined #evergreen |
| 14:02 |
csharp |
it ran for an hour and killed itself after the 3600 second timeout |
| 14:03 |
csharp |
in our case it's chewing on ~25K circs |
| 14:11 |
berick |
wow, slow query |
| 14:12 |
csharp |
what seems to be taking the most time is moving them from "pending" to "collected" |
| 14:12 |
bshum |
csharp: Just the one A/T event def? |
| 14:12 |
csharp |
bshum: yep |
| 14:12 |
bshum |
We fortunately (or unfortunately) have one preoverdue for each library. So we batched them by granularities |
| 14:12 |
bshum |
When we discovered that timeout breakage |
| 14:12 |
bshum |
I guess you can't do that with the one only |
| 14:13 |
csharp |
I've increased the timeout to 2 hours - I'm now watching them move |
| 14:16 |
csharp |
if worse comes to worst, we can just stay on legacy notice scripts, but I'd like to move everything to A/T |
| 14:23 |
jboyer-isl |
csharp: do your database logs tell any tales? Is the select taking a while (indexes) or just the commit (db tuning, blech) |
| 14:24 |
jboyer-isl |
Also, is that indicative of a normal volume or is that a backlog of things, and finally, how many notices are running on that granularity? Perhaps multiples could be split. |
| 14:30 |
csharp |
jboyer-isl: I'll check out the db logs - good idea. The volume looks to be slightly higher than typical - other (legacy) files from the past week contain 16K and 19K of notices |
| 14:32 |
jboyer-isl |
I'm sure patrons would also prefer they not get the same notice twice, once scripted and once A/T'd, heh. |
| 14:34 |
csharp |
jboyer-isl: yeah - I'm hoping to transition off of scripted notices and onto a/t |
| 14:34 |
csharp |
looks like there's a select query that's taking 2+ seconds per event - I'm going to look into speeding that up |
| 14:35 |
jboyer-isl |
I know, that was a reference to the potential cause of the backlog. (Although I suppose you've probably got your timing settings setup so that's not really an issue. ;) ) |
| 14:36 |
jboyer-isl |
That's no good, is it generic enough you can paste it? |
| 14:37 |
csharp |
I think so |
| 14:38 |
csharp |
http://pastie.org/10684276 |
| 14:41 |
csharp |
that's taking 2.5 seconds for every event |
| 14:43 |
Dyrcona |
csharp: You have indexes on asset.call_number and asset.circ_lib? |
| 14:43 |
* Dyrcona |
figures you do. |
| 14:43 |
jeff |
csharp: this is during which stage of the process? |
| 14:44 |
csharp |
Dyrcona: yeah, and it's using those indexes according to explain analyze |
| 14:44 |
csharp |
(which is taking far less time to perform for some reason) |
| 14:44 |
Dyrcona |
csharp: Well, guess the low hanging fruit is gone. :) |
| 14:44 |
csharp |
jeff: moving the event from collecting to collected |
| 14:46 |
csharp |
ooh - interesting: Planning time: 2.247 ms |
| 14:48 |
jeff |
0.002 seconds |
| 14:48 |
csharp |
oh |
| 14:48 |
csharp |
duh |
| 14:48 |
berick |
fyi, the timeout won't affect moving from collecting to collected |
| 14:48 |
csharp |
I was seeing 2.2 seconds |
| 14:49 |
berick |
the timeout (if we're talking about the same thing) only affects the initial query |
| 14:49 |
berick |
after that, it's just a very long running series of tiny api calls and transactions |
| 14:49 |
berick |
none of which would individually time out |
| 14:49 |
csharp |
the timeout I'm talking about is my $opt_max_sleep = 3600; |
| 14:50 |
csharp |
in action_trigger_runner.pl |
| 14:50 |
berick |
oh, different timeout then. /me looks at that |
| 14:51 |
jeff |
csharp: what does your environment for this event look like? circ_lib.billing_address, usr, target_copy.call_number.record.simple_record? |
| 14:52 |
berick |
csharp: a better approach might be to give each of the long-running event defs their own granularity. that way each has its own lock file and runs completely independently of the others. |
| 14:53 |
csharp |
jeff: hmm - looks like I don't have any entries for "Event Environment" |
| 14:53 |
berick |
then use --granularity and --granularity-only when using a_t_runner.pl |
| 14:53 |
csharp |
berick: unfortunately, I'm already using granularity here :-/ |
| 14:54 |
berick |
wait, A/T runner died / killed itself after an hour? |
| 14:54 |
csharp |
berick: yep |
| 14:54 |
berick |
that's just an error |
| 14:55 |
berick |
an explosion |
| 14:55 |
csharp |
ok |
| 14:55 |
berick |
it won't just kill itself |
| 14:55 |
csharp |
berick: thanks for that info - I'll stop chasing the red herring |
| 14:56 |
csharp |
jeff: I think your question has got me on a better track - I'll report back after checking on environment stuff |
| 14:56 |
berick |
the 1 exception is the initial "what am I processing" query taking longer than the $req_timeout, in which case the events would never even get created |
| 14:57 |
csharp |
well, the good news is that I'm working this out now on a test server rather than doing it next week when everythings on fire ;-) |
| 15:05 |
Dyrcona |
@praise csharp |
| 15:05 |
* pinesol_green |
You don't want to get mixed up with someone like csharp. csharp is a loner, Dottie. A rebel. |
| 15:06 |
bshum |
Memory killers |
| 15:08 |
berick |
there are things about csharp you wouldn't understand. |
| 15:08 |
jeff |
I am wondering what part of the process is generating that specific query, but shouldn't dig further right now. |
| 15:57 |
|
jlitrell joined #evergreen |
| 17:07 |
|
mmorgan left #evergreen |
| 17:20 |
|
Devin_ joined #evergreen |
| 18:37 |
|
bmills joined #evergreen |
| 21:02 |
|
rashma_away joined #evergreen |
| 21:03 |
|
b_bonner joined #evergreen |
| 21:03 |
|
mnsri joined #evergreen |
