| Time |
Nick |
Message |
| 01:56 |
|
eady joined #evergreen |
| 06:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:11 |
|
rjackson_isl joined #evergreen |
| 07:56 |
|
mantis joined #evergreen |
| 08:38 |
|
rfrasur joined #evergreen |
| 08:45 |
|
mmorgan joined #evergreen |
| 08:46 |
|
jvwoolf joined #evergreen |
| 08:52 |
|
Dyrcona joined #evergreen |
| 09:08 |
|
Keith-isl joined #evergreen |
| 09:41 |
|
jvwoolf joined #evergreen |
| 10:46 |
Dyrcona |
So, this is a thing that Evergreen/OpenSRF does: Jul 2 10:00:59 cron open-ils.storage: [INFO:13693:System.pm:125:] server: restarting after fatal crash... |
| 10:52 |
jvwoolf |
Dyrcona: Yikes... |
| 10:57 |
Dyrcona |
Except, it doesn't look like the restart was successful. Unfortunately, I didn't get in and look at things until after someone else restarted the storage service. |
| 10:58 |
Dyrcona |
We've had storage crash on this server before. It seems to be the fine generator that triggers it, but I don't know how. |
| 10:59 |
mmorgan |
One more reason to eliminate fines ;-) |
| 11:02 |
Dyrcona |
:) |
| 11:02 |
Dyrcona |
It is running just fine right now. These crashes seem random. Everything is going great, then boom. |
| 11:06 |
Dyrcona |
I wonder if this is a case of a drone replacing the listener that berick has mentioned in the past? |
| 11:07 |
Dyrcona |
I was told by the person who restarted the storage service that "the uptime for the storage service was the same as all the other services." |
| 11:10 |
jeff |
I've not seen that message as a side effect in the scenario where "fork fails, listener becomes drone". |
| 11:13 |
jeff |
Dyrcona: that log message should be preceeded by a message with an error from $@" |
| 11:13 |
jeff |
$logger->error("server: died with error $@"); |
| 11:13 |
jeff |
(the restarting after fatal crash message is logged at info level) |
| 11:14 |
jeff |
OpenSRF/System.pm has the logging and while / eval loop if you hadn't looked at it yet. |
| 11:16 |
csharp_ |
@band add OpenSRF Fatal Crash |
| 11:16 |
pinesol |
csharp_: Band 'OpenSRF Fatal Crash' added to list |
| 11:16 |
jeff |
after logging the "restarting" message, it should sleep(2) and then the while loop should start over, doing an eval{$server->run} again. Was it erroring repeatedly every 2 seconds? |
| 11:22 |
csharp_ |
mmorgan++ # bug tagging |
| 11:22 |
csharp_ |
thanks for bringing bug 1884261 back to my attention |
| 11:22 |
pinesol |
Launchpad bug 1884261 in Evergreen "fm_IDL.xml needs cleanup" [Low,New] https://launchpad.net/bugs/1884261 |
| 11:23 |
csharp_ |
I was creating a report this morning on an obscure source and thinking about the need for better labels everywhere |
| 11:23 |
mmorgan |
:) |
| 11:25 |
* mmorgan |
is reviewing bugs this am, bringing a number of back to my own attention, but my attention is now getting too crowded :-/ |
| 11:26 |
* csharp_ |
knows the feeling far too well |
| 11:34 |
Dyrcona |
jeff: Jul 2 10:00:58 cron open-ils.storage: [ERR :13693:System.pm:123:] server: died with error Use of freed value in iteration at /usr/lib/x86_64-linux-gnu/perl/5.26/IO/Select.pm line 71. |
| 11:34 |
Dyrcona |
Yay for Perl? |
| 11:35 |
Dyrcona |
@band add Lipstick and Hate |
| 11:35 |
pinesol |
Dyrcona: Band 'Lipstick and Hate' added to list |
| 11:38 |
Dyrcona |
I have recently been coming closer and closer to the conclusion that Perl5 is a bad choice of server programming language. |
| 11:42 |
jeff |
we've experienced that in Evergreen code before, fixed as bug 1691563 |
| 11:42 |
pinesol |
Launchpad bug 1691563 in Evergreen 3.0 ""Use of freed value in iteration" error during adjust to zero" [Undecided,Fix released] https://launchpad.net/bugs/1691563 |
| 11:42 |
jeff |
but this appears to be in IO::Select (is that part of IO::Socket?) |
| 11:42 |
jeff |
(no, not part of IO::Socket) |
| 11:56 |
|
jihpringle joined #evergreen |
| 11:59 |
csharp_ |
working on bug 1932051 ... I have set up a similar test to what berick describes in https://bugs.launchpad.net/evergreen/+bug/1896285/comments/6 |
| 11:59 |
pinesol |
Launchpad bug 1896285 in Evergreen 3.5 "Use batch methods for multi-row grid actions" [Medium,Fix released] |
| 11:59 |
pinesol |
Launchpad bug 1932051 in Evergreen "AngularJS Add to Item Bucket Generating too many simultaneous requests" [High,New] https://launchpad.net/bugs/1932051 - Assigned to Chris Sharp (chrissharp123) |
| 11:59 |
csharp_ |
still maxxing out the 5/5 open-ils.actor drones when I add 100 items to a new bucket |
| 11:59 |
csharp_ |
that's with scant understanding of how promises work in AngJS |
| 12:01 |
berick |
csharp_: can look more later, but at a glance add a 'return' in front of egCore.net.request |
| 12:03 |
csharp_ |
berick: thanks - I'll try that |
| 12:13 |
csharp_ |
sigh... nope still not working :-( |
| 12:36 |
berick |
csharp_: and you've confirmed open-ils.actor.container.item.create is the problematic API? (seems likely) |
| 12:42 |
berick |
actually, yeah, just confirmed here |
| 12:46 |
berick |
csharp_: try this: https://gist.github.com/berick/4c49eeb6c0ea3c91ca800ab6b398f2be |
| 13:06 |
csharp_ |
berick: I'm still seeing it max out with your gist applied :-( |
| 13:08 |
berick |
hmm, i verified each request was completing before the next was fired... |
| 13:09 |
csharp_ |
this is with Firefox on Fedora, cache cleared manually (though shouldn't have to with dev tools open)... |
| 13:09 |
csharp_ |
double-checked that the file was in place |
| 13:11 |
csharp_ |
https://csharp-master.gapines.org/js/ui/default/staff/cat/bucket/copy/app.js |
| 13:11 |
berick |
csharp_: what happens when you add these 2 console lines? |
| 13:11 |
berick |
https://gist.github.com/berick/56fd9565f1167b82afcf2b75eeb842b8 |
| 13:12 |
berick |
it should alternate between sending and receiving. |
| 13:12 |
berick |
if you get a pile of 'sending' first, then it's still a problem |
| 13:15 |
|
jvwoolf joined #evergreen |
| 13:15 |
csharp_ |
huh - not seeing the console messages at all - so something's not right |
| 13:16 |
csharp_ |
I'm in item status, loading 100 barcodes, clicking Actions -> Add to Item Bucket |
| 13:16 |
csharp_ |
https://csharp-master.gapines.org/js/ui/default/staff/cat/bucket/copy/app.js shows the console.log additions :-/ |
| 13:16 |
berick |
csharp_: ooohhh |
| 13:17 |
berick |
i think there are 2 copies of that function |
| 13:17 |
csharp_ |
heh |
| 13:17 |
* csharp_ |
finds ice to put on head |
| 13:18 |
berick |
csharp_: see service.add_copies_to_bucket in circ/services/item.js |
| 13:19 |
berick |
we were editing the one inside the bucket interface |
| 13:19 |
berick |
but the item status UI uses this other function |
| 13:19 |
berick |
of course, they should both be fixes |
| 13:19 |
berick |
*fixed |
| 13:19 |
berick |
ditto the container item delete calls |
| 13:20 |
csharp_ |
ok - that makes sense - I can see the services version getting called into Item Status but not the bucket one |
| 13:30 |
|
jihpringle joined #evergreen |
| 13:30 |
Dyrcona |
Hmm. I'm wondering why the reporter child processes daemonize themselves. If it is just to set the process name, that seems like overkill to me. |
| 14:38 |
Dyrcona |
Experimentation with the reporter suggests that setting umask(0) isn't such a hot idea for a daemon written in Perl. |
| 14:40 |
Dyrcona |
It makes the output files world writable, which isn't what we'd normally want, and I suspected that it would do this because default permissions when writing files with Perl. |
| 14:51 |
JBoyer |
Anytime umask is 0 you should end up with 777 or 666 perms. Looking at that section it looks like you should set it to 0 to be absolutely certain you don't inherit an unknown value. So we can either specify the mode every single time open is called (not great) or we can specifically set it to a known-good value for our purposes, I imagine 022 is fine for that. |
| 14:53 |
Dyrcona |
JBoyer: Yes. The logic for a program written in C is set umask to 0, then use your own permission mask when creating files from a daemon process. umask is typically 022 so setting it to that may not be a bad idea. |
| 14:54 |
Dyrcona |
I've reverted that commit for my current test. I wasn't seeing anything in syslog, but now, I'm not seeing anything to stderr, either, so maybe there were no log messages, yet. |
| 14:56 |
Dyrcona |
In production, there are almost always some messages about not being able to draw some figure or another. |
| 14:57 |
Dyrcona |
Ha! Timing: Jul 2 14:57:08 jasontest Clark Kent reporting: ** daily LOST ITEMS PAYMENTS: [WARN:7388:clark-kent.pl:40:] Reporter warning: Couldn't draw /openils/var/web/reporter/6236/72235/204102/report-data.html.bar.gif : No attribute 'shading'#012No data sets or points at /openils/bin/clark-kent.pl line 835. |
| 14:58 |
JBoyer |
I forgot that perl's open doesn't accept that kind of mode, we should definitely just set the umask to what we want rather than trying to swap to sysopen. |
| 14:58 |
JBoyer |
Ah, and that error, I assume that's a 0 results (or at least 0 rows with numbers?) result? |
| 14:58 |
Dyrcona |
JBoyer: Yeah. I think we'll have to use 022. |
| 14:59 |
Dyrcona |
Something like that, but the good thing is that message went to syslog. |
| 15:00 |
JBoyer |
Maybe that can be cleaned up when pie charts are either finished or throw out. ;) |
| 15:04 |
Dyrcona |
So, now, I'm testing with my daemonize changes back in place to see if that unintentionally closes the connection to syslog. |
| 15:05 |
Dyrcona |
I'll change the umask from 0 to 022 before I update the bug. |
| 15:07 |
Dyrcona |
I'm not so sure that the individual reporter processes need to daemonize themselves. |
| 15:11 |
Dyrcona |
Ah, but the fact that they daemonize themselves means that they keep running even if the parent process dies, no? |
| 15:11 |
* Dyrcona |
tries it to see what happens. |
| 15:12 |
Dyrcona |
Yeahp. They're still chugging away. |
| 15:13 |
Dyrcona |
Think I'll leave that alone. |
| 15:13 |
Dyrcona |
Progress has been made. :) |
| 15:28 |
Dyrcona |
So, warnings from clark-kent.pl when run as a daemon are going to syslog. When run from a command line, they should go to syslog as well as to the console, with a couple of exceptions that just go to the console. |
| 15:54 |
|
jwoodard joined #evergreen |
| 16:05 |
JBoyer |
Dyrcona++ |
| 16:33 |
alynn26 |
Dyrcona ++ |
| 16:50 |
|
jvwoolf left #evergreen |
| 17:23 |
|
mmorgan left #evergreen |
| 18:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
