| Time |
Nick |
Message |
| 01:03 |
|
Stompro joined #evergreen |
| 01:20 |
|
book` joined #evergreen |
| 01:32 |
|
StomproJ joined #evergreen |
| 01:46 |
|
book` joined #evergreen |
| 06:57 |
|
geoffsams joined #evergreen |
| 07:00 |
|
rangi joined #evergreen |
| 07:00 |
|
rangi joined #evergreen |
| 07:04 |
|
BigRig joined #evergreen |
| 07:20 |
|
ningalls joined #evergreen |
| 07:30 |
|
_bott_ joined #evergreen |
| 07:31 |
|
dkyle joined #evergreen |
| 07:48 |
|
rjackson_isl joined #evergreen |
| 07:50 |
csharp |
note to self: figure out why the reporter.classic_item_list view does seq scans on basically every table it touches |
| 07:52 |
csharp |
'explain select * from reporter.classic_item_list' -> http://pastie.org/9892086 |
| 07:52 |
csharp |
unfortunately, I'll be AFK most of the day, so today is not the day I can dig :-/ |
| 07:53 |
csharp |
I just had one of those middle-of-the-night thoughts that 2 hours is too damn long for a report to run |
| 08:00 |
|
ericar joined #evergreen |
| 08:04 |
|
book` joined #evergreen |
| 08:04 |
jboyer-isl |
csharp: You and me both. (We have some "special" reports out there that have 2 different circ totals columns. HOURS of fun running those.) |
| 08:04 |
jboyer-isl |
And yes, seq scans for everyone. |
| 08:13 |
|
mrpeters joined #evergreen |
| 08:16 |
csharp |
jboyer-isl: I'm really interested in optimizing that view - item reports seem to be the main drag right now |
| 08:16 |
|
_bott_ joined #evergreen |
| 08:17 |
csharp |
we've been getting cancellations even after increasing the streaming delay to 2 hours |
| 08:18 |
csharp |
I was recommended in #postgresql to turn feedback on - one of the devs asked me to see how many updates/deletes we have per second, and he seemed to think that the rate was slow enough not to worry about bloat as much |
| 08:20 |
|
graced joined #evergreen |
| 08:20 |
csharp |
(mentioning for the logs that the general concern with having hot_standby_feedback = on is that the master can't vacuum rows that are needed by queries on the standby, which risks bloat) |
| 08:20 |
|
akilsdonk joined #evergreen |
| 08:21 |
csharp |
anyway, so rather than just working around the long reports times, I'm interested now in shortening them ;-) |
| 08:21 |
|
collum joined #evergreen |
| 08:25 |
jboyer-isl |
The number of seq scans in that paste is gross, can I see the sql that you explained so I can compare here? |
| 08:26 |
jboyer-isl |
I can't believe that some of those worked out that way if all of your indexes are in place. |
| 08:46 |
|
julialima_ joined #evergreen |
| 08:46 |
|
remingtron joined #evergreen |
| 08:47 |
|
Stompro joined #evergreen |
| 08:54 |
|
maryj joined #evergreen |
| 09:06 |
|
abowling joined #evergreen |
| 09:06 |
|
mmorgan joined #evergreen |
| 09:27 |
kmlussier |
Good morning #evergreen |
| 09:28 |
kmlussier |
@coffee [someone] |
| 09:28 |
* pinesol_green |
brews and pours a cup of Kenya Peaberry Thika Gethumbwini, and sends it sliding down the bar to dcook |
| 09:28 |
mmorgan |
Good Morning! |
| 09:31 |
maryj |
Mornin'! |
| 09:49 |
gmcharlt |
howdy folks - I'm going to be writing a patch so that osrf_control can act as an NRPE plugin |
| 09:49 |
gmcharlt |
one question - is anybody going anything that's dependent on the current format of the output from osrf_control --diagnostic ? |
| 09:59 |
|
yboston joined #evergreen |
| 10:06 |
jboyer-isl |
gmcharlt: would it make more sense to improve the checks that opensrf's check_osrf_services performs? (and install it by default) |
| 10:06 |
jboyer-isl |
(Or I suppose check_osrf_services could call osrf_control --diagnostic to get it's data...) |
| 10:08 |
gmcharlt |
jboyer-isl: the latter would be my preference; basically, have osrf_control be capable of identifying all known failure modes |
| 10:09 |
|
mdriscoll joined #evergreen |
| 10:09 |
gmcharlt |
that way, we don't end up with drift |
| 10:09 |
gmcharlt |
between it and the NRPE check script |
| 10:11 |
csharp |
jboyer-isl: http://pastie.org/9892341 - this is the view definition for reporter.classic_item_list |
| 10:11 |
jboyer-isl |
I suppose it's already doing all of the info lookups anyway (drone counts / limits) Sounds like a fine plan. |
| 10:12 |
jboyer-isl |
csharp: Oh, wow. I thought it was a specific report that someone ran, I didn't realize the view itself caused that. |
| 10:12 |
csharp |
yep |
| 10:13 |
csharp |
I modified the view a couple of times - it's possible my modifications have caused the planner problems |
| 10:13 |
csharp |
I can verify that |
| 10:14 |
csharp |
oooh - I didn't know about osrf_control --diagnostic |
| 10:15 |
gmcharlt |
csharp: so, my current plan of attack |
| 10:15 |
gmcharlt |
1. teach osrf_control --diagnostic to recognize --service to do service-specific checks |
| 10:15 |
gmcharlt |
2. teach it about a couple more failure modes |
| 10:15 |
gmcharlt |
3. give it away to emit NRPE-compatible check and perfdata output (and return values) |
| 10:15 |
gmcharlt |
4. hopefully speed it up a bit |
| 10:16 |
csharp |
gmcharlt: that sounds awesome |
| 10:16 |
|
maryj joined #evergreen |
| 10:21 |
hopkinsju |
Morning all. I'm looking for a tool to help with SIP debugging. It would be nice to have a front end - graphical or otherwise - that abstracted the various types of SIP requests into more human friendly interfaces, took care of checksumming, etc. |
| 10:21 |
hopkinsju |
I don't suppose anyone knows of a tool like this? |
| 10:22 |
csharp |
hopkinsju: not that I've seen - given my knowledge of SIP2 development, I think that would be something we'd (the Evergreen community) would probably have to develop |
| 10:23 |
hopkinsju |
I wonder if someone at Overdrive has something they would share. They use SIP2 a whole lot, and seem to really know what they're talking about when I've spoken to them. |
| 10:27 |
csharp |
hopkinsju: so are you thinking of parsing log messages? |
| 10:29 |
hopkinsju |
csharp: No, I was more thinking about interactive debugging |
| 10:29 |
csharp |
hopkinsju: so how would the interface be getting data if not from logs? |
| 10:30 |
* csharp |
found this Q&A from atz regarding a standard log analyzer: http://www.faqoverflow.com/libraries/594.html |
| 10:31 |
kmlussier |
Does the acq.fiscal_year table serve any purpose other than providing a default value for the order record upload form? |
| 10:31 |
jboyer-isl |
hopkinsju: I just so happen to have one of those. Runs on Windows (still just a cmd window) and isn't very user friendly, but it does have a message list so you don't have to guess, and it can do checksumming. |
| 10:31 |
hopkinsju |
jboyer-isl: There we go! |
| 10:31 |
kmlussier |
I'm mostly curious to know if there are any problems with staff starting to set up next year's funds if we don't have an entry in that table. |
| 10:31 |
csharp |
hopkinsju: also, mrpeters has access to 3M's ancient SIP emulator that still works with a DLL tweak ;-) |
| 10:32 |
mrpeters |
i think its on admin01 csharp...maybe we can move it to a public location |
| 10:32 |
hopkinsju |
Those are exactly the problems I was hoping to solve. |
| 10:33 |
mrpeters |
I think its WINVCR32.dll or something....comes right up with google |
| 10:33 |
jboyer-isl |
remind me of your email and I'll find a way to get it to you. |
| 10:33 |
hopkinsju |
justin mobiusconsortium.org |
| 10:33 |
mrpeters |
i can throw it on my google drive |
| 10:33 |
|
kbutler joined #evergreen |
| 10:34 |
mrpeters |
3M SIP2 Development Kit -- http://goo.gl/BFcJQM |
| 10:34 |
mrpeters |
SC_Emulator/Settings.sc needs edited for your parameters |
| 10:35 |
mrpeters |
Program/SCEmul.exe is the one you want to use, I beleive. The missing dll msvcrtd.dll is included in that package so it runs fine on Win 7. |
| 10:51 |
csharp |
jboyer-isl: well, I *can* confirm that my changes are what made reporter.classic_item_list slow |
| 10:52 |
csharp |
not sure what the problem is, though |
| 10:52 |
* csharp |
has to leave, so will look at it again later. |
| 10:56 |
|
sarabee joined #evergreen |
| 10:57 |
phasefx |
random aside, I totally forgot there was a little xul app in the OpenSRF source :) |
| 11:00 |
goood |
phasefx: wow... yeah. and it dates from a time when inter-opensrf communication was authenticated! |
| 11:01 |
goood |
also, from before we have object vivication ... |
| 11:05 |
hopkinsju |
mrpeters: I got your .rar from Google Drive, but the dll doesn't seem to be in there. |
| 11:05 |
mrpeters |
really? ok let me reup |
| 11:08 |
mrpeters |
http://goo.gl/oTZoFm -- 3M SIP2 Development Kit with MSVCRTD.DLL fix |
| 11:08 |
mrpeters |
sorry, i had it in my extracted version, and had an old rar sitting there and assumed it had the dll, my bad |
| 11:09 |
* dbs |
notes that public logs of distributing software that is supposed to be licensed directly from 3M is probably a bad combination. |
| 11:09 |
mrpeters |
hmm, well, it was given to me in here |
| 11:09 |
dbs |
Evergreen works because people respect its license (generally); as much as we might not like 3M's license and requirements, it's hypocritical to ignore them and redistribute their software. |
| 11:11 |
mrpeters |
ok dan, go scrub the logs then -- its been shared in here probably a hundred times. I can find no license terms in my installed copy. |
| 11:11 |
dbs |
I'm not going to scrub the logs. |
| 11:11 |
dbs |
No license = All rights reserved by default. |
| 11:13 |
mrpeters |
well, ill wait for my lawsuit then....the software hasnt been touched in about 16 years....im not too concerned. And im far from the only person to share it here. I get your point, and you made your statement for the logs. Lets just move on. |
| 11:21 |
|
vlewis joined #evergreen |
| 11:23 |
|
vlewis_ joined #evergreen |
| 11:28 |
gmcharlt |
eeevil: thoughts on using Proc::ProcessTable rather than ps/pgrep and friends? |
| 11:28 |
gmcharlt |
for osrf_control, to be clear |
| 11:29 |
gmcharlt |
pro: less forking, less dependency on shell constructs, and looks like it works on BSD |
| 11:29 |
goood |
gmcharlt: my thought is "it's gotta be faster to look in /proc with opendir than to use qx//' ;) (which is what I assume proc::processtable does) |
| 11:29 |
gmcharlt |
con: yet another dependency |
| 11:30 |
goood |
I'm for it. perl modules are cheap to install |
| 11:30 |
gmcharlt |
goood: and yep, I've glanced and that's what it does (in C, even) |
| 11:33 |
* bshum |
idly wonders about libproc-processtable-perl versions compared to what's in CPAN |
| 11:35 |
goood |
bshum: portable to where? |
| 11:35 |
bshum |
goood: That's just the name of the package for Debian/Ubuntu |
| 11:36 |
bshum |
I was just comparing, out of curiosity |
| 11:45 |
|
bmills joined #evergreen |
| 11:56 |
|
sandbergja joined #evergreen |
| 11:56 |
|
Stompro joined #evergreen |
| 12:23 |
|
book` joined #evergreen |
| 12:28 |
|
buzzy joined #evergreen |
| 12:29 |
|
bmills joined #evergreen |
| 12:32 |
|
mglass joined #evergreen |
| 12:37 |
jboyer-isl |
So... 040.schema.asset.sql has 2 indexes on asset.copy (call_number) and one of them has available in the name. Missing "WHERE status in (0,7)" in the definition? |
| 12:38 |
jboyer-isl |
Not certain where that index would be used off hand. |
| 12:48 |
|
rjackson_isl joined #evergreen |
| 12:48 |
gmcharlt |
goood: http://git.evergreen-ils.org/?p=working/OpenSRF.git;a=shortlog;h=refs/heads/collab/gmcharlt/better_osrf_control_diagnostic |
| 13:02 |
kmlussier |
Woo hoo! Booked my hotel room for the conference. :) |
| 13:08 |
|
nhilton joined #evergreen |
| 13:13 |
phasefx |
berick: I was trying a variant of the script in bug#1268619 to test websockets prior to installing anything EG (including dojo). And I found I had to include opensrf_ws.js manually, and after calling .send(), I also had to call .session.send_ws() on the request object. Does that sound crazy? |
| 13:16 |
phasefx |
http://paste.lisp.org/display/145683 |
| 13:19 |
phasefx |
the manual part doesn't sound crazy, the path is hard-coded in opensrf.js |
| 13:21 |
phasefx |
okay <rubber ducking>, so if I change the transport from OSRF_TRANSPORT_TYPE_WS_SHARED to OSRF_TRANSPORT_TYPE_WS, I can just do req.send() and it works |
| 13:24 |
berick |
phasefx: OSRF_TRANSPORT_TYPE_WS_SHARED is currently disabled. send_ws() is the underyling handler for OSRF_TRANSPORT_TYPE_WS. |
| 13:24 |
berick |
so even though it was set to OSRF_TRANSPORT_TYPE_WS_SHARED, you were forcing it to run as non-shared by calling send_ws() directly |
| 13:25 |
berick |
when you changed to OSRF_TRANSPORT_TYPE_WS, the send_ws() was no longer needed |
| 13:25 |
berick |
(since it gets called under the covers) |
| 13:31 |
phasefx |
berick: thanks man, that makes sense |
| 13:48 |
|
book` joined #evergreen |
| 14:11 |
|
Tony__ joined #evergreen |
| 14:22 |
csharp |
okay, looks like it's the join to extend_reporter.full_circ_count that's causing the crappy performance of reporter.classic_item_list |
| 14:24 |
csharp |
looks like that view does seq scans on both circulation and aged_circulation |
| 14:25 |
csharp |
with that COALESCE there, I don't think there's a way around that though |
| 14:27 |
dbwells |
csharp: So did this come about via bug #1208572? |
| 14:27 |
pinesol_green |
Launchpad bug 1208572 in Evergreen 2.4 "reporter.classic_item_list not counting aged circulations" (affected: 1, heat: 6) [Undecided,Fix released] https://launchpad.net/bugs/1208572 |
| 14:28 |
csharp |
dbwells: yep |
| 14:29 |
csharp |
and since then, I've been taking for granted that reports based on that view just take forever |
| 14:29 |
csharp |
this morning I started wondering if there is a way to speed those up, and here I am ;-) |
| 14:32 |
berick |
that's what you get for thinking about stuff |
| 14:32 |
csharp |
berick: yep!! |
| 14:34 |
berick |
next thing you know you're hanging out in #evergreen hustlin for indexes |
| 14:34 |
csharp |
haha |
| 14:35 |
* csharp |
queues up https://www.youtube.com/watch?v=UOg_8hCC4u4 |
| 14:43 |
csharp |
goood: / eeevil: any thoughts on how one might avoid seq scans on action.circulation/action.aged_circulations when using extend_reporter.full_circ_count? It looks to me that the right indexes (target_copy) are already present on both tables :-/ |
| 14:44 |
csharp |
alternately, if there's a static table that records active circs + aged circs + legacy circs, even better |
| 14:47 |
* csharp |
notes that the indexes are btree and wonders if another type of index would help too |
| 14:50 |
|
book` joined #evergreen |
| 15:01 |
* csharp |
also wonders if this is more a configuration tuning issue |
| 15:06 |
|
book` joined #evergreen |
| 15:19 |
|
book` joined #evergreen |
| 15:25 |
|
book` joined #evergreen |
| 15:29 |
|
julialima_ joined #evergreen |
| 15:37 |
dbwells |
csharp: If you feel like trying something try this as a different take on structuring the full_circ_count view: http://pastie.org/9893003 |
| 15:37 |
dbwells |
It isn't broadly going to be an optimization, but it definitely gives a better plan in certain cases. |
| 15:43 |
|
tony__ joined #evergreen |
| 15:43 |
|
tony__ left #evergreen |
| 15:56 |
csharp |
dbwells++ # that helps |
| 16:04 |
|
akilsdonk joined #evergreen |
| 16:07 |
|
gsams joined #evergreen |
| 16:14 |
csharp |
dbwells: actually, I'm having to fight with COALESCE-ing those values to get the circ count value to appear - thanks for the pointer for a faster possible solution, though ;-) |
| 16:14 |
* csharp |
calls it a day - gonna tackle this next week |
| 16:16 |
jboyer-isl |
csharp: fight not, copy pasta: http://pastie.org/9893118 |
| 16:16 |
jboyer-isl |
Still does a seq scan on asset.copy and unit though. :-/ |
| 16:17 |
csharp |
jboyer-isl++ |
| 16:18 |
csharp |
actually using indexes only in my case |
| 16:18 |
csharp |
btw, for the logs, looks like random_page_cost is also a relevant setting, especially if you've got tonloads of RAM and SSDs |
| 16:20 |
jboyer-isl |
Ah. Makes sense. |
| 16:24 |
|
mdriscoll left #evergreen |
| 16:28 |
|
nhilton_ joined #evergreen |
| 16:41 |
csharp |
amazing... with dbwells's fix with jboyer-isl's modifications, my nearly two-hour query dropped to 4 seconds |
| 16:45 |
|
nhilton joined #evergreen |
| 16:48 |
dbwells |
csharp: Sorry for being lazy on the COALESCE, I only did a quick test on lower ids which had a legacy_circ_count entry (which should be, I think, the only number needing a COALESCE). |
| 16:56 |
dbwells |
This should work: http://pastie.org/9893249 |
| 17:01 |
dbwells |
I also kept COUNT(*) over COUNT(distinct id) because 1) there are no joins involved, so DISTINCT doesn't do anything for us here, and 2) COUNT(*) is generally recommended when simply counting rows, as it lets the planner pick whatever it thinks is best (which will probably be 'id', but anyway... at least that's what I've been told). |
| 17:05 |
dbwells |
In any case, the changes I'm advocating will not affect performance in any measurable way, I think it just helps readability. |
| 17:06 |
|
vlewis_ joined #evergreen |
| 17:09 |
|
mmorgan left #evergreen |
| 17:16 |
pinesol_green |
Incoming from qatests: Test Success - http://testing.evergreen-ils.org/~live/test.html <http://testing.evergreen-ils.org/~live/test.html> |
| 17:30 |
csharp |
dbwells: excellent - thank you |
| 17:31 |
csharp |
I'll file a bug on this just to get it on the record |
| 17:32 |
dbwells |
sounds like a plan |
| 17:48 |
csharp |
bug 1419172 created |
| 17:48 |
pinesol_green |
Launchpad bug 1419172 in Evergreen "extend_reporter.full_circ_count view unusably slow on large datasets" (affected: 1, heat: 6) [Undecided,New] https://launchpad.net/bugs/1419172 |
| 18:55 |
kmlussier |
exit |
| 19:18 |
|
bmills joined #evergreen |
| 20:58 |
|
nhilton joined #evergreen |
| 21:14 |
|
bmills joined #evergreen |
| 22:39 |
|
book` joined #evergreen |
| 22:52 |
|
nhilton joined #evergreen |
| 23:30 |
|
book` joined #evergreen |
| 23:31 |
|
bmills joined #evergreen |
