| Time |
Nick |
Message |
| 01:28 |
|
Jillianne joined #evergreen |
| 06:40 |
|
rlefaive joined #evergreen |
| 07:14 |
|
rjackson_isl joined #evergreen |
| 07:49 |
|
kmlussier joined #evergreen |
| 07:55 |
kmlussier |
Good morning #evergreen! |
| 07:57 |
kmlussier |
phasefx: Do you know if there's something wrong with the live tester. It hasn't run since Wednesday afternoon. |
| 08:07 |
|
rlefaive joined #evergreen |
| 08:40 |
|
mmorgan joined #evergreen |
| 08:42 |
|
Dyrcona joined #evergreen |
| 08:46 |
|
Enjabain joined #evergreen |
| 08:48 |
Dyrcona |
@later tell jeff Do you think we should bug that marc_export behavior we saw yesterday? |
| 08:48 |
pinesol_green |
Dyrcona: The operation succeeded. |
| 08:49 |
jeff |
perhaps. i tend to wait until i have a little more of an idea what's wrong, beyond just symptoms. i realize that there are disadvantages to that, of course... :-) |
| 08:51 |
jeff |
i'm looking at another encoding issue where the migration-tools extract_holdings script seems to be taking in marcxml in utf8 and emitting iso-8859-1 call numbers. |
| 08:52 |
jeff |
almost entirely unrelated, and i'm not even confident in that summary of the symptoms yet. |
| 08:54 |
csharp |
jeff: Dyrcona: for people like me who weren't following along in real time, a bug report with a summary of even just the symptoms would be great |
| 08:55 |
* csharp |
says as a fellow marc_export user |
| 08:55 |
Dyrcona |
csharp: Yeah, I think so, too. I'm not sure it's a marc_export bug, though. |
| 08:55 |
* jeff |
files a bug with his symptoms |
| 08:55 |
Dyrcona |
I'm writing an email to open-ils-dev with something related that I've seen. |
| 08:56 |
jeff |
"marc_export creating MARC data that yaz-marcdump dislikes" |
| 08:56 |
Dyrcona |
I think it's Encode.pm or possibly MARC::Charset with perl >= 5.20. |
| 08:57 |
|
jvwoolf joined #evergreen |
| 09:04 |
Dyrcona |
I will test something before I send that email. |
| 09:09 |
jeff |
okay, yep. extract_holdings in this environment emits iso-8859-1 in the main holdings file, and i can override this by adding an encoding to the open() call, like this: |
| 09:09 |
jeff |
open HOLDINGS, '>:encoding(UTF-8)', ($c->{prefix} . "-HOLDINGS.pg"); |
| 09:11 |
jeff |
(which may or may not be the best solution) |
| 09:11 |
jeff |
I should try and reproduce this marc_extract issue with concerto records. |
| 09:14 |
csharp |
hmm - why would "select id from asset.copy;" require a seq scan? |
| 09:14 |
jeff |
are you seeing the seq scan in EXPLAIN or EXPLAIN ANALYZE? |
| 09:15 |
csharp |
explain |
| 09:16 |
jeff |
csharp: are you passing any WHERE clause? if you're asking for every id, there doesn't seem to be a performance hit in a seq scan -- it needs all the ids anyway. |
| 09:17 |
Dyrcona |
jeff: You're saying the marc_export issue is with -i ? |
| 09:17 |
Dyrcona |
Or is that something else you're talking about? |
| 09:18 |
jeff |
Dyrcona: nope, something else. i'm not talking about marc_export with -i / --items yet. |
| 09:18 |
csharp |
jeff: well, I'm troubleshooting a query based on asset.copy that's doing seq scans all over the place and I've been removing joins incrementally to debug - I ended with asset.copy and the seq scan is still there: http://pastebin.com/i0H7Het0 |
| 09:18 |
Dyrcona |
OK. You sometimes get ahead of me, so... :) |
| 09:18 |
jeff |
Dyrcona: my unexpected iso-8859-1 comments from earlier this morning were in relation to the extract_holdings tool in the ESI migration-tools |
| 09:19 |
Dyrcona |
Ah, OK. I've not used that one. |
| 09:19 |
csharp |
explain analyze: http://pastebin.com/GBdqmXWw |
| 09:20 |
Dyrcona |
jeff: I'm dumping all records with no item info in usmarc format to see what I get. |
| 09:20 |
Dyrcona |
I expect to either run out of RAM or have a file that yaz-marcdump doesn't like. |
| 09:20 |
csharp |
maybe this is the way it always worked, but I'm surprised, obviously :-/ |
| 09:20 |
jeff |
csharp: if you add a WHERE clause to limit to a certain id, or ids between X and Y or greater than Z... does the seq scan go away? |
| 09:21 |
Dyrcona |
Right now, it looks like running of RAM is on the table. |
| 09:21 |
Dyrcona |
Mem: 3.9G 3.7G 106M 1.3M 60M 4.2M |
| 09:21 |
Dyrcona |
Swap: 4.0G 2.5G 1.5G |
| 09:21 |
Dyrcona |
o |
| 09:21 |
jeff |
csharp: given a query that returns all ids, i don't think a seq scan is a poor plan. |
| 09:21 |
csharp |
jeff: yes it does go away - so I'm chasing a red herring |
| 09:21 |
csharp |
thanks for the suggestions |
| 09:21 |
* Dyrcona |
revises his draft email. |
| 09:23 |
|
yboston joined #evergreen |
| 09:23 |
Dyrcona |
And the VM freezes as OOM killer kicks in. |
| 09:23 |
jeff |
and then the horns kicked in... and my shoes started to squeak. |
| 09:24 |
Dyrcona |
Well OOM killer hasn't gotten to it, yet. |
| 09:24 |
Dyrcona |
but bash can't fork another process. |
| 09:24 |
Dyrcona |
This was all it took: marc_export -a -e UTF-8 > all.mrc |
| 09:25 |
Dyrcona |
Something's up with Perl >= 5.20. |
| 09:25 |
Dyrcona |
and all.mrc is empty. |
| 09:26 |
Dyrcona |
I'm going to try by feeding marc_export with equivalent query output from psql. |
| 09:27 |
Dyrcona |
Then, I see Perl debugger sessions in my future. |
| 09:28 |
Dyrcona |
Hmm.. I might be barking up the wrong tree.... It could be DBI/Pg that has the issue. |
| 09:31 |
Dyrcona |
http://pastebin.com/qGLndVaU seems to be headed in the same direction. |
| 09:37 |
Dyrcona |
And, yes, almost there. |
| 09:40 |
Dyrcona |
So, now, I'm dumping the query output to a file. |
| 09:40 |
Dyrcona |
See what that does. |
| 09:41 |
Dyrcona |
It rather quickly creates a 21MB file of 2.7 million bib record ids, so the issue is definitely in marc_export and friends. |
| 09:42 |
Dyrcona |
And pipe the file to marc_export looks like I'm headed for another crash. |
| 09:43 |
Dyrcona |
I could dump about 51 to 52 thousand records yesterday. |
| 09:43 |
Dyrcona |
So, looks like I'll need to split my output up until I figure this out. |
| 09:44 |
Dyrcona |
"I know I used to send you 1 file, but an O/S upgrade kinda busted my tools. Here's 50 files instead." |
| 09:45 |
Dyrcona |
I suppose I could just cat them all together in the end, eh? |
| 09:45 |
|
abowling left #evergreen |
| 09:45 |
jeff |
that is one of the nice things about binary marc files, yes. |
| 09:49 |
Dyrcona |
I should try with -f XML. |
| 09:49 |
Dyrcona |
I haven't done that, yet. |
| 09:49 |
Dyrcona |
I suspect the result will be the same. |
| 09:50 |
Dyrcona |
Then, I'll split my file of ids up into batches of 50,000 and give those a go. |
| 09:52 |
Dyrcona |
This poor, litle VM is taking a lot of abuse this morning. |
| 09:54 |
Dyrcona |
I wonder if 20G is enough room for 2.7 million MARCXML records? :) |
| 09:56 |
Dyrcona |
It's weird, though. It seems almost like whatever is happening messes with Perl's output buffering. |
| 09:57 |
Dyrcona |
I'd expect to get some output as Perl flushes it buffers every 4 to 64K. |
| 10:01 |
phasefx |
kmlussier: test.evergreen-ils.org is out of disk space; I'll see if I can delete some stuff |
| 10:01 |
kmlussier |
phasefx: Thanks for looking into it! phasefx++ |
| 10:01 |
|
rlefaive_ joined #evergreen |
| 10:07 |
jeff |
bug 1671845 |
| 10:07 |
pinesol_green |
Launchpad bug 1671845 in Evergreen "marc_export creating MARC data that yaz-marcdump dislikes" [Undecided,New] https://launchpad.net/bugs/1671845 |
| 10:09 |
csharp |
jeff++ |
| 10:18 |
jeff |
added one comment. |
| 10:31 |
Dyrcona |
So, for some reason, doing 50,000 produces output and doesn't lead to runaway buffering. |
| 10:31 |
Dyrcona |
But, doing the whole database does. |
| 10:31 |
Dyrcona |
I don't know what the limit might be. |
| 10:37 |
Dyrcona |
Boy, this is gonna be "fun." |
| 10:59 |
Dyrcona |
jeff++ # I finally got around to reading the bug report. |
| 11:14 |
|
brahmina joined #evergreen |
| 11:20 |
|
khuckins__ joined #evergreen |
| 11:21 |
|
rlefaive joined #evergreen |
| 11:26 |
* Dyrcona |
goes to make another iced tea while his test runs. |
| 11:34 |
csharp |
@tea Dyrcona |
| 11:34 |
* pinesol_green |
brews and pours a pot of Dragon Pearl Jasmine, and sends it sliding down the bar to Dyrcona (http://ratetea.com/tea/harney/dragon-pearl-jasmine/1320/) |
| 12:11 |
pinesol_green |
News from qatests: Test Failure <http://testing.evergreen-ils.org/~live> |
| 12:12 |
kmlussier |
:( |
| 12:19 |
kmlussier |
egDate? |
| 12:22 |
berick |
ah |
| 12:22 |
berick |
daylight savings i bet |
| 12:22 |
berick |
the classic "2 days" is not always 48 hours. |
| 12:22 |
berick |
kmlussier: if we ignore it until monday it will go away (/me ducks) |
| 12:23 |
kmlussier |
Sounds like a good plan to me! At least until they do away with all this clock-changing nonsense. |
| 12:24 |
berick |
i'd say in this case egDate is fine, the unit test is faulty |
| 12:24 |
berick |
(well, and a comment in egDate that mimics the unit test) |
| 12:55 |
* csharp |
totally forgot that the time changes on Sunday |
| 12:56 |
Dyrcona |
Yeah. |
| 12:56 |
Dyrcona |
Two days = 48 hours give or take an hour or two. |
| 13:24 |
|
hbrennan joined #evergreen |
| 13:53 |
kmlussier |
I noticed this last week and then promptly forgot about it again. |
| 13:54 |
kmlussier |
When signing off on and merging bug bug 1308090, I never checked or added anything for the required reingest for the upgrade script. It therefore didn't make it to the 2.12 upgade script. |
| 13:54 |
pinesol_green |
Launchpad bug 1308090 in Evergreen "sorting of name headings with relator codes " [Wishlist,Fix released] https://launchpad.net/bugs/1308090 |
| 13:56 |
kmlussier |
What's the best way to fix this? Should I add it to the 1006 upgrade script now or is there a better way to handle it? |
| 13:57 |
Dyrcona |
Well, you're release manager, and theoretically no one has run that script yet in production. |
| 13:57 |
Dyrcona |
I prefer not to mess with existing scripts, but I think I've done so once or twice. |
| 13:58 |
kmlussier |
Dyrcona: Yes, I'm release manager, but I like to rely on the wisdom of the community. :) |
| 13:58 |
Dyrcona |
Of course! |
| 13:59 |
kmlussier |
And the community has been very generous in that regard. |
| 13:59 |
Dyrcona |
For the ingest, I'd just stick \echo lines in the bottom of the upgrade script. |
| 14:00 |
Dyrcona |
There are some previous examples. |
| 14:00 |
kmlussier |
At a minimum, I should open a Launchpad bug. |
| 14:00 |
kmlussier |
Dyrcona: Yes, I've done one once before for jlitrell's code. I'm pretty sure I mangled it up a few times before I got it right. |
| 14:00 |
Dyrcona |
In this case, I wouldn't make a new upgrade script. |
| 14:01 |
kmlussier |
But, looking at the fix, I'm thinking a full reingest is needed, right? Because it affects browse and facet entries for anything that has an author. |
| 14:04 |
Dyrcona |
Oh. But I'd still have it echo that you need a full ingest. It can be perturbing to run an upgrade script that takes hours. |
| 14:05 |
* kmlussier |
nods. |
| 14:06 |
Dyrcona |
So, I just replaced a couple of hundred lines of Perl with two configuration files that requires a custom module or two with a 50 or so line shell script including blank lines and comments. |
| 14:06 |
Dyrcona |
It's maybe 30 lines of actual code. |
| 14:08 |
Dyrcona |
@dunno search Go away |
| 14:08 |
pinesol_green |
Dyrcona: 1 found: #13: "Go away, or I'll replace you with a very small..." |
| 14:08 |
Dyrcona |
@dunno get 13 |
| 14:08 |
pinesol_green |
Dyrcona: Dunno #13: "Go away, or I'll replace you with a very small shell script!" (added by Dyrcona at 04:39 PM, October 02, 2012) |
| 14:08 |
Dyrcona |
:) |
| 14:12 |
kmlussier |
berick++ bug 1671904 |
| 14:12 |
pinesol_green |
Launchpad bug 1671904 in Evergreen "egDate unit test fails on time change boundary" [Undecided,New] https://launchpad.net/bugs/1671904 |
| 14:31 |
Dyrcona |
I love it when git diff tells me two lines are different, but I don't see and it doesn't look like whitespace changes, either. |
| 14:31 |
Dyrcona |
Although git diff -w doesn't show this particular difference. |
| 14:32 |
berick |
no one expects the zero-width space |
| 14:35 |
Dyrcona |
true... |
| 14:35 |
Dyrcona |
It's all comment lines in a crontab file, so no biggie. |
| 14:40 |
* Dyrcona |
renames a few local branches. |
| 15:39 |
|
Jillianne joined #evergreen |
| 15:41 |
kmlussier |
I've added a branch at working/collab/kmlussier/2_12RC_release_note_additions where release note changes can be made. |
| 16:01 |
|
dkyle joined #evergreen |
| 16:31 |
pinesol_green |
News from qatests: Test Failure <http://testing.evergreen-ils.org/~live> |
| 16:55 |
jeff |
i think that marc_export with --all and --items may exclude bibs with zero holdings, vs marc_export --all |
| 17:03 |
|
mmorgan left #evergreen |
| 17:18 |
jeff |
yup, confirmed. |
| 18:02 |
|
jvwoolf joined #evergreen |
| 18:18 |
jeffdavis |
I asked the other day about using the new nginx proxy setup for different OPAC skins on separate subdomains. Turns out the default nginx config works just fine for this, as long as you remember to adjust the port numbers in *all* your Apache vhosts. *shakes head at self* |
| 18:58 |
|
Enjabain joined #evergreen |
| 21:25 |
|
jvwoolf joined #evergreen |
| 23:28 |
pinesol_green |
[evergreen|Bill Erickson] LP#1671904 egDate unit test handles time change - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=482c984> |
