Evergreen ILS Website

IRC log for #evergreen, 2013-07-24

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
01:29 Mark__T joined #evergreen
01:57 sseng_ joined #evergreen
01:58 BigRig joined #evergreen
04:28 rri_ joined #evergreen
04:28 wjr__ joined #evergreen
04:28 dbs_ joined #evergreen
04:29 pmurray` joined #evergreen
04:29 jeff___ joined #evergreen
04:29 mtj_- joined #evergreen
05:11 ktomita joined #evergreen
06:08 sseng joined #evergreen
06:08 ktomita_ joined #evergreen
06:08 fparks joined #evergreen
06:11 dconnor joined #evergreen
06:39 jeff___ joined #evergreen
07:26 collum joined #evergreen
07:37 jboyer-isl joined #evergreen
08:09 misilot joined #evergreen
08:10 Dyrcona joined #evergreen
08:21 pmurray joined #evergreen
08:27 kbeswick joined #evergreen
08:35 akilsdonk_ joined #evergreen
08:35 mrpeters joined #evergreen
08:41 finnx joined #evergreen
08:48 mmorgan joined #evergreen
08:51 kmlussier joined #evergreen
09:00 Meliss joined #evergreen
09:05 moodaepo_nb joined #evergreen
09:08 ericar joined #evergreen
09:19 Dyrcona Grr. pg_restore ran for over 24 hours.
09:20 Dyrcona I had to kill CREATE INDEX metabib_full_rec_index_vector_idx ON metabib.real_full_rec USING gist (index_vector); for it to finally finish.
09:20 Dyrcona I'm now running the above via psql.
09:20 dbs_ yikes.
09:36 * Dyrcona wonders if he should run upgrade scripts while that is going on.
09:52 yboston joined #evergreen
09:52 mschell joined #evergreen
09:56 dbs joined #evergreen
09:57 * dbs could have sworn we were going to cut over to GIN instead of GIST a few releases back
10:02 kayals_ joined #evergreen
10:18 jeff gmcharlt++ for libxml goodness in MARC::File::XML
10:26 Dyrcona Hey! That create index that ran since 11:27 yesterday, finished in about an hour after I killed it and restarted it.
10:26 Dyrcona bad comma!
10:27 mcooper joined #evergreen
10:36 rfrasur joined #evergreen
10:42 dboyle joined #evergreen
10:56 Meliss1 joined #evergreen
10:56 zerick joined #evergreen
11:04 kayals joined #evergreen
11:35 jeffdavis dbs: I believe you're right about that. We've been using GIN indexes for a while now FWIW.
11:36 bshum dbs: Maybe it's some orphaned branch somewhere.
11:37 Dyrcona Maybe its just our database that may not have had all the indexes recreated at every upgrade?
11:38 Dyrcona recreated/redefined....
11:43 phasefx joined #evergreen
11:45 jdouma joined #evergreen
11:57 acoomes joined #evergreen
12:00 smyers_ joined #evergreen
12:24 rfrasur There are few things nicer than having someone be both pleasant AND knowledgeable on the phone.
12:26 tsbere rfrasur: Pleasant, knowledgeable, and permitted by their company rules to help you, perhaps? I find the first two fairly often with the third absent. :(
12:27 rfrasur tsbere: I know!  But it just happened.
12:27 * rfrasur has renewed faith in humanity.
12:35 rfrasur But then I tried to deal w/ adobe reader and microsoft word and realized why those moments are so rare.
12:36 rfrasur @hate Microsoft
12:36 pinesol_green rfrasur: The operation succeeded.  rfrasur hates Microsoft.
12:36 rfrasur @hate Adobe
12:36 pinesol_green rfrasur: The operation succeeded.  rfrasur hates Adobe.
12:38 stevenyvr2 joined #evergreen
12:49 mllewellyn joined #evergreen
13:15 jhpringle joined #evergreen
13:16 _bott_ I think related to bug 976054, I'm not getting URI only bibs to show up in bookbags.  Can anyone else reproduce this?
13:16 pinesol_green Launchpad bug 976054 in Evergreen "Tpac - not showing deleted bibs in my lists" (affected: 2, heat: 10) [Wishlist,Confirmed] https://launchpad.net/bugs/976054
13:18 kmlussier _bott_: I think there may be a bug on that. Let me look.
13:19 linuxpoet left #evergreen
13:19 kmlussier _bott_: No, it's a little different. That bug was related to adding those records to bookbags. https://bugs.launchpad.net/evergreen/+bug/979038
13:19 pinesol_green Launchpad bug 979038 in Evergreen "tpac: Records with a located URI cannot be added to a list" (affected: 1, heat: 6) [Low,Confirmed]
13:21 _bott_ I think both bugs are related.  The item is entered into the table, but it's not visible via TPAC
13:22 _bott_ Much like 976054, which notes a title, "no longer had any items attached".
13:23 kmlussier _bott_: Sure, that makes sense. I was just coming to the same conclusion. I must have thought the problem was adding the item when, in fact, I just wasn't seeing it there. Never checked in the database to see what was happening.
13:23 * Dyrcona dies a little more inside.....
13:25 * rfrasur wonders if one of our access points is dying or if Comcast is having issues.
13:26 kmlussier _bott_: What version of Evergreen are you running? I just tried it on 2.3 and didn't have any trouble.
13:27 _bott_ Ah, 2.2'ish.
13:28 _bott_ I'll dig into any related pieces I can find.
13:32 jeff_ oops:  j => $copy->{call_numner},
13:33 kayals joined #evergreen
13:33 jeff_ "use strict" did not help me there.
13:33 dbs Ah. The return of the Numenorian king.
13:33 gmcharlt dbs++
13:34 jeff_ dbs++
13:36 phasefx remember tolkien-ring networks? <ducks>
13:40 bshum dbs: We also hid subfield 6 as part of work in bug 1085554 which was released for 2.3.4 according to the tracker.
13:40 pinesol_green Launchpad bug 1085554 in Evergreen "TPAC - hide subfield 6 and 8 from record title display" (affected: 2, heat: 12) [Medium,Fix released] https://launchpad.net/bugs/1085554
13:40 zed1 joined #evergreen
13:41 bshum But dbs++ for replying faster than I could :)
13:41 dbs bshum: yeah, I was looking at the 2.3 branch too and thinking "Man, I don't even know if we're talking TPAC or JSPAC at this point"
13:46 moodaepo_nb joined #evergreen
13:56 Evergreen joined #evergreen
13:57 Evergreen I'm working to get EDI working on our Evergren install, and I'm getting a Address Already in Use error, no matter what port I give it when I start edi_webrick.bash.  Any ideas what is wrong?
14:00 jeff_ Given an evergreen database with postgresql encoding UTF-8, i I see the following string in psql output, am I correct in thinking that this is the XML as it appears in the db, and this is not psql showing me an escaped version of the data? Montalv&#xE1;n
14:01 jeff_ Evergreen: I'm not well-versed in EDI, but I can stab in the dark until someone else comes along -- what ports have you tried to use?
14:01 bshum Evergreen: I think the default port is 9191?  So do you have stuff running on that port already for that server?
14:02 Evergreen 9191, 9391, 9193, 9292, and so on.  Nothing is running on those ports.
14:02 bshum And also, curious where you changed the port value.  I envisioned edi_webrick.cnf for the config options, but I haven't changed it either.
14:02 jeff_ Evergreen: what is the exact text of the error?
14:02 * tsbere once saw that kind of error when security layers said "you can't open ports for the world to connect to unless I have been told you can" - selinux might do that?
14:02 Evergreen before i start edi_webrick, 9191 is not in use
14:03 Evergreen TCPServer Error: Address already in use - bind(2)
14:04 Evergreen To spell it out more, using the default port 9191, nothing is running.  Then when I start edi_webrick, it shows that is running on 9191, but then it throws the error that the port is in use, and it won't let me connect to port 9191 using the test_client.pl or telnet
14:05 Evergreen It's almost like it's tripping over itself.
14:06 Dyrcona Evergreen: netstat -an will tell you what ports are in use on your server. Maybe something is using 9191 and you aren't aware of it.
14:07 kbeswick joined #evergreen
14:07 rfrasur joined #evergreen
14:08 Evergreen That's what I thought as well.  But netstat -an shows nothing on 9191, nor does lsof -i :9191
14:08 rfrasur (who is Evergreen?)
14:09 tsbere Thought: Is it configured to listen twice, and the second time is being stepped on by the first?
14:10 Dyrcona rfrasur: I'm going to guess someone from Asbury Seminary.
14:10 * rfrasur nods
14:10 bshum That was going to be my guess too, given the recent EDI questions on the list.
14:10 Evergreen Yes, that's me
14:10 Evergreen typo
14:11 pgardella there we go
14:11 rfrasur ahh :-)
14:11 pgardella tsbere: how would i know if it was configured to listen twice?  I'm using it striaght out of the box
14:12 tsbere pgardella: No clue, we don't use that functionality right now.
14:12 pgardella :)
14:12 * tsbere is just guessing based on experience with other services
14:12 bshum pgardella: Out of the box, ours just "worked".  I've never seen that problem before.
14:15 pgardella well, I reran install.sh and now even though I get the same error, I can at least connect to taht port
14:20 bshum pgardella: Just curious, which version of Evergreen are you working with?
14:20 pgardella 2.3.2
14:21 * dbs wonders if there was a fix to EDI stuff in the subsequent 7 point releases
14:27 dbs Acq: EDI omnibus bugfix package from af46b5cc031 was January, 2013
14:27 pinesol_green [evergreen|Lebbeous Fogle-Weekley] Acq: EDI omnibus bugfix package - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=af46b5c>
14:28 dbs which probably would have been 2.3.3
14:28 pgardella Ah, good point.  Thanks.  We'll try that!
14:29 dbs In general, it's tougher on us if bugs are reported for versions that aren't up to date :/
14:29 zed1 left #evergreen
14:29 pgardella Yeah, I know.  Sorry. :(
14:30 zed1 joined #evergreen
14:30 bshum Looks like omnibus was linked to 2.3.4 actually.  Though 2.3.3 had some EDI reader cleanup too.
14:31 bshum I've also got this vague recollection that there's some custom branch for edi that I had to checkout manually somewhere along the line
14:31 bshum I'm looking for old references
14:31 bshum But that might only apply for new EDI invoicing work in 2.4.
14:35 jeff_ hah.
14:35 jeff_ use encoding 'utf8';
14:35 jeff_ completely messing me up today.
14:36 jeff_ (not needed by my code, and deprecated in 5.19 for seemingly good reasons)
14:42 jeff_ hrm. Montalv&#xE1;n becomes Montalv&#xE2;an after a MARC::File::XML -> yaz-marcdump journey.
14:48 Dyrcona jeff_: Are e1 and e2 valid utf-8 codepoints?
14:48 pgardella left #evergreen
14:48 jeff_ yeah, but the second is wrong. still digging.
14:49 jeff_ looks like i needed to pass explicit encoding to new_from_xml
14:49 Callender_ joined #evergreen
14:50 dbs jeff_: there should be a bunch of examples sprinkled through Evergreen of how to do it; eeevil and I tried to make things consistent, once upon a time
14:50 jeff_ in an evergreen database, is any biblio.record_entry.marc data with leader position 09 set to ' ' an error? it's not possible to have MARC8 in marcxml, is it?
14:51 gmcharlt jeff_: correct, Leader/09 should be set to 'a' throughout
14:51 jeff_ dbs: yeah. i'm getting some of my clues from marc_export -- much credit to everyone who worked on that.
14:51 Callender__ joined #evergreen
14:51 Dyrcona jeff_: I don't think e1 or e2 are valid as single characters in UTF-8. I think they start longer sequences.
14:52 jeff_ Dyrcona: U+00E1 and U+00E2 seem to be valid on their own. maybe U+00E1 != &#xE1; ?
14:52 Dyrcona jeff_: No, it isn't.
14:52 RoganH joined #evergreen
14:52 csharp tt2++
14:52 Callender joined #evergreen
14:52 jeff_ anyway, passing UTF-8 to new_from_xml seems to be working, and i'm going to exclude those records where the leader/09 isn't 'a' -- we'll see how many there are.
14:52 bshum Found it.  There was some special changes to the install process for EDI with bug 1030041
14:52 pinesol_green Launchpad bug 1030041 in Evergreen "ORDER record copy-level data for "enriched" EDI " (affected: 1, heat: 8) [Undecided,Fix released] https://launchpad.net/bugs/1030041
14:53 bshum To grab a custom git checkout for openils-mapper from http://git.evergreen-ils.org/?p=wor​king/random.git;a=shortlog;h=refs/h​eads/collab/berick/openils-mapper
14:53 jeff_ Dyrcona: ah. there's another place I'm wrong. :-)
14:53 bshum berick: Do I poke you further on that one?  :)
14:53 Dyrcona I mean I don't think U+00E1 equal &#xe1;
14:54 Dyrcona jeff_: I'm guessing you have MARC-8 or ISO-8859-1 input here.
14:56 Dyrcona U+00E1 = &#xc3;&#xa1;
14:57 berick bshum: hmm, the code all lives on github now (since that's where the original code lives)  https://github.com/berick/openils-ma​pper/tree/GIR-segments-for-copy-data
14:58 Dyrcona The property entity is &aacute;
14:58 gmcharlt Dyrcona: I'm not so sure about that -- I think &#xE1; means the same as &x00E1; -- that's part of the range where iso-8859-1 and Unicode overlap
14:58 Dyrcona proper
14:58 gmcharlt &#x00E1; that is
14:59 Dyrcona gmcharlt: I'm going by  http://www.utf8-chartable.de/unicode-u​tf8-table.pl?start=128&amp;number=128&​amp;names=-&amp;utf8=0x&amp;htmlent=1
14:59 Dyrcona You are probably correct, just the same.
14:59 gmcharlt yeah, &#xNNNN; are numeric character references in XML, not octet references
14:59 Dyrcona The 00 may be stripped in the entity, not sure.
15:00 bshum berick: Ah gotcha, I didn't have the updated branch URL in my old logs.
15:01 gmcharlt Dyrcona: yep, I believe it is
15:02 * jeff bangs on the reporting db with select marc from biblio.record_entry where marc ~ '.*<leader>......... .*' limit 1;
15:03 jeff no rows.
15:04 * jeff revises to select marc from biblio.record_entry where marc !~ '.*<leader[^>]*>.........a.*' limit 1;
15:05 Dyrcona I was erroneously expecting the entity to correspond with the utf-8 hex code.
15:06 jeff 25996 records do not match '.*<leader[^>]*>.........a.*'
15:06 jeff hrm.
15:07 gmcharlt Dyrcona: it does, leading zeroes just get truncated
15:08 jeff only 3447 of those records are not-deleted.
15:08 jeff ugh.
15:09 bshum berick: Are we pointing at your new branch on github during the install though?  Just thinking of the stock setup.
15:11 Dyrcona jeff: One of your records is likely bib -1, if that makes you feel any better. ;)
15:12 jeff_ heh
15:12 jeff it does.
15:13 Dyrcona It has no leader, just an empty <record /> with namespace, at least in my database.
15:14 Dyrcona gmcharlt++ # for bearing with me in a moment of confusion on my part
15:15 gmcharlt Dyrcona: my pleasure; character set issues have given me a ton of headaches over the years, and I'm happy to help NOT spread the pain around
15:16 Callender joined #evergreen
15:17 Dyrcona gmcharlt: Speaking of which, you said a while ago that you were going to release a new version of MARC::Charset. I see 1.34 is still the latest on CPAN.
15:18 Dyrcona Those changes that I picked from sourceforge worked for me, except the Hangul records as you noted.
15:18 * dbs was thinking about updating Fedora packages sometime in the next few weeks
15:18 gmcharlt Dyrcona: yep, sudden loss of tuits for some Cyrllic corrections I'm working on -- release soonish, though
15:18 Dyrcona gmcharlt++
15:19 dbs gmcharlt++
15:20 berick bshum: i don't recall offhand if any of the EDI configuration/install is documented.  (it's been a while)
15:20 Dyrcona I find it amusing that when I tell someone we're having charset problems with MARC, they always ask, "Is it Cyrillic?"
15:20 Dyrcona I usually have to say, "No, its Vietnamese."
15:20 Dyrcona ;)
15:20 bshum berick: Yeah it's in the docs.  I guess we need to adjust them slightly to point at the new github location.
15:20 bshum For 2.4 and 2.3 too, I'll bet.
15:20 berick yeah
15:24 bshum I'm tackling something with ou proximity quirkiness; but I'll mock up a branch to change the docs later this evening maybe.
15:24 berick bshum: well, i could push it all back to the evregreen repo.  i threw it on github under the hopes I could get it merged back into mbklein's original repo, but in the long run i don't think it's a reasonable approach.
15:34 jeff more good news: specifying the encoding to new_from_xml also gives a heck of a speed boost, since it was transcoding to marc8 before. :P
15:34 jeff i really thought that the defaults were properly set, and that i had verified that.
15:41 jeff hah. went from about 600 reconds/sec to 20,000 records/sec
15:42 Dyrcona jeff: Do you know how many times MARC::Record->new_from_xml() appears in the EG code?
15:42 jeff no, but i'm about to find out, either from you or from ack.
15:43 jeff new_from_xml appears 124 times
15:43 jeff (in master)
15:43 Dyrcona So, there are 124 opportunities for improvement. :)
15:44 jeff now of course a lot of that is not live code, being upgrade scripts, etc.
15:44 Dyrcona Right.
15:44 jeff Dyrcona: this speed is something that the Evergreen code I was looking at as an example (marc_export) already was taking advantage of.
15:44 Dyrcona Multiple upgrades to the same database functions.
15:44 jeff but there could be others. :-)
15:45 Dyrcona I see a few that inlcude the encoding as a parameter.
15:45 Dyrcona Maybe it isn't that big of a deal.
15:45 jeff whenever i wait for a selfcheck to run the MODS xslt to get the title of each of my 38 holds, i consider ways to improve that performance
15:45 jeff (unrelated to new_from_xml)
15:46 jeff like, caching the most commonly used mods values, or delaying/backgrounding the xslt from happening during the initial patron info SIP requests...
15:47 Dyrcona Most of the uses of it in perl modules specify the encoding.
15:47 jeff so yeah, i can do a full export of the in-scope bibs + holdings in just over 8 minutes now. previously it was taking 2.5 - 3 hours.
15:49 Dyrcona jeff: I think I need to check a couple of my export scripts now....
15:50 Dyrcona Ah well, already specifying UTF-8.....
15:53 jeff_ for context, that's just a subset of our records: 222,861 holdings across 165,247 bibs.
15:54 jeff_ i need to exclude some more items, actually. forgot to exclude age hold protected stuff, which won't be in scope here.
15:54 * jeff_ looks for an existing db-level "ahp is in effect"
15:56 dbs if those usages of new_from_xml() that don't specify encoding use " use MARC::File::XML ( BinaryEncoding => 'utf8', ... " then they should still be fine.
15:59 dbs unless I'm misunderstanding. of course I'm also looking at MARC::File::XML 0.93
16:00 dbs Actually, even if they don't specify that, they should default to utf-8. Huh.
16:00 * dbs makes a mental note to compare with MARC-XML 1.0.1
16:06 kmlussier joined #evergreen
16:08 dbs looks the same. hmm.
16:14 * gmcharlt was kinda hoping it would, since the main point of MARC-XML 1.0.1 was to get Moar Speedz without changing behavior
16:17 dbs gmcharlt: does what jeff is saying make any sense to you? I don't see how he could get an automatic transcode to marc8 happening unless he explicitly set the default encoding to marc8 :/
16:19 gmcharlt dbs: it current will auto-transcode if the Leader/09 is not 'a'
16:19 gmcharlt *currently
16:25 dbs gmcharlt: ah, right. so bad incoming XML
16:25 tmccanna joined #evergreen
16:25 tmccanna left #evergreen
16:28 jeff_ it seemed to be doing that in cases other than where ldr/09 was not 'a'
16:29 jeff_ i can probably give a test case.
16:32 jeff_ i'm not a huge fan of age hold protection rules / policies, so i appreciate the aliased / abbreviated name of config.rule_age_hold_protect.
16:32 jeff_ AND (crahp.age IS NULL OR AGE(NOW(), acp.create_date)) > crahp.age)
16:33 gmcharlt jeff_: I'll await with interest
17:25 jeff_ ah.
17:25 jeff_ having this: use MARC::File::XML ( BinaryEncoding => 'UTF-8' );
17:26 jeff_ then somewhere after an eval having this:         import MARC::File::XML; # reset SAX parser so that one bad record doesn't kill the entire export
17:26 mmorgan left #evergreen
17:28 jeff_ even if it's never hit, because the eval test never fires, the use is still parsed/executed...
17:30 jeff_ so i used marc_export as an example, but not enough of it.
17:31 dbs gmcharlt: I've pushed MARC::File::XML packages for 1.0.1 to Fedora rawhide and 19 testing; in a few days they should be part of stable
17:31 gmcharlt dbs++
17:31 jeff_ and now i know what to look for next time.
17:32 jeff_ i wonder if that's what was causing the marc8 transcoding to show up in profiling on larger batches.
17:32 jeff_ *dig*, *dig*
17:33 jeff and of course, since sax isn't in play in recent versions, the whole reinit step might not be required -- if it ever was?
17:33 jeff_ dunno.
17:35 finnx left #evergreen
17:38 * dbs wonders idly if that "import MARC::File::XML" should be "import MARC::File::XML ( BinaryEncoding => 'UTF-8' )"
17:40 jeff in marc_import? it doesn't much matter because the new_from_xml supplies an encoding arg.
17:41 jeff oh heck.
17:41 jeff yeah, that's another thing -- i was doing two uses, not one use and one import.
17:42 jeff which is why the second one (which should have been import if i was copying marc_export) was being parsed at BEGIN time rather than when it should -- only when the eval fell through.
17:42 dbs heh
17:45 jeff anyway, i don't have any issues to point at at this time, i've re-learned a bunch of things i've forgotten or overlooked, and i may do more digging on the profiling that seemed to point at marc8, in light of what i now know.
17:45 jeff oh.
17:47 jeff yeah, that was due to the encoding being reset to marc8.
17:47 jeff all mysteries solved! time to go pack more boxes!
17:50 gmcharlt at least you're not opening them, so hopefully no more mysteries this evening as well ;)
17:54 berick *shudder* at opening a box and a marc8 blob pops out
17:56 berick marc8 blob => http://perla94.edublogs.org/files​/2012/03/abyss-blob1-1b364p8.jpg
17:57 gmcharlt berick: pass the mental bleach, would ya?
17:57 mcooper joined #evergreen
18:16 zed1 left #evergreen
18:24 acoomes joined #evergreen
18:36 jeff_ gmcharlt: only mystery surrounding boxes is "where am i going to put all these?"
18:37 jeff_ "hello, i'd like to inquire about your storage units. size? do they come in 'tardis'?"
18:39 gmcharlt "why yes, for the low fee of one Triganic Ningi"
22:12 stevenyvr2 joined #evergreen
22:12 stevenyvr2 left #evergreen
23:44 mcooper joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat