Evergreen ILS Website

IRC log for #evergreen, 2014-02-14

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
00:18 bshum jeff: Definitely warmer and less snowy than CT.  Or so I hear.
01:24 pinesol_green [evergreen|Mike Rylander] LP1046026 - Fix duplicate rows in holds pull list - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=dc58382>
01:30 bshum @later tell eeevil Question about bug 1224597, the LP description and git commit message are a little vague on the pro's of reordering to put INNER JOINs before potential OUTER ones.  For my own curiosity & learning (I'm sure it's probably a good idea) but what are the benefits to this?
01:30 pinesol_green bshum: The operation succeeded.
01:30 gsams joined #evergreen
01:32 bshum @later tell dbwells Just a quick opinion on https://bugs.launchpad.net/evergreen/+bug/1255172 where we have a potential string change that's got a 2.5 target.  I'm 100% fine with the language change for clarity, but backporting string changes seems bad.  Thoughts/opinions?
01:32 pinesol_green bshum: The operation succeeded.
01:32 pinesol_green Launchpad bug 1255172 in Evergreen ""Alias" in Patron Search needs to be labeled "Holds Alias" for clarity/consistency" (affected: 1, heat: 6) [Low,In progress] - Assigned to Ben Shum (bshum)
01:50 bshum @later tell tsbere Unable to pick the change in https://bugs.launchpad.net/evergreen/+bug/1016204 due to conflicts when 7a7c229db was put in. Probably a simple fix, but tossing it back to you.
01:50 pinesol_green bshum: The operation succeeded.
01:50 pinesol_green Launchpad bug 1016204 in Evergreen "backdating checkins to the future should not be allowed" (affected: 1, heat: 6) [Medium,Triaged]
01:58 bshum @later tell berick If I'm reading the notes on https://bugs.launchpad.net/evergreen/+bug/1265047, it is just a temporary measure to up the time limit while we work on making vandelay matching faster in the future?  If I read that right, then I think I'd be happy to push that through for you tomorrow.
01:58 pinesol_green bshum: The operation succeeded.
01:58 pinesol_green Launchpad bug 1265047 in Evergreen 2.5 "Vandelay queued record creation can timeout, causing queue creation failure" (affected: 1, heat: 6) [Undecided,New]
02:33 pinesol_green [evergreen|Jeff Godin] Add button to update expire date field - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=6807a06>
02:33 pinesol_green [evergreen|Jeff Godin] Release notes for "Update Expire Date" button - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=c317a6e>
03:41 jeff joined #evergreen
07:42 Dyrcona joined #evergreen
07:47 tsbere__ joined #evergreen
08:26 berick bshum: re:1265047 exactly.
08:48 jl- joined #evergreen
08:55 mmorgan joined #evergreen
08:59 dluch joined #evergreen
09:26 eeevil bshum: I'll look
09:31 jeff bshum++
09:34 eeevil bshum: PG will try different join orderings during the planning phase, but each additional join adds a whole new pile of possible orders.  once there are enough that brute force planning is likely to take a very long time, it switches to a genetic algo. at that point, the user-supplied order becomes important because it's used as a guide to finding a good plan. since we know the shape of the data, and know that there are indexes to use on those two
09:34 eeevil tables, we can help the planner out by telling it about a good order for those joins.  in search, we can get into the "too many joins" situation if we have a long string of subqueries (explicit bool ops, different classes, or paren-enclosed expressions). that patch is attempting to protect against some worst-case plans
09:35 Callender joined #evergreen
09:36 eeevil bshum: it'll show no difference for simple queries (other than reducing maintenance by making the query a little easier to read by grouping together joins we always do) but /might/ help in pathologically complex queries
09:47 gsams joined #evergreen
09:49 kbeswick joined #evergreen
09:49 yboston joined #evergreen
10:07 BigRig joined #evergreen
10:31 ericar joined #evergreen
10:44 gdunbar joined #evergreen
10:50 eeevil csharp: you around?
11:12 tsbere So, in playing around with call number sorting I created this: http://pastebin.com/94U8XBzE - The "one main function with two wrappers" is to keep the two acting as closely as possible outside of how they handle dewey pieces (I think some of MVLC's issues lately are related to drift between the dewey and generic normalizers). This also allows numbers to sort properly (aka, 10 after 9, not before 2). Any thoughts?
11:14 phasefx tsbere: do you have some easily shareable comparisons of output?
11:14 tsbere phasefx: Gimme a min
11:16 phasefx I don't have a good handle on how dewey and generic differ at the moment
11:19 tsbere Hmmm
11:20 * tsbere notes an issue with the true/false bit, he tested with 1/0 in a pl file but forgot the DB doesn't do that with boolean values
11:22 tsbere easily fixed. <_<
11:26 tsbere phasefx: I added example callnumbers to the end of the paste and what the original and "new" variants spit out for them
11:26 phasefx cool deal
11:44 mrpeters joined #evergreen
11:45 mrpeters left #evergreen
11:46 dbwells tsbere: If we're going to move in this direction, I'd love to see the dewey right padding done away with.  Is there a case where it would sort differntly than the space alone would?
11:50 ktomita_ joined #evergreen
11:51 phasefx I think showing some examples of sorting under the different regimes could help, including the cases you're trying to fix
11:52 tsbere dbwells: I think the main issue there is the extra digits on a lone number. Changing it to not do anything special with existing and adding a .0 (instead of .000000000000000) would, I think, deal with all the sort issues for dewey in that case.
11:53 dbwells I agree
11:53 tsbere dbwells: I was mainly being consistent with numbers in this case otherwise. ;)
11:57 * phasefx likes the wrapper notion; just doesn't think generic was ever well defined
11:57 tsbere phasefx: Some of the sorting fixes are a bit of a PITA, given that one such "issue" with us is with things like Fic/King, but mixed generic and dewey (with no digits in the call numbers at all) on the pull list
11:59 * phasefx nods
12:00 * tsbere adds some new examples to the list to help show ordering differences
12:06 * tsbere notes that he didn't get different ordering for the original generic/dewey when used consistently, will need to build an example for the mixed issue MVLC saw
12:07 jihpringle joined #evergreen
12:11 tsbere phasefx: Really hard to come up with things beyond pointing at the differences in handling of / on the original generic and dewey normalizers, plus the original dewey using underscores instead of spaces (which generic had changed, I believe, due to decimals coming before "whole numbers" in call number browse when it used underscores as well)
12:11 tsbere The fact that the change in question was not applied to dewey as well may technically be a bug itself.
12:13 phasefx another thing worth looking at / considering is how the equivalents in Koha have evolved, if at all.  May want to try to keep the two projects on the same page here
12:15 phasefx I did that with my last call number sorting fix; but my head is out of the game now
12:19 mcooper joined #evergreen
13:06 ericar joined #evergreen
13:15 jl- I just got this error: psql:load_concerto.sql:17: ERROR:  Field 012 must have indicators (use ' ' for empty indicators) at /usr/share/perl5/MARC/File/SAX.pm line 113
13:15 jl- CONTEXT:  PL/Perl function "maintain_901"
13:15 jl- any suggetions on troubleshooting this?
13:20 jl- dbs
13:54 jeff jl-: you should examine your source records, but i expect you have a record with a 012 field that lacks any value for ind1 or ind2.
13:55 jeff jl-: you'll need to repair those issues before attempting to insert them.
13:56 ktomita joined #evergreen
13:58 jeff good: <datafield tag="010" ind1=" " ind2=" ">
13:58 jeff bad: <datafield tag="010" ind1="" ind2="">
13:58 jeff also bad: <datafield tag="010">
13:59 jeff jl-: though I also note that tag 012 isn't defined in MARC21. I'm not familiar with your specifics in terms of if your data follows another standard. You may have challenges there also, or you may simply have a small bit of invalid data in general.
14:07 ktomita_ joined #evergreen
14:10 jl- jeff: any idea how I can repair / find that particular record
14:10 jl- I ran the records.xml through yaz-marcdumper
14:11 jeff you should be able to search for an 012 field, assuming it's uncommon in your data
14:12 jcamins jl-: if the 012 isn't uncommon enough for you to just look for a field 012, a little script that uses MARC::Batch and prints the title of every record it hits will get you there... it'll crash after hitting the invalid record, and you can just go to the record after the one that was last printed out.
14:15 jl- hmm
14:15 jl- 200,000 records so 012 might be common
14:15 jl- I can do a grep
14:17 jeff you could also try to grep for ind1="" or ind2=""
14:17 jeff but jcamins' suggestion is worth doing. it will help you catch the next issue.
14:18 ktomita joined #evergreen
14:29 jl- jeff: all tags are ind1=" " or ind2=" "
14:30 jl- none are ind1=""
14:30 jl- how can I look for the field 12 more specifically?
14:30 jl- grep -c '012" yields wayy too many results
14:33 Dyrcona I'd do what jcamins suggests, but I'd print the record position in the file as a number, basically just incrementing a count variable as each record is read and printing that.
14:34 jl- I need to find that record first
14:34 Dyrcona jl-: grep "tag='012'"
14:35 Dyrcona What jcamins suggested is another way to find that record and others that have problems.
14:35 jcamins Dyrcona: the problem with the count is that it's a nuisance to find the nth record in an XML file.
14:36 Dyrcona jcamins: It can be, but when you know how to use sed, it's easy.
14:36 Dyrcona ;)
14:36 jcamins Dyrcona: for XML?
14:36 * jcamins doesn't know how to do that.
14:37 jl- 012 Dyrcona I have too many of those :)
14:37 jcamins Well... unless I collapse each record into a single line.
14:37 Dyrcona jcamins: sed works just fine over multiple lines.
14:38 Dyrcona jl-: Can you program in Perl?
14:38 jcamins Dyrcona: that would be an example of something about sed which I did not know. I may have to find a use to test this.
14:38 jcamins *reason
14:39 jl- grep -c '\="012"' ship_all.xml
14:39 jl- 3111
14:40 Dyrcona @marc 012
14:40 pinesol_green Dyrcona: unknown tag 012
14:48 Dyrcona jl-: This should get rid of all 012 tags in your file: sed -e '/<datafield tag="009"/,/<\/datafield>/ d' < ship_all.xml > ship_fixed.xml
14:48 Dyrcona oh...
14:48 Dyrcona I forgot the change the tag when I pasted.
14:48 Dyrcona sed -e '/<datafield tag="012"/,/<\/datafield>/ d' < ship_all.xml > ship_fixed.xml
14:49 Dyrcona Then, try loading ship_fixed.xml.
14:51 Dyrcona You might want to get rid of 009, also. That's not a valid field either.
14:54 stevenyvr joined #evergreen
15:01 Polonel joined #evergreen
15:01 jl- hmm
15:01 jl- grep -c 'datafield tag="012"' marcxml.xml
15:01 jl- 3111
15:02 jl- grep -c /record marcxml.xml
15:02 jl- 237543
15:02 Polonel I get the following error after updating from 2.3.2 to 2.5.2 http://paste2.org/mZGtk4sf
15:02 jl- they're not in line format Dyrcona so removing the line is currently not an option
15:03 jl- I could write a script that will delete everything from <record> to </record> if it has a datafield=012
15:06 jl- hmm
15:06 jl- so I ran the file through yaz-marcdumper (again)
15:07 jl- now I'm attempting to load it
15:07 jl- hopefully it will work
15:07 jl- but yeah
15:07 jl- I should bring it into line format
15:07 jl- even tho I did import ~50,000 and they weren't in line format
15:07 jcamins jl-: line format?
15:08 Dyrcona jl-: The sed command that I posted says, delete all lines beginning with the first line matching tag="012" and ending with the next line that matches </datafield> inclusive.
15:08 jcamins Apparently sed works over multiple lines.
15:08 Dyrcona Yes, and what I shared is really simple stuff.
15:09 jl- Dyrcona: ok thx I will try it if this load doesn't work
15:10 pastebot "Dyrcona" at 64.57.241.14 pasted "sed script I used to clean a particular batch of MARC records in XML" (7 lines) at http://paste.evergreen-ils.org/21
15:11 Dyrcona jcamins: There's an example of something a little more complicated to work on multiple lines with sed, but still pretty simple.
15:11 Polonel Anyone know the casue of the error message? http://paste2.org/mZGtk4sf
15:12 jcamins Dyrcona: Ooh, that's very cool!
15:12 jcamins I should learn enough sed to use sed for that sort of thing rather than Perl.
15:14 Dyrcona Polonel: A lot of things, but my guess is a bad or not loaded field mapper. Have you run autogen.sh on the server?
15:14 Polonel yes
15:14 Polonel I followed the steps of upgrading
15:14 Dyrcona But, wait, autogen.sh wouldn't affect the perl.
15:15 bshum https://bugs.launchpad.net/evergreen/+bug/1095307
15:15 pinesol_green Launchpad bug 1095307 in Evergreen "TPAC fails randomly, either has no css style or with Internal Server Error" (affected: 1, heat: 6) [Undecided,Invalid]
15:15 bshum That error gets noted there and paxed seemed to indicate that it was an order of operations step.  Restarting Evergreen, then restarting apache
15:16 jl- Dyrcona: no 009 fields but I think you're right about the 012 those are bad
15:16 phasefx ~troubleshoot
15:16 pinesol_green If you're having trouble getting Evergreen to work, please follow this guide for isolating the problem: http://evergreen-ils.org/dokuwiki/doku.ph​p?id=troubleshooting:checking_for_errors
15:16 Polonel this is not my case. I have made sure eg was running and restarted apache with the same error
15:57 ktomita_ joined #evergreen
16:01 jl- Dyrcona: next obstacle, psql:load_concerto.sql:17: ERROR:  Field sou must have indicators (use ' ' for empty indicators) at /usr/share/perl5/MARC/File/SAX.pm line 104
16:01 jl- CONTEXT:  PL/Perl function "maintain_901"
16:04 jl- jcamins or jeff
16:05 Dyrcona jl-: If you have tags like sou in your file, they aren't valid, either.
16:05 Dyrcona jl-: What ILS did these records come from?
16:05 jl- root@evergreendev:/openils/Evergreen-ILS​-2.5.1/Open-ILS/tests/datasets/sql# grep -c 'tag="sou"' bibs_concerto.sql
16:05 jl- 1
16:06 jl- Dyrcona: to be honest, the more I am working on them the more I think these records are shit.. they are from voyager but it also depends on the librarians how clean
16:06 jl- they handle their records
16:06 Dyrcona jl-: All MARC records are shit.
16:06 jl- I was able to import the first 50000
16:06 jl- :)
16:07 jl- then I was like.. yay.. finally everything clean
16:07 jl- little did I know
16:07 Dyrcona If you modify the sed command that I shared earlier, you can get rid of these tags, too.
16:08 jl- yeah
16:08 jl- appearently there's only 1
16:15 jl- Dyrcona: it's a <controlfield tag=
16:15 jl- can I just remove that control field
16:15 jl- or do I need to remove the entire record
16:15 jl- not sure what it will do to the logic of the record
16:15 Dyrcona just remove the controlfield tag.
16:15 Dyrcona It will improve the logic of the record.
16:16 Dyrcona if you remove it.
16:16 Dyrcona Well.... assuming the record has any logic to begin with.
16:16 jl- heh
16:18 fparks are you around eeevil?
16:22 jl- Dyrcona: hopefully this run will do it, have a great weekend
16:23 Dyrcona One can always hope. :)
16:23 Dyrcona Have a good weekend, yourself, jl-.
17:20 mmorgan left #evergreen
17:45 jcamins jl-: that's the good thing about the script I suggested. Run it, fix things, and repeat until it all works.
18:31 jboyer-UPG joined #evergreen
19:02 hbrennan joined #evergreen
19:05 hbrennan Preparing for upgrade ... Where do report templates save? I want to make sure our custom reports are safe.
19:07 bshum hbrennan: Report templates live in the database.
19:08 bshum Outputs are in generally safe places from upgrades.
19:09 bshum Like erm /openils/var/web/reporter/..
19:12 hbrennan Generally.. haha
19:13 hbrennan We have very few, and I don't even use them, which is why I just made sure MY stuff was backed up (OPAC)
19:13 hbrennan Oh boy... web/reporter.. is a lot of numbers
19:14 bshum hbrennan: Yeah, so those are just the outputs, meaning the actual excel, html, etc. that gets generated for each report.
19:14 bshum Happened to me before where I left those behind on an old server where I 'upgraded' to a new one somewhere else.
19:15 bshum Otherwise, all the templates themselves should be safely stored away in the database.  reporter.templates (or something like that)
19:15 bshum An upgrade shouldn't touch anything in the reporter folder for outputs.
19:15 hbrennan Okie dok. Yeah, this is just a Evergreen upgrade. Same equipment.
19:15 bshum The only thing might be just the definitions changing slightly as things evolve.  Either fixing old broken fields or adding new things.
19:16 bshum But so far as I know, upgrades don't tend to break too many things with our report templates.
19:16 hbrennan That shouldn't be noticed. We're really only done things like counts
19:16 bshum Then you're probably fine on the reports front.
19:17 hbrennan I can't wait until March 19 when I'll finally get an intro to reports
19:17 bshum Just don't be like one of my libraries' directors and try *running* late night reports during an upgrade.
19:17 hbrennan haha, okay
19:17 bshum Which was a fun conversation to be had at 2 am in the morning...
19:17 hbrennan We shut down our server last Friday for a power outage, and didn't notice that the reporter wasn't turned on until today
19:17 hbrennan so we're not frequent users of reports :)
19:17 bshum Ahh, hehe, been there before too :)
19:18 bshum Upgrading is exciting.
19:18 bshum 2.5?
19:18 hbrennan Our first one! Yep, from 2.3.4 to 2.5
19:18 hbrennan Our catalogers will be soooo much happier with the acq
19:18 bshum Sweet.  Happy thoughts for you and your folks then.
19:19 hbrennan Thanks! Next Wednesday is the big day. Was suppose to be this week, but that power outage didn't help things
19:22 jcamins How you know you spend too much time on IRC: when you see "next Wednesday is the big day," your immediate thought is "oh my goodness, congratulations! I had no idea you were getting married!"
19:22 hbrennan haha
19:22 jasonb-iph joined #evergreen
19:22 hbrennan If it hadn't already happened, I'm sure I would have announced it here
19:29 pinesol_green [evergreen|Chris Sharp] Change "Alias" to "Holds Alias" in Patron Search. - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=0ec0ddc>
19:29 pinesol_green [evergreen|Thomas Berezansky] Assume that future checkins are junk input - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=351a827>
19:36 kktaus joined #evergreen
19:41 kktaus hello -- is there a url where one can read about evergreen 508 compliance or evergreen's accessibility in general?
19:41 pinesol_green [evergreen|Mike Rylander] Keep standard INNER joins above possible OUTERs - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=b37a259>
19:41 pinesol_green [evergreen|Bill Erickson] Vandelay: increase queued item creation timeout - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=59a81f5>
19:43 bshum kktaus: Hi there, it's a quiet Friday evening in the US (plus Valentine's Day) so probably not too many folks around right now.
19:44 bshum kktaus: I'm not sure there's a page per say explaining what we have in detail right now.  Could be worth sending a short email question to the general mailing list.
19:44 kktaus bhsum: i understand,
19:44 kktaus nice idea
19:44 hbrennan Still afternoon in Alaska! :)
19:45 kktaus bhsum: thank you!
19:45 bshum hbrennan++ # it's always early somewhere :)
19:46 hbrennan But not for most. I seem to always have my questions at 6:30pm Eastern time
19:52 hbrennan kktaus: There are things mentioned on Launchpad about accessibility. You could probably do a search there and come up with more conversations on it, but here's one: https://bugs.launchpad.net/evergreen/+bug/1155278
19:52 pinesol_green Launchpad bug 1155278 in Evergreen "TPAC: Accessibility audit compliance" (affected: 3, heat: 14) [Undecided,Fix released]
19:55 eeevil and this: https://bugs.launchpad.net/evergreen/+bug/1268636
19:55 pinesol_green Launchpad bug 1268636 in Evergreen "Additional TPAC accessibility (WCAG) improvements" (affected: 1, heat: 6) [Undecided,Fix committed]
19:56 hbrennan Well that's MUCH better than the one I found eeevil++
19:57 eeevil well, i work with berick, so i have an advantage ;)
19:57 eeevil on finding his fine work, anyway
19:58 hbrennan I am still trying to figure out the techniques for searching in Launchpad
19:58 hbrennan it's tricky
19:58 hbrennan How's Round Two of winter?
20:01 bshum Searching in Launchpad is probably a lost cause.
20:01 hbrennan So I've heard.
20:02 hbrennan But it still must be done
20:37 eeevil i use google. prefix with:  site:launchpad.com evergreen
20:38 hbrennan eeevil: good idea. Since moving to the new website, I started using google to search that too... but for some reason I never thought about doing it for launchpad
20:38 hbrennan Thanks! eeevil++
20:39 hbrennan Can't a google search box be integrated rather easily? Or no?
20:39 hbrennan It seems everyone and their mom has one on their blog
20:39 eeevil np. but i have to credit berick for that, too ;)
20:40 hbrennan Okay then, berick++ too
20:53 hbrennan Hey eeevil: since you're so buddy buddy with berick, do you know when the two serial control modules are being merged?
20:54 eeevil heh. you'll have to poke dbwells and senator about that
20:54 hbrennan Ohhhh, that was Dan and Lebbous, not Dan and Bill
20:55 hbrennan woops, real names - oh no!
20:55 hbrennan and I spelled one wrong!
22:17 jasonb-iph joined #evergreen
22:25 jasonb-iph joined #evergreen
22:42 jl- ERROR:  duplicate key value violates unique constraint "asset_call_number_label_once_per_lib"
22:42 jl- DETAIL:  Key (record, owning_lib, label, prefix, suffix)=(1, 4, ML 2533 B34 C1,
22:43 jl- how can I search for this asset call number in my records.xml?
22:43 jl- considering these numbers/letters aren't right next to each other
23:56 mrpeters joined #evergreen
23:58 mrpeters left #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat