Evergreen ILS Website

IRC log for #evergreen, 2017-03-13

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
01:58 NawJo joined #evergreen
02:22 NawJo joined #evergreen
03:22 NawJo joined #evergreen
05:00 pinesol_green News from qatests: Test Success <http://testing.evergreen-ils.org/~live>
07:26 rjackson_isl joined #evergreen
07:33 agoben joined #evergreen
08:41 mmorgan joined #evergreen
08:52 bos20k joined #evergreen
09:08 csharp whew! my DST jetlag is way worse this year for some reason :-/
09:08 rhamby morning folks
09:10 Dyrcona joined #evergreen
09:18 kmlussier joined #evergreen
09:37 rlefaive_ joined #evergreen
09:49 jvwoolf joined #evergreen
10:02 mmorgan1 joined #evergreen
10:06 maryj joined #evergreen
10:10 dbs csharp: right there with you
10:22 kmlussier Good morning #evergreen!
10:23 kmlussier I just talked with gmcharlt, and we're looking at getting any code for the release candidate in by the end of the day tomorrow. I'm hoping to get some attention on a few bugs before the release.
10:24 kmlussier This is mostly a repeat, with a couple of changes, to what I posted in here last week.
10:25 kmlussier I set bug 1522644 as a high priority for the release because of concerns I have with staff being able to see the holds for patrons that they do not have permission to view.
10:25 pinesol_green Launchpad bug 1522644 in Evergreen "webclient: Transfer title holds issues" [Medium,New] https://launchpad.net/bugs/1522644
10:25 kmlussier Nope, that's the wrong bug. Hold on...
10:25 kmlussier bug 1670250
10:25 pinesol_green Launchpad bug 1670250 in Evergreen "Web client: Circ staff users unable to view all checked-out items and other permission issues" [High,New] https://launchpad.net/bugs/1670250
10:26 kmlussier The transfer title holds bug I pasted above is something I've already signed off on, but I added a commit that needs a signoff. The same is true for bug 1621178
10:26 pinesol_green Launchpad bug 1621178 in Evergreen "webclient: Copy status field missing from column pickers" [Medium,Confirmed] https://launchpad.net/bugs/1621178
10:26 kmlussier If somebody could sign off on those commits, we could get those in for the release.
10:27 Bmagic Dyrcona: yes, we are using 16.04, and haven't seen or heard about MARC issues, but that doesn't mean anything really. Maybe I could actively hunt down issues.
10:30 * berick looks at bug 1522644
10:30 pinesol_green Launchpad bug 1522644 in Evergreen "webclient: Transfer title holds issues" [Medium,New] https://launchpad.net/bugs/1522644
10:38 NawJo joined #evergreen
10:41 kmlussier Also, let me know if there is anything you would like me to review to get into this release. At the moment, I'm feeling pretty good about where we are with the release.
10:43 berick kmlussier: is this a new bug.. testing 1522644, I notice when viewing holds, if I click Next or Previous to change records, it does not refresh the holds list to match the new record.
10:43 berick unrelated to 1522644, of course, just noticed it
10:45 * kmlussier checks
10:46 berick i can look, just thought it might sound familiar
10:48 kmlussier berick: Looks like that's an existing bug. I don't think I've seen that one in LP yet. Good catch!
10:48 berick ok, i'll open a bug
10:48 berick thanks
10:49 pinesol_green [evergreen|Victoria Lewis] LP#1522644: webclient: Transfer title holds issues - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=961970f>
10:49 pinesol_green [evergreen|Kathy Lussier] LP#1522644: Make Mark for Hold Transfer option consistent with other options - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=fca751f>
10:49 Dyrcona Bmagic: I've pretty much only noticed issues with exporting records.
10:51 kmlussier berick++
10:59 * jeff ponders classes of copy notes
11:00 jeff unless it's an alert message, it's mostly buried. if there's an existing alert message and you need to add some temporary alerting note, you end up having to edit the current note, etc.
11:00 berick time for asset.copy_standing_penalty /me ducks
11:00 jeff something closer to how standing penalties (even though the "penalty" part doesn't really apply)
11:01 jeff yeah. jinx.
11:01 berick csharp: have you experimented at all with hold targeter --skip-viable?
11:01 dbs Dyrcona: Bmagic: issues with exporting records? I had a record with iso-8859-1 chars in it that blew things up real good, I ended up exporting as UTF8 to XML and then changing the file encoding with vim to avoid corruption
11:02 jeff our missing pieces workflow can collide with existing alert_message and/or the missing pieces note gets left/forgotten once the item is checked back in... if there were a note of type "missing pieces", you could clean it up on (overridden) checkin.
11:02 csharp berick: no, I have not - we're just running it with "--parallel 3" nightly at this point
11:02 Dyrcona dbs: On Perl 5.20 and 5.22, I've seen marc_export and a similar program eat all of the RAM on a system when exporting records as UTF-8 USMARC and MARCXML.
11:03 dbs yikes
11:03 jeff dbs: bug 1671845 has a bit more context.
11:03 pinesol_green Launchpad bug 1671845 in Evergreen "marc_export creating MARC data that yaz-marcdump dislikes" [Medium,Confirmed] https://launchpad.net/bugs/1671845
11:03 Dyrcona I also sent an email to the development list last week about the memory issue.
11:03 jeff dbs: (that bug isn't focused on the memory exhaustion bit that Dyrcona is chasing)
11:04 berick csharp: gotcha.  just curious.
11:04 csharp berick: what would be the ideal use case for that?
11:07 berick csharp: it's meant to act as a repair tool.  the idea would be to run the regular targeter w/ a, say, 24hr retarget interval.  then run a secondary targeter (or bounce between them, whatever) with a more aggressive retarget interval (6, 12 hours) in --skip-viable mode, so that it can fix holds that target bogus copies or have no targets.
11:07 dbs jeff: "healthy appearing" could be the issue, if one of the quotes or accents is actually iso-8859-1, maybe. reproducing with Concerto would be super
11:07 dbs jeff: but yes, those symptoms appear identical to what I was seeing
11:08 berick csharp: a la Matrix Sentinels ;)
11:12 berick csharp: in any event, if all goes as planned, we'll be deploying the new targeter next month.  likely doing a 48-hour regular / 24-hour --skip-viable arrangement.  will let you know how it goes.
11:17 csharp berick: awesome - we were thinking about moving from regular 24 to 48 - maybe following your likely plan would be good for us too
11:18 berick for bug 1670250, I'm kind of surprised we create stock groups with VIEW_USER perms at the system/branch levels.  I was under the impression that such strick view-user perms was not a viable arrangement.
11:18 pinesol_green Launchpad bug 1670250 in Evergreen "Web client: Circ staff users unable to view all checked-out items and other permission issues" [High,New] https://launchpad.net/bugs/1670250
11:18 berick csharp: cool
11:18 csharp the current dilemma is between accommodating the fact that libraries often take > 1 day to pull holds vs. PINES policy that libraries pull all holds daily
11:20 berick csharp: could keep the policy, and split the difference w/ a 36 hour retarget interval.  a little buffer.
11:20 csharp hmm - that's a thought
11:20 berick csharp: so, you just run the targeter once a day?
11:20 csharp right
11:21 berick interesting
11:21 csharp (following miker's advice during our crazy upgrade to PG 9.4)
11:22 berick in that case, you're effectively doing a 24->48 hour window.  assuming midnight run, holds placed at say 2am won't be re-targeted until 46 hours later.
11:25 sandbergja joined #evergreen
11:25 csharp hmm - we'll talk about options
11:26 miker I think, and correct me if I'm wrong csharp, the problem is holds placed at 10am landing after the morning's print of the pull list and getting retargetted at 10am the next day /before/ that day's pull list. with a midnight retargetting, you get an effective average age of, say, 36 hours and skip the problem I described at the front of this long ramble :)
11:27 berick oh yeah, i wasn't knocking it.  just curious how people are doing things
11:27 csharp we're currently running it at 22:45
11:28 csharp miker: and, yeah, the "where did the hold go?" tickets are basically nonexistent since the change
11:28 berick not running the targeter during the normal pull list time seems like a really good idea.
11:29 miker it makes pulling holds a little more predictable
11:32 Bmagic dbs Dyrcona jeff - this is specifically the bash command? Or exporting via vandelay?
11:33 Dyrcona Bmagic: I have no idea about Vandelay, but the problem seems to involve MARC::Record somehow.
11:33 Dyrcona Well, both problems.
11:34 Bmagic alright
11:36 Bmagic I have a handful of custom perl scripts that extract the marc on a regular basis on 16.04. Let me take a look at the one from March
11:41 khuckins__ joined #evergreen
11:44 jeff confirmed symptoms with sample bibs from --load-sample-all. commenting on bug.
12:02 dbs jeff++
12:03 mmorgan joined #evergreen
12:05 mllewellyn joined #evergreen
12:06 brahmina joined #evergreen
12:18 JBoyer joined #evergreen
12:31 dbs jeff: extracting that one record into an XML file and running 'yaz-marcdump -i marcxml -o marc -t marc8 > badrecord.mrc && yaz-marcdump -t utf8 badrecord.mrc' results in no badness, so it doesn't seem to be a problem with the record itself
12:32 * tsbere stares at the eject options on a new windows server and cringes at the fact that the *boot drive* is on the list
12:33 jeff bug 1671845 updated with more comments and sample files
12:33 pinesol_green Launchpad bug 1671845 in Evergreen "marc_export creating MARC data that yaz-marcdump dislikes" [Medium,Confirmed] https://launchpad.net/bugs/1671845
12:33 Dyrcona joined #evergreen
12:40 rfrasur joined #evergreen
12:40 Jillianne joined #evergreen
12:41 rfrasur graced: when you come back, can you ping me here or on FB, if I've gotten disconnected from IRC?
12:44 graced rfrasur: yes'm?
12:46 dbs jeff: hmm, in marc_export "$marc = MARC::Record->new_from_xml($r->marc(), $Marque::config->option_value('encoding'), ..." - would that mean if you're asking for MARC8 output that it's reading the MARCXML from the database (which would always be in UTF8) as MARC8?
12:46 * dbs half-concentrating here
12:47 dbs http://search.cpan.org/~gmcharlt/MARC-XML​-1.0.3/lib/MARC/File/XML.pm#new_from_xml([$encoding,_$format]) suggests its okay, I think
12:51 jeff dbs: from my read of the MARC::Record::XML function's documentation, it's purely a "give me a record with this encoding" thing. While I have encountered "MARC-8 encoded MARCXML" at least once in the wild, it isn't something that I think anyone wants to encourage as being a Thing. :-)
13:14 Dyrcona I'd like to point out that this started being a problem at Perl 5.20 or so.
13:14 Dyrcona I think MARC::Record and/or MARC::Charset need fixes, not marc_export.
13:15 Dyrcona Encode.pm has likely changed on us, again.
13:17 dbs Dyrcona: I'm on ubuntu 14.04 with perl 5.18 fwiw
13:20 Dyrcona I never noticed it on 14.04, but doesn't mean it didn't happen and I was unaware.
13:21 Dyrcona The memory issues start with Perl 5.20. Try a marc_export -a sometime.
13:27 JBoyer Before I waste any more time on ghosts from the past, is there any way to tie a grocery bill to a user or workstation?
13:32 mmorgan JBoyer: Looks like the only thing you get is the billing_location. Unless someone has provided something useful in the note.
13:33 JBoyer Thats what I was thinking / afraid of. Thanks for lending an additional set of eyes.
13:34 Dyrcona I say, "Wishlist it."
13:35 mmorgan What Dyrcona said :)
13:35 Dyrcona It gets asked from time to time.
13:35 JBoyer My wishlist for the money schema starts with lighter fluid and a match. :p It would be good to track some of the existing shortcomings though.
13:42 Dyrcona :)
13:43 jeff JBoyer: can't remember if you were around last week when we were talking about bug 1671150 -- i did eventually realize why you were seeing the behavior you were (unaccent dictionary also needs to be in the search path or explicitly specified)
13:43 pinesol_green Launchpad bug 1671150 in Evergreen "Unqualified references in evergreen.unaccent_and_squash lead to index creation failures with pg_restore" [Undecided,In progress] https://launchpad.net/bugs/1671150 - Assigned to Jeff Godin (jgodin)
13:44 JBoyer I was, and if no one has signed off on your changes yet I'm planning to test them soon and do so. (Last week we had a migration that had me out of the office for a while)
13:44 jeff i'll let you know when i push a branch. ideally later today, we'll see.
13:46 jeff JBoyer: as someone with a decent amount of rows in actor.usr who has likely manually created those indexes a few times now, do you have any opinion on dropping and re-creating them vs trying to create them outside of a transaction and warning to ignore the resulting warning/error output as "normal"?
13:46 jeff if the indexes don't take that long to create, i'm almost leaning toward drop/create.
13:47 JBoyer Were I writing the upgrade script I'd probably lean on a DO block. Running a little anonymous pl/pgsql is a good way to deal with things that may or may not need doing.
13:49 JBoyer If you don't want to do that though, I'd say drop/create to minimize error messages. (only warnings for dropping things that don't exist vs duplicate index errors that can be ignored)
13:54 Dyrcona Hmm. ISO-8859-1 is supposed to be a subset of UTF-8 isn't it? Or, rather UTF-8 a superset of ISO-8859-1?
13:55 jeff no to both.
13:55 jeff they encode ascii characters the same way.
13:55 Dyrcona OK, then. I guess I misunderstood something that I skimmed recently. :)
13:56 Dyrcona Or, maybe I dreamt it/made it up. My brain is bad like that.
13:56 Dyrcona :)
13:57 jeff if you're talking about dec 0 through 127 / hex 0x00 through 0x7f, the encoding is identical between iso-8859-1 / Latin-1 and UTF-8.
13:58 Dyrcona No, I was think of most of the upper nibble characters.
13:58 Dyrcona thinking, even.
13:58 Dyrcona I was just plain wrong.
14:00 * Dyrcona mumbles "Not enough time...not enough time."
14:07 * dbs mumbles along with Dyrcona
14:07 jeff copyright symbol (U+00A9 or &copy; in html entity terms) is single byte 0xa9 in iso-8859-1, but multi byte 0xc2 0xa9 in UTF-8
14:07 Dyrcona Yeah.
14:08 Dyrcona That's the great thing about character sets: there are so many to choose from.
14:08 Dyrcona And, you can mix and match. :)
14:09 Dyrcona Not that you should...but in real life...with real data...
14:09 Dyrcona I love the records that come up "short" because they have some Windows smart quote in a field. Part of the multibyte sequence is a record terminator.
14:10 * jeff nods
14:10 jeff there was a recent patch for MARC::File::XML to try and handle those better.
14:10 jeff i haven't tested to see how yaz tools handle it
14:11 jeff oh, nevermind -- outstanding pull request from tsbere, actually: https://github.com/perl4lib/marc-perl/pull/4
14:12 jeff though there's something else similar that i saw elsewhere... hrm.
14:14 Dyrcona Writing your MARC record splitter in Perl is remarkably simple.
14:14 Dyrcona I keep words... :)
14:14 jeff and this: https://rt.cpan.org/Public​/Bug/Display.html?id=70169
14:14 Dyrcona Anyway, since I'm messing with marc_export stuff lately, I'd like to make some improvements.
14:15 Dyrcona Namely, implement bug 1350916.
14:15 pinesol_green Launchpad bug 1350916 in Evergreen "Use -i in marc_export exclude records with located URIs.should at least include records with located URIs" [Wishlist,Triaged] https://launchpad.net/bugs/1350916
14:16 Dyrcona And, I'd like to make org_unit_descendants aware. (I don't think it is, currently.)
14:16 jeff and this is unrelated but what i was actually picturing in my mind: https://github.com/perl4lib/marc-perl/pull/3
14:16 Dyrcona And, instead or using openils.reporter-store settings it should use openisl.reporter settings by default.
14:16 jeff yeah, i ran into the "--all doesn't mean all" also, figured i might add a wishlist for that.
14:17 Dyrcona I have a local branch for the reporter-store thing.
14:18 jeff er, wait. i'm wrong on the all != all bit.
14:18 Dyrcona The org_unit_descendants thing should be easy.
14:18 Dyrcona --all doesn't mean all when -i is specified.
14:18 Dyrcona I thought that was what you meant.
14:18 jeff i meant that i ran into the "--all --items" gives only records with holdings, which can be far fewer records than you get with "--all"
14:18 Dyrcona Yeah.
14:19 jeff yeah, that's exactly what i meant.
14:19 Dyrcona That's a bit trickier to fix, I think.
14:19 Dyrcona Because of how --items works.
14:20 Dyrcona Well, maybe I'll Lp my two features that don't have bugs while I wait on a dump that is doing too much. :)
14:20 jeff in my case i worked around it with dumping "--all --items" then dumping based on output of a query for NOT deleted AND id > 0 AND acp being null
14:20 Dyrcona I just realized I'm items for more libraries than I need.
14:20 Dyrcona dumping items....
14:20 Dyrcona Yeah, that's crufty, but works.
14:21 Dyrcona For what I'm doing, org_unit_descendants would be very handy, so I may implement that today.
14:27 jeff amusing tidbit, going back to ISO-8859-1 / UTF-8: if you have two pairs of files (each in UTF-8 and ISO-8859 encoding) containing copyright symbol with no newline and copyright symbol with newline, the UTF-8 pair are detected by file as:
14:28 jeff copy.txt:            UTF-8 Unicode text
14:28 jeff copy-bare.txt:       UTF-8 Unicode text, with no line terminators
14:28 jeff first file is three bytes, second is two bytes (no newline)
14:28 jeff on the latin-1 set:
14:28 jeff copy-latin.txt:      ISO-8859 text
14:29 jeff copy-bare-latin.txt: very short file (no magic)
14:29 Dyrcona :)
14:29 jeff because the second file is only one byte, "no magic" :-)
14:29 Dyrcona Yeah.
14:29 Dyrcona Throw a BOM on the front and really freak it out.
14:30 Dyrcona UTF-32-le
14:30 Dyrcona Storage is cheap, right?
14:31 jeff three bytes becomes 12 bytes, sure :-)
14:31 jeff larger than all the other files combined. :-)
14:32 Dyrcona Whee! Typos:  * [new branch]      master -> mater
14:37 JBoyer Whoo-eee, I tell you what, that's a software project, right there.
14:38 JBoyer (Mater jokes may not have that much pull in this channel on further reflection.)
14:38 jeff JBoyer++
14:38 csharp JBoyer++
14:40 dbwells JBoyer: !repyt sdrawkcab tseb s'dlroW  ;)
14:40 csharp dbwells++
14:41 JBoyer dbwells++
14:41 JBoyer :D
14:44 jeff ++sllewbd
14:46 Dyrcona heh
14:47 Dyrcona That's what happens when you try push a branch other than the one you currently have checked out. :)
14:56 rlefaive joined #evergreen
15:03 rlefaive joined #evergreen
15:27 Dyrcona And, I think marc_export spits out waiting for input when it shouldn't but I need to check on that.
15:28 Dyrcona Ah, my bad. I forgot about that with my new feature.
15:29 Dyrcona It was too easy. :)
15:43 kmlussier gmcharlt: While you were gone, I noticed a reingest need that had been missed in the beta release. bug 1671936
15:43 pinesol_green Launchpad bug 1671936 in Evergreen "1006 upgrade script needs reingest instructions" [High,New] https://launchpad.net/bugs/1671936
15:44 kmlussier Well, I'm not sure a discussion between Dyrcona and me necessarily qualifies as consensus, but nobody objected.
15:55 Dyrcona Hm... I noticed I'm getting two 852$b in my exports with -i.
15:56 Dyrcona Sure enough, it's in the code twice. Was that on purpose, I wonder?
15:57 bshum Isn't one for the org unit for the call number vs. copy?
15:57 bshum Certain vendors always hated when we did that... :)
15:57 JBoyer That's how the import format is defined. first is call, second is item. (or vise-versa, but that's the reason.)
15:58 dbs bshum: yes, I removed one of them locally as a result :)
15:58 Dyrcona Well, maybe we should remove on for export.
15:58 Dyrcona s/on/one/
15:58 Dyrcona I don't remember writing it that way, but it was a few years ago. ;)
15:59 Dyrcona Also, maybe someone else added it. I haven't checked.
15:59 Dyrcona My --descendants option seems to be working, though.
15:59 Dyrcona I'll Lp it tomorrow.
16:00 Dyrcona I'm going to try combining the two libraries that I'm testing with. It's supposed to work with more than 1. I want to make sure.
16:02 Dyrcona These are the two libraries that export records for EDS.
16:09 jeff and yes, the 852 has two $b subfields. i left those and added the 999 tags that had been used in the mapping from one library's previous EDS catalog. :-)
16:09 jeff i think i also added logic to remove any existing 999 fields even if not doing an --items export, since their imported bibs still have the legacy 999 at the moment.
16:10 Dyrcona OK.
16:11 Dyrcona Y'know, I don't think there is a way to pipe in ids and get holdings for just certain libraries on those bibs.
16:11 Dyrcona Not sure how common that would be.
16:16 jeff i have that need, but haven't tried to accomplish that with marc_export. our melcat extracts use a different-but-similar process.
16:17 jeff we have to be incremental there, because full extracts are not a routine thing in melcat.
16:18 miker Dyrcona: the two $b's were from the very first days, pre-marc_export, fwiw. blame me :)
16:19 Dyrcona No blame. I just thought it was odd and didn't remember doing it.
16:19 Dyrcona jeff: I added the --since option for incremental updates.
16:19 miker actually, blame PINES ;) ... the format followed as closely as possible their previous output format, with the addition of the 2nd $b to hold the potential difference between owning/circ lib
16:20 Dyrcona But, never needed that pipe with items for a specific location, and realized that when I tried it recently, it wasn't doing what I thought.
16:21 jeff Dyrcona: yeah, but the melcat system has such an aversion to full extracts that we need to do weird things like account for a shelving location changing opac_visible on a copy on a bib as dirtying the export flag for that bib.
16:21 jeff Dyrcona: i don't think marc_export is going to grow those features, or probably shouldn't. :-)
16:21 Dyrcona OK. That's more complicated.
16:21 Dyrcona :)
16:23 Dyrcona Well, I'm satisfied --descendants works for me. I'll put in in production for tomorrow night's monthly export.
16:27 Dyrcona tramp++
17:01 pinesol_green News from qatests: Test Success <http://testing.evergreen-ils.org/~live>
17:07 mmorgan left #evergreen
18:44 mllewellyn left #evergreen
19:08 Jillianne joined #evergreen
19:28 csharp @blame PINES
19:28 pinesol_green csharp: PINES WILL PERISH UNDER MAXIMUM DELETION! DELETE. DELETE. DELETE!
22:26 JBoyer joined #evergreen
23:20 genpaku joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat