Evergreen ILS Website

IRC log for #evergreen, 2017-08-31

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
00:31 pinesol_green [evergreen|Cesar Velez] LP#1695029-Webstaff Fix Patron Registration page never loading - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=688e8e1>
00:31 pinesol_green [evergreen|Bill Erickson] LP#1695029 Patron reg. supports bool opt-in defaults - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=edc503b>
05:00 pinesol_green News from qatests: Test Failure <http://testing.evergreen-ils.org/~live>
06:57 agoben joined #evergreen
07:15 rjackson_isl joined #evergreen
07:55 kmlussier joined #evergreen
07:55 csharp (reading scrollback) nginx may be a factor in my vandelay issue
07:55 * csharp hadn't thought of that
07:56 csharp latest results: I can see the cached object, but after importing a couple of records, the process dies because it can't see the file path anymore
08:16 JBoyer joined #evergreen
08:18 Dyrcona joined #evergreen
08:57 _adb joined #evergreen
08:58 collum joined #evergreen
09:17 yboston joined #evergreen
09:21 * Bmagic waves at yboston
09:50 jvwoolf joined #evergreen
10:29 berick csharp: after importing a couple of records, meaning it fails inside of open-ils.acq.process_upload_records ?  Or does it fail while it's actually uploading the records to WWW/Vandelay?
10:39 berick csharp: fwiw, just created a PO from a pile of new marc records (i.e. using vandelay to import) and had no issues.  using nginx.  master as of yesterday.  single server / no NFS.
10:43 ningalls joined #evergreen
11:38 csharp berick: thanks for the feedback/confirmation - I'll try to get back in the vandelay headspace after lunch and report back
11:46 msh joined #evergreen
11:48 msh Hi everyone! I'm IT support for our library which uses Evergreen. Do you know if it is possible to view the history for everyone who has ever checked out an item?
11:52 Christineb joined #evergreen
11:55 kmlussier msh: When you say view the history, are you talking about seeing it in the staff client, generating a report or something else?
11:56 bshum Since they said "ever", I guess a report or direct database query would probably get the best results.
11:56 bshum Viewing in the staff client, there are generally limitations applied on how many past recent checkout users you can view
11:56 msh kmlussier: I meant in the Staff Client. So, suppose they want to generate a list of everyone who has checked out item XXXX
11:57 msh But it is possible?
11:57 msh To a limited extent, I mean?
11:58 msh To clarify more: I do not actually use the software, I just help maintain the system it runs on. My librarian insists that viewing any kind of check out history for an item is impossible. There have been times where such information would have been very useful.
11:59 kmlussier msh: From item status, you can view recent circulations for a particular item. There is a library setting where you can configure the maximum number of circulations to show there.
12:00 csharp msh: there are also processes that anonymize circulation data, so if those are running (up to the site administrator) that will affect what's available to staff users too
12:01 abneiman msh: from the documentation -- http://docs.evergreen-ils.org/2.11/itemstatus.html -- or a report can be run to give a history.  Keeping in mind, of course, csharp's point about potentially anonymized data.
12:02 kmlussier Oh, wow! Those are my screenshots. I don't even remember doing those.
12:02 csharp msh: also (outside of the tech focus of this channel) ethical/privacy concerns should be honored too, of course
12:02 abneiman kmlussier++ #creative patron names
12:02 kmlussier abneiman: That's how I recognized them. :)
12:03 bshum abneiman++ # finding good docs
12:04 msh Thank you all! I'm amazed at how many helpful responses I received.
12:04 abneiman msh: csharp has another good point about privacy concerns.  Some state laws are also very strict about use of library circulation data, so, proceed with caution.
12:05 msh abneiman: Understood. I'm thinking that that may be the reason why my librarian can't get this information. We are a school so there are probably laws about this data.
12:05 jihpringle joined #evergreen
12:10 kmlussier miker / gmcharlt: Would either of you have tuits to rebase bug 1676608 against current master?
12:10 pinesol_green Launchpad bug 1676608 in Evergreen "Copy Alert Persistence and Suppression Matrix" [Wishlist,New] https://launchpad.net/bugs/1676608
12:10 kmlussier I should have time to test it this afternoon. There were conflicts in a lot of files, which made me nervous about resolving them myself.
12:11 Dyrcona msh: You can't really get it from the staff client. You can really only see the last two or so who have checked out an item.
12:11 miker kmlussier: I can take a look, yes
12:11 kmlussier miker++
12:11 Dyrcona msh: It may be possible to generate a report, but I'm not a reports guru.
12:11 kmlussier Dyrcona: You can see more if the OU setting is set high enough. There are two tabs. One only shows two, the other shows more.
12:12 Dyrcona kmlussier: I've never seen that setting be very high, and if you set it to high, the client will timeout retrieving the data.
12:13 bshum And also you'd probably be dealing with the ability of a single OU to view more circs for any item across the organization, not just the ones that belong to them.
12:13 bshum So probably not good for that privacy stuff
12:14 bshum Report sounds safer to me
12:14 bshum (relatively)
12:15 Dyrcona Sounds like a report is wanted. Because that item status interface is only meant to show the last handful for purposes of seeing who might have damaged the item or similar.
13:01 csharp berick: huh - my file upload from this morning *did* in fact work - though it took nearly 30 mins to process
13:01 csharp (more arguments for async vandelay)
13:02 csharp 53 lineitems with 1 copy each
13:02 berick that ain't right
13:03 csharp yeah
13:03 berick my cruddy vm did 10 LIs with 3 copies each in a few seconds
13:05 berick about 10 seconds for import and po activation combined
13:06 Dyrcona VMs are faster than real hardware, I find. Does it all in RAM if it fits.
13:06 berick guessing my small DB has something to do with that too, though
13:07 berick but still, 30 mins is not OK
13:07 Dyrcona Well, and nothing else going on, yeah. :)
13:07 berick that too
13:07 csharp nothing much going on on my test server at 6 a.m. either ;-)
13:08 Dyrcona Test servers are just slow.... Witness my circulation aging attempts. :)
13:08 Dyrcona Some kind of unwritten law of computer science. :)
13:08 csharp well, I haven't looked lately, but I know we have complaints about slow loading in acq vandelay in general
13:08 csharp but I agree that 30 mins is kinda crazy
13:09 Dyrcona Well, loading bibs is not fast, even if you circumvent vandelay and do it in the database.
13:09 kmlussier Yikes! 30 mins!
13:09 Dyrcona Some god-awful number of complex triggers.
13:09 csharp honestly "this might take 30 mins" wasn't on my radar as an explanation for "this appears not to be working at all"
13:09 berick no, 30 mins == broken
13:10 Dyrcona For 53 records, yeah, 30 minutes is broken.
13:10 Dyrcona Should only take 1 to 2 minutes straight into the db.
13:10 berick so there's your explanation, csharp.  it's just broken!
13:12 Dyrcona On our old db hardware it would take about 1 to 2 seconds for all the triggers to do their thing.
13:12 Dyrcona I added a timing option to one of my load scripts to spit out how long it took to process each record.
13:13 * csharp runs off to email pines libs that it's broken; rides off into sunset
13:14 berick i hear the bahamas are nice
13:14 Dyrcona hurricane season....
13:14 csharp ooh - that sounds way better than staying in town and upgrading
13:14 Dyrcona It does, though.
13:15 Dyrcona They have Internet in the Bahamas, don't they? ;)
13:20 csharp http://evergreen-ils.org/~​csharp/vandelay-upload.log - for anyone interested
13:21 csharp is "Message processing duration" in seconds or ms?
13:22 csharp oh I guess seconds
13:22 berick yeah, seconds
13:22 berick yeah, creating those queued bib recs is slow
13:23 Dyrcona TBH, I eschew Vandelay. I've always considered it to be too slow.
13:23 * csharp would love to not bother with it but our acq libs need it
13:24 berick csharp: maybe time to explain/analyze one of the those queued_bib_record INSERTs
13:25 berick updates taking as long too.  that might be easier to test.
13:27 Dyrcona Yeah....
13:29 Dyrcona Probably some trigger is taking way too long.
13:30 berick look for the trigger that calls pg_sleep(16)
13:30 Dyrcona heh
13:30 csharp yeah, definitely taking some time in the db
13:30 csharp duration: 18791.423 ms, duration: 16223.722 ms, etc.
13:34 Dyrcona These are tiny records, too, with no 856s.
13:35 Dyrcona Is this creating copies at the same time?
13:35 csharp yes
13:35 berick well, acq copies, not vandelay copies
13:35 berick so, not relevant to the insert
13:36 ohiojoe joined #evergreen
13:36 csharp Trigger zz_match_bibs_trigger: time=19216.105 calls=1
13:36 Dyrcona Right.
13:39 berick wonder how many rows are inserted into vandelay.bib_match for each of these records.
13:40 Dyrcona Also, how long does vandelay.measure_record_quality take? Looks like it could be run twice.
13:41 Dyrcona Ah. I see why it's called twice. Different records.
13:44 miker kmlussier: ok, I have a rebased branch that I think is fully resolved.  However, if testing goes well, I think a good bit of squashing is in order. pushing the branch now....
13:46 miker kmlussier: http://git.evergreen-ils.org/?p=working/Everg​reen.git;a=shortlog;h=refs/heads/collab/miker​/lp1676608_copy_alerts-rebased-again-20170831
13:50 csharp FOR test_result IN SELECT * FROM
13:50 csharp vandelay.match_set_test_marcxml(match_set, NEW.marc, match_bucket) LOOP
13:50 csharp ^^ appears to be the bottleneck
13:51 csharp gonna figure out what's going on there
13:51 berick csharp: see how many matches are getting created.  if it's a very broad match, it could be inserting a thousand match rows
13:52 berick select count(*) from vandelay.bib_match where queued_record = vl_rec_id
13:52 csharp berick: from what I can tell, it only looped once
13:52 kmlussier Thanks miker!
13:53 berick csharp: ok, good, that's an easy one to rule out.
13:53 csharp yeah, it only entered the loop one time
13:54 csharp RAISE_NOTICE++
13:59 annagoben joined #evergreen
14:03 csharp yeah, the select query constructed by vandelay.match_set_test_marcxml is definitely where it's slowing
14:03 csharp now to try and piece it together manually for EXPLAIN-ing
14:04 csharp apparently adding my log file to lupin pushed it over 80% disk usage :-0
14:06 bshum @blame csharp
14:06 pinesol_green bshum: csharp is NOT CONNECTED TO THE NETWORK!!!
14:09 csharp definitely true of me today ;-)
14:09 Dyrcona ;)
14:09 Dyrcona @blame Dyrcona
14:09 pinesol_green Dyrcona: Dyrcona is NOT CONNECTED TO THE NETWORK!!!
14:09 Dyrcona hm.....
14:09 Dyrcona @blame Ice cream
14:09 pinesol_green Dyrcona: Ice cream stole bradl's tux doll!
14:10 Dyrcona And, now I want some ice cream.
14:12 bshum I had ice cream yesterday.  Home made mint chocolate chip.
14:12 bshum Or wait, maybe it was the day before, hmm
14:12 bshum What day is today?
14:14 Dyrcona heh
14:15 * bshum is ready for his weekend
14:20 * csharp is upgrading PINES this weekend, so no weekend
14:20 csharp though hoping to move holiday to next Friday, $DEITY willing
14:22 csharp resulting query: https://pastebin.com/XwZSSSkn
14:23 csharp and that's like basically impossible to test :-/
14:23 kmlussier Ice cream should never be blamed for anything. It is only a force for good.
14:23 kmlussier @praise ice cream
14:23 * pinesol_green I come to praise ice cream, not to bury them.
14:23 csharp @blame ice cream for mah belly
14:23 pinesol_green csharp: ice cream forgot to give the gerbils their chocolate-frosted sugar bombs for mah belly
14:24 csharp actually, in my case it's probably pizza and nachos
14:24 kmlussier csharp: See! It's never the ice cream.
14:24 * kmlussier is hungry now.
14:25 * csharp ponders what "$2" is in that query
14:25 Bmagic Anyone know which database table the "Work Log" uses?
14:26 csharp Bmagic: I think that's stored per workstation
14:26 csharp like locally
14:26 Bmagic ah, that might explain some things
14:27 Dyrcona Yeah, it's stored locally.
14:27 csharp ok FOR rec IN EXECUTE query_ USING tags_rstore, svf_rstore LOOP
14:28 csharp so tags_rstore and svf_rstore
14:28 Dyrcona csharp: $2 would generally be the second argument to the function that runs that query.
14:29 csharp so I guess record_xml
14:34 csharp I think it's actually svf_rstore, which contains vandelay.extract_rec_attrs(record_xml);
14:34 csharp jeez
14:39 csharp got it - those variables contain hashes of data and I can see them when I raise a notice
14:39 csharp svf_rstore is a massive wall of text, but I can see bib_level and item_type, so all's good
14:43 csharp explain analyze: https://pastebin.com/NwMBfEY4
14:51 * csharp stares at metabib.record_attr_flat
14:51 berick csharp: what's in the vandelay.match_set_point rows for the match_set in question?
14:51 berick mostly just curious, but could be helpful
14:53 csharp berick: sorry - my eyeballs are rolling around in my head - trying to find where to get what you're asking for :-)
14:54 csharp ok got it
14:54 csharp just a sec
14:55 csharp berick: https://pastebin.com/9B88HWxF
14:55 miker csharp: so, first, the query constructor is not being very smart when some of the mfr matches are to NULL (those joins should be dropped)... and second, vandelay's using metabib.record_attr_flat, which is a view over a view over a view over ... etc ... over the new-ish attr_vector stuff, and very slow. so, I think import matching is just going to need a rewrite sooner rather than later. which is not an answer that helps you, of course :(
14:56 csharp miker: well it does help me not keep chasing my tail ;-)
14:56 csharp miker: thanks
14:56 miker I suppose there is that :)
14:56 berick miker++
14:57 * csharp sets helpdesk ticket status to "Deal With It" and closes it
14:57 csharp probably worth a bug report
14:57 miker bug++
15:06 bshum bug-- # giving bugs karma seems dangerous.  Don't feed the gremlins...
15:08 kmlussier bug_reporting++
15:08 tsbere I dunno. If a bug gets enough positive karma does it evolve into a feature?
15:08 berick @karma bug
15:08 pinesol_green berick: Karma for "bug" has been increased 1 time and decreased 1 time for a total karma of 0.
15:09 bshum tsbere++ # that was funny :)
15:09 kmlussier Ha ha! tsbere++
15:09 bshum @quote add <tsbere> "I dunno. If a bug gets enough positive karma does it evolve into a feature?"
15:09 pinesol_green bshum: The operation succeeded.  Quote #173 added.
15:11 Dyrcona One person's feature is another person's bug. ;)
15:11 Dyrcona And, yeah, I've seen bug reports along those lines: Feature X is broken because it doesn't do Y. Feature X was deliberately designed not to do Y.
16:00 Bmagic I am chasing down an checkin issue. The copy is on the holds shelf, but is being checked in with the modifier clear holds shelf. Following the opensrf logs:  the copy is attempted to fill 50 holds but cant because of ABHP. The final step is:
16:00 Bmagic circulator: copy is on-holds-shelf, but there is no hold - reshelving
16:01 Bmagic editor[1|234234] asset.copy.update ....... - request en-US open-ils.cstore.direct.asset.copy.update 123123 - Returning method exception with message: An unknown server error occurred
16:01 berick to the cstore / sql logs!
16:02 Bmagic editor[1|234234] request error open-ils.cstore.direct.asset.copy.update : 123123 : Exception: OpenSRF::DomainObject::oilsMethodException 2017-08-31T14:44:55 OpenILS::Utils::CStoreEditor /usr/local/share/perl/5.22.1/Op​enILS/Utils/CStoreEditor.pm:465 <400>  No active transaction -- required for UPDATE
16:02 Bmagic postgres didn't have an error
16:02 berick ah
16:02 berick timeout maybe?
16:03 Bmagic checking cstore log, didnt check that yet
16:03 Bmagic nothing
16:06 Bmagic unless this counts
16:06 Bmagic DBIx::ContextualFetch::db=H​ASH(0xa06d2f0)->disconnect invalidates 1 active statement handle (either destroy statement handles or call finish on them before disconnecting) at /usr/local/share/perl/5.22.1/OpenILS/Ap​plication/Storage/Driver/Pg/storage.pm line 11.
16:08 Bmagic berick: a timeout on the staff client? I am waiting awhile during this checkin
16:08 Bmagic due to all of the holds
16:08 berick Bmagic: one thing to look for are API calls to not-cstore (usually storage) that take longer than the cstore drone keepalive timeout (usually 6 seconds).
16:09 berick for example, if the API call to open-ils.storage.action.hold​_request.nearest_hold.atomic took longer than 6 seconds, the open transaction w/ cstore will timeout
16:09 Bmagic yeah, it was more than 6 seconds for sure
16:10 Bmagic ok, so, is there a solution? Perhaps your hold targeter improvements could address this? We were talking about this issue at last year's hack-away - ABHP causing checking to hang because of the sort order of the fillable holds
16:11 berick ABHP?
16:12 Bmagic age based hold protection
16:13 berick one solution is to port open-ils.storage.action.hold​_request.nearest_hold.atomic to a Circ.pm (or whatever) utility function that uses cstore under the covers.  that would bypass the cstore drone timeout.
16:14 Bmagic alright, so it's offically a bug then?
16:14 berick and no the new hold targeter has no effect here.  this is a separate API call that only exists in open-ils.storage today.
16:14 berick Bmagic: didn't you code some sorting improvements?
16:14 Bmagic I didn't
16:14 Bmagic We found the lines of code though :)
16:15 berick ah
16:16 berick i think the sorting improvements would be a bonus regardless
16:16 berick maybe start there?
16:21 Bmagic Sounds good. I think we figured out that the query to take into account the age based hold protection would be expensive, but I will take a look at it again
16:30 berick oh, i don't remember that part
16:32 sandbergja joined #evergreen
16:49 Christineb joined #evergreen
17:00 pinesol_green News from qatests: Test Failure <http://testing.evergreen-ils.org/~live>
17:22 pinesol_green [evergreen|blake] LP1655158 Patron Search by Date of Birth - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=baf337b>
17:42 jvwoolf left #evergreen
20:22 Jillianne joined #evergreen
23:03 gmcharlt hmm, I think that test failure is basically a problem with the test itself
23:05 gmcharlt besides dropping that static redefinition of unapi.mmr_mra, there isn't a reliable ordering imposed on source_list, nor does it seems that the specific order matters

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat