| Time |
Nick |
Message |
| 00:31 |
pinesol_green |
[evergreen|Cesar Velez] LP#1695029-Webstaff Fix Patron Registration page never loading - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=688e8e1> |
| 00:31 |
pinesol_green |
[evergreen|Bill Erickson] LP#1695029 Patron reg. supports bool opt-in defaults - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=edc503b> |
| 05:00 |
pinesol_green |
News from qatests: Test Failure <http://testing.evergreen-ils.org/~live> |
| 06:57 |
|
agoben joined #evergreen |
| 07:15 |
|
rjackson_isl joined #evergreen |
| 07:55 |
|
kmlussier joined #evergreen |
| 07:55 |
csharp |
(reading scrollback) nginx may be a factor in my vandelay issue |
| 07:55 |
* csharp |
hadn't thought of that |
| 07:56 |
csharp |
latest results: I can see the cached object, but after importing a couple of records, the process dies because it can't see the file path anymore |
| 08:16 |
|
JBoyer joined #evergreen |
| 08:18 |
|
Dyrcona joined #evergreen |
| 08:57 |
|
_adb joined #evergreen |
| 08:58 |
|
collum joined #evergreen |
| 09:17 |
|
yboston joined #evergreen |
| 09:21 |
* Bmagic |
waves at yboston |
| 09:50 |
|
jvwoolf joined #evergreen |
| 10:29 |
berick |
csharp: after importing a couple of records, meaning it fails inside of open-ils.acq.process_upload_records ? Or does it fail while it's actually uploading the records to WWW/Vandelay? |
| 10:39 |
berick |
csharp: fwiw, just created a PO from a pile of new marc records (i.e. using vandelay to import) and had no issues. using nginx. master as of yesterday. single server / no NFS. |
| 10:43 |
|
ningalls joined #evergreen |
| 11:38 |
csharp |
berick: thanks for the feedback/confirmation - I'll try to get back in the vandelay headspace after lunch and report back |
| 11:46 |
|
msh joined #evergreen |
| 11:48 |
msh |
Hi everyone! I'm IT support for our library which uses Evergreen. Do you know if it is possible to view the history for everyone who has ever checked out an item? |
| 11:52 |
|
Christineb joined #evergreen |
| 11:55 |
kmlussier |
msh: When you say view the history, are you talking about seeing it in the staff client, generating a report or something else? |
| 11:56 |
bshum |
Since they said "ever", I guess a report or direct database query would probably get the best results. |
| 11:56 |
bshum |
Viewing in the staff client, there are generally limitations applied on how many past recent checkout users you can view |
| 11:56 |
msh |
kmlussier: I meant in the Staff Client. So, suppose they want to generate a list of everyone who has checked out item XXXX |
| 11:57 |
msh |
But it is possible? |
| 11:57 |
msh |
To a limited extent, I mean? |
| 11:58 |
msh |
To clarify more: I do not actually use the software, I just help maintain the system it runs on. My librarian insists that viewing any kind of check out history for an item is impossible. There have been times where such information would have been very useful. |
| 11:59 |
kmlussier |
msh: From item status, you can view recent circulations for a particular item. There is a library setting where you can configure the maximum number of circulations to show there. |
| 12:00 |
csharp |
msh: there are also processes that anonymize circulation data, so if those are running (up to the site administrator) that will affect what's available to staff users too |
| 12:01 |
abneiman |
msh: from the documentation -- http://docs.evergreen-ils.org/2.11/itemstatus.html -- or a report can be run to give a history. Keeping in mind, of course, csharp's point about potentially anonymized data. |
| 12:02 |
kmlussier |
Oh, wow! Those are my screenshots. I don't even remember doing those. |
| 12:02 |
csharp |
msh: also (outside of the tech focus of this channel) ethical/privacy concerns should be honored too, of course |
| 12:02 |
abneiman |
kmlussier++ #creative patron names |
| 12:02 |
kmlussier |
abneiman: That's how I recognized them. :) |
| 12:03 |
bshum |
abneiman++ # finding good docs |
| 12:04 |
msh |
Thank you all! I'm amazed at how many helpful responses I received. |
| 12:04 |
abneiman |
msh: csharp has another good point about privacy concerns. Some state laws are also very strict about use of library circulation data, so, proceed with caution. |
| 12:05 |
msh |
abneiman: Understood. I'm thinking that that may be the reason why my librarian can't get this information. We are a school so there are probably laws about this data. |
| 12:05 |
|
jihpringle joined #evergreen |
| 12:10 |
kmlussier |
miker / gmcharlt: Would either of you have tuits to rebase bug 1676608 against current master? |
| 12:10 |
pinesol_green |
Launchpad bug 1676608 in Evergreen "Copy Alert Persistence and Suppression Matrix" [Wishlist,New] https://launchpad.net/bugs/1676608 |
| 12:10 |
kmlussier |
I should have time to test it this afternoon. There were conflicts in a lot of files, which made me nervous about resolving them myself. |
| 12:11 |
Dyrcona |
msh: You can't really get it from the staff client. You can really only see the last two or so who have checked out an item. |
| 12:11 |
miker |
kmlussier: I can take a look, yes |
| 12:11 |
kmlussier |
miker++ |
| 12:11 |
Dyrcona |
msh: It may be possible to generate a report, but I'm not a reports guru. |
| 12:11 |
kmlussier |
Dyrcona: You can see more if the OU setting is set high enough. There are two tabs. One only shows two, the other shows more. |
| 12:12 |
Dyrcona |
kmlussier: I've never seen that setting be very high, and if you set it to high, the client will timeout retrieving the data. |
| 12:13 |
bshum |
And also you'd probably be dealing with the ability of a single OU to view more circs for any item across the organization, not just the ones that belong to them. |
| 12:13 |
bshum |
So probably not good for that privacy stuff |
| 12:14 |
bshum |
Report sounds safer to me |
| 12:14 |
bshum |
(relatively) |
| 12:15 |
Dyrcona |
Sounds like a report is wanted. Because that item status interface is only meant to show the last handful for purposes of seeing who might have damaged the item or similar. |
| 13:01 |
csharp |
berick: huh - my file upload from this morning *did* in fact work - though it took nearly 30 mins to process |
| 13:01 |
csharp |
(more arguments for async vandelay) |
| 13:02 |
csharp |
53 lineitems with 1 copy each |
| 13:02 |
berick |
that ain't right |
| 13:03 |
csharp |
yeah |
| 13:03 |
berick |
my cruddy vm did 10 LIs with 3 copies each in a few seconds |
| 13:05 |
berick |
about 10 seconds for import and po activation combined |
| 13:06 |
Dyrcona |
VMs are faster than real hardware, I find. Does it all in RAM if it fits. |
| 13:06 |
berick |
guessing my small DB has something to do with that too, though |
| 13:07 |
berick |
but still, 30 mins is not OK |
| 13:07 |
Dyrcona |
Well, and nothing else going on, yeah. :) |
| 13:07 |
berick |
that too |
| 13:07 |
csharp |
nothing much going on on my test server at 6 a.m. either ;-) |
| 13:08 |
Dyrcona |
Test servers are just slow.... Witness my circulation aging attempts. :) |
| 13:08 |
Dyrcona |
Some kind of unwritten law of computer science. :) |
| 13:08 |
csharp |
well, I haven't looked lately, but I know we have complaints about slow loading in acq vandelay in general |
| 13:08 |
csharp |
but I agree that 30 mins is kinda crazy |
| 13:09 |
Dyrcona |
Well, loading bibs is not fast, even if you circumvent vandelay and do it in the database. |
| 13:09 |
kmlussier |
Yikes! 30 mins! |
| 13:09 |
Dyrcona |
Some god-awful number of complex triggers. |
| 13:09 |
csharp |
honestly "this might take 30 mins" wasn't on my radar as an explanation for "this appears not to be working at all" |
| 13:09 |
berick |
no, 30 mins == broken |
| 13:10 |
Dyrcona |
For 53 records, yeah, 30 minutes is broken. |
| 13:10 |
Dyrcona |
Should only take 1 to 2 minutes straight into the db. |
| 13:10 |
berick |
so there's your explanation, csharp. it's just broken! |
| 13:12 |
Dyrcona |
On our old db hardware it would take about 1 to 2 seconds for all the triggers to do their thing. |
| 13:12 |
Dyrcona |
I added a timing option to one of my load scripts to spit out how long it took to process each record. |
| 13:13 |
* csharp |
runs off to email pines libs that it's broken; rides off into sunset |
| 13:14 |
berick |
i hear the bahamas are nice |
| 13:14 |
Dyrcona |
hurricane season.... |
| 13:14 |
csharp |
ooh - that sounds way better than staying in town and upgrading |
| 13:14 |
Dyrcona |
It does, though. |
| 13:15 |
Dyrcona |
They have Internet in the Bahamas, don't they? ;) |
| 13:20 |
csharp |
http://evergreen-ils.org/~csharp/vandelay-upload.log - for anyone interested |
| 13:21 |
csharp |
is "Message processing duration" in seconds or ms? |
| 13:22 |
csharp |
oh I guess seconds |
| 13:22 |
berick |
yeah, seconds |
| 13:22 |
berick |
yeah, creating those queued bib recs is slow |
| 13:23 |
Dyrcona |
TBH, I eschew Vandelay. I've always considered it to be too slow. |
| 13:23 |
* csharp |
would love to not bother with it but our acq libs need it |
| 13:24 |
berick |
csharp: maybe time to explain/analyze one of the those queued_bib_record INSERTs |
| 13:25 |
berick |
updates taking as long too. that might be easier to test. |
| 13:27 |
Dyrcona |
Yeah.... |
| 13:29 |
Dyrcona |
Probably some trigger is taking way too long. |
| 13:30 |
berick |
look for the trigger that calls pg_sleep(16) |
| 13:30 |
Dyrcona |
heh |
| 13:30 |
csharp |
yeah, definitely taking some time in the db |
| 13:30 |
csharp |
duration: 18791.423 ms, duration: 16223.722 ms, etc. |
| 13:34 |
Dyrcona |
These are tiny records, too, with no 856s. |
| 13:35 |
Dyrcona |
Is this creating copies at the same time? |
| 13:35 |
csharp |
yes |
| 13:35 |
berick |
well, acq copies, not vandelay copies |
| 13:35 |
berick |
so, not relevant to the insert |
| 13:36 |
|
ohiojoe joined #evergreen |
| 13:36 |
csharp |
Trigger zz_match_bibs_trigger: time=19216.105 calls=1 |
| 13:36 |
Dyrcona |
Right. |
| 13:39 |
berick |
wonder how many rows are inserted into vandelay.bib_match for each of these records. |
| 13:40 |
Dyrcona |
Also, how long does vandelay.measure_record_quality take? Looks like it could be run twice. |
| 13:41 |
Dyrcona |
Ah. I see why it's called twice. Different records. |
| 13:44 |
miker |
kmlussier: ok, I have a rebased branch that I think is fully resolved. However, if testing goes well, I think a good bit of squashing is in order. pushing the branch now.... |
| 13:46 |
miker |
kmlussier: http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/miker/lp1676608_copy_alerts-rebased-again-20170831 |
| 13:50 |
csharp |
FOR test_result IN SELECT * FROM |
| 13:50 |
csharp |
vandelay.match_set_test_marcxml(match_set, NEW.marc, match_bucket) LOOP |
| 13:50 |
csharp |
^^ appears to be the bottleneck |
| 13:51 |
csharp |
gonna figure out what's going on there |
| 13:51 |
berick |
csharp: see how many matches are getting created. if it's a very broad match, it could be inserting a thousand match rows |
| 13:52 |
berick |
select count(*) from vandelay.bib_match where queued_record = vl_rec_id |
| 13:52 |
csharp |
berick: from what I can tell, it only looped once |
| 13:52 |
kmlussier |
Thanks miker! |
| 13:53 |
berick |
csharp: ok, good, that's an easy one to rule out. |
| 13:53 |
csharp |
yeah, it only entered the loop one time |
| 13:54 |
csharp |
RAISE_NOTICE++ |
| 13:59 |
|
annagoben joined #evergreen |
| 14:03 |
csharp |
yeah, the select query constructed by vandelay.match_set_test_marcxml is definitely where it's slowing |
| 14:03 |
csharp |
now to try and piece it together manually for EXPLAIN-ing |
| 14:04 |
csharp |
apparently adding my log file to lupin pushed it over 80% disk usage :-0 |
| 14:06 |
bshum |
@blame csharp |
| 14:06 |
pinesol_green |
bshum: csharp is NOT CONNECTED TO THE NETWORK!!! |
| 14:09 |
csharp |
definitely true of me today ;-) |
| 14:09 |
Dyrcona |
;) |
| 14:09 |
Dyrcona |
@blame Dyrcona |
| 14:09 |
pinesol_green |
Dyrcona: Dyrcona is NOT CONNECTED TO THE NETWORK!!! |
| 14:09 |
Dyrcona |
hm..... |
| 14:09 |
Dyrcona |
@blame Ice cream |
| 14:09 |
pinesol_green |
Dyrcona: Ice cream stole bradl's tux doll! |
| 14:10 |
Dyrcona |
And, now I want some ice cream. |
| 14:12 |
bshum |
I had ice cream yesterday. Home made mint chocolate chip. |
| 14:12 |
bshum |
Or wait, maybe it was the day before, hmm |
| 14:12 |
bshum |
What day is today? |
| 14:14 |
Dyrcona |
heh |
| 14:15 |
* bshum |
is ready for his weekend |
| 14:20 |
* csharp |
is upgrading PINES this weekend, so no weekend |
| 14:20 |
csharp |
though hoping to move holiday to next Friday, $DEITY willing |
| 14:22 |
csharp |
resulting query: https://pastebin.com/XwZSSSkn |
| 14:23 |
csharp |
and that's like basically impossible to test :-/ |
| 14:23 |
kmlussier |
Ice cream should never be blamed for anything. It is only a force for good. |
| 14:23 |
kmlussier |
@praise ice cream |
| 14:23 |
* pinesol_green |
I come to praise ice cream, not to bury them. |
| 14:23 |
csharp |
@blame ice cream for mah belly |
| 14:23 |
pinesol_green |
csharp: ice cream forgot to give the gerbils their chocolate-frosted sugar bombs for mah belly |
| 14:24 |
csharp |
actually, in my case it's probably pizza and nachos |
| 14:24 |
kmlussier |
csharp: See! It's never the ice cream. |
| 14:24 |
* kmlussier |
is hungry now. |
| 14:25 |
* csharp |
ponders what "$2" is in that query |
| 14:25 |
Bmagic |
Anyone know which database table the "Work Log" uses? |
| 14:26 |
csharp |
Bmagic: I think that's stored per workstation |
| 14:26 |
csharp |
like locally |
| 14:26 |
Bmagic |
ah, that might explain some things |
| 14:27 |
Dyrcona |
Yeah, it's stored locally. |
| 14:27 |
csharp |
ok FOR rec IN EXECUTE query_ USING tags_rstore, svf_rstore LOOP |
| 14:28 |
csharp |
so tags_rstore and svf_rstore |
| 14:28 |
Dyrcona |
csharp: $2 would generally be the second argument to the function that runs that query. |
| 14:29 |
csharp |
so I guess record_xml |
| 14:34 |
csharp |
I think it's actually svf_rstore, which contains vandelay.extract_rec_attrs(record_xml); |
| 14:34 |
csharp |
jeez |
| 14:39 |
csharp |
got it - those variables contain hashes of data and I can see them when I raise a notice |
| 14:39 |
csharp |
svf_rstore is a massive wall of text, but I can see bib_level and item_type, so all's good |
| 14:43 |
csharp |
explain analyze: https://pastebin.com/NwMBfEY4 |
| 14:51 |
* csharp |
stares at metabib.record_attr_flat |
| 14:51 |
berick |
csharp: what's in the vandelay.match_set_point rows for the match_set in question? |
| 14:51 |
berick |
mostly just curious, but could be helpful |
| 14:53 |
csharp |
berick: sorry - my eyeballs are rolling around in my head - trying to find where to get what you're asking for :-) |
| 14:54 |
csharp |
ok got it |
| 14:54 |
csharp |
just a sec |
| 14:55 |
csharp |
berick: https://pastebin.com/9B88HWxF |
| 14:55 |
miker |
csharp: so, first, the query constructor is not being very smart when some of the mfr matches are to NULL (those joins should be dropped)... and second, vandelay's using metabib.record_attr_flat, which is a view over a view over a view over ... etc ... over the new-ish attr_vector stuff, and very slow. so, I think import matching is just going to need a rewrite sooner rather than later. which is not an answer that helps you, of course :( |
| 14:56 |
csharp |
miker: well it does help me not keep chasing my tail ;-) |
| 14:56 |
csharp |
miker: thanks |
| 14:56 |
miker |
I suppose there is that :) |
| 14:56 |
berick |
miker++ |
| 14:57 |
* csharp |
sets helpdesk ticket status to "Deal With It" and closes it |
| 14:57 |
csharp |
probably worth a bug report |
| 14:57 |
miker |
bug++ |
| 15:06 |
bshum |
bug-- # giving bugs karma seems dangerous. Don't feed the gremlins... |
| 15:08 |
kmlussier |
bug_reporting++ |
| 15:08 |
tsbere |
I dunno. If a bug gets enough positive karma does it evolve into a feature? |
| 15:08 |
berick |
@karma bug |
| 15:08 |
pinesol_green |
berick: Karma for "bug" has been increased 1 time and decreased 1 time for a total karma of 0. |
| 15:09 |
bshum |
tsbere++ # that was funny :) |
| 15:09 |
kmlussier |
Ha ha! tsbere++ |
| 15:09 |
bshum |
@quote add <tsbere> "I dunno. If a bug gets enough positive karma does it evolve into a feature?" |
| 15:09 |
pinesol_green |
bshum: The operation succeeded. Quote #173 added. |
| 15:11 |
Dyrcona |
One person's feature is another person's bug. ;) |
| 15:11 |
Dyrcona |
And, yeah, I've seen bug reports along those lines: Feature X is broken because it doesn't do Y. Feature X was deliberately designed not to do Y. |
| 16:00 |
Bmagic |
I am chasing down an checkin issue. The copy is on the holds shelf, but is being checked in with the modifier clear holds shelf. Following the opensrf logs: the copy is attempted to fill 50 holds but cant because of ABHP. The final step is: |
| 16:00 |
Bmagic |
circulator: copy is on-holds-shelf, but there is no hold - reshelving |
| 16:01 |
Bmagic |
editor[1|234234] asset.copy.update ....... - request en-US open-ils.cstore.direct.asset.copy.update 123123 - Returning method exception with message: An unknown server error occurred |
| 16:01 |
berick |
to the cstore / sql logs! |
| 16:02 |
Bmagic |
editor[1|234234] request error open-ils.cstore.direct.asset.copy.update : 123123 : Exception: OpenSRF::DomainObject::oilsMethodException 2017-08-31T14:44:55 OpenILS::Utils::CStoreEditor /usr/local/share/perl/5.22.1/OpenILS/Utils/CStoreEditor.pm:465 <400> No active transaction -- required for UPDATE |
| 16:02 |
Bmagic |
postgres didn't have an error |
| 16:02 |
berick |
ah |
| 16:02 |
berick |
timeout maybe? |
| 16:03 |
Bmagic |
checking cstore log, didnt check that yet |
| 16:03 |
Bmagic |
nothing |
| 16:06 |
Bmagic |
unless this counts |
| 16:06 |
Bmagic |
DBIx::ContextualFetch::db=HASH(0xa06d2f0)->disconnect invalidates 1 active statement handle (either destroy statement handles or call finish on them before disconnecting) at /usr/local/share/perl/5.22.1/OpenILS/Application/Storage/Driver/Pg/storage.pm line 11. |
| 16:08 |
Bmagic |
berick: a timeout on the staff client? I am waiting awhile during this checkin |
| 16:08 |
Bmagic |
due to all of the holds |
| 16:08 |
berick |
Bmagic: one thing to look for are API calls to not-cstore (usually storage) that take longer than the cstore drone keepalive timeout (usually 6 seconds). |
| 16:09 |
berick |
for example, if the API call to open-ils.storage.action.hold_request.nearest_hold.atomic took longer than 6 seconds, the open transaction w/ cstore will timeout |
| 16:09 |
Bmagic |
yeah, it was more than 6 seconds for sure |
| 16:10 |
Bmagic |
ok, so, is there a solution? Perhaps your hold targeter improvements could address this? We were talking about this issue at last year's hack-away - ABHP causing checking to hang because of the sort order of the fillable holds |
| 16:11 |
berick |
ABHP? |
| 16:12 |
Bmagic |
age based hold protection |
| 16:13 |
berick |
one solution is to port open-ils.storage.action.hold_request.nearest_hold.atomic to a Circ.pm (or whatever) utility function that uses cstore under the covers. that would bypass the cstore drone timeout. |
| 16:14 |
Bmagic |
alright, so it's offically a bug then? |
| 16:14 |
berick |
and no the new hold targeter has no effect here. this is a separate API call that only exists in open-ils.storage today. |
| 16:14 |
berick |
Bmagic: didn't you code some sorting improvements? |
| 16:14 |
Bmagic |
I didn't |
| 16:14 |
Bmagic |
We found the lines of code though :) |
| 16:15 |
berick |
ah |
| 16:16 |
berick |
i think the sorting improvements would be a bonus regardless |
| 16:16 |
berick |
maybe start there? |
| 16:21 |
Bmagic |
Sounds good. I think we figured out that the query to take into account the age based hold protection would be expensive, but I will take a look at it again |
| 16:30 |
berick |
oh, i don't remember that part |
| 16:32 |
|
sandbergja joined #evergreen |
| 16:49 |
|
Christineb joined #evergreen |
| 17:00 |
pinesol_green |
News from qatests: Test Failure <http://testing.evergreen-ils.org/~live> |
| 17:22 |
pinesol_green |
[evergreen|blake] LP1655158 Patron Search by Date of Birth - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=baf337b> |
| 17:42 |
|
jvwoolf left #evergreen |
| 20:22 |
|
Jillianne joined #evergreen |
| 23:03 |
gmcharlt |
hmm, I think that test failure is basically a problem with the test itself |
| 23:05 |
gmcharlt |
besides dropping that static redefinition of unapi.mmr_mra, there isn't a reliable ordering imposed on source_list, nor does it seems that the specific order matters |
