Evergreen ILS Website

IRC log for #evergreen, 2014-02-25

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
02:04 dbwells_ joined #evergreen
07:14 ALB joined #evergreen
07:25 timlaptop joined #evergreen
07:29 tfaile joined #evergreen
07:36 rjackson-isl joined #evergreen
07:36 Shae joined #evergreen
07:52 collum joined #evergreen
08:11 ALB_ joined #evergreen
08:18 kbeswick joined #evergreen
08:19 akilsdonk joined #evergreen
08:33 bshum Hmm
08:43 mmorgan joined #evergreen
08:46 Dyrcona joined #evergreen
08:47 dbs uh oh. when bshum is cogitating, something is afoot.
08:48 bshum Oh no, I'm just trying to hang together some data for a library that's joining us and going live today.
08:48 bshum First new lib in a couple years.  Too rusty :)
08:51 dbs Are they getting libinfo pages yet? :)
08:52 bshum dbs: Heh, unfortunately not.  I have to spend more time teasing out what else is new alongside all of that to make sure it doesn't break stuff for production.
08:53 bshum That said, I did trial run libinfo on my test server.  I think the only thing I want to tweak locally for us is how the mailing address gets displayed and maybe removing the parent links (most of it doesn't help the single branch systems)
08:56 jeff in the case of the parent links, would you see benefit to removing the human-visible links but leave the microdata?
08:56 dbs a) There's no microdata, it's RDFa </pedant>
08:56 dbs b) Probably not, as the search engines get suspicious when they're fed info that humans don't see.
08:57 dbs c) JSON-LD might be an option in that case.
08:57 jeff dammit, i KNEW i'd get it wrong.
08:57 jeff if i had gone with my initial instinct of "linked data" instead of "microdata", would I have been more accurate? :-)
08:58 dbs Structured data is probably most accurate, linked data also more accurate
08:58 jeff thanks. :-)
08:58 dbs bshum: I should probably build in OU visibility to those parent links if I didn't already
08:59 jeff good morning, eg-dev-db! it's been... er, a little less than three hours since we last spoke. Good to see you again!
08:59 bshum dbs: Ah, you know I think that might help.
09:00 bshum Now that you mention it.
09:01 Polonel joined #evergreen
09:01 jeff @later tell eeevil alas, looks like not even a restore as the evergreen user can save you from pg_restore's explicit search paths.
09:01 pinesol_green jeff: The operation succeeded.
09:03 dbs so... schema-qualify all the functions, or put everything into public schema
09:03 * dbs votes for the former
09:03 jeff likewise.
09:04 jeff and eevil already voted last night!
09:09 bshum I like that.
09:12 jeff i'm mildly curious if a schema-only followed by a data-only restore would circumvent the issue, but that's a bit of a tangent.
09:19 graced joined #evergreen
09:23 Dyrcona jeff: We run a script after a restore to fix things. I believe that I have shared it before.
09:23 * jeff nods
09:29 jeff oh those wacky pg_dump/pg_restore antics!
09:31 jeff view definitions with timestamptz get normalized somewhere in the process of restoring then dumping (then technically "restoring" to diff)
09:31 jeff nothing stock in evergreen, so it just serves to bring attention to some old views that can be removed at some point in the near future. :-)
09:32 * jeff stops pulling at these threads and goes to pick up a different thread
09:33 yboston joined #evergreen
09:48 jboyer-isl If the polls are still open I'm all for fully qualified function names. Needing to use an "evergreen" user to get expected functionality is essentially the same as requiring a specific installation prefix...
09:59 Dyrcona Now, why would we want to do that? ;)
10:15 fparks joined #evergreen
10:24 ktomita_ joined #evergreen
11:02 mrpeters joined #evergreen
11:13 ktomita joined #evergreen
11:13 fparks joined #evergreen
11:13 sseng joined #evergreen
11:17 sciani joined #evergreen
11:24 csharp jeff: I have ended up doing 'ALTER ROLE evergreen SET search_path = blah' in our case
11:27 jeff and in your experience have you found that doing this before a pg_restore resolves your search_path issues when creating the two authorities indexes in question?
11:27 RoganH joined #evergreen
11:29 afterl joined #evergreen
11:32 csharp jeff: I can say that having done that last month for our production upgrade gave us an error-free pg_restore experience ;-)
11:33 csharp I can't speak to the specific indexes, but I believe so, yeah
11:33 jeff and were those indexes present before the pg_dump? ;-)
11:33 * jeff grins
11:34 csharp I'm pretty sure I used 'script' to log everything - let me look
11:35 csharp nope - I just 'script'-ed the actual upgrade scripts
11:36 jeff do you have the indexes authority.by_heading_and_thesaurus and authority.by_heading ?
11:38 csharp hmm - no, in fact, we don't :-/
11:39 jeff aha! :-)
11:39 csharp I will say that we have been happily running 2.5.1 since 1/22 without them ;-)
11:39 csharp is there documentation somewhere about what they do?
11:40 jeff they are both indexes on authority.record_entry
11:41 jeff any queries or functions that use authority.normalize_heading or authority.simple_normalize_heading would likely benefit from them.
11:42 csharp hmm
11:42 Dyrcona csharp: There's a script for that. :)
11:43 csharp Dyrcona: oh?
11:43 jeff Dyrcona refers to a script which re-creates the two indexes, used post-pg_restore
11:43 csharp ah - I see
11:46 jihpringle joined #evergreen
11:47 Dyrcona I was just looking to see if I could find it at paste.evergreen-ils.org, but gave up. :)
11:49 csharp well, I'm now running the CREATE INDEX commands on my test db to time them
11:49 Dyrcona Cool.
11:49 csharp I don't see much in the code though that actually references the normalize_heading functions
11:49 csharp which must be why we haven't noticed ;-)
11:49 Dyrcona I think our script also sets the function search path, too.
11:51 pastebot "Dyrcona" at 64.57.241.14 pasted "Errors on pg_restore" (8 lines) at http://paste.evergreen-ils.org/25
11:51 Dyrcona csharp: You should errors like the above on pg_restore.
11:51 csharp looks like search_authority_by_simple_normalize_heading is defined as a subroutine in Open-ILS/perlmods/lib/OpenILS/A​pplication/Search/Authority.pm but it doesn't appear to be called anywhere
11:52 csharp I'm sure that happened to us too then
11:52 csharp I do remember seeing "2 errors ignored", so I ignored them too ;-)
11:53 csharp happily it wasn't crucial ;-)
11:53 Dyrcona :)
11:54 Dyrcona You probably won't see that last line. That comes from setting the path in evergreen db from postgres db.
12:07 bshum mmorgan++ # more feedback!
12:09 Bmagic joined #evergreen
12:10 Dyrcona Hmm. I just remembered why our script set the database search path.
12:11 Dyrcona We don't login to the evergreen database as user evergreen.
12:17 shadowspar joined #evergreen
12:20 jwoodard joined #evergreen
12:26 csharp eeevil: I'll take a look at your hold proximity patches
12:26 csharp (for bug 1272316, for context)
12:26 pinesol_green Launchpad bug 1272316 in Evergreen "much slower holds processing in 2.4+" (affected: 9, heat: 38) [High,Confirmed] https://launchpad.net/bugs/1272316
12:27 eeevil csharp: great. I'm getting that in place in another large test system... MORE DATA POINTS!
12:27 csharp excellent
12:28 eeevil jeff: bah! (restore) ... I have a feeling things are getting worse, not better, with more modern pg_restore binaries
12:30 jeff eeevil: all the more reason to move toward explicit qualification. of course, then there's potential for different search paths over time to have resulted in conflicting (or at least duplicate) functions in a live evergreen database, which might not be an issue until explicit schema qualification comes along...
12:31 eeevil jeff: yeah, we'll have to 1) qualify function names and 2) CREATE OR REPLACE the functions ... not in that order, obv
12:31 jeff eeevil: overall, seems to be a good band-aid to rip off, and perhaps a helpful script in extras to help identify things that may be problematic for a particular database.
12:32 tsbere drop them all and re-create them where we want them! ;)
12:34 jeff once 2) and 1) above are done, a script could be optionally run for "hey, we found these evergreen functions hanging out in public -- you might want to kill them off!" :-)
12:35 dbwells re: the current billing discussion, I'm wondering if anyone can shed light on the usefulness of "work" and "goods" payments.  I mean, I can imagine what they mean, but I wonder if they are really broadly useful.
12:35 csharp dbwells: since those are PINES things, as far as I can tell, I'll see how well-used they are here
12:35 jeff libraries that use them will have them in their database and will likely want to preserve that, as well as preserve whatever use they are using them for. "work" i would imagine as "volunteer at the library, we forgive some fines" and goods as in "food for fines" programs.
12:36 csharp s/PINES things/originally PINES things?
12:36 dbwells Locally, we abuse "work" payments to mean something else.
12:38 jeff we've been using "Forgive" for partial voiding.
12:38 jeff (a la negative conditional balances)
12:38 csharp goods (15454) is far more commonly used than work (1653) over the 2013 calendar year
12:39 Sato`kun joined #evergreen
12:40 dbwells csharp: So your libraries actually accept "goods" for payment of fines?  Like trading in candy bars or livestock or something?  Or does it mean something else?
12:40 kmlussier I imagine some libraries use it for canned food drives.
12:41 csharp dbwells: it's usually canned goods, like a "food for fines" type thing
12:41 dbwells Ah, makes sense :)
12:41 csharp though, I wonder in certain of our rural libraries if livestock would be acceptable ;-)
12:42 dbwells I know from my goat farming youth that they can be worth a pretty penny :)
12:42 csharp but, goods constitute less than 1% of our total payments over the same time period, so there's that
12:42 mmorgan jeff: So when you "Forgive", do you have problems with negative balances if the forgiven bills get voided later?
12:42 csharp mmorgan: yep
12:43 dbwells csharp: good to know
12:43 csharp mmorgan: I know you weren't asking me, but yeah, we just dealt with that with voiding snowday fines :-)
12:44 csharp a library had forgiven them before I went through and voided them (with the intention of catching ones they had missed)
12:44 mmorgan csharp: I think that's true with all of us :)
12:44 csharp they said "void" but meant "forgive"
12:44 jeff mmorgan: in our most common workflows, no. we're running code that handles auto-forgive of overdues on Lost, re-billing of forgiven overdues on later checkin of a Lost item, etc. It's a working branch attached to a bug that was made (mostly, at least) superfluous by the conditional negative balances work.
12:46 jeff My hope would be to test and ensure that the conditional negative balance work does indeed make our Forgive-not-Void work obsolete.
12:48 mmorgan Ah, so you have been circumventing the built in Evergreen behavior for a while.
12:49 jeff when it comes to lost item handling, yes. we'd like to bring things back into the fold, since other efforts with slightly differing requirements came along. I believe some of the code from our working branch became part of the conditional negative balances branch.
12:50 Dyrcona1 joined #evergreen
12:51 jeffdavis Where does osrf_control come from?
12:51 jeff jeffdavis: recent versionf of OpenSRF
12:52 jeffdavis Sorry, I mean, I don't see it (or a template for it or anything) in the 2.6 source.
12:52 dbwells jeff: it replaces osrf_ctl.sh in OpenSRF 2.3+.  It is part of OpenSRF, not Evergreen.
12:52 dbwells jeffdavis: it replaces osrf_ctl.sh in OpenSRF 2.3+.  It is part of OpenSRF, not Evergreen.
12:53 dbwells jeff: sorry :)
12:53 jeffdavis ah, that explains it. Thanks.
12:53 jeff jeffdavis: the OpenSRF makefile links osrf_control to opensrf-perl.pl
12:53 jeffdavis I was looking in the EG source. :)
12:53 jeff jeffdavis: so there is no file matching osrf_control\* in OpenSRF repo, if you were looking there next. :-)
12:54 * jeffdavis skips `find`, proceeds directly to `grep`
12:55 jeff in the OpenSRF repo, you'll find the file that becomes opensrf-perl.pl (the target of the osrf_control symlink) at bin/opensrf-perl.pl.in
12:55 Dyrcona1 jeffdavis: If you want to see the source of osrf_control.sh just find opensrf-perl.pl and read that.
12:55 Dyrcona1 err... I spelled that wrong.
12:56 jeff and fwiw, I don't think that there's an osrf_control.sh :-)
12:56 jeff er, Dyrcona just realised that. :-)
12:56 Dyrcona Yeah, I just said that.
12:56 jeffdavis Yeah, I'm slowly accommodating myself to the freaky new world of suffix-less scripts
12:57 jeff tabs vs spaces is just a distraction from the real debate: underscore vs hyphen.
12:58 jeff dff00df1cc0d2730cd332c6086ee908e
12:58 akilsdonk joined #evergreen
13:01 Dyrcona My understanding is that most people use suffixes on scripts as a hint to their text editor.
13:04 akilsdonk_ joined #evergreen
13:05 gsams joined #evergreen
13:09 jihpringle joined #evergreen
13:09 mrpeters jeff:  did you happen to get my email last week Re: Jasper Reports?
13:16 jeff no, I don't think so.
13:16 * jeff looks
13:17 jeff mrpeters: alas, you ended up in my spam folder. unusual. sorry about that!
13:17 jeff > Be careful with this message. Many people marked similar messages as spam.
13:17 mrpeters no problem!
13:17 mrpeters ls
13:18 dbwells joined #evergreen
13:19 mceraso dbwells: Just tested the 2.5.3 release candidate using Ubuntu 12.04 LTS with great success
13:19 Sato`kun joined #evergreen
13:20 jeff prediction didn't come true: echo -n '/me waits for the typographers to come along and point out his mis-use of "hyphen"' | md5sum
13:20 dbwells mceraso++ # thanks!
13:20 mceraso On to testing 2.6 beta now...
13:20 bshum mceraso++
13:21 mrpeters i said "ls" dammit!
13:21 mrpeters haha
13:22 Dyrcona jeff: hypen, dash, en dash, or em dash? Perhaps you want a reverse solidus?
13:22 Dyrcona mrpeters: bshum is working on fixing the breakage with ls in IRC. :)
13:23 bshum Dyrcona: mrpeters: Yeah, just go file a bug report / feature request with the supybot folks :)
13:23 bshum :)
13:23 mrpeters haha
13:23 csharp @ls
13:23 pinesol_green csharp: http://wonder-tonic.com/geocitiesizer/content.ph​p?theme=2&amp;music=6&amp;url=evergreen-ils.org
13:24 csharp @dunno
13:24 pinesol_green csharp: MARC still isn't dead yet, alas
13:25 Dyrcona @dunno get 7
13:25 pinesol_green Dyrcona: Dunno #7: "You probably want hard-boiled eggs." (added by Dyrcona at 05:11 PM, July 02, 2012)
13:26 Bmagic joined #evergreen
13:33 csharp eeevil: wow - that patch dropped processing time from 24 seconds to 6 seconds for a bib with 224 items attached
13:34 bshum Fancy!
13:34 csharp it was the test item I sent you the logs for
13:34 csharp gonna test some more examples I've gotten
13:34 eeevil csharp: great.  can you confirm that the output is exactly as you expect? action.hold_copy_map contents, action.hold_request for that one, etc?
13:35 kmlussier csharp: That's awesome!
13:35 eeevil csharp: and that's on top of senator's previous patch, yes?
13:35 csharp eeevil: correct
13:35 eeevil csharp: related: how long did it take to run the proximity pre-calc update in the upgrade script? (aprox)
13:36 eeevil (approximately, I mean)
13:36 csharp well... wait - I may be mixing up my test environments
13:36 eeevil heh
13:36 csharp it ran for about 20-25 mins, I think
13:36 eeevil ok. that's not too bad
13:36 csharp and that's on a legacy (underpowered) db server
13:36 csharp old frankenweezie11, actually ;-)
13:37 eeevil and in the worst case it handles the lack of that gracefully in the code, so you'd have one final slow overnight run before the effects are truly felt
13:37 eeevil FRANKENWEEZIE!!!!!!
13:37 csharp eeevil: yeah - they're still kicking ;-)
13:38 * csharp may release bradl's tux doll on the day we retire those good old DL580G5s
13:41 snowkitteh__ joined #evergreen
13:42 bshum FREEDOM!
13:47 csharp eeevil: yes, I can confirm that senator's patches are included
13:55 csharp yeah - definite improvements - a hold that took 65 seconds in production takes 14 seconds on my test server
13:56 csharp still a long time, www-wise, but way better
13:57 eeevil csharp: you don't happen to have pre-2.5 timings for an equiv hold, do you?
13:58 csharp eeevil: unfortunately not
13:59 eeevil I'd honestly be very surprised if a 600+ copy hold was ever faster than the post-patch version
14:03 csharp eeevil: yeah - probably not
14:03 csharp you know how upgrades bring up old problems as if they're new ;-)
14:03 eeevil csharp: I HAVE NO IDEA WHAT YOU ARE TALKING ABOUT
14:04 eeevil csharp / bshum: pushing a rebased version of that with senator's commits embedded below mine in a moment
14:04 csharp eeevil++
14:04 bshum eeevil: Cool, I'm testing out the differences between our test system (with senator's stuff + yours now) vs. production
14:06 bshum 10 seconds in test, ~28 in production
14:06 bshum For one of the sample bibs giving some grief
14:07 eeevil http://git.evergreen-ils.org/?p=working/Everg​reen.git;a=shortlog;h=refs/heads/collab/miker​/prox-calc-speedup_plus_aouas-speedup_rebased FTR
14:09 csharp bshum: nice
14:13 stevenyvr joined #evergreen
14:23 kbutler joined #evergreen
14:27 jboyer-dc joined #evergreen
14:32 mceraso dbwells: Tested 2.6 Beta Preview using Ubuntu 12.04 LTS. Works just fine!
14:33 dbwells mceraso++ # thanks again!
15:17 floadam joined #evergreen
15:42 bshum csharp: You didn't add it to the bug, but I assume your signoff branch is official?
15:42 bshum For holds
15:42 pinesol_green` joined #evergreen
15:42 jeff dear apache: why aren't you starting?
15:42 bshum jeff: It's probably because it's actually named apache2.
15:42 berick @dunno search apache
15:42 pinesol_green berick: 1 found: #4: "Try restarting apache."
15:42 jeffdavi1 joined #evergreen
15:43 mrpeters left #evergreen
15:44 jeff strace to the rescue
15:44 jeffdavis joined #evergreen
15:46 csharp bshum: got interrupted, yeah - was going to add that to the bug
15:47 csharp done
15:47 bshum csharp: All good, just making sure you get your name on there too.  I'm probably going to edit the commit lines to include the LP # before I commit it.
15:47 csharp bshum: good - thanks
15:49 jeff well drat. running tpac on a non-standard port seems unsupported.
15:49 * bshum wants to say there's an LP for that
15:49 bshum But maybe I'm just dreaming ghostly LP bugs now.
15:49 jeff i think there's one close.
15:49 jeff tangent for now.
15:53 bshum Calling 0869
15:54 bshum Well actually, before I do... eeevil, are you cutting a 2.4 maintenance release?  (wondering about backports given that dbwells already cut 2.5.3)
16:06 stevenyvr joined #evergreen
16:37 jeff pondering possible wishlist bug: permit sorting list ("bookbag") html views by date added to list and/or position in list (which would need support for the "pos" column in container.biblio_record_entry_bucket_item)
16:38 jeff might need an index on create_time and/or pos
16:42 eeevil jeff: only for really big lists ... REALLY big
16:42 eeevil like, 10k+
16:42 jeff oh, because the items being sorted would already be small and have been selected by an existing index.
16:42 jeff handy, that.
16:42 jeff logical, even!
16:42 eeevil otherwise the "by bucket" index will be the index to use
16:42 eeevil right
16:43 jeff so my thought was to modify the QP driver to allow for a new sort_filter (which would be valid if and only if this was a container search)
16:44 eeevil plus that table is soooo skinny, and sorting is on int's, which are just about as fast as you can do (faster than an index+heap lookup)
16:44 eeevil well, we have a container-specific filter already
16:44 eeevil could we add a parameter to that?
16:45 eeevil hrm..
16:45 eeevil no
16:45 eeevil you're right
16:48 kmlussier joined #evergreen
16:49 eeevil jeff: you mean a new acceptable sort axis (like create_date, which is not dynamic) but is ignored when there's no container() filter in use at the top of the query?
16:49 jeff right.
16:50 eeevil like: sort(container.pos)
16:50 jeff something like:
16:50 jeff } elsif ($sort_filter eq 'bucket_item_create_date') { # XXX: Need to also ensure that this is a container search?
16:50 jeff $rank = "FIRST((SELECT create_date FROM container.biblio_record_entry_bucket_item cbrebi WHERE cbrebi.target_biblio_record_entry = m.source))";
16:50 jeff
16:50 jeff (completely untested stab)
16:50 jeff er, i'd also need to select the bucket id -- oops. :-)
16:50 eeevil ah, well, I think we've already joined container.biblio_record_entry_bucket_item in the main query
16:51 jeff that too. i planned to revisit that after ensuring that i can actually do what i think i can do.
16:51 * eeevil is not looking at the code, mind
16:51 kmlussier_ joined #evergreen
16:51 mceraso joined #evergreen
16:51 bshum joined #evergreen
16:51 jeff QP newb here.
16:52 jeff i can probably just set $rank to ci.create_date
16:52 jeff (as well as ensure that i'm not trying to sort on that unless we're doing a container search)
16:52 eeevil jeff: ah ... well, it's a subselect. we'll need to surface create_date and pos, it seems
16:52 sunnybunbun joined #evergreen
16:54 eeevil I think we'll want more integration between flatten() and toSQL() WRT the container filter.  the filter is handled by the former, sorting by the latter
16:54 eeevil but I think it's totally doable
16:56 jeff looks like i'd need to have the value of $filter_alias at the point where I'm setting $rank...
16:57 eeevil jeff: it's not an exact conceptual match, but look at how $uses_bre (and friends) works ... something similar can be checked in the sorting if-block
16:57 afterl left #evergreen
16:57 eeevil well, the join needs to surface the pos and create_date columns, but after that it's just another if-branch where we're picking a sorter
16:58 eeevil container join, I mean
16:59 eeevil hrm... well, we'll still need a distinct-on (or surface the entry id and look up the sort col in a SELECT subquery, like the others)
16:59 eeevil to avoid duplicated container entries
16:59 eeevil anyway, it's very doable.
16:59 eeevil jeff++
17:00 jeff i'll take a stab at create_date first and see how it goes. i'm going to need to digest a bit more of QP first.
17:03 eeevil jeff: sounds good. if you find yourself adding more than, say, 10-15 lines of code, make sure you haven't wandered too far into the forest of fangorn
17:05 jeff a good metric.
17:07 jeff # vim:et:ts=4:sw=4:fanghorn_ratio=.04
17:10 gmcharlt @quote add <jeff> # vim:et:ts=4:sw=4:fanghorn_ratio=.04
17:10 pinesol_green gmcharlt: Error: You must be registered to use this command. If you are already registered, you must either identify (using the identify command) or add a hostmask matching your current hostmask (using the "hostmask add" command).
17:11 gmcharlt @quote add <jeff> # vim:et:ts=4:sw=4:fanghorn_ratio=.04
17:11 pinesol_green gmcharlt: The operation succeeded.  Quote #78 added.
17:16 hbrennan joined #evergreen
17:16 mmorgan left #evergreen
17:44 dcook joined #evergreen
17:59 Dyrcona joined #evergreen
18:03 jihpringle joined #evergreen
22:47 stevenyvr joined #evergreen
23:27 zerick joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat