Evergreen ILS Website

IRC log for #evergreen, 2017-11-24

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
01:31 Jillianne joined #evergreen
06:32 pinesol_green News from qatests: Test Success <http://testing.evergreen-ils.org/~live>
09:03 _adb joined #evergreen
09:31 plux jeffdavis: still having issues with pingest .I believe it’s on me and not the script but missing something. Forgot yesterday was Thanksgiving there!
09:40 jeff plux: are you able to paste your command and output (minus any information you deem sensitive) here? http://paste.evergreen-ils.org/
09:41 plux just working on that now….
09:41 rlefaive joined #evergreen
09:43 pastebot "plux" at 64.57.241.14 pasted "pingest.pl errors: If I run th" (27 lines) at http://paste.evergreen-ils.org/944
09:43 rlefaive joined #evergreen
09:45 jeff plux: what version of Evergreen is in use with this database?
09:46 plux so this is 3.1 upgrade from 2.12.4
09:47 plux the pingest.pl is from the 3x fork
09:47 plux I checked the function(s) in the db and they appear to have all the updated elements
09:48 jeff did you remove some of the lines of output, or are you only getting errors on records 915197, 915198, and 915199?
09:48 plux I just grabbed a few….the others were/are the same
09:48 jeff okay, just making sure.
09:49 plux I can also replicate it by piping in a small test file with small number of bib ids….
09:51 * jeff nods
09:52 plux it seems like the query is passed to postgres without the values being parsed…wondering if the db server is missing some perl class or postgres mod that might impact the prepare statement
09:55 jeff there was a change to pingest to use named parameters in the call to the function metabib.reingest_metabib_field_entries()
09:56 jeff in psql, the syntax for a named argument would be something like function( argument := 'value')
09:58 jeff it looks like pingest.pl is using a perl-style syntax to prepare the query, using => instead of := -- which is probably entirely correct.
09:58 jeff but I've not delved into that particular scenario / syntax before.
09:59 jeff the problem seems to be that the => isn't being changed into a := before being sent to postgresql... or there's some other scenario where postgresql would handle the => without issue, and it works for Dyrcona but not you.
09:59 jeff plux: what version of postgresql is this, and what distro/version of linux?
10:00 plux postgres 9.4 and ubuntu 16.04
10:01 jeff can you check the version of some perl libs with these commands?
10:01 jeff perl -MDBD::Pg -e 'print $DBD::Pg::VERSION, "\n"'
10:01 jeff perl -MDBI -e 'print $DBI::VERSION, "\n"'
10:01 plux pretty sure it’s 3.07 but I’ll check
10:03 plux DBD:Pg  is 3.7.0 and DBI is 1.634
10:08 rlefaive joined #evergreen
10:12 Dyrcona joined #evergreen
10:14 Dyrcona plux: Looks like I forgot that I added db connection options to pingest.pl.
10:17 jeff ooh! that's it.
10:18 jeff postgresql 9.4 still uses := for named function argument notation.
10:18 jeff and not =>
10:18 jeff so the recent changes to pingest only support postgresql >= 9.5
10:19 jeff branch on its way!
10:20 Dyrcona I thought that went back to 9.1.
10:22 plux there was a package for postgresql-9.4-preprepare on xenial so I tried adding that to the mix yesterday but it doesn’t seem to have changed any of this behaviour
10:22 plux may be time to upgrade to 9.5 in any event
10:26 jeff Dyrcona: https://github.com/Dyrcona/​evergreen_utilities/pull/2
10:28 jeff sorry, i was wandering down the wrong path forgetting that the => in pingest.pl was just inside a string, not actually being interpreted by perl/DBI/DBD::Pg. i would have noticed the issue earlier. :-)
10:35 jeff Dyrcona: named notation goes back to at least 9.0, but the syntax didn't support arg => 'value' until 9.5
10:36 jeff Dyrcona: previous to that, => was an hstore operator, and postgresql has been warning about => for a few versions now, because of the upcoming use of it in 9.5 and above for named notation in functions.
10:40 Dyrcona Thanks, jeff. I merged it to master and to the evergreen-3.0-compatibility branch.
10:41 plux just tested on this system (jeff’s first master version) and it’s working on this db
10:41 jeff rather, hstore defined => as a user-defined operator back in 9.1 (and depending on various things, you could still have an old-style hstore contrib hanging around in newer databases, but you'll trip on it if you try to upgrade to 9.5)
10:41 jeff plux: yay!
10:41 jeff Dyrcona: thanks!
10:43 plux Drycona: jeff: thanks to both….nice script
10:53 Dyrcona Wonder if I should just make a 2.12 branch and merge the 3.0 branch into master at this point?
10:59 jeff what, no desire to teach pingest to auto-detect the version based on the database? ;-)
11:00 jeff at least one of the changes in master mentions 3.0 -- is that a commit that works for both, or is current master already >=3.0-only.
11:00 jeff s/\./?/
11:01 plux just tested the 3.0 branch and it’s working well on this 9.4 db
11:01 plux so both worked
11:08 Dyrcona Current master works on 3.0 and lower, but you don't have the option to skip the display ingest.
11:09 Dyrcona If you try to run the 3.0 branch on a 2.12 or older db, the ingests fail because there's no skip_display option.
11:09 Dyrcona If it had been added at the end of the other options, I may have a workable solution.
11:13 * dbs wonders if we would benefit from directory organization like 00/10/00/00/s or 00/10/00/00/00100000_s or the like to offer up a cover image for record # 100000 instead of potentially millions of cover images in a single directory
11:14 dbs our cover images are in an NFS share which undoubtedly complicates things. around 100K so far.
11:17 dbs also mused about enabling multiple images per record to provide a gallery (for things like puppets or physical material that benefit from more than just one image)
11:17 jeff dbs: yes, i have thought for a while now that this would be helpful.
11:18 jeff (not the multiple-images thing, the hashed dirs thing)
11:18 jeff multiple images would be nice also.
11:18 dbs that's promising :)
11:19 jeff we ended up offloading local cover images to nginx with fallback to evergreen for an AddedContent provider query when not found.
11:19 dbs mmm, I like that approach
11:21 dbs fewer requests for Evergreen, faster response time, win win win
11:23 jeff not sure if mostly-sequential filenames means we'd want to break up based on a simple hash of the filename or reversing the filename... that would probably just make things oppositely-lopsided.
11:23 jeff might not matter, as long as the leaf dirs don't end up with too many files in them.
11:24 jeff probably doesn't matter. isn't the same as a hash scheme like you'd use with memcached to fairly-weight.
11:31 dbs yeah, 3 (small, medium, large) * 100 in a single directory, no problem; even with galleries you're not likely to get beyond 1000 per directory
11:34 jeff and at present, the small/medium/large division is at the higher level
11:35 dbs yep, that's always kind of bugged me :)
11:36 dbs but I could live with it to make minimal changes if desired
12:30 jihpringle joined #evergreen
14:05 rlefaive joined #evergreen
14:14 Dyrcona joined #evergreen
14:47 rlefaive joined #evergreen
18:30 pinesol_green News from qatests: Test Success <http://testing.evergreen-ils.org/~live>
19:21 beanjammin joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat