| Time |
Nick |
Message |
| 01:31 |
|
Jillianne joined #evergreen |
| 06:32 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 09:03 |
|
_adb joined #evergreen |
| 09:31 |
plux |
jeffdavis: still having issues with pingest .I believe it’s on me and not the script but missing something. Forgot yesterday was Thanksgiving there! |
| 09:40 |
jeff |
plux: are you able to paste your command and output (minus any information you deem sensitive) here? http://paste.evergreen-ils.org/ |
| 09:41 |
plux |
just working on that now…. |
| 09:41 |
|
rlefaive joined #evergreen |
| 09:43 |
pastebot |
"plux" at 64.57.241.14 pasted "pingest.pl errors: If I run th" (27 lines) at http://paste.evergreen-ils.org/944 |
| 09:43 |
|
rlefaive joined #evergreen |
| 09:45 |
jeff |
plux: what version of Evergreen is in use with this database? |
| 09:46 |
plux |
so this is 3.1 upgrade from 2.12.4 |
| 09:47 |
plux |
the pingest.pl is from the 3x fork |
| 09:47 |
plux |
I checked the function(s) in the db and they appear to have all the updated elements |
| 09:48 |
jeff |
did you remove some of the lines of output, or are you only getting errors on records 915197, 915198, and 915199? |
| 09:48 |
plux |
I just grabbed a few….the others were/are the same |
| 09:48 |
jeff |
okay, just making sure. |
| 09:49 |
plux |
I can also replicate it by piping in a small test file with small number of bib ids…. |
| 09:51 |
* jeff |
nods |
| 09:52 |
plux |
it seems like the query is passed to postgres without the values being parsed…wondering if the db server is missing some perl class or postgres mod that might impact the prepare statement |
| 09:55 |
jeff |
there was a change to pingest to use named parameters in the call to the function metabib.reingest_metabib_field_entries() |
| 09:56 |
jeff |
in psql, the syntax for a named argument would be something like function( argument := 'value') |
| 09:58 |
jeff |
it looks like pingest.pl is using a perl-style syntax to prepare the query, using => instead of := -- which is probably entirely correct. |
| 09:58 |
jeff |
but I've not delved into that particular scenario / syntax before. |
| 09:59 |
jeff |
the problem seems to be that the => isn't being changed into a := before being sent to postgresql... or there's some other scenario where postgresql would handle the => without issue, and it works for Dyrcona but not you. |
| 09:59 |
jeff |
plux: what version of postgresql is this, and what distro/version of linux? |
| 10:00 |
plux |
postgres 9.4 and ubuntu 16.04 |
| 10:01 |
jeff |
can you check the version of some perl libs with these commands? |
| 10:01 |
jeff |
perl -MDBD::Pg -e 'print $DBD::Pg::VERSION, "\n"' |
| 10:01 |
jeff |
perl -MDBI -e 'print $DBI::VERSION, "\n"' |
| 10:01 |
plux |
pretty sure it’s 3.07 but I’ll check |
| 10:03 |
plux |
DBD:Pg is 3.7.0 and DBI is 1.634 |
| 10:08 |
|
rlefaive joined #evergreen |
| 10:12 |
|
Dyrcona joined #evergreen |
| 10:14 |
Dyrcona |
plux: Looks like I forgot that I added db connection options to pingest.pl. |
| 10:17 |
jeff |
ooh! that's it. |
| 10:18 |
jeff |
postgresql 9.4 still uses := for named function argument notation. |
| 10:18 |
jeff |
and not => |
| 10:18 |
jeff |
so the recent changes to pingest only support postgresql >= 9.5 |
| 10:19 |
jeff |
branch on its way! |
| 10:20 |
Dyrcona |
I thought that went back to 9.1. |
| 10:22 |
plux |
there was a package for postgresql-9.4-preprepare on xenial so I tried adding that to the mix yesterday but it doesn’t seem to have changed any of this behaviour |
| 10:22 |
plux |
may be time to upgrade to 9.5 in any event |
| 10:26 |
jeff |
Dyrcona: https://github.com/Dyrcona/evergreen_utilities/pull/2 |
| 10:28 |
jeff |
sorry, i was wandering down the wrong path forgetting that the => in pingest.pl was just inside a string, not actually being interpreted by perl/DBI/DBD::Pg. i would have noticed the issue earlier. :-) |
| 10:35 |
jeff |
Dyrcona: named notation goes back to at least 9.0, but the syntax didn't support arg => 'value' until 9.5 |
| 10:36 |
jeff |
Dyrcona: previous to that, => was an hstore operator, and postgresql has been warning about => for a few versions now, because of the upcoming use of it in 9.5 and above for named notation in functions. |
| 10:40 |
Dyrcona |
Thanks, jeff. I merged it to master and to the evergreen-3.0-compatibility branch. |
| 10:41 |
plux |
just tested on this system (jeff’s first master version) and it’s working on this db |
| 10:41 |
jeff |
rather, hstore defined => as a user-defined operator back in 9.1 (and depending on various things, you could still have an old-style hstore contrib hanging around in newer databases, but you'll trip on it if you try to upgrade to 9.5) |
| 10:41 |
jeff |
plux: yay! |
| 10:41 |
jeff |
Dyrcona: thanks! |
| 10:43 |
plux |
Drycona: jeff: thanks to both….nice script |
| 10:53 |
Dyrcona |
Wonder if I should just make a 2.12 branch and merge the 3.0 branch into master at this point? |
| 10:59 |
jeff |
what, no desire to teach pingest to auto-detect the version based on the database? ;-) |
| 11:00 |
jeff |
at least one of the changes in master mentions 3.0 -- is that a commit that works for both, or is current master already >=3.0-only. |
| 11:00 |
jeff |
s/\./?/ |
| 11:01 |
plux |
just tested the 3.0 branch and it’s working well on this 9.4 db |
| 11:01 |
plux |
so both worked |
| 11:08 |
Dyrcona |
Current master works on 3.0 and lower, but you don't have the option to skip the display ingest. |
| 11:09 |
Dyrcona |
If you try to run the 3.0 branch on a 2.12 or older db, the ingests fail because there's no skip_display option. |
| 11:09 |
Dyrcona |
If it had been added at the end of the other options, I may have a workable solution. |
| 11:13 |
* dbs |
wonders if we would benefit from directory organization like 00/10/00/00/s or 00/10/00/00/00100000_s or the like to offer up a cover image for record # 100000 instead of potentially millions of cover images in a single directory |
| 11:14 |
dbs |
our cover images are in an NFS share which undoubtedly complicates things. around 100K so far. |
| 11:17 |
dbs |
also mused about enabling multiple images per record to provide a gallery (for things like puppets or physical material that benefit from more than just one image) |
| 11:17 |
jeff |
dbs: yes, i have thought for a while now that this would be helpful. |
| 11:18 |
jeff |
(not the multiple-images thing, the hashed dirs thing) |
| 11:18 |
jeff |
multiple images would be nice also. |
| 11:18 |
dbs |
that's promising :) |
| 11:19 |
jeff |
we ended up offloading local cover images to nginx with fallback to evergreen for an AddedContent provider query when not found. |
| 11:19 |
dbs |
mmm, I like that approach |
| 11:21 |
dbs |
fewer requests for Evergreen, faster response time, win win win |
| 11:23 |
jeff |
not sure if mostly-sequential filenames means we'd want to break up based on a simple hash of the filename or reversing the filename... that would probably just make things oppositely-lopsided. |
| 11:23 |
jeff |
might not matter, as long as the leaf dirs don't end up with too many files in them. |
| 11:24 |
jeff |
probably doesn't matter. isn't the same as a hash scheme like you'd use with memcached to fairly-weight. |
| 11:31 |
dbs |
yeah, 3 (small, medium, large) * 100 in a single directory, no problem; even with galleries you're not likely to get beyond 1000 per directory |
| 11:34 |
jeff |
and at present, the small/medium/large division is at the higher level |
| 11:35 |
dbs |
yep, that's always kind of bugged me :) |
| 11:36 |
dbs |
but I could live with it to make minimal changes if desired |
| 12:30 |
|
jihpringle joined #evergreen |
| 14:05 |
|
rlefaive joined #evergreen |
| 14:14 |
|
Dyrcona joined #evergreen |
| 14:47 |
|
rlefaive joined #evergreen |
| 18:30 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 19:21 |
|
beanjammin joined #evergreen |
