| Time |
Nick |
Message |
| 03:16 |
|
devted joined #evergreen |
| 03:17 |
|
dluch joined #evergreen |
| 03:17 |
|
Bmagic joined #evergreen |
| 04:04 |
|
Christineb joined #evergreen |
| 04:04 |
|
ejk joined #evergreen |
| 04:04 |
|
jeffdavis joined #evergreen |
| 04:04 |
|
jeff joined #evergreen |
| 04:04 |
|
pinesol joined #evergreen |
| 04:04 |
|
rhamby joined #evergreen |
| 04:04 |
|
awitter joined #evergreen |
| 04:04 |
|
phasefx joined #evergreen |
| 04:04 |
|
drigney joined #evergreen |
| 04:04 |
|
book` joined #evergreen |
| 04:04 |
|
stephengwills joined #evergreen |
| 04:04 |
|
dluch joined #evergreen |
| 04:04 |
|
Bmagic joined #evergreen |
| 04:05 |
|
bshum joined #evergreen |
| 04:05 |
|
troy__ joined #evergreen |
| 04:05 |
|
kip joined #evergreen |
| 04:05 |
|
eby joined #evergreen |
| 04:05 |
|
abneiman joined #evergreen |
| 04:05 |
|
laurie joined #evergreen |
| 04:05 |
|
jvwoolf joined #evergreen |
| 04:05 |
|
jonadab joined #evergreen |
| 04:05 |
|
miker joined #evergreen |
| 04:05 |
|
lisacarlucci joined #evergreen |
| 04:05 |
|
akilsdonk_ joined #evergreen |
| 04:05 |
|
devted joined #evergreen |
| 04:06 |
|
dbs joined #evergreen |
| 04:06 |
|
dickreckard joined #evergreen |
| 04:06 |
|
egbuilder joined #evergreen |
| 04:06 |
|
alynn26_away joined #evergreen |
| 04:06 |
|
jeff_ joined #evergreen |
| 04:07 |
|
yeats joined #evergreen |
| 04:07 |
|
cesardv joined #evergreen |
| 04:07 |
|
csharp joined #evergreen |
| 04:07 |
|
yar joined #evergreen |
| 04:08 |
|
genpaku joined #evergreen |
| 04:25 |
|
devted joined #evergreen |
| 04:27 |
|
Bmagic joined #evergreen |
| 04:49 |
|
JBoyer joined #evergreen |
| 04:49 |
|
jweston joined #evergreen |
| 04:49 |
|
gmcharlt joined #evergreen |
| 04:49 |
|
berick joined #evergreen |
| 06:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:24 |
|
rjackson_isl_hom joined #evergreen |
| 08:11 |
|
collum joined #evergreen |
| 08:29 |
|
mmorgan joined #evergreen |
| 08:32 |
|
mantis joined #evergreen |
| 08:34 |
|
Dyrcona joined #evergreen |
| 08:55 |
|
collum joined #evergreen |
| 10:52 |
miker |
jeffdavis: from last night, is that a pingest run, or a specialized external loader, or something else? |
| 10:54 |
jeffdavis |
INSERT INTO biblio.record_entry (id, marc, last_xact_id) SELECT id, marc, 'IMPORT' FROM setup.staging_records_import WHERE id = 3; |
| 10:55 |
jeffdavis |
miker: ^ running 4 separate scripts containing that kind of thing in 4 different terminal windows |
| 10:55 |
jeffdavis |
(no overlapping id's among the files) |
| 10:56 |
miker |
ah ... well, I will look into that. but note that they'll likely be (effectively, partially) serialized due to browse ingest regardless, because of the unique index there |
| 10:57 |
miker |
jeffdavis: new migration? (congrats if so! :) ) |
| 11:03 |
jeffdavis |
Setting up a server with some custom training data, actually. We do it semi-regularly with the same dataset. IIRC 4 parallel jobs was chosen as a "safe" number for avoiding deadlocks. |
| 11:04 |
jeffdavis |
Our guy who does migrations is off Fridays, but I want to follow up with him about this since he uses MARC import tools that are rather less crude. :) |
| 11:05 |
miker |
jeffdavis: it may be that, for situations like yours where you need to load the data FAST, a "load without symspell, then generate sysmspell data" would be best. the 3.7 symspell side-loader should make that more or less straight forward |
| 11:06 |
miker |
like, disable some triggers, load as you did before, generate the dictionary, reenable triggers |
| 11:09 |
miker |
the side-loader is intentionally single threaded to make it simple while being as fast as a single CPU will go, but there's room in the world for a map-reduce-on-memcache version that could leverage All The Cores, I think |
| 11:10 |
* miker |
tries to resist buildng that right now |
| 11:10 |
jeffdavis |
I'll try disabling the triggers as a comparison. If it goes well, maybe we can add an "ingest.disable_symspell" type flag. |
| 11:11 |
jeffdavis |
I also want to do some Vandelay loads to make sure they're unaffected, they're already quite slow. We should get to that over the next few days I think. |
| 11:11 |
miker |
jeffdavis++ |
| 11:15 |
* miker |
mumbles something about queued ingest and making imports faster and authority propagation possible... |
| 11:17 |
Dyrcona |
jeffdavis++ |
| 11:39 |
jeffdavis |
Disabling maintain_symspell_entries_tgr on *_field_entry tables does the trick. Now to try adding that flag... |
| 11:44 |
Dyrcona |
So, I'm writing Perl tests for Lp 1923076. Looks like I have it working for open-ils.actor.user.hold_requests.count. I'll go through miker's list and add tests for the backend calls. |
| 11:44 |
pinesol |
Launchpad bug 1923076 in Evergreen 3.6 "Incorrect typing when encoding some values as JSON" [Medium,Confirmed] https://launchpad.net/bugs/1923076 |
| 11:46 |
Dyrcona |
Using some the B module to determine if a value is an integer or not. |
| 11:46 |
Dyrcona |
Oops. Bad edit... |
| 11:47 |
jeff |
should we be testing that, or testing the result of the JSON encode? |
| 11:47 |
jeff |
(I don't know) |
| 11:56 |
Dyrcona |
jeff: The responses go through JSON encode in OpenSRF. Look at https://bugs.launchpad.net/evergreen/+bug/1923076/comments/2 and try the srfsh command (adding the missing comma after the authtoken). On an affected, but unpatched system, you'll see the responses are strings. |
| 11:56 |
pinesol |
Launchpad bug 1923076 in Evergreen 3.6 "Incorrect typing when encoding some values as JSON" [Medium,Confirmed] |
| 11:57 |
Dyrcona |
Meahwhile...It's lunch time for me. |
| 12:08 |
|
jihpringle joined #evergreen |
| 12:38 |
Dyrcona |
So, I think I've come across one that I can't test on concerto, but I'm pretty sure it needs a patch: open-ils.circ.hold.clear_shelf.process |
| 12:51 |
miker |
Dyrcona++ |
| 12:58 |
Dyrcona |
miker: There are a couple that I'm sure don't need fixes, but I'm unsure about this one: Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Driver/Pg/storage.pm:401: return scalar(@fm_nodes); |
| 12:59 |
Dyrcona |
I had trouble find a context where it is used, so I moved on. |
| 12:59 |
|
sandbergja joined #evergreen |
| 13:01 |
miker |
Dyrcona: IMO, when in doubt, cast it |
| 13:01 |
Dyrcona |
OK, that makes sense. |
| 13:02 |
miker |
because the only time it /actually/ matters is when we shove it into json and want it to be numbery |
| 13:02 |
miker |
and in every perl-only case, DWIM does what we mean :) |
| 13:03 |
miker |
(until they decide to follow php8 and, apparently, roku and make "0" truthy) |
| 13:05 |
Dyrcona |
All right. |
| 13:13 |
Dyrcona |
So, one of these appears in a commented-out section of code. I'm going to delete as dead code. |
| 13:18 |
Dyrcona |
Y'know, I doubt they'll go the way of making "0" truthy in Perl 5. The whole reason we're doing this is an optimization to make boolean lookups on scalar faster. |
| 13:18 |
Dyrcona |
Famous last words, eh... :) |
| 13:18 |
Dyrcona |
Perl-- |
| 13:25 |
|
collum joined #evergreen |
| 13:59 |
Dyrcona |
I'm drawing a blank on how to lookup records with ISBNs in the database. |
| 14:02 |
|
sandbergja joined #evergreen |
| 14:02 |
Dyrcona |
Oh, nice. Here's one that isn't zero: "superpage_size":"1000" |
| 14:03 |
Dyrcona |
I guess we should just wrapp all calls to scalar with int.... Perl is so broken, now..... |
| 14:04 |
Dyrcona |
That's not coming from scalar, though. It's coming from the search_hash->{check_limit} which is passed as argument somewhere. |
| 14:05 |
Dyrcona |
So, when are we gonna reimplement this in C++? /me ducks..... |
| 14:13 |
* jeff |
eyes Cpanel::JSON::XS and tries to get past the Cpanel prefix |
| 14:21 |
Dyrcona |
Perl-- |
| 14:35 |
jeff |
Cpanel::JSON::XS appears to already do the "right" thing and only treats a simple scalar as a string when the string is different. Put another way, quoting from the docs, "numbers with temporary strings which represent the same number are here treated as numbers, not strings" |
| 14:35 |
jeff |
(I don't think we should switch to Cpanel::JSON::XS right now.) |
| 14:36 |
jeff |
But it also supports passing an additional argument to encode_json, so you can write a spec of what your encoded JSON should look like. |
| 14:37 |
Dyrcona |
I think we should ditch Perl, tbh. Seems like every few years we play whack-a-mole with nonsense like this. |
| 14:44 |
Dyrcona |
I'm testing Cpanel::JSON::XS on buster. Looks like all of the necessary changes are limited to 1 file. |
| 14:46 |
Dyrcona |
I still get some numbers that are strings, but the superpage size one is fixed. I'm gonna see if my tests pass. |
| 14:47 |
Dyrcona |
The tests that I have written pass with Cpanel::JSON::XS on the unpatched buster VM. I think switching to Cpanel::JSON::XS is easier than ditching Perl. |
| 14:49 |
Dyrcona |
So, looking at the output of these tests and some other calls, I'd say we have bigger problems than calling scalar on empty arrays and passing the results through JSON::XS. |
| 14:54 |
jeff |
Do you mean that you think Cpanel::JSON::XS fixes more things than we were focusing on? |
| 14:56 |
Dyrcona |
Yes, I do, but it still isn't perfect. I'm updating the bug. |
| 14:57 |
* jeff |
nods |
| 14:58 |
|
mantis left #evergreen |
| 15:00 |
Dyrcona |
jeff: https://bugs.launchpad.net/evergreen/+bug/1923076/comments/4 |
| 15:00 |
pinesol |
Launchpad bug 1923076 in Evergreen 3.6 "Incorrect typing when encoding some values as JSON" [Medium,Confirmed] |
| 15:00 |
Dyrcona |
Without getting into too much detail. |
| 15:05 |
jeff |
do we rely on the value for superpage being a number? did its type change between perl versions or between JSON libs? |
| 15:13 |
Dyrcona |
I haven't tried it on Perl 5.26. |
| 15:13 |
Dyrcona |
I'm not sure how much of an issue all of that is. |
| 15:35 |
Dyrcona |
So, the superpage_size shows up as a string on Perl 5.26 (Ubuntu 18.04). |
| 15:47 |
Dyrcona |
Cpanel::JSON::XS seems to fix the issues we're concerned about in the bug. |
| 16:01 |
|
sandbergja joined #evergreen |
| 16:12 |
abneiman |
dluch++ # yay preconferences! |
| 16:51 |
|
collum joined #evergreen |
| 16:56 |
miker |
well, he's gone for the week, but I don't see any reason to touch numbers-as-strings except where they might reasonably be 0, TBH |
| 16:58 |
miker |
also, if cpanel::json::xs is doing a lot more work, I'd hate to make things slower (though the speed is probably not noticeable), and that's a significant deep change... |
| 17:00 |
jeff |
arguably, scalar(@empty-lexical) is the only place we've found change in behavior. |
| 17:13 |
jeff |
if we flip every numeric string to a JSON number, we'll probably shoot ourselves in the foot in one or two places. :-) |
| 17:15 |
|
mmorgan left #evergreen |
| 18:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 21:07 |
|
jvwoolf1 joined #evergreen |
