| Time |
Nick |
Message |
| 07:09 |
|
collum joined #evergreen |
| 07:24 |
|
kworstell-isl joined #evergreen |
| 07:40 |
|
cbrown joined #evergreen |
| 08:09 |
|
dguarrac joined #evergreen |
| 08:25 |
|
BDorsey joined #evergreen |
| 08:37 |
|
mmorgan joined #evergreen |
| 08:44 |
|
redavis joined #evergreen |
| 09:24 |
Bmagic |
I've captured the two sql queries, one for "salem witch trials" and one for "salem witch trials" -oo. The cache busting one returns results and the non-cache buster doesn't return results. The interesting difference between the two resulting constructed sql queries is: /* virtual field addition */ doesn't exist in the "bad" query |
| 09:24 |
Bmagic |
see comparison: https://dropbox.mobiusconsortium.org/pickup/salemwithctrials.html |
| 09:38 |
mmorgan |
Bmagic: If only 1 search term, there's no benefit to searching the virtual fields. At least I think that's what this says: https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/Application/Storage/Driver/Pg/QueryParser.pm;hb=e14804c4f02ffe2f94c35a260ead2af84a7c6d68#l1329 |
| 09:39 |
Bmagic |
hmmm |
| 09:39 |
Bmagic |
we're getting really close |
| 09:40 |
Bmagic |
it seems that the issue is: "salem witch trials" doesn't appear exactly like that in metabib.keyword_field_entry, no results. But if we look at the other metabib tables, we can find it |
| 09:44 |
Bmagic |
so, do we understand that the only way to make QueryParser include the virtual indexes, is when there is more than one search term? Because I think* my problem is getting the QueryParser to use the virtual index. I'd like it to include it for keyword searches |
| 09:49 |
Bmagic |
It's starting to make sense. If I search for "salem" (with quotes), I get results, however, the results are not highlighted. If I throw in "-ppp" at the end, I get highlighted results and, actually, more relevant results. If I remove the quotes and only search for salem, I get the same results as I do when searching with "salem" -ppp |
| 09:54 |
Bmagic |
I thought it was my problem and only a problem on this one database, but I don't think so. I just think that this database is small enough, that there aren't any matching results for "salem witch trials" in metabib.keyword_field_entry compared to other databases. So it's looking more like a corner case bug? |
| 09:57 |
|
cbrown joined #evergreen |
| 09:57 |
Bmagic |
mmorgan: keyword searching noble for "salem witch trials" vs. "salem witch trials" -asdf yeilds the same number of results oddly, but the former doesn't produce highlighting, and the latter produces the results in a different relevance order (improved results IMO) |
| 10:29 |
|
Dyrcona joined #evergreen |
| 10:37 |
Bmagic |
wb |
| 10:40 |
Dyrcona |
@blame [band] for The Armenian Regression |
| 10:40 |
pinesol |
Dyrcona: Broken as Designed stole bshum's tux doll! for The Armenian Regression |
| 10:41 |
Bmagic |
oh! mmorgan: and it explains why title searches turn up results! Because the virtual index that isn't included in "salem witch trials" doesn't include metabib.title_field_entry |
| 10:44 |
mmorgan |
Bmagic: I ran the query you posted directly in the database on our system and got zero results. |
| 10:44 |
mmorgan |
But, when I search "salem witch trials" in our opac, I get 240 results. So there's something different about the generated query between the 2 systems. |
| 10:45 |
Dyrcona |
Bmagic: So the cache busting isn't cache busting. It's adding another term to the search? |
| 10:45 |
Bmagic |
yeah! I was tracing down the cache, trying to figure out where I needed to delete the cache, but capturing the queries told the story |
| 10:46 |
Bmagic |
Here's the smoking gun: https://dropbox.mobiusconsortium.org/pickup/salemwithctrials.html |
| 10:46 |
Dyrcona |
Also are you saying that your keyword search does not include title_field_entry? That's fixable, I think. |
| 10:46 |
Bmagic |
it actually does include title_field_entry IF* you search for more than one term. And since "salem witch trials" is quoted, it's counted as a single term |
| 10:47 |
Dyrcona |
OK. |
| 10:47 |
Bmagic |
take a gander at that comparison output |
| 10:47 |
Dyrcona |
Yeah, I have the diff open. That smells like a bug. |
| 10:48 |
Dyrcona |
What Evergreen release are we talking about? |
| 10:48 |
Bmagic |
3.13.0 |
| 10:48 |
Bmagic |
I'm writing it up to the dev list at the moment |
| 10:49 |
Dyrcona |
I can try my 3.13 and 3.14 systems. See what I get. I wonder if this is related to the indexing mess? |
| 10:49 |
Bmagic |
the coded value thing? |
| 10:49 |
Dyrcona |
I say 3.14, 'cause updated it yesterday. ;) |
| 10:54 |
Bmagic |
bug 2073561 |
| 10:54 |
pinesol |
Launchpad bug 2073561 in Evergreen "Incorrect content in the config.coded_value_map with ctype audience after applying the upgrade script from 3.12.3 to 3.13.0" [Medium,Confirmed] https://launchpad.net/bugs/2073561 - Assigned to Jason Stephenson (jstephenson) |
| 10:54 |
Dyrcona |
You're looking at the OPAC here, not the staff catalog? |
| 10:55 |
Bmagic |
right, just patron opac |
| 10:55 |
|
JBoyer joined #evergreen |
| 10:55 |
Bmagic |
though, I find the same issue on the staff side |
| 10:55 |
Dyrcona |
Yeah, that bug may be involved. |
| 10:56 |
|
cbrown_ joined #evergreen |
| 10:57 |
Dyrcona |
So, I may have cheated. I first did salem witch trials unquoted as a keyword search and got 277 results. I then did "salem witch trials" as a keyword search and got 180. This was on 3.13.3. I'll try the one that's almost up to date with main, but it was doing an ingest yesterday. |
| 10:58 |
Dyrcona |
We're in luck, the ingest finished overnight. |
| 10:59 |
Dyrcona |
Quoted "salem witch trials" as a keyword search got me 183 results. I have deleted/fixed all of the erroneous coded_value_map entries that I could find and done an ingest on both databases. Maybe that makes a difference? |
| 11:00 |
Dyrcona |
Have we ever outright recalled an update before? |
| 11:00 |
Bmagic |
I expect that you will get results because your database contains matching "salem witch trials" in metabib.keyword_field_entry. But if you can capture the actual query, you should find that it's only* consulting the metabib.keyword_field_entry table |
| 11:01 |
Dyrcona |
How did you capture the query? Database logging? |
| 11:01 |
Bmagic |
database logging, yep |
| 11:01 |
Dyrcona |
I suppose I can flip the switch to log everything if it didn't run long enough. |
| 11:01 |
Bmagic |
log_min_duration_statement = 0 |
| 11:01 |
Dyrcona |
There's another one that will do it, too, IIRC. |
| 11:02 |
Bmagic |
restart postgres, resetart app server, echo '' > /var/log/postgresql/postgresql-15-main.log |
| 11:02 |
Bmagic |
run the search both ways, one with the buster and one without. copy the log to another place so it doesn't get too polluted with more output |
| 11:05 |
Dyrcona |
I don't think you have to restart Pg if you set log_min_statement_duration while it is running. |
| 11:06 |
Bmagic |
ok, I did it anyways because I could |
| 11:06 |
Dyrcona |
If you change the config file, you have to reload, I think. This is one you can set while connected with psql, if I understand the documentation. |
| 11:11 |
Dyrcona |
Well, that almost worked. I think I got cached search results. I'll use a different phrase. |
| 11:11 |
Bmagic |
during my test, I just stopped memcached |
| 11:13 |
Dyrcona |
I'll restart memcached. It looks like the setting didn't take. I tried a different search and it wasn't logged. Maybe it only changes the setting for the current Pg process when you set it dynamically? |
| 11:14 |
Bmagic |
yeah, that could be it. If you make a setting change via psql cli, it might be just for that session? |
| 11:17 |
Dyrcona |
OK. that appeared to work. |
| 11:22 |
Bmagic |
well, jeez, I thought I could reproduce it on a concerto set but I can't. Though, when I capture the two queries, I can confirm that the virtual index is only included when there is a second term after the quoted string. Though, on concerto, I can't find a bib search that only appears in a title search and not in keyword |
| 11:23 |
Dyrcona |
I'm finding everything but this search query.... |
| 11:26 |
Dyrcona |
I guess it grabbed results from cache even though I restarted memcached. |
| 11:27 |
mmorgan |
Bmagic: On the system where keyword "salem witch trials" returns no results, what do the keyword entries look like for a record that comes up in a title search? |
| 11:27 |
Dyrcona |
So, we'll reboot the db server and the Evergreen vm. |
| 11:27 |
mmorgan |
select * from metabib.keyword_field_entry where source = 37937 |
| 11:28 |
Bmagic |
mmorgan: yeah |
| 11:32 |
Bmagic |
comparing to concerto has be scratching my head, so I started comparing the settings in config.metabib_field and config.metabib_class and I found a difference: config.metabib_class has keyword "combined" set to true on my problem database, and set to false on vanilla concerto..... So! I'm flipping that back to false and reingesting |
| 11:32 |
Dyrcona |
Maybe it will work better if I try this from the Corwin house down the street? |
| 11:32 |
Bmagic |
mmorgan: I used that exact example when I looked at the rows in keyword_field_entry. And there are rows in there for that bib. But I assumed the phrase "salem witch trials" wasn't in the index vector exactly like that |
| 11:33 |
|
Christineb joined #evergreen |
| 11:33 |
sleary |
redavis do you want to distinguish bug fixes that involve string changes vs those that don't for the remaining items on your 3.14 spreadsheet, so we can prioritize the stringy things this week? |
| 11:35 |
Dyrcona |
Bmagic: FWIW: combined is 'f' for keyword in my upgraded database. |
| 11:35 |
redavis |
sleary, yes please. |
| 11:36 |
Bmagic |
Dyrcona: ty, this should make a difference in the rows appearing in keyword_field_entry after reingest |
| 11:39 |
Dyrcona |
And, I was able to find my previous search in the logs after restarting. I should have just searched log for "salem" the first time. It was there before the restart. |
| 11:47 |
sleary |
redavis on the Confirmed and SO tab, all are bugfixes, and I think the only one with string changes is 1770979. There are competing branches there and it would be great to have some input from either of the branch authors. |
| 11:49 |
redavis |
Can we get gmcharlt and berick in a room together? |
| 11:50 |
redavis |
It looks like berick's branch has the signoff. |
| 11:50 |
* berick |
looks |
| 11:51 |
berick |
i think my patch was lacking |
| 11:51 |
berick |
i removed the signedoff tag |
| 11:52 |
berick |
i love showing up and moving things in the wrong direction |
| 11:53 |
redavis |
lol, I'm not sure what to think about that (currently simultaneously evaluating pesticide labels...unrelated) |
| 11:58 |
sleary |
redavis things in the w/o signoffs tab that involve strings: 1848398, 1555766, 1895695, 2018941 (minimal though), 2065344, 2080438, 1988085. You may have decided to punt some of those as new features, though. |
| 12:02 |
sleary |
berick++ # clearing clutter is the right direction! |
| 12:02 |
* sleary |
has some clutter to remove, too... |
| 12:03 |
redavis |
sleary, I kind of colorcoded the tickets in that tab. It should be noted that I'm not in charge of the world, but I was the yellow ones would be eligible and the purpose would be punted down the road. |
| 12:04 |
sleary |
redavis++ |
| 12:04 |
redavis |
If they got appropriate signoffs, of course. |
| 12:05 |
Bmagic |
you guys! it was the combined=true flag (needed to be set to false). The quoted search term is now providing results as I would expect. Though there might* be a bug with the highlighting when searching with quoted strings |
| 12:05 |
redavis |
Bmagic++ |
| 12:05 |
redavis |
Congratulations!! |
| 12:06 |
mmorgan |
Bmagic++ |
| 12:06 |
sleary |
Bmagic++ # nice! |
| 12:06 |
Bmagic |
I think I set it to true when chasing down the other* issue, having to do with the branch visibility, and forgot that my solution for that problem wasn't related to that flag, ended up being call number visibility cache |
| 12:06 |
Bmagic |
Geez Louise |
| 12:07 |
Bmagic |
well, I learned stuff |
| 12:08 |
|
jihpringle joined #evergreen |
| 12:18 |
sleary |
redavis I'm wrong, 1848398 only removes strings. In any case, it now has my signoff |
| 12:19 |
redavis |
sleary++ |
| 12:22 |
redavis |
All of the remaining uncommitted but signed off tickets look like bugfixes rather than features. Dyrcona, can we include those if they get committed between now and Friday? |
| 12:24 |
|
jvwoolf joined #evergreen |
| 12:26 |
Dyrcona |
Bug fixes are pretty much always OK. Though we want to be mindful of string changes so the translators get a chance to do some work. |
| 12:26 |
redavis |
Dyrcona++ |
| 12:26 |
redavis |
sleary, is that helpful? |
| 12:29 |
Bmagic |
just setup a new server from main this morning. Man it's looking really nice |
| 12:29 |
redavis |
Bmagic++ |
| 12:30 |
Bmagic |
Color mode switcher, wow. Buckets in search, wow, better contrasting colors on the staff client, wow |
| 12:31 |
redavis |
It's going to be a pretty amazing release. |
| 12:33 |
Dyrcona |
Hopefully, the last 3.x release, but time will tell. |
| 12:35 |
sleary |
redavis++ |
| 12:35 |
sleary |
super helpful for me to prioritize things this week, yes, thank you! |
| 12:39 |
sleary |
That thing we did in Buckets where you can now put a button right up next to the title? That's available everywhere, if you want to take advantage of it in other modules. I will write up how to do so... any minute now |
| 12:39 |
redavis |
sleary++ Dyrcona++ #TeamWorkMarksTheDreamWork (I read a think about capitalized hashtags this morning and now I'm doing it forever) |
| 12:41 |
* sleary |
hopes people limit themselves to ONE button up there and don't go crazy with this... |
| 12:41 |
|
cbrown joined #evergreen |
| 12:41 |
redavis |
That should probably be included in the writeup. Restraint is sometimes challenging. |
| 13:09 |
Dyrcona |
I think I wanna fix up NCIPServer during the Hack-A-Way. Do you think I can convert it from an (obsolete) Dancer app to mod_perl in about 2 days? |
| 13:32 |
* Dyrcona |
meanders off in search of .... |
| 13:39 |
|
kworstell-isl joined #evergreen |
| 14:58 |
|
Dyrcona joined #evergreen |
| 16:18 |
|
jvwoolf left #evergreen |
| 16:46 |
|
Dyrcona joined #evergreen |
| 16:59 |
|
kworstell-isl joined #evergreen |
| 17:15 |
|
mmorgan left #evergreen |
