| Time |
Nick |
Message |
| 07:43 |
|
kworstell-isl joined #evergreen |
| 08:06 |
|
cbrown joined #evergreen |
| 08:20 |
|
BDorsey joined #evergreen |
| 08:42 |
|
mmorgan joined #evergreen |
| 09:02 |
|
dguarrac joined #evergreen |
| 09:03 |
|
Dyrcona joined #evergreen |
| 09:18 |
|
redavis joined #evergreen |
| 09:19 |
redavis |
Ruth question for the day. This question has likely been asked and answered approximately 10 years ago. Can PostGreSQL and ElasticSearch work together in some way? |
| 09:22 |
Dyrcona |
What do you mean by "work together?" |
| 09:32 |
redavis |
Well, I'm not entirely sure. Obviously there's a whole bucket of information in this question that qualifies as "I don't know what I'm talking about." But, of course, EVGILS stores data in Pgres, but I know, in terms of bibliographic search, ElasticSearch has some benefits. Is it nonsense to have bibliographic/maybe_authorities/bib_component_of_acq in that db framework and the rest in pgres? |
| 09:32 |
redavis |
I think my real question is "is this nonsense?" |
| 09:33 |
redavis |
Hmmm, I guess even if it wasn't nonsense, that could break other things. |
| 09:41 |
JBoyer |
Elasticsearch would be a separate thing just for searching most likely, it's not necessarily good at doing "database stuff." At least berick and possibly others have looked at using ES for OPAC / client catalog searches, which is where it would excel. |
| 09:41 |
berick |
right, for our setup, elasticsearch works entirely as an add-on. nothing changed in PG itself except for some ES config tables. |
| 09:42 |
redavis |
berick, are you using it as an add-on right now with KCLS? |
| 09:43 |
berick |
redavis: oh yeah, for a few years now |
| 09:44 |
redavis |
So, this might be a loaded question that starts a fight in the EG underground, but are there cases against it? does it impact your connection with Bibliocommons in any way? why isn't it part of stock EG? |
| 09:44 |
* redavis |
isn't trying to start a fight in the EG underground, so apologies for any toe stompin' that might have just occurred. |
| 09:45 |
berick |
heh |
| 09:45 |
berick |
i can't fully answer that. from my perspective, it's a absolute win |
| 09:46 |
berick |
the implementation could probably use some punching up; there are some unimplemented features, all do-able though |
| 09:46 |
redavis |
berick++ JBoyer++ Dyrcona++ |
| 09:47 |
berick |
it is one more thing to maintain -- a separate piece of software -- but thus far it's been trivial to maintain on our end |
| 09:48 |
redavis |
Thanks all. I'm reading stuff and got stuck in the "why" spiral. Gonna let the questions hang in the IRCverse. It does seem that if it is a not-negligible improvement in catalog search efficiency, that is a good thing. |
| 09:49 |
JBoyer |
One thing that's specific to ES is their change of license because of AWS. It's no big deal for self-hosted installations but hosting providers are a bit wary of using it now (and Amazon has of course, forked it). Solr, which is used by VuFind and Aspen could do similar things, though I think it may take more setup than ES? |
| 09:50 |
berick |
yeah been meaning to look at OpenSearch (the amazone one) |
| 09:51 |
redavis |
JBoyer, do you think the time spent would be worth the payoff in terms of Solr (or OpenSearch)? |
| 09:53 |
JBoyer |
OS is a drop-in replacement at the moment. I haven't looked into the differences in what ES and Solr offer to know how much is even involved. Koha uses ES and those 2 big discovery layers use Solr, so both are certainly capable. |
| 09:53 |
redavis |
++ |
| 09:53 |
JBoyer |
Just depends on who wants to dev what and which communit(ies) we want to be able to ask for help. |
| 09:55 |
redavis |
Alright, I think y'all have answered my questions to the level that I can retain and not get bogged down with ill-conceived plans to take over the whole wide world. |
| 09:55 |
redavis |
Thank you. |
| 10:17 |
Bmagic |
redavis: I've been dying to get ES or it's equivalent. I brought it up a few times at various meetings and hack-a-ways. TADL showed how impressive it was, berick ended up making a branch for it, then I wanted to take that work and make it a YAOUS for the OPAC. |
| 10:19 |
Bmagic |
I've, sadly, not had time to loop back to that as our workload hasn't decreased :) |
| 10:20 |
csharp_ |
redavis: the way most of those discovery-layer-ish addons work is they receive data from the Postgres layer periodically (via uploads or triggers or something) and store that data in their own "databases" (usually something like SOLR, which is NoSQL storage) that are far faster to retrieve - I say this never having administered anything like that |
| 10:21 |
csharp_ |
so whatever PostgreSQL is doing in real time doesn't have anything to do with ElasticSearch or vice-versa |
| 10:22 |
redavis |
BMagic and csharp_ that all makes sense. I'm letting a few things roll around in my head with limited freedom since I don't want to start tilting at windmills... |
| 10:22 |
csharp_ |
Bmagic: we're interested too, especially since berick has pioneered it |
| 10:22 |
csharp_ |
so many windmills |
| 10:23 |
redavis |
indeed. And I'm got some significent Dutchness in me, so it's almost pathological to be drawn in by the windmills. |
| 10:23 |
Bmagic |
:) |
| 10:23 |
csharp_ |
@band add Significant Dutchness |
| 10:23 |
redavis |
s/I'm/I've |
| 10:23 |
pinesol |
csharp_: Band 'Significant Dutchness' added to list |
| 10:25 |
Bmagic |
I attended the Google conference last Sept. and I spoke with the Elasticsearch booth for a little while. I asked them about the licensing issue and how it may or may not be applicable to the Evergreen project. They didn't have a straight answer but took my info and said they'd be in touch.... I probably need to follow up |
| 10:27 |
Dyrcona |
If you were hosting Aspen also, I'd suggest getting into Solr. That way there is only 1 to support. But... I suppose one could always add ES to Aspen... :) |
| 10:28 |
Rogan |
I need a face melting emoji to reply to Dyrcona's comment there |
| 10:30 |
Rogan |
joking aside, I've been working with Aspen on a mostly application end user level but hoping to dig into the guts more |
| 10:32 |
* Dyrcona |
has PRs added to Aspen. :) I have yet to set up a test Aspen installation, but it's on my list of things that I'll (probably) never get to. |
| 10:35 |
redavis |
Y'all are the best ever. Also, pretty sure my husband wasn't expecting a long diatribe aboue databases, indexes, and search when he asked how my studying for the master gardener class was going. |
| 10:35 |
berick |
ooh, Master Gardener |
| 10:35 |
berick |
nice |
| 10:35 |
redavis |
Also, I WANT to throw Aspen and Solr into all of this, but it looks a LOT like a windmill. |
| 10:37 |
redavis |
berick, you know how it is...start reading about humidity and soluble salts and it inevitably leads to catalog efficiencies and full stack development. |
| 10:38 |
berick |
@band add The Soluble Salts |
| 10:38 |
pinesol |
berick: Band 'The Soluble Salts' added to list |
| 10:38 |
berick |
redavis: if I had a nickel... |
| 10:38 |
* redavis |
laughs |
| 10:38 |
Dyrcona |
redavis++ |
| 10:38 |
redavis |
berick++ |
| 10:38 |
Dyrcona |
Gardening and full stack development do have many similarities. |
| 10:39 |
csharp_ |
redavis: keep saying stuff that leads to @band entries! |
| 10:39 |
* csharp_ |
is here for the quality content |
| 10:41 |
JBoyer |
I haven't pointed a browser at an ES instance but I will say building queries in Solr's web UI is great for troubleshooting. |
| 10:41 |
redavis |
Dyrcona, they do. And it's also a healthy way for me to "parallel process" without ending up in an "ADHD miasmic brain chaos." |
| 10:41 |
redavis |
JBoyer++ |
| 10:52 |
redavis |
https://mccue.dev/pages/8-16-24-just-use-postgres This last line is gold. |
| 10:54 |
Dyrcona |
Indeed, that last line is gold. That article also implies that we should be using Elastic Search because search is a huge part of what we do. |
| 10:56 |
Dyrcona |
That whole thing is littered with nuggets. "The artist formerly known as Redis..." |
| 10:57 |
redavis |
Yes. At least partially. Or that was my takeaway. That PG is appropriate because of its flexibility and scalability, but ES is fer realz for search. |
| 10:57 |
redavis |
I almost bit on that line, Dyrcona, but needed to stay focused. Ish. |
| 11:13 |
Bmagic |
redavis++ # nice article pull |
| 11:13 |
csharp_ |
redavis++ |
| 11:14 |
* csharp_ |
moves PINES production to Google Sheets immediately |
| 11:14 |
csharp_ |
"someone on the internet said it's good" |
| 11:15 |
Bmagic |
lol |
| 11:16 |
Bmagic |
"Fake quotes will ruin the internet" - Benjamin Franklin |
| 11:16 |
csharp_ |
speaking of postgres, we consistently see deadlocks when trying to update mulitple bibs in the same transaction (PG 11 currently, upgrading soon) and I don't see a bug on it - anyone else have evidence of this kind of thing too? |
| 11:16 |
redavis |
Haha!! |
| 11:17 |
redavis |
(not "Haha!!" to PG lockups) |
| 11:17 |
csharp_ |
a stackexchange post pointed me to https://www.postgresql.org/docs/current/transaction-iso.html#XACT-SERIALIZABLE - but I'm wondering if it's more appropriate to code defensively further up in the Perl layer |
| 11:18 |
csharp_ |
well, no |
| 11:18 |
redavis |
opppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp0 |
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
csharp_ |
because it happens with direct DB stuff |
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
Bmagic |
oh no, Ruth dropped a book on her keyboard again |
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
|
| 11:18 |
redavis |
(that was a cat) |
| 11:18 |
mmorgan |
Does she have a cat? |
| 11:18 |
mmorgan |
Hah! |
| 11:18 |
csharp_ |
holy cats! |
| 11:18 |
redavis |
It was a cat on a book that then moved to the keyboard |
| 11:18 |
Bmagic |
quick, alt tab |
| 11:19 |
redavis |
So...all of those things were in play |
| 11:19 |
csharp_ |
the most library thing ever |
| 11:19 |
* mmorgan |
is familiar :) |
| 11:19 |
* redavis |
laughs |
| 11:20 |
pinesol |
News from commits: Docs: point release notes for 3.13.3 and 3.12.6 <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=ebfdd4821c06fd46502495b3f37a86abd7bf18bb> |
| 11:20 |
redavis |
Awwww yeah. |
| 11:20 |
redavis |
abneiman++ |
| 11:20 |
mmorgan |
csharp_: We haven't seen deadlocks since bug 1931737 was fixed. |
| 11:20 |
pinesol |
Launchpad bug 1931737 in Evergreen 3.8 "Did you mean breaks parallel reingest and causes deadlocks when loading/overlaying bib records in the client" [High,Fix released] https://launchpad.net/bugs/1931737 |
| 11:20 |
mmorgan |
abneiman++ |
| 11:22 |
redavis |
mmorgan, Bmagic, once y'all get your branches, I'll send out an email that the merge pause has been lifted. |
| 11:22 |
Bmagic |
redavis++ # rock'n roll branch'n emailer |
| 11:23 |
* mmorgan |
is just going into a meeting, will have to grab it after. |
| 11:23 |
Dyrcona |
csharp_: I have not seen that on my test systems, but granted, I don't update a whole lot of bibs all at once. We probably should consider forcing serialized updates on bibs. |
| 11:23 |
berick |
csharp_: see also the description of bug 1880726. |
| 11:23 |
pinesol |
Launchpad bug 1880726 in Evergreen "MARC Batch Edit Angular Port" [Wishlist,Fix released] https://launchpad.net/bugs/1880726 |
| 11:23 |
redavis |
I ain't got a lot of skillz, but I can email like a master...typer (with typos included) |
| 11:24 |
berick |
my understanding is this can happen with multi-bib update transactions and should be avoided, but I suspect it's more nuanced than that. |
| 11:24 |
Dyrcona |
I also have the did you mean code disabled |
| 11:26 |
Dyrcona |
Pg 11+ will do updates in parallel if it "thinks" it can get away with it, and it's not super smart at figuring that out. |
| 11:27 |
Bmagic |
our functions don't teach it very well about the cascading affects of data manipulations down the stack |
| 11:48 |
|
dmoore joined #evergreen |
| 12:00 |
|
jihpringle joined #evergreen |
| 12:21 |
JBoyer |
oppppppppppppppppppppp... made me think of The Matrix, but set in the midwest. |
| 12:21 |
* redavis |
laughs |
| 12:21 |
redavis |
There is everything good and right about that...except that I haven't met Keanu Reeves yet, and that's tragic. |
| 12:22 |
* redavis |
did get the 3rd BRZRKR recently though. Not the same as a real Keanu Reeves, but stil good. |
| 12:36 |
JBoyer |
I threw that article a quick skim and hit this vomment: "MySQL has much better and stable replication features than Postgres. That's one area MySQL kicks Postgres out of the park." |
| 12:37 |
redavis |
"vomment" is now my word of the campaign season. Thank you. |
| 12:38 |
JBoyer |
I had to close the window because I don't have days to school fools. Also, my blood pressure. (Also, lol at the guy bringing up sequences. I used one a single time, couldn't mysqldump | mysql because it's not smart enough to sequence things correctly.) |
| 12:58 |
|
BDorsey_ joined #evergreen |
| 13:01 |
abneiman |
vomment is my new favorite neologism / portmanteau |
| 13:01 |
abneiman |
JBoyer++ |
| 13:01 |
|
cbrown_ joined #evergreen |
| 13:07 |
JBoyer |
\o/ |
| 13:16 |
Dyrcona |
rebasing a branch on rel_3_13 and I don't think I've ever had conflicts with release notes before. |
| 13:17 |
Dyrcona |
Oh, that's why. It's the wrong branch! |
| 13:17 |
redavis |
mysqldump is funny if you can think like an 11 year old (I can). |
| 13:18 |
JBoyer |
If the phrase fits, mysql ... |
| 13:18 |
redavis |
JBoyer++ |
| 13:19 |
redavis |
best. comeback. ever. (I needed to employ a 13 year old brain for that though...and it was...*chef's kiss) |
| 13:46 |
Dyrcona |
If I wanted to add a message to the Angular staff client, something like "Training server do not use for live transactions," that would appear on every screen, where I should look to do that? |
| 13:47 |
Dyrcona |
Well, I do want to add such a message, but I'm not sure others here want it just yet. |
| 13:53 |
Dyrcona |
Maybe just in the equivalent of staff/t_splash.tt2... That's where we put it for AngularJS. |
| 13:54 |
csharp_ |
@band add Artoo TT2 |
| 13:54 |
pinesol |
csharp_: Band 'Artoo TT2' added to list |
| 13:55 |
Bmagic |
is there a trick in Evergreen to remove "CLAIMS_RETURNED" from their account for an item that is clearly in good standing. Other patrons have checked it out many times since the CLAIMS_RETURNED incident. But the item remains on the patron's account |
| 13:56 |
redavis |
I think it has to be removed directly from the DB |
| 13:56 |
JBoyer |
Bmagic, Does that specific circ have a checkin date? I think that plus the status is what is being looked for to collect those. |
| 13:57 |
Dyrcona |
FYI: I think I found where I want to add my training server message: ./Open-ILS/src/eg2/src/app/staff/splash.component.html |
| 13:58 |
|
jihpringle joined #evergreen |
| 13:58 |
Bmagic |
JBoyer: yes, there is a checkin date, but xact_finish is null. I suppose the chore is to update xact_finish to something other than null |
| 13:58 |
Dyrcona |
Could be it was never checked in... That happens. Someone could have forced a copy update, or if you're counting copies by "barcode" you might have a deleted/reinstated copy or reused barcode on your hands. |
| 13:58 |
JBoyer |
yeah, that may do it. (provided there isn't a fine-related reason to leave it null) |
| 13:59 |
Dyrcona |
If there are no fines, and you're sure about the actual copy, I'd update xac_finish to the checkin_time and set stop_fines to CHECKIN (I think that's correct....) |
| 14:01 |
Dyrcona |
On my message thing: Now, I wonder where I should put the message. Maybe in a div in the "container" div? |
| 14:08 |
Dyrcona |
Yeah, I probably want it as a "row" between the logo and the splash_nav div. |
| 14:12 |
Dyrcona |
Uh, no... I probably want it below where the title displays in splash_nav. |
| 14:12 |
|
BDorsey__ joined #evergreen |
| 14:13 |
|
cbrown joined #evergreen |
| 14:13 |
Dyrcona |
I'll just paste the text below the h2 with a line break and see what happens. |
| 14:14 |
Dyrcona |
OK, in a <p class="text-center">...>/p> |
| 14:14 |
Dyrcona |
Modulo the typos in that off the cuff thing. |
| 14:36 |
Dyrcona |
Nope. That was the wrong place. It appears at the top of each splash page column.... |
| 14:43 |
Dyrcona |
So, I put it in the logo div and it's much better. |
| 14:44 |
* eeevil |
reads up a bit ... csharp_, I'm working on a concurrency improvement WRT ingest-time browse entry management, which is where you can see deadlocks today. tl;dr: the solution that I implemented for that pingest bug linked earlier will work for browse entries as well. but, tuits. hopefully in the next week or two I'll have a branch to share |
| 14:57 |
csharp_ |
eeevil++ # great to hear! |
| 14:58 |
csharp_ |
deadlocks aren't presenting massive problems day to day, but we occasionally hit them in the wild from acquisitions/vandelay uploads |
| 15:00 |
|
halloy1408 joined #evergreen |
| 15:02 |
|
halloy1408 joined #evergreen |
| 15:05 |
|
adam_reid joined #evergreen |
| 15:09 |
|
Stompro joined #evergreen |
| 15:17 |
Dyrcona |
Hm... I should probably update the Redis branch on the server where I'm testing it. |
| 15:24 |
Dyrcona |
I wonder if I can just install it while things are running. Well, I guess I'll find out. |
| 15:30 |
Dyrcona |
That seemed to work. |
| 15:30 |
Dyrcona |
Nothing has gone boom so far.... |
| 16:54 |
|
sandbergja joined #evergreen |
| 16:56 |
sandbergja |
Funny story regarding postgres full-text search vs. Solr: we just launched a new tool last week that uses both postgres search and solr |
| 17:12 |
|
mmorgan left #evergreen |
| 18:21 |
|
adam_reid joined #evergreen |
| 22:07 |
|
adam_reid joined #evergreen |
| 23:01 |
|
StomproJ joined #evergreen |
