| Time |
Nick |
Message |
| 03:29 |
|
mrisher joined #evergreen |
| 03:56 |
|
mrisher joined #evergreen |
| 04:07 |
|
mrisher joined #evergreen |
| 05:40 |
|
mrisher joined #evergreen |
| 06:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:35 |
|
rfrasur joined #evergreen |
| 08:03 |
|
dbwells joined #evergreen |
| 08:14 |
|
Dyrcona joined #evergreen |
| 08:24 |
|
stompro_ joined #evergreen |
| 08:36 |
|
mmorgan joined #evergreen |
| 08:44 |
|
mantis1 joined #evergreen |
| 09:23 |
Dyrcona |
plprofiler is still only Python 2.x....... :( |
| 09:33 |
csharp |
we will be finding obsolete Python 2 stuff for years |
| 09:33 |
pinesol |
[evergreen|Bill Erickson] LP1869906 Angular staff cat browse links - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=4d864ff> |
| 09:43 |
Dyrcona |
Well, the database extension won't build. It looks like the Makefile depends on stuff installed on the maintainer's machine that isn't found elsewhere. |
| 09:44 |
Dyrcona |
csharp: Evergreen does a lot of obsolete things, too. :) |
| 09:45 |
Dyrcona |
Oh! "This means that it can be compiled and installed when checked out into the contrib directory of the PostgreSQL source tree." |
| 09:45 |
Dyrcona |
This is turning into actual work..... |
| 09:55 |
Dyrcona |
Cloning the PostgreSQL git repository, checking out 9.6 stable, linking the plprofiler clone to the contrib directory, setting USE_PGXS=1, and typing make almost works. |
| 09:56 |
Dyrcona |
It can't find it's own local header, which should be easy enough to fix, but I see that it is trying to include development headers from /usr/include/postgresql/12/server, and I want it to build for 9.6... I'm not so sure that this is worth it.... |
| 09:58 |
Dyrcona |
Ah, no... It's failing cause it can't find /usr/include/postgresql/12/server.... |
| 09:59 |
* Dyrcona |
returns to DDG. |
| 10:04 |
Dyrcona |
Bingo! We have a winner: make PG_CONFIG=/usr/lib/postgresql/9.6/bin/pg_config USE_PGXS=1 |
| 10:17 |
|
rfrasur joined #evergreen |
| 10:28 |
|
sandbergja joined #evergreen |
| 10:36 |
Dyrcona |
Now that I have the output, I'm not sure how useful it is.... |
| 10:38 |
Dyrcona |
Only thing it tells me is what I already knew: Updating an authority record spends way too much time in the database. |
| 10:42 |
Dyrcona |
Basically, it can take up to 5 seconds to ingest a bib record on this db server, so when you update someone like Jackie Chan with a 100 or so associated bibs, you can forget about it finishing if done from the client. |
| 10:44 |
Dyrcona |
Scarily enough, it can spend 0.2 seconds on an IF NOT FOUND! |
| 10:47 |
|
dbwells joined #evergreen |
| 10:48 |
Dyrcona |
Actually, it's disturbing how long PostgreSQL can spend on a simple IF. |
| 10:50 |
Dyrcona |
I should profile this on Pg 10, 11, & 12, too. |
| 10:53 |
Bmagic |
I've noticed the same when using "not" logic |
| 10:56 |
Dyrcona |
I found another IF NOT FOUND where the longest execution took 6.7 seconds (6,688,141 microseconds). |
| 10:57 |
Dyrcona |
This makes me wonder if the performance for that one line includes the whole IF/ELSE block? |
| 10:58 |
Dyrcona |
Well, no, that can't be because this one has a nested IF that can take up to 2 seconds, and that time, plus the query that ran before would be over 7 seconds, but that's assuming that the longest iteration of each block happened at the same time. |
| 11:00 |
Dyrcona |
Yeah, reading through the documentation and looking at the examples more closely, the numbers for the IF lines are for that line only! |
| 11:00 |
jeff |
Bmagic: NOT IN is on the "Don't Do This" list: https://wiki.postgresql.org/wiki/Don't_Do_This#Don.27t_use_NOT_IN |
| 11:00 |
jeff |
Bmagic: or were you talking about a different NOT? |
| 11:01 |
Dyrcona |
jeff: I know, and I'm talking about "PERFORM .... ; IF NOT FOUND....." With the IF NOT FOUND line taking longer than the PERFORM! |
| 11:02 |
Dyrcona |
My takeway: Don't use IF in PostgreSQL functions on 9.6, at least. |
| 11:04 |
Dyrcona |
I'll generate another profile of undoing the delete with a title and description and share that if anyone else wants to see what I'm talking about. |
| 11:04 |
Dyrcona |
s/delete/update/ |
| 11:04 |
|
sandbergja joined #evergreen |
| 11:04 |
Dyrcona |
I have a "revert" SQL with the original authority record marc handy. |
| 11:08 |
Bmagic |
jeff: yeah, mostly that. I can confirm that you shoild do it, lol. |
| 11:08 |
Bmagic |
Shouldn't* |
| 11:10 |
Dyrcona |
This time it was faster, probably because of database cache, but still too long.... |
| 11:14 |
Dyrcona |
Bleh.. Google Drive is useless for sharing this. |
| 11:14 |
Dyrcona |
It shows the HTML. |
| 11:17 |
Dyrcona |
OK. Here we go: https://www.sigio.com/~jason/revertchan.html |
| 11:17 |
Dyrcona |
Not sure how long I'll leave that up. |
| 11:21 |
Dyrcona |
Hmm. The more often that I do it, the faster it gets. :) |
| 11:25 |
Dyrcona |
Here are the results of doing the original update again, after undoing them: https://www.sigio.com/~jason/redochan.html |
| 11:26 |
Dyrcona |
And, here's the output from the cold update that shows the IFs taking a long time: https://www.sigio.com/~jason/firstchan.html |
| 11:31 |
Dyrcona |
BTW, I also verified that we have all of the expected indexes for Evergreen 3.2.8 this morning. In fact, we have a few extra indexes. |
| 11:36 |
Dyrcona |
Is the above Lp worthy? Out catalogers haven't been able to update Jackie Chan's authority record in production via the staff client because of the time it takes to update the 117 attached bibliographic records. I can update it successfully via SQL in the test database. |
| 11:42 |
Dyrcona |
I can try turning some of the NOTs into positive logic to see if that helps any.... |
| 11:44 |
Dyrcona |
Though on the one I'm eyeballing, I think it's the = ANY( attr_list) that contributes the most to the run time. |
| 11:55 |
|
mrisher joined #evergreen |
| 12:05 |
csharp |
too bad that NOT IN is the default construction created by the reporter for "Not in list" |
| 12:09 |
|
jihpringle joined #evergreen |
| 12:14 |
csharp |
NOT IN (list,of,values,...) is mostly safe unless you might have a null in the list (via a parameter or otherwise). So it's sometimes natural and even advisable to use it when excluding specific constant values from a query result. |
| 12:15 |
csharp |
I guess that's usually the case in the reporter, so there's that |
| 12:15 |
csharp |
(usually not any nulls in the lists we create) |
| 12:25 |
|
stompro_ joined #evergreen |
| 12:35 |
|
eady joined #evergreen |
| 12:47 |
Dyrcona |
I think there's an index or two that might help the performance slightly. |
| 13:11 |
Dyrcona |
Well, maybe not so much. |
| 13:14 |
Dyrcona |
Looks like our biggest performance hit comes from using IF NOT.... |
| 13:29 |
pinesol |
[evergreen|Jason Stephenson] LP1813088: Coerce Values to String in egGridValueFilter - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=be3abe4> |
| 13:51 |
Dyrcona |
Whee! One of my brick drones has two pcrud drones using 12GB of virt and 7.2GB of RES! |
| 13:53 |
JBoyer |
csharp, I can see it being less of a problem in the reporter where a suboptimal query delays your report a little bit, but it's significantly worse in a trigger based function that can be / is called frequently in quick succession, *especially* if something has a timeout waiting on it. |
| 13:53 |
JBoyer |
(There's a timeout on the reporter, but if your report is running for multiple hours it's not because of NOT IN (...) :) |
| 13:54 |
pinesol |
[evergreen|Jason Stephenson] LP1787636: Remove right justification in grid view - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=7dc13e0> |
| 13:54 |
Dyrcona |
The last log entries for these drones was at 10:23 am. |
| 13:55 |
Dyrcona |
What happens if I kill them? |
| 13:55 |
berick |
Dyrcona: killing a drone won't kill the listener if that's what you're wondering |
| 13:56 |
Dyrcona |
Nope not that so much. |
| 13:56 |
Dyrcona |
I'm assuming that whatever was communicating with them has given up after 3.5 hours.... |
| 13:57 |
Dyrcona |
Y'know, before I kill the next one, I'm going to see what's happening in the database. |
| 13:58 |
pinesol |
[evergreen|Daniel Pearl] LP1787636: Sort Libraries without Holdings In Holdings View - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=e0d7705> |
| 13:58 |
Dyrcona |
Interestingly, nothing is running at the moment. |
| 14:00 |
berick |
Dyrcona: would be curious to know wht the last API call was to those drones |
| 14:01 |
Dyrcona |
open-ils.pcrud open-ils.pcrud.search.au.atomic "REDACTED",{"id":{"!=":null}},{} |
| 14:01 |
Dyrcona |
So, yeah... |
| 14:02 |
Dyrcona |
Both of them, same authtoken.... |
| 14:06 |
csharp |
@who wanted to retrieve ALL THE USERS? |
| 14:06 |
pinesol |
felicia wanted to retrieve ALL THE USERS. |
| 14:10 |
Dyrcona |
Well, I know who actually did it. Logged in, first thing, search for all of the patrons in the database, apparently give up, and start doing acq. |
| 14:35 |
Dyrcona |
Back to Jackie Chan, it still takes 2 minutes to update the authority record even with everything cached. |
| 15:12 |
pinesol |
[evergreen|Jeff Davis] LP#1865533: save changes on Edit Hold in My Account - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=69f9ec7> |
| 16:03 |
|
sandbergja joined #evergreen |
| 16:16 |
|
mantis1 left #evergreen |
| 16:49 |
|
mantis1 joined #evergreen |
| 16:55 |
|
jihpringle joined #evergreen |
| 17:06 |
|
mmorgan left #evergreen |
| 18:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:11 |
|
abowling joined #evergreen |
| 19:45 |
|
Stompro joined #evergreen |
| 20:06 |
|
mantis1 joined #evergreen |
| 20:25 |
|
mantis1 joined #evergreen |
| 20:27 |
|
mantis1 left #evergreen |
| 22:10 |
|
stompro_ joined #evergreen |
| 23:19 |
|
sandbergja joined #evergreen |
