| Time |
Nick |
Message |
| 00:56 |
|
collum joined #evergreen |
| 01:20 |
|
collum joined #evergreen |
| 01:57 |
|
collum joined #evergreen |
| 02:00 |
|
book` joined #evergreen |
| 02:35 |
|
collum joined #evergreen |
| 03:11 |
|
collum joined #evergreen |
| 03:30 |
|
collum joined #evergreen |
| 04:09 |
|
collum joined #evergreen |
| 04:46 |
|
collum joined #evergreen |
| 05:23 |
|
collum joined #evergreen |
| 05:44 |
|
collum joined #evergreen |
| 06:20 |
|
collum joined #evergreen |
| 06:27 |
|
collum joined #evergreen |
| 08:04 |
|
BDorsey joined #evergreen |
| 08:04 |
|
redavis joined #evergreen |
| 08:30 |
|
Dyrcona joined #evergreen |
| 08:42 |
|
mmorgan joined #evergreen |
| 08:45 |
|
mantis joined #evergreen |
| 08:49 |
|
dguarrac joined #evergreen |
| 09:02 |
Dyrcona |
"Mild Picante." Either it's mild or it's picante. It's not both. :) |
| 09:14 |
|
kmlussier joined #evergreen |
| 09:14 |
kmlussier |
Dyrcona: Hot, but not too hot? |
| 09:14 |
kmlussier |
Speaking of hot things... |
| 09:15 |
kmlussier |
@coffee [someone] |
| 09:15 |
* pinesol |
brews and pours a cup of Ethiopia Yirgacheffe, and sends it sliding down the bar to troy |
| 09:15 |
kmlussier |
@tea [someone] |
| 09:15 |
* pinesol |
brews and pours a pot of Earl Grey Decaffeinated Black Tea, and sends it sliding down the bar to phasefx (http://ratetea.com/tea/bigelow/earl-grey-decaf/87/) |
| 09:24 |
Dyrcona |
So, I'm thinking about proposing a new service: open-ils.auditor. It would be used to manage auditor tables and do things like delete old rows, and possibly vacuum the tables. The more that I think about it, though, the trickier it gets. |
| 09:25 |
* Dyrcona |
wonders if it is even possible to do a vacuum through cstore? I've never tried. |
| 09:34 |
Rogan |
I'd joke you should just rewrite cstore if necessary but even I'm not that flippant. |
| 09:34 |
Dyrcona |
Rogan++ |
| 09:37 |
|
collum joined #evergreen |
| 09:38 |
Dyrcona |
At worst, cstore would probably just require a new method to call a process other than a stored procedure or function. |
| 09:41 |
Dyrcona |
If creating a new service seems impractical, I am considering adding some database functions that can purge old auditor table data and vacuum the tables. We have processes doing this via SQL, and I'm sure that most sites do. It could be useful added to the schema and then a cronscript could be scheduled to trigger them. |
| 09:42 |
Dyrcona |
The database functions would likely get added as part of the new schema anyway. Storage calls could be added to run them if cstore can't. |
| 09:45 |
|
sandbergja joined #evergreen |
| 09:47 |
Dyrcona |
hmm... yeah. Add a parameter to the stored procedure(s) to run the vacuum if true. Have storage calls to run the stored procedures. open-ils.auditor methods just call the storage methods. And none of this should be exposed to the public router. |
| 09:48 |
Dyrcona |
srfsh scripts could be run via cron to periodically clean up the auditor tables. |
| 10:19 |
JBoyer |
Something I've had in the back of my mind for a while is a housecleaning script that could handle cleanup for multiple things: auditor, report output retention, offline uploads, removing expired staged patrons, etc. No reason it couldn't just quiz opensrf.settings for the db connection info and go about whatever business we decide to give it. |
| 10:20 |
JBoyer |
That's about as far as I've taken the idea though, because *gestures around to a chorus of buzzing noises* |
| 10:22 |
Dyrcona |
JBoyer: yeah. I'm only thinking about auditor right now because we had a deadlock with an auditor cleanup script this week. it seems like something we ought to include with Evergreen if someone wants to use it or not. The other cleanups would be good, too. |
| 10:24 |
Dyrcona |
I could see the reporter cleanup being its own thing with settings in open-ils.reporter in opensrf.xml. I didn't think about adding settings for the auditor cleanup there, but it might make sense. |
| 10:24 |
* Dyrcona |
hunts for the cwmars script to remove old report output. |
| 10:25 |
Dyrcona |
Also, I get the buzzing noises, too. I've got 4 different conversations going on right now, including this one. :) |
| 10:29 |
|
kworstell-isl joined #evergreen |
| 10:30 |
|
redavis_reboot joined #evergreen |
| 10:34 |
pinesol |
News from commits: LP#2053245: fix Angular staff client test failure <https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=9aa81e3a89ff8631f5e026081a069e033a719d9e> |
| 10:51 |
|
stephengwills joined #evergreen |
| 10:56 |
|
collum joined #evergreen |
| 11:07 |
Dyrcona |
One thing that has come up in internal CW MARS discussion is that we need different intervals to clean data from different auditor tables. We keep 13 months on the biblio record entry history because we use it for a report. We keep 6 months on most tables. Then there's the hold request history where we need to keep 3 months or less because we purge certain holds at 3 months for privacy reasons. (NB: 6 months for most things was consi |
| 11:07 |
Dyrcona |
ered a decent compromise between privacy and needing data to investigate problems reported by members.) |
| 11:09 |
Dyrcona |
So, configuring that becomes a question. My first thought was org unit setting, but now I think a new table is required. It's not really an org. unit thing. It's global, and it needs to map a table to a duration. There could be a default setting somewhere. |
| 11:11 |
Dyrcona |
A new configuration table means a new staff client interface, though it could be built from fm-editor (or whatever it is called). |
| 11:13 |
Dyrcona |
Hm... If I add open-ils.auditor, I could also add an interface to create/drop auditor tables, and that could take care of setting up the config entries..... |
| 11:14 |
Bmagic |
Dyrcona++ |
| 11:15 |
Dyrcona |
<gopher>This is gonna run into money.</gopher> :) |
| 11:15 |
|
collum joined #evergreen |
| 11:15 |
Bmagic |
paper is looking better and better |
| 11:16 |
Dyrcona |
Bmagic: This doesn't solve our immediate problem, but I'm going to solve our immediate problem with some of this in mind, so the transition will be easy if my scribbles here ever become reality. |
| 11:16 |
Bmagic |
I'm excited |
| 11:17 |
Dyrcona |
Yeah, when people ask me about library automation lately (which they almost never do), I usually reply with "What's wrong with pen and paper?" |
| 11:17 |
Bmagic |
lol |
| 11:17 |
Bmagic |
answer "you can't erase pen" |
| 11:18 |
Dyrcona |
Well, use a pencil... :) |
| 11:19 |
Bmagic |
Knowing that's going to be their reply, you'll have to update your standard reply with pencil :) |
| 11:19 |
Dyrcona |
Yeah, guess I will. |
| 11:19 |
* mmorgan |
muses on 'pen'~'hardcoded' |
| 11:20 |
Bmagic |
$disks_price > $paper_price ? 'use paper' : 'use computers' |
| 11:21 |
Dyrcona |
Have you factored XML into the $disks_price? :) |
| 11:22 |
Bmagic |
$disks_price > ($paper_price - ($pencil_use * 5) + ($pen_use *-10) ) ? 'use paper' : 'use computers'; |
| 11:22 |
Bmagic |
there, I did it |
| 11:22 |
Dyrcona |
Heh. |
| 11:24 |
Dyrcona |
pen: write once read many. pencil: write many, read many, erase until the paper gets a hole. (Don't mention erasable ink. The paper wears out too fast.) |
| 11:24 |
Bmagic |
use 100 pound :) |
| 11:25 |
Dyrcona |
Wow! How do carry that around? :) Also, postage.... :) |
| 11:26 |
Bmagic |
https://en.wikipedia.org/wiki/Grammage |
| 11:27 |
Dyrcona |
I know... It's four times as thick as 25 pound bond, which is nice stuff. Most is 20 pound. |
| 11:28 |
Dyrcona |
The joke is, it is heavier paper. :) |
| 11:28 |
Bmagic |
:) |
| 11:28 |
Bmagic |
looks like we have a new variiable for our equation |
| 11:28 |
Dyrcona |
Yeahp. |
| 11:30 |
* Dyrcona |
is concerned that the cafe downstairs has not sent out a menu for today, nor have they said they are closed. I hope that means that they are open and the menu is the same as yesterday. |
| 11:30 |
Bmagic |
I hope so too :) |
| 11:31 |
Dyrcona |
Serious question: Where should a table to track auditor table retention go? The config schema? |
| 11:31 |
Bmagic |
I've never seen such a thing, let me know if you find it |
| 11:32 |
Dyrcona |
Well, I am going to make it. |
| 11:32 |
|
redavis joined #evergreen |
| 11:32 |
Dyrcona |
config.auditor_data_retention ( id sequence ..., table_name text, retention_interval interval); |
| 11:33 |
Dyrcona |
Also, how should a default interval be set, and should there even be a default value? |
| 11:34 |
Bmagic |
2 years is pretty conservative I would say |
| 11:34 |
Bmagic |
the real answer should be a function of table size rather than age |
| 11:34 |
Dyrcona |
Well, I was thinking more along the lines of an internal flag or org setting or a table name of 'default' or special id in the table. |
| 11:35 |
Dyrcona |
Bmagic: The idea is to give local sites a place where they can choose and set their values. |
| 11:35 |
Bmagic |
why delete the last 10 rows in the table even though they are 2 years old? |
| 11:36 |
Bmagic |
when I say last 10 rows, I mean the table has a total of 10 rows in it and after delete, it'd be an empty table |
| 11:36 |
Dyrcona |
Well, there are issues of privacy and relevance in some cases. For us, it was not about table size. |
| 11:36 |
Bmagic |
that's true. I'm speaking with a devops hat on |
| 11:36 |
Dyrcona |
Yeah, I get that. If the table is empty after delete, then you probably don't need that auditor any more. |
| 11:37 |
Dyrcona |
If the audited data hasn't changed in 2 years, then the table is probably obsolete. |
| 11:37 |
Stompro |
We normally purge by keeping the last x entries for each record, so that might be another option. |
| 11:38 |
Bmagic |
yeah, which takes us to a neat idea: completely automated auditor. Audit the whole database, and through time, trim the audit tables down to just those tables that are getting changed a lot |
| 11:38 |
Dyrcona |
Stompro: That's a lot more complicated. |
| 11:38 |
Dyrcona |
Bmagic: No. |
| 11:39 |
Bmagic |
haha |
| 11:39 |
Dyrcona |
Sometimes too much automation is too much automation. :) |
| 11:39 |
* eeevil |
lobs https://www.postgresql.org/docs/6.3/c0503.htm at Dyrcona and Bmagic and runs away |
| 11:40 |
Bmagic |
it's more like what people probably want. They don't know they want the history until they do. |
| 11:40 |
Dyrcona |
eeevil++ |
| 11:40 |
Bmagic |
eeevil++ |
| 11:41 |
Dyrcona |
Bmagic | Stompro: I'm thinking of adding some basic functionality to Evergreen that would be off by default. If sites want to continue to use their current solutions, they will be forced into the Evergreen way. |
| 11:41 |
Bmagic |
Dyrcona: sounds good to me. This is an area of the system that is lacking IMO |
| 11:42 |
Dyrcona |
That said, an option to keep the last X records could be useful. It could be set to 0. I wonder what kind of impact it would have on performance during clean up, though. |
| 11:43 |
Dyrcona |
"It could be set to 0." Means that sites that just want to clear by age could set it to 0. Other sites could set it to whatever value they want. I'm also thinking it isn't necessarily an either/or. You clean up by date, but still keep the last X records even if they're older. That's where it gets complicated and might get slow. |
| 11:44 |
berick |
we do a combo of delete older than X and keep at most Y. just on the copy auditor for now since it's a beast. |
| 11:44 |
eeevil |
fwiw, we do have precedent for "time horizon, or min number of old rows, whichever retains more" in the circ aging setup, IIRC. I personally care more about the last X edits of a row than the edits overs the last Y months, from a troubleshooting perspective |
| 11:45 |
Dyrcona |
Stompro++ berick++ eeevil++ Bmagic++ |
| 11:45 |
Dyrcona |
So, for the community code, I'll have to make a provision to keep x rows per id. |
| 11:46 |
* Dyrcona |
isn't promising anything gets done timely either. |
| 11:47 |
Stompro |
We normally get rid of everything older than x, and also trim based on number of entries for asset.copy, just to trim down those popular items that have circulated a ton recently. |
| 12:00 |
|
jihpringle joined #evergreen |
| 12:06 |
|
kworstell-isl joined #evergreen |
| 12:27 |
|
collum joined #evergreen |
| 13:07 |
|
kworstell-isl joined #evergreen |
| 13:16 |
|
kworstell-isl joined #evergreen |
| 13:40 |
csharp_ |
@band add Classic Rookie Mistake |
| 13:40 |
pinesol |
csharp_: Band 'Classic Rookie Mistake' added to list |
| 13:44 |
csharp_ |
@band is my favorite band, I don't care what [someone] says |
| 13:44 |
csharp_ |
pinesol: ahem |
| 13:44 |
pinesol |
csharp_: I just wanted to let you know that you mean a lot to me |
| 13:44 |
csharp_ |
@band is my favorite band, I don't care what [someone] says |
| 13:45 |
csharp_ |
pinesol: your dum |
| 13:45 |
pinesol |
csharp_: You're going to have to answer to the Coca-Cola company. |
| 13:45 |
Dyrcona |
Oof... I'm trying to move some acq auditor tables from the acq schema to the auditor schema, and the existing table columns do not agree with the columns in the table created by create_auditor(). |
| 13:46 |
csharp_ |
ah, I guess update_auditors() doesn't know about acq tables? |
| 13:46 |
Dyrcona |
[band] is my favorite band, and I don't care what pinesol says. |
| 13:46 |
csharp_ |
@band |
| 13:46 |
pinesol |
csharp_: Amish Cheese |
| 13:46 |
Dyrcona |
Well, that doesn't work either. |
| 13:47 |
Dyrcona |
@who is my favorite [band] |
| 13:47 |
pinesol |
csharp_ is your favorite Cookie Smuggler. |
| 13:47 |
Dyrcona |
Ha! |
| 13:47 |
csharp_ |
@band rules |
| 13:47 |
pinesol |
csharp_: Faraday Cage |
| 13:47 |
csharp_ |
huh - doesn't like any text after the command |
| 13:47 |
csharp_ |
I forgot about Cookie Smuggler |
| 13:48 |
csharp_ |
kmlussier++ |
| 13:48 |
kmlussier |
csharp_: What, what? Did I do something to get karma? |
| 13:49 |
csharp_ |
kmlussier: you smuggled cookies into the EG conference, so yeah! |
| 13:49 |
Dyrcona |
csharp_: update_auditors only looks at the auditor schema. I'm trying to move these old audit tables because I consider their existence to be a mistake. |
| 13:50 |
kmlussier |
Oh, lol. Still living off of my good deeds from nine months ago. |
| 13:50 |
Dyrcona |
kmlussier++ tuits++ |
| 13:51 |
* Dyrcona |
should open a Lp bug to eliminate the acq auditor functions. |
| 13:53 |
Dyrcona |
csharp_: I could adapt fix_columns to a do block that hits these two tables. That might work for my purposes. |
| 13:59 |
Dyrcona |
Well, my test database is now a mess. |
| 14:01 |
Dyrcona |
Grr. It's not that simple.... Of course it's not that simple...... |
| 14:05 |
Dyrcona |
Well, that did not fix anything. |
| 14:06 |
Dyrcona |
because columns are still in different orders...... |
| 14:12 |
Dyrcona |
Ok. It's working now. I have to specify the columns in the order that they appear in the new auditor table. |
| 14:19 |
Dyrcona |
ugh... I'm going to have to do this again on a different schema version. |
| 14:20 |
|
_collum joined #evergreen |
| 14:23 |
Dyrcona |
So, the insert ... select takes longer when you don't truncate the source table first. |
| 14:29 |
Dyrcona |
Looks like the schema hasn't really changed between the two releases that I have test data for. |
| 14:33 |
csharp_ |
gmcharlt_: Rogan: in the Zoom waiting room for board meeting in case you can see this |
| 14:33 |
csharp_ |
or kmlussier or redavis |
| 14:33 |
Rogan |
csharp_: I'll mention it |
| 14:33 |
csharp_ |
Rogan: thank you |
| 14:41 |
|
collum joined #evergreen |
| 14:49 |
|
collum joined #evergreen |
| 15:03 |
Dyrcona |
heh. Test it on a database that hasn't had a refresh in over 6 months and all the rows disappear! That's a 100% space savings on that 1 table alone. :) |
| 15:17 |
Bmagic |
space good; cougar bad |
| 15:20 |
jeff |
space good. |
| 15:20 |
jeff |
sadly, this update wasn't nearly as exciting as it could have been: Introducing new Space Manager capabilities in Google Chat |
| 15:22 |
* Dyrcona |
got the Stim upgrade.... Aahhhhh.... |
| 15:29 |
Dyrcona |
@decide tonight or next week |
| 15:29 |
pinesol |
Dyrcona: go with tonight |
| 15:30 |
Dyrcona |
Why not? It won't make a difference. |
| 15:31 |
Dyrcona |
I wonder if shoving 8.8 million rows around will wreak havoc on the streaming replication.... Maybe I should delete most of the rows first, or will that make a difference, since the delete also needs to be replicated. |
| 15:41 |
Dyrcona |
Yeah. I think we should do away with acq.create_acq_auditor. |
| 15:46 |
|
collum joined #evergreen |
| 15:49 |
Dyrcona |
Bmagic: I've scheduled the SQL update to move the acq audit tables for 9:00 PM EST tonight. We'll see if PostgreSQL loses its mind, thought I doubt it. |
| 15:49 |
Dyrcona |
s/thought/though/ |
| 15:51 |
Bmagic |
Dyrcona: https://imgur.com/a/05CngrU |
| 15:51 |
Dyrcona |
heh. |
| 15:52 |
Dyrcona |
Well, it's tested as far as it works.... I'm just not sure if the streaming replication is gong to like it. |
| 15:52 |
Dyrcona |
I mean, I ran it on 4 databases..... :) |
| 15:53 |
Bmagic |
alright, I retract my link |
| 15:54 |
|
mantis left #evergreen |
| 16:05 |
|
jihpringle joined #evergreen |
| 16:15 |
* Dyrcona |
prepares to go home. |
| 16:15 |
Dyrcona |
Catch ya'll later! |
| 16:17 |
|
collum joined #evergreen |
| 16:43 |
|
collum joined #evergreen |
| 16:45 |
|
stephengwills left #evergreen |
| 16:57 |
|
mmorgan1 joined #evergreen |
| 17:09 |
|
mmorgan1 left #evergreen |
| 17:36 |
Bmagic |
does anyone have an automated process for "clearing" vandelay.session_tracker rows that are stuck "active" ? |
| 17:37 |
Bmagic |
I was thinking about a cron job to simply update the state to "error" when the session is older than say, 1 week, but never got switched to complete |
| 17:52 |
|
kmlussier left #evergreen |
| 18:06 |
|
collum joined #evergreen |
| 18:24 |
|
collum joined #evergreen |
| 18:32 |
|
jihpringle joined #evergreen |
| 19:36 |
|
jihpringle joined #evergreen |
