| Time |
Nick |
Message |
| 02:26 |
|
remingtron_ joined #evergreen |
| 04:03 |
|
Stompro joined #evergreen |
| 04:05 |
|
Stompro joined #evergreen |
| 04:31 |
|
book` joined #evergreen |
| 04:31 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 06:40 |
|
rlefaive joined #evergreen |
| 07:13 |
|
rjackson_isl joined #evergreen |
| 08:37 |
|
mmorgan joined #evergreen |
| 08:45 |
|
bos20k joined #evergreen |
| 08:46 |
remingtron |
ejk: Did anyone respond to your OpenSRF gateway question from yesterday? |
| 08:48 |
remingtron |
ejk: if you have code you can share, I bet someone would take a quick look at it. |
| 08:50 |
ejk |
Thanks, not in the office yet but I'll try to find something later this morning. I think the key is figuring out how to format the objects when they are method arguments |
| 08:56 |
|
_adb joined #evergreen |
| 09:04 |
|
kmlussier joined #evergreen |
| 09:14 |
|
yboston joined #evergreen |
| 09:15 |
|
Dyrcona joined #evergreen |
| 09:19 |
Bmagic |
Am I hearing things or do I remember someone telling me they had a SQL script that really removes rows from biblio.record_entry ? |
| 09:29 |
|
Dyrcona joined #evergreen |
| 09:33 |
bshum |
Bmagic: Dyrcona wrote one for Bibliomation when I was there, by really remove we mean dropping constraints and getting rid of rows for bibs with nothing interesting attached to them |
| 09:34 |
bshum |
So I guess it depends on what you mean by "really remove" too :) |
| 09:34 |
bshum |
And for what rows |
| 09:34 |
Dyrcona |
What I wrote for Bibliomation only removed bibs with no copies, etc. They had some issues from migration. |
| 09:36 |
rhamby |
That's pretty straight forward, just drop the rule protecting deletion. |
| 09:36 |
rhamby |
then re-enable it when you're done |
| 09:36 |
* Dyrcona |
needs to check the logs. |
| 09:37 |
rhamby |
or if you choose to not re-enable it you can live dangerously and see if anything running does deletes instead of updates .... (I don't like living dangrously) |
| 09:37 |
Dyrcona |
rhamby: It's not that simple. :) |
| 09:38 |
Dyrcona |
There are constraints that prevent deletion, not just the trigger. |
| 09:38 |
rhamby |
true, but none are hard to disable |
| 09:38 |
Dyrcona |
if you don't also delete everything that references the row, you're going to have serious breakage. |
| 09:39 |
Dyrcona |
Well, what happens is, you spend several hours playing whack-a-mole with the various constraints and things that need to be disabled and deleted. |
| 09:39 |
rhamby |
sure, but it's not terribly hard, I"ve done it when removing org units and their bibs |
| 09:40 |
Dyrcona |
I don't recommend it. |
| 09:40 |
Dyrcona |
Storage is cheap. :) |
| 09:41 |
rhamby |
if you do have to do it there are steps you'll want to do it in, it's simpler if there aren't volumes or copies but if there are you just have to do some things in particular order |
| 09:42 |
|
_bott_ joined #evergreen |
| 09:45 |
rhamby |
I thought I remembered it and I do, the Equinox migration repo has org unit removal scripts that contain examples of isolating and removing bibs. it's for whole org units but could easily be split up and adapted. |
| 09:45 |
csharp |
has anyone written a script that parses 'osrf_control --diagnostic' that could be used for nagios-ish monitoring? |
| 09:46 |
Dyrcona |
rhamby: I'll check that repository for those scripts later. We might find those handy. |
| 09:47 |
rhamby |
Dyrcona: they are, I periodically update them as I do projects. I just pushed a few updates yesterday in fact though I admit I mostly update them as I run into things rather than do comprehensive review of possible schema changes. |
| 09:47 |
Dyrcona |
I'd like to recommend the migration_tools repo. We use the staged bib loader for our quarterly Backstage updates. |
| 09:47 |
Dyrcona |
The above was a general statement for the benefit of the channel. :) |
| 09:50 |
rhamby |
:) |
| 09:53 |
csharp |
hmm - maybe just 'osrf_control --diagnostic | grep ERR' and alert if it comes back with something? |
| 09:53 |
Dyrcona |
csharp: That might work. |
| 10:08 |
Bmagic |
Dyrcona rhanby - My reasons is to lighten up the total size for transport. I can't be certain but I have a theory that we could significantly reduce the volume of our cold backups if we weren't tugging a ton of useless data. |
| 10:08 |
Bmagic |
It would mean less bandwidth, faster backup times, and just all around easier to work with |
| 10:09 |
Bmagic |
The deleted=true represents half of biblio.record_entry |
| 10:11 |
Dyrcona |
Bmagic: What it would really mean is less data in metabib.real_full_rec and a smaller metabib_full_rec_index_vector_idx which leads to faster restore times. :) |
| 10:11 |
Bmagic |
rhamby: bre rows are connected all over the place, metabib and friends, asset, action, action_trigger, etc. |
| 10:12 |
rhamby |
bmagic: yep, but you can delete things in proper order, as I said I've done it and it's even in the scripts |
| 10:13 |
Bmagic |
Awesome, where do I go to take a peak? migration_tools ? |
| 10:13 |
Dyrcona |
And, I can send you the script I did for bibliomation. |
| 10:14 |
Bmagic |
sweet! Thanks guys |
| 10:14 |
Bmagic |
rhamby++ |
| 10:14 |
Bmagic |
Dyrcona++ |
| 10:14 |
|
Christineb joined #evergreen |
| 10:14 |
* Bmagic |
waves at Christineb |
| 10:15 |
rhamby |
bmagic: yeah, look at the migration tools repo, it's named something obvioius like 'remove_ou' or something like that, it's a series of scripts i |
| 10:15 |
Bmagic |
ty |
| 10:15 |
rhamby |
also note that I can not type correctly today /ug |
| 10:16 |
rhamby |
if I recall correctly the last time I used them was against 2.11 so there shouldn't be anything major in the schema at that time to roadblock them |
| 10:16 |
Bmagic |
even better |
| 10:17 |
Dyrcona |
I think it is OK for me to share the script as a gist. It's a few years old so it may miss a few things. |
| 10:18 |
pinesol_green |
[evergreen|Bill Erickson] LP#1708510 Postgres 9.6 full text search config - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=aa139de> |
| 10:18 |
Dyrcona |
I just did a count and as of 7/31, deleted bibs make up about 1/4 of our bre table. |
| 10:19 |
Dyrcona |
I counted on a copy not the actual production database. |
| 10:19 |
kmlussier |
Maybe I'm the only one who doesn't know this, but where exactly is the migration tools repo? |
| 10:20 |
rhamby |
kmlussier http://git.esilibrary.com/?p=migration-tools.git;a=summary |
| 10:20 |
kmlussier |
rhamby: Thanks! |
| 10:21 |
mmorgan |
kmlussier: You weren't the only one who didn't know ;-) |
| 10:21 |
mmorgan |
rhamby++ |
| 10:25 |
Dyrcona |
Bmagic: https://gist.github.com/Dyrcona/e9b69ac5281502b6755a80447af50504 |
| 10:25 |
Bmagic |
Dyrcona: TY! |
| 10:30 |
Bmagic |
ha! I have 111 rows in bre where id< -1 and deleted='t' |
| 10:30 |
Dyrcona |
Bmagic: yeah, I'm not sure how that happens, but I've seen those bib ids < -1 before. |
| 10:31 |
Bmagic |
precats? |
| 10:31 |
Dyrcona |
At one point, I think I knew what was doing it. |
| 10:31 |
Dyrcona |
yeah, maybe precats or something like that. |
| 10:31 |
berick |
there's 1 precat bib they all use. probably not that. |
| 10:31 |
Bmagic |
oh yeah, that's -1 |
| 10:32 |
Bmagic |
starting on -2, the system creates dummy bibs for some reason, which I used to know.... |
| 10:34 |
Bmagic |
Dyrcona: your select statement identified 859k bibs out of 2 million. That is almost 50% of our bibs! I seriously believe that cutting these out of our database would be a significant improvement. |
| 10:34 |
Dyrcona |
Yes, I say that could well be worth it. |
| 10:37 |
* berick |
is looking forward to purging a huge pile of action_trigger data |
| 10:37 |
Dyrcona |
Instead of using the HAVING clause, I'd probably do that with and <tablename> is null in the where clause today. |
| 10:37 |
Dyrcona |
FYI: That script is from March of 2014, so some things have likely changed since then as far as triggers and rules. |
| 10:45 |
|
agoben joined #evergreen |
| 10:51 |
* csharp |
finds 293,926 bibs with select from Dyrcona's gist |
| 11:20 |
Dyrcona |
phasefx++ |
| 11:20 |
Dyrcona |
phasefx: I answered Eva off list, but I originally thought the message was going to the list. |
| 11:21 |
Dyrcona |
Unfortunately, I have not had time to work on Lp 1131238. |
| 11:21 |
pinesol_green |
Launchpad bug 1131238 in Evergreen "Move MARC templates into the database" [Wishlist,Triaged] https://launchpad.net/bugs/1131238 |
| 11:22 |
phasefx |
Dyrcona: thanks man; never enough tuits :) |
| 11:25 |
Dyrcona |
Or the tuits are the wrong shape. :) |
| 11:27 |
|
StomproJ joined #evergreen |
| 11:31 |
berick |
these 4-dimensional hyper-cube tuits are great, but most of the code is inaccessible |
| 11:38 |
Dyrcona |
heh. |
| 11:42 |
Dyrcona |
"So much time and so little to do. Wait a minute. Strike that. Reverse it. " |
| 11:44 |
|
kmlussier joined #evergreen |
| 12:00 |
|
khuckins joined #evergreen |
| 12:12 |
gsams |
Bmagic: You have more deleted records than I have records total. |
| 12:13 |
Bmagic |
gsams: bibs happen |
| 12:14 |
gsams |
yeah, we've got about 15% in that boat here |
| 12:30 |
Dyrcona |
I haven't actually checked. I'm sure most of our deleted bibs had copies or uris at some time. |
| 12:46 |
abowling |
apt-key add - |
| 12:46 |
abowling |
^and it's friday. d'oh! |
| 12:50 |
Dyrcona |
Whee! |
| 13:27 |
|
yeogai joined #evergreen |
| 13:52 |
gmcharlt |
the web staff serials branch now has a pull request; https://bugs.launchpad.net/evergreen/+bug/1708291 |
| 13:52 |
pinesol_green |
Launchpad bug 1708291 in Evergreen "webstaff: serials module" [Wishlist,New] - Assigned to Galen Charlton (gmc) |
| 13:52 |
kmlussier |
gmcharlt++ |
| 13:53 |
csharp |
created a simple Nagios check for OpenSRF Diagnostics: http://git.evergreen-ils.org/?p=contrib/pines.git;a=blob;f=nagios/check_osrf;h=395d107be79a104e6257efd9d6c45e131d6d4d26;hb=HEAD |
| 13:55 |
_adb |
csharp++ |
| 13:55 |
Dyrcona |
csharp++ |
| 14:05 |
_adb |
we have a different nagios check in place. i inherited it, i didn't write it. it looks a lot more complicated than yours: https://gist.github.com/anonymous/5ca7804d3b43aef1c17decbf2448b0c5 |
| 14:23 |
csharp |
_adb: interesting |
| 14:24 |
* csharp |
suspects miker or berick authorship |
| 14:25 |
* berick |
takes 2 steps back |
| 14:25 |
csharp |
heh |
| 14:26 |
_adb |
well, it hasn't caused any problems :-) i have no idea where it came from originally |
| 14:28 |
_adb |
https://github.com/evergreen-library-system/OpenSRF/blob/master/examples/nagios/check_osrf_services looks about right. tsbere++ |
| 14:36 |
csharp |
tsbere++ |
| 14:36 |
tsbere |
I remember that. I think we kept having listeners stop listening on us, but to other checks they looked fine. |
| 15:01 |
|
khuckins_ joined #evergreen |
| 15:02 |
|
sandbergja joined #evergreen |
| 15:14 |
|
kmlussier joined #evergreen |
| 15:29 |
|
kmlussier joined #evergreen |
| 15:34 |
|
StomproJosh joined #evergreen |
| 15:40 |
|
khuckins__ joined #evergreen |
| 16:25 |
gmcharlt |
The tracking page for next week's feedback fest is ready - https://wiki.evergreen-ils.org/doku.php?id=dev:3.0:feedback_fest_2 |
| 16:26 |
Dyrcona |
gmcharlt++ |
| 16:28 |
abneiman |
gmcharlt++ |
| 16:32 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 17:04 |
|
mmorgan left #evergreen |
| 17:04 |
gmcharlt |
https://evergreen-ils.org/evergreen-3-0-development-update-13-let-the-fest-begin-again/ |
| 19:38 |
|
JBoyer_alt joined #evergreen |
| 20:03 |
|
remingtron_ joined #evergreen |
