| Time |
Nick |
Message |
| 03:50 |
|
akilsdonk joined #evergreen |
| 05:01 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 07:14 |
|
rjackson_isl joined #evergreen |
| 07:18 |
|
JBoyer joined #evergreen |
| 07:33 |
|
agoben joined #evergreen |
| 07:50 |
phasefx |
morning. SSL cert expired for evergreen-ils.org |
| 08:26 |
|
collum joined #evergreen |
| 08:27 |
csharp |
looks like you can set up a cron to automatically renew the cert |
| 08:30 |
phasefx |
sweet |
| 08:38 |
csharp |
okay - the cert is renewed |
| 08:38 |
csharp |
I'll set up a cron that just does that every 90 days |
| 08:41 |
csharp |
actually the docs recommend running twice a day |
| 08:43 |
csharp |
done |
| 08:44 |
csharp |
set to run at 7:04 a.m. and 7:04 p.m. |
| 08:47 |
|
mmorgan joined #evergreen |
| 08:50 |
phasefx |
csharp++ |
| 09:01 |
|
Dyrcona joined #evergreen |
| 09:11 |
|
kmlussier joined #evergreen |
| 09:29 |
|
maryj joined #evergreen |
| 09:29 |
|
yboston joined #evergreen |
| 09:31 |
|
jvwoolf joined #evergreen |
| 09:34 |
Dyrcona |
RE e-commerce: Is there a current diagram that shows all cardholder data flows across systems and networks? |
| 09:36 |
bshum |
csharp++ # cert fixing |
| 10:01 |
berick |
Dyrcona: not aware of any diagrams. the card data goes from the browser to the EG server, then to the e-com provider. (assuming non-Stripe setup) |
| 10:03 |
|
mmorgan1 joined #evergreen |
| 10:03 |
|
Christineb joined #evergreen |
| 10:04 |
jeff |
fewer diagrams! more code removal! |
| 10:30 |
|
mmorgan joined #evergreen |
| 10:53 |
|
bos20k joined #evergreen |
| 11:05 |
remingtron |
What's the proper way to submit a pull request to OpenSRF? I don't see a working repo. I just have two simple README changes. |
| 11:05 |
bshum |
remingtron: There is a working repo |
| 11:06 |
bshum |
http://git.evergreen-ils.org/?p=working/OpenSRF.git;a=summary |
| 11:06 |
bshum |
And there's a Launchpad for OpenSRF too |
| 11:06 |
bshum |
https://launchpad.net/opensrf |
| 11:06 |
remingtron |
bshum: thanks! I'll start with a launchpad bug, then see if I have access to the working repo. |
| 11:06 |
Dyrcona |
And if you have access to the Evergreen working repo, you have access to the OpenSRF working repo. |
| 11:06 |
bshum |
To my recollection, if you have working repo access for Evergreen, OpenSRF is granted too |
| 11:06 |
bshum |
Jinx |
| 11:06 |
Dyrcona |
:P |
| 11:08 |
remingtron |
bshum: Dyrcona: thanks for double-confirming |
| 11:09 |
Dyrcona |
remingtron: NP, and if for some reason you find that you don't have access to the OpenSRF repo, let me know and I can fix it. |
| 11:10 |
remingtron |
Dyrcona: will do, thanks |
| 11:17 |
Dyrcona |
It would surprise me if you don't have access. |
| 11:31 |
remingtron |
Dyrcona: it worked fine |
| 11:33 |
Dyrcona |
Yeah.... |
| 11:33 |
|
sandbergja joined #evergreen |
| 11:34 |
Dyrcona |
Basically, if you have access to the working repos, you have access to working/Evergreen, working/OpenSRF, working/SIPServer, working/NCIPServer, working/random, and working/Hatch. |
| 11:46 |
|
brahmina joined #evergreen |
| 12:04 |
|
jihpringle joined #evergreen |
| 12:07 |
|
mmorgan joined #evergreen |
| 12:15 |
|
hbrennan joined #evergreen |
| 12:34 |
Bmagic |
This might seem like a dumb question, how can I get an idea of how postgres is doing? Like health, indexes, memory needs? I am seeing queries taking a long time, but not all queries. There is no swap space, which is something new. |
| 12:35 |
Bmagic |
Does postgres do well when the memory is full, write to disk and bring needed items into memory, etc? Or does it let the OS manage that? Without swap space, I could see that being an issue |
| 12:58 |
bshum |
Bmagic: So maybe pg stats? https://www.postgresql.org/docs/9.3/static/view-pg-stats.html |
| 12:59 |
Bmagic |
ah! I missed that |
| 12:59 |
Bmagic |
oh, lol, I thought it was a binary. Yeah, the table. I've looked at that |
| 13:02 |
bshum |
Ah, gotcha |
| 13:02 |
bshum |
And you've probably read gmcharlt's https://galencharlton.com/blog/2010/05/database-server-disk-space-usage-in-evergreen/ |
| 13:02 |
bshum |
I find that if your DB is bigger than your memory, it's going to strike the disk far often than not, and that slows stuff down tremendously |
| 13:02 |
bshum |
But some tables and indexes are more important than others |
| 13:03 |
* bshum |
doesnt' think speed is all that important to check the audit tables |
| 13:05 |
Bmagic |
I am considering adding swap to the server. I wonder what impact that would have while postgres is running. |
| 13:10 |
Bmagic |
DB size on disk is 143GB. Memory on the machine is 102GB. Clearly it is going to disk. Luckily, we have a disk that performs at 90k iops. top shows a io wait time of 0.0. I'm not sure what is really going on. Some tables perform better than others. |
| 13:11 |
berick |
Bmagic: how much ram is cached? ('cached Mem' in top) |
| 13:12 |
Bmagic |
99GB |
| 13:12 |
berick |
ok, good, just making sure you weren't out of accessible ram |
| 13:12 |
Bmagic |
it says 90GB available |
| 13:12 |
berick |
Bmagic: you do want to have some swap space on the server, though. |
| 13:13 |
Bmagic |
berick: I have read conflicting things about that |
| 13:18 |
berick |
Bmagic: do tell |
| 13:21 |
Bmagic |
I'll be the first to admit, there are things going on with the kernel and postgres that I just plain don't understand. Maybe the answer is having swap space but a tweak to /proc/sys/vm/swappiness instead? |
| 13:26 |
berick |
Bmagic: yeah, maybe, i mean certainly the goal is not to rely heavily on swap, but having no swap at all seems extreme. |
| 13:27 |
Bmagic |
It's the default for VM images now from Amazon/Google - no swap |
| 13:27 |
berick |
interesting |
| 13:29 |
Dyrcona |
No swap on a workstation makes sense, but a server should still have some. |
| 13:29 |
Dyrcona |
Just my opinion.... |
| 13:30 |
bshum |
I kept my swap, but just made it smaller than the "suggested" defaults from the Ubuntu installer. |
| 13:30 |
* berick |
sees the aws host his irc is running from does not have swap |
| 13:31 |
berick |
Dyrcona: right... that's kind of what I'm thinking. |
| 13:32 |
Dyrcona |
But, maybe it does no good in the "Cloud..." |
| 13:32 |
jeff |
on an instance where your swap partition lives on the network... |
| 13:37 |
csharp |
Bmagic: I recommend check_postgres: https://bucardo.org/wiki/Check_postgres - which you can use with Nagios (and probably other monitoring frameworks) to monitor things |
| 13:37 |
csharp |
Bmagic: also pg_top (known as "pgtop" in the debian/ubuntu repos) |
| 13:38 |
Bmagic |
thanks yall! |
| 13:38 |
berick |
csharp++ # actual stuff to look a |
| 13:38 |
berick |
t |
| 13:42 |
Bmagic |
I have a query that is taking 30+ minutes (I haven't seen it finish) - it's a clark kent query. Other reports finish instantly. But this one referencing action.hold_request, action.circulation, asset.copy, actor.org_unit, action.hold_transit_copy just will not finish |
| 13:44 |
Bmagic |
combine that behavior with another query that I saw take an enormous amount of time, I think there is a database issue of some kind. The no swap space is something that I can see right away, but I wonder.. Will adding swap space make these queries work? I doubt it |
| 13:45 |
jeff |
have you looked at pg_stat_activity for more information on your current long-running query? |
| 13:47 |
Bmagic |
jeff: I did, but I don't see anything helpful there |
| 13:50 |
jeff |
for the query in question, is pg_stat_activity.waiting = t? |
| 13:51 |
Bmagic |
waiting='f' |
| 13:51 |
berick |
Bmagic: it's easy to build queries in the reporter that are pathologically slow. not saying you shouldn't try to make them faster, but a long-running reporter query does not necessarily mean there's a problem w/ the DB. |
| 13:52 |
jeff |
especially if the reporter query is new and... problematic to begin with. :-) |
| 13:52 |
Bmagic |
berick: I generally agree, but I have been seeing this behavior on this new db server, where it didn't come to my attention on the old db server |
| 13:52 |
berick |
Bmagic: gotcha |
| 13:52 |
Bmagic |
The template in question is old and used heavily |
| 13:52 |
jeff |
and you're sure this particular report query is not a new one? |
| 13:52 |
jeff |
ah. |
| 13:53 |
Dyrcona |
Bmagic: Different version of Pg? |
| 13:53 |
Bmagic |
same version |
| 13:53 |
Bmagic |
same database files in fact |
| 13:54 |
Bmagic |
I mean, the file system in /var/lib/postgresql was mirrored |
| 13:54 |
* jeff |
nods |
| 13:54 |
Dyrcona |
Yeah. |
| 13:55 |
Bmagic |
I suppose I can drop the swap on there and "See what happens" - I am worried that adding swap space live (via swapon) could do something bad to postgres while running |
| 13:56 |
Dyrcona |
It shouldn't. |
| 13:57 |
Bmagic |
I tend to agree, it shouldn't. It's just that I didn't write the swapon software, and really have no idea what it does. In god we trust I suppose |
| 13:58 |
Dyrcona |
It just says, "Hey, kernel! Here's some (more) swap to use." |
| 13:58 |
bshum |
But who hasn't crashed their database server whilst "tweaking"? It's like a rite of passage Bmagic, embrace it! |
| 13:58 |
Bmagic |
lol |
| 13:58 |
hbrennan |
haha |
| 13:58 |
Bmagic |
I can't wait |
| 13:59 |
berick |
considering how my usable ram you have, i'd be surprised if adding swap helped anything. |
| 13:59 |
bshum |
"it'll be fine" -- famous last words |
| 13:59 |
Bmagic |
stop hitting refresh on your web browser...... |
| 14:10 |
csharp |
https://i.ytimg.com/vi/UoStgb4P6xU/hqdefault.jpg |
| 14:11 |
remingtron |
csharp: that was a little underwhelming |
| 14:11 |
csharp |
heh |
| 14:11 |
csharp |
http://knowyourmeme.com/memes/this-is-fine |
| 14:12 |
|
abowling joined #evergreen |
| 14:12 |
remingtron |
csharp: ah, makes much more sense now |
| 15:02 |
|
mmorgan1 joined #evergreen |
| 15:10 |
Bmagic |
an hour later, it's using 472k of the swap lol |
| 15:10 |
Bmagic |
and the flowers are still standing |
| 15:12 |
Bmagic |
I'll bet my postgresql.conf settings needs a tweak |
| 15:13 |
Bmagic |
constraint_exclusion = on, checkpoint_completion_target = 0.9, effective_cache_size = 160GB, work_mem = 224MB, wal_buffers = 8MB, shared_buffers = 8GB |
| 15:17 |
Bmagic |
this page probably needs some TLC https://wiki.evergreen-ils.org/doku.php?id=documentation:performance |
| 15:25 |
Dyrcona |
It probably does, and I'm not so sure about the turn hyperthreading off suggestion. |
| 15:30 |
bshum |
That was mostly cause it was doing something funny with our particular version of PG and Ubuntu at the time |
| 15:30 |
bshum |
And I think the #postgresql guys were telling me to avoid it |
| 15:30 |
bshum |
https://www.postgresql.org/message-id/53F4F36E.6050003@agliodbs.com |
| 15:30 |
bshum |
For reference |
| 15:30 |
bshum |
But hey that was two years ago |
| 15:30 |
bshum |
Maybe things are better now? |
| 15:32 |
Dyrcona |
Well, there seems to be a difference of opinion... Bruce doesn't necessarily agree with Josh B. And pgbench is not necessarily a normal load.... ;) |
| 15:32 |
* bshum |
shrugs |
| 15:32 |
Dyrcona |
Also, I just googled this topic again and much of what is coming up is that newer Intel HT is much better and it adds to performance, but not as much as a full CPU. |
| 15:37 |
Dyrcona |
It is difficult to find a definitive answer. |
| 15:53 |
|
mmorgan joined #evergreen |
| 15:54 |
Bmagic |
Ok, so I think I might have found an issue with this query |
| 15:54 |
pastebot |
"Bmagic" at 64.57.241.14 pasted "SELECT * FROM (SELECT "002ba6d" (16 lines) at http://paste.evergreen-ils.org/33 |
| 15:55 |
Bmagic |
I think* the action.circulation table needs to be connected to asset.copy to prevent the (each row in action.circulation duplicating for each row in asset.copy) |
| 15:57 |
* tsbere |
isn't sure what that is supposed to check in the first place, so can't really comment on what is wrong ;) |
| 15:57 |
Bmagic |
The title of the report is: Checkins by Date and Checkin Library |
| 15:58 |
tsbere |
Ok. Then each circulation/copy combo should exist. Why are transits in there? |
| 15:59 |
Dyrcona |
Yeah, that query is some kind of broken. :) |
| 15:59 |
Bmagic |
the source table is transit_copy |
| 15:59 |
tsbere |
Why? |
| 15:59 |
Dyrcona |
Why is the ...... |
| 15:59 |
Dyrcona |
:) |
| 15:59 |
tsbere |
Checkins don't always generate transits after all. |
| 15:59 |
Bmagic |
I didn't construct the template |
| 15:59 |
Bmagic |
I really dont know, but yeah, I think it's the template (even though it has been running for two years) |
| 16:00 |
Dyrcona |
In that case, I'd find out what the actual goal of the report is and make a new template. |
| 16:00 |
Bmagic |
I have been chasing the database issues or lack there of, on the assumption that this worked before and now it doesnt. But the query isn't looking stellar. The question still remains, why did this work before |
| 16:01 |
Dyrcona |
How do you know it did work before? |
| 16:01 |
tsbere |
I would start by assuming it didn't work before, they just thought it did. Go from there ;) |
| 16:02 |
Bmagic |
the template was created in 2014.... so I assumed |
| 16:03 |
Dyrcona |
It *may* be retrieving the correct information, but the performance is your main concern at the moment, yes? |
| 16:03 |
Bmagic |
but, lol, select * from reporter.schedule where report in(....) yields NOTHING. |
| 16:03 |
Dyrcona |
Ah, in that case.... ;) |
| 16:04 |
Bmagic |
Dyrcona: yes I am concerned about performance, and I had this hard example to work from... It turns out, it's not a hard example |
| 16:05 |
Dyrcona |
That reminds me.... I wanted to bug someone here about a query that they were running on our reports database last week that used about 150GB in pgsql_tmp before we killed it. |
| 16:09 |
Dyrcona |
"SELECT * FROM (SELECT ...)" always looks funny to me. :) |
| 16:23 |
tsbere |
Dyrcona: Sometimes it is useful to apply an extra filter. |
| 16:23 |
tsbere |
Especially on things like window functions |
| 16:23 |
Dyrcona |
Right, but it still looks funny. |
| 16:28 |
mmorgan |
Does the reshelving completer add a row to the auditor.asset_copy_history table? |
| 16:28 |
tsbere |
I believe it does not |
| 16:28 |
tsbere |
I could be wrong though |
| 16:29 |
* mmorgan |
is also suspecting it does not. |
| 16:33 |
mmorgan |
Nope. definitely does not add a row to copy history. |
| 16:35 |
tsbere |
mmorgan: I was wrong, for MVLC it does. Maybe someone modified your auditor function. |
| 16:36 |
mmorgan |
Hmm. Don't think so. Looking more closely. |
| 16:36 |
tsbere |
mmorgan: It is the status changed time update trigger that skips on Reshelving -> Available |
| 16:39 |
mmorgan |
Ah. Ok. And also the edit_date is not updated. But the row is indeed in the auditor table. |
| 16:40 |
mmorgan |
And the audit_user and audit_ws are NULL for that row. |
| 16:41 |
tsbere |
Well, yea. The reshelving completer doesn't log in. ;) |
| 16:43 |
mmorgan |
tsbere++ |
| 16:45 |
dbwells |
We have some fairly recent rows in actor.usr with "deleted = 't'" rather than the full purge. Does anyone know offhand what process might set the deleted flag in actor.usr? |
| 16:45 |
bshum |
dbwells: Merging? |
| 16:46 |
bshum |
Is the card field empty? I thought merging user A and B if A becomes lead would mean B gets "deleted" and stuff reassociated |
| 16:46 |
bshum |
I don't think it completely nullifies the row otherwise. I remember usrname staying put, for example |
| 16:46 |
bshum |
And causing annoyances when people tried to re-use ones that were stuck on deleted users |
| 16:47 |
bshum |
"Obliterate" that's the word I remember. |
| 16:47 |
bshum |
It isn't obliterated |
| 16:47 |
dbwells |
could be. If so, do you know why it doesn't purge, or is it just something never changed after purge was added? |
| 16:48 |
berick |
i think it was just never changed |
| 16:48 |
* bshum |
doesn't remember when purge was added |
| 16:48 |
berick |
it should probably purge |
| 16:48 |
bshum |
+1 |
| 16:49 |
dbwells |
I agree, that would be nice from where I am sitting. |
| 16:49 |
Dyrcona |
dbwells: You're not using the alt_patron_summary branch, right? |
| 16:50 |
dbwells |
No |
| 16:50 |
Dyrcona |
OK, then. |
| 16:51 |
Dyrcona |
I asked, 'cause it doesn't purge patrons, AFAICT. |
| 16:53 |
dbwells |
Well, I will throw something on LP to record the idea of purge-on-merge behavior, as it seems desirable to at least some present folk. |
| 16:54 |
berick |
dbwells++ |
| 17:01 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 17:01 |
|
jvwoolf left #evergreen |
| 17:03 |
|
mmorgan left #evergreen |
| 17:54 |
|
abowling left #evergreen |
| 17:55 |
Bmagic |
So, emailing checkout receipts. I see the action trigger but not seeing rows getting added to action_trigger.event after checkout. I need to set something up somewhere? |
| 18:15 |
bshum |
Bmagic: Are they opted in? |
| 18:15 |
bshum |
I only ask cause in 3.3.4 Email checkout receipts in https://evergreen-ils.org/documentation/release/RELEASE_NOTES_2_11.html it says the patrons opt in, so I assume there's a setting that's checked |
| 18:16 |
* bshum |
doesn't know how the trigger is defined |
| 18:18 |
bshum |
Yeah, there's an opt in setting field check for circ.send_email_checkout_receipts |
| 21:28 |
Bmagic |
bshum: yep, opted in prior to finishing checkout |
| 23:55 |
bshum |
Bmagic: And the user definitely has an email address associated with their account too? Hmm |
