| Time |
Nick |
Message |
| 06:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 06:19 |
|
jonadab joined #evergreen |
| 07:14 |
|
rjackson_isl_hom joined #evergreen |
| 07:44 |
|
pinesol` joined #evergreen |
| 07:46 |
|
rjackson_isl_hom joined #evergreen |
| 07:53 |
|
collum joined #evergreen |
| 08:42 |
|
mantis1 joined #evergreen |
| 09:10 |
|
Dyrcona joined #evergreen |
| 09:11 |
Dyrcona |
Well, I tried resetting my Chrome settings to the default and tabs are still not dragging in the same window. They always pop out as soon as i drag. I should check on my other laptop to see if it is happening in Chromium there. |
| 09:15 |
Dyrcona |
Hm. I unchecked "Use System Title Bar and Borders" and it works again. However, I have always used the system tittle bar and borders, so looks like it was a change in Gnome on Ubuntu that broke Chrome. |
| 09:15 |
Dyrcona |
@blame Canonical |
| 09:15 |
pinesol |
Dyrcona: Canonical musta been an Apple employee. |
| 09:15 |
Dyrcona |
Ha! |
| 09:29 |
csharp_ |
so this has probably been answered before, but why would we send log messages to stderr at all (after initial development) - if warn "Caught error from 'run' method: $e\n"; happens, wouldn't we want that logged to syslog? |
| 09:34 |
|
jvwoolf joined #evergreen |
| 09:35 |
|
rfrasur joined #evergreen |
| 09:42 |
|
Keith-isl joined #evergreen |
| 09:53 |
rjackson_isl_hom |
Good morning y'all! Hoping for some much needed assistance in support for the Evergreen Indiana consortium. Having connectivity issues to the db and seeing long running queries with SET NAMES 'UTF-8'; and SELECT auditor.clear_audit_info(); |
| 09:53 |
rjackson_isl_hom |
Are those things that can be killed off with no lasting consequences? |
| 09:56 |
rjackson_isl_hom |
and should they ever show in list of running queries for more than a few seconds? |
| 09:58 |
rjackson_isl_hom |
selecting currently running transactions that have been running for more than 5 minutes currently yields 451 rows |
| 09:58 |
rjackson_isl_hom |
guessing that is way high |
| 10:03 |
Dyrcona |
Yes, that is way high. I'd pg_cancel_backed the long-running queries and keep monitoring it for a while. |
| 10:04 |
Dyrcona |
Don't panic if they don't go away immediately. Cancel let's the process clean up any temp space it is using and that can take a while sometimes. |
| 10:04 |
rjackson_isl_hom |
Dyrcona++ |
| 10:05 |
Dyrcona |
If it happens again, you might want to consider a vacuum analyze (or even full) on some tables. |
| 10:07 |
Dyrcona |
FWIW, we run a vacuum analyze on the whole database at 2:30 am every day. |
| 10:07 |
csharp_ |
rjackson_isl_hom: when looking at pg_stat_activity, filter for "where state <> 'idle'" |
| 10:08 |
Dyrcona |
Yeah, that, too. |
| 10:08 |
csharp_ |
the ones with all the SET NAMES and auditor.clear_audit_info() are showing the last command performed on the connect |
| 10:08 |
csharp_ |
er.. connection |
| 10:08 |
csharp_ |
it doesn't mean those are still actually running queries |
| 10:09 |
rjackson_isl_hom |
OK panic attack number 1 of 50 alleviated as they were all idle... |
| 10:10 |
csharp_ |
select * from pg_stat_activity where state <> 'idle' and now() - query_start > '1 minute'; or similar might help you see long-running ones |
| 10:11 |
csharp_ |
of course, adjust that interval to suit |
| 10:11 |
miker |
Dyrcona: re chrome tabs, I have the same behavior as you but I'm using KDE, so ... I think it's chrome what broke chrome, still. I haven't tried system title bar and borders yet, though. trying that now |
| 10:11 |
Dyrcona |
I'm going to paste something that I got from JBoyer and then modified slightly for the output. |
| 10:12 |
csharp_ |
rjackson_isl_hom: also, look into pg_top (deb is named "pgtop" in the Debian/Ubuntu repos) - it's a "top"-like output for postgres |
| 10:12 |
csharp_ |
shows you the top queries and number of processes running |
| 10:12 |
csharp_ |
as well as system resources |
| 10:13 |
csharp_ |
usage would be something like "PGUSER=evergreen pg_top" (or whatever your actual PG user is) |
| 10:13 |
csharp_ |
might need .pgpass configured for that |
| 10:14 |
miker |
Dyrcona: *sadface* that didn't "fix" chrome for me |
| 10:14 |
Dyrcona |
https://pastebin.com/AA0aLagR <- Reports run time of all non-idle queries that you care about, ordered by runtime. |
| 10:14 |
Dyrcona |
miker: My other laptop has KDE, but I don't use it as much. I'll give it a try later. |
| 10:15 |
Dyrcona |
I had to turn off System Titlebar to fix it on Ubuntu with "Gnome" |
| 10:17 |
csharp_ |
is this the thing where the window in gnome won't resize correctly and can't be dragged to a second monitor? |
| 10:17 |
csharp_ |
or is this specifically a Chrome thing? |
| 10:19 |
Dyrcona |
I think it is specifically a Chrome thing. Dragging tabs in my IRC client works just fine, and it uses GTK+. |
| 10:19 |
miker |
for me, it acts like chrome dropping a feature where you could drag and reorder tabs (and drag them between chrome windows). can't do that at all now, they pop out of the window when you start to drag them and don't pop in to other windows |
| 10:19 |
csharp_ |
ah |
| 10:19 |
csharp_ |
I'm pretty much a FF user - only use Chrome for certain things |
| 10:19 |
Dyrcona |
My symptoms were the same as miker's. |
| 10:20 |
Dyrcona |
I use both... Chrome for work and FF for personal, signed in to different Google accounts. |
| 10:20 |
csharp_ |
@ana too many google accounts |
| 10:20 |
pinesol |
csharp_: Cool! Young, comatose gnat |
| 10:20 |
berick |
heh |
| 10:20 |
csharp_ |
@ana TOO MANY SECRETS |
| 10:20 |
pinesol |
csharp_: Necrotomy seats |
| 10:20 |
csharp_ |
pinesol: close |
| 10:20 |
pinesol |
csharp_: Have you tried throwing it across the room? |
| 10:21 |
berick |
chrome + xfce behaving normally for me fwiw |
| 10:21 |
berick |
updated yesterday |
| 10:21 |
* Dyrcona |
checks for updates. |
| 10:21 |
rjackson_isl_hom |
Dyrcona that query gives me 9 rows total and they are short durations so stress level is coming down more |
| 10:22 |
miker |
from the fount of all truth, redit, "Just searched it up, they say its a Linux problem it will be fixed in the next update which will be soon. Im using the beta version and its working perfectly" |
| 10:22 |
Dyrcona |
rjackson_isl_hom: Good to hear. |
| 10:22 |
rjackson_isl_hom |
now if I could figure out why autorenewals have failed 2 nights in a row and the job ends up creating "no connection to the server" errors out the wazoo |
| 10:22 |
Dyrcona |
miker: :) |
| 10:23 |
csharp_ |
I also don't like using Chrome because it's managed centrally by our IT dept nowadays and I feel like I'm being watched (more than normal) :-) |
| 10:23 |
rjackson_isl_hom |
autorenewals run on our utility server which gets rebooted nightly before the major crons are kicked off |
| 10:23 |
Dyrcona |
I have kernel , gdb, and ruby updates that I'll install later. |
| 10:24 |
rjackson_isl_hom |
last night failed and there were only 6K and the night before had over 11K |
| 10:24 |
Dyrcona |
rjackson_isl_hom: Does it work if you run it manually later? |
| 10:24 |
rjackson_isl_hom |
I reran yesterday multiple times - adjusted all accounts with over 50 renewals manually as well |
| 10:24 |
rjackson_isl_hom |
we made it thru about 80% by eod |
| 10:24 |
csharp_ |
rjackson_isl_hom: can you share the full error from the log (scrubbed of any sensitive info of course)? |
| 10:26 |
Dyrcona |
rjackson_isl_hom: Triple check the max_stanza_size in ejabberd, too. We've had that conversation before, but maybe it needs to be larger still? |
| 10:27 |
Dyrcona |
It's not just number of items out, long subtitles can do it, too, if there are enough. One thought would be to modify the templates so that subtitles are not included. |
| 10:29 |
Dyrcona |
Another thought from the last time, it should be possible to filter out staff accounts that typically have large numbers of items out with a filter on the hook based on profile group if you're using that for staff. |
| 10:30 |
Dyrcona |
Assuming it is a continuation of the previous problem and not something new. |
| 10:35 |
|
terranm joined #evergreen |
| 10:44 |
Dyrcona |
Hitting Caps lock instead of Shift while typing a password is annoying. |
| 11:14 |
|
mmorgan joined #evergreen |
| 11:22 |
|
jihpringle joined #evergreen |
| 11:49 |
|
jihpringle joined #evergreen |
| 12:00 |
csharp_ |
https://pastebin.com/J780AXq8 - as much log data as I can currently get (reasonably) around the open-ils.actor drop issues |
| 12:06 |
|
terranm joined #evergreen |
| 13:12 |
|
Keith_isl joined #evergreen |
| 13:19 |
jeffdavis |
I don't know what I'm talking about, but could it be a file descriptor thing? |
| 13:19 |
jeffdavis |
i.e. ejabberd hitting the file descriptor limit |
| 13:24 |
Dyrcona |
See, I love how Evergreen basically says it was kicked out and ejabberd says the other end went away. jeffdavis could be on to something. Could be a system or resource limit is being hit. |
| 13:26 |
Dyrcona |
Unfortunately, "Jabber Exception " is vague. Looking at this line might reveal something interesting: XMPPReader.pm:262 |
| 13:29 |
Dyrcona |
Eh, not so much. It basically got 0 data back from ejabberd and is disconnecting. |
| 13:38 |
jeffdavis |
the file descriptor thing comes from this 14-year-old article saying ejabberd can be a hog that way: https://metajack.im/2008/09/23/file-descriptors-are-yummy-or-common-pitfalls-of-ejabberd/ - this was linked in an old LP bug somewhere |
| 13:39 |
jeffdavis |
just seems kinda weird that OpenSRF can connect to the Jabber server but gets 0 bytes when it tries to read data, but again, I don't know anything about this stuff |
| 13:45 |
Dyrcona |
Yeah, well, sockets are files. |
| 13:46 |
Dyrcona |
It happens on first read, too. Looks like it might connect, send, get nothing back, and die. |
| 13:47 |
Dyrcona |
There are ways to find out how many file descriptors a process is using. |
| 13:58 |
Dyrcona |
I should read that post more carefully before commenting further. It has been a day for switching taks, skimming over problem descriptions, getting the gist, and missing an important detail here or there. |
| 13:58 |
|
andrisi joined #evergreen |
| 14:04 |
|
jvwoolf1 joined #evergreen |
| 14:25 |
rjackson_isl_hom |
status update on autorenewals issue - tried another rerun after deleting all entries with a state of error and they ran to completion. No other changes were made :) :( |
| 14:39 |
terranm |
New Devs meeting in 20 minutes over at https://wiki.evergreen-ils.org/doku.php?id=newdevs:meetings:agenda-2022-01-19 --- we're going to be talking about wins and losses with our respective recent upgrades if anybody wants to join in |
| 14:42 |
|
Christineb joined #evergreen |
| 14:47 |
Dyrcona |
Hmm. I see messages in the PostgreSQL main log that concern me, but things seem to be working normally for replication: cp: cannot create regular file '/usr/backups/database2/wal/0000000100003E5D000000CB': No such file or directory |
| 14:48 |
Dyrcona |
/usr/backups/database2/wal/ exists and is getting new files.... |
| 14:48 |
Dyrcona |
Also, my understanding is wal log shipping is redundant with streaming replication, so I'm not too bothered by it. |
| 14:48 |
Dyrcona |
Back to seeing if I can find the insert error. |
| 14:50 |
Dyrcona |
Oh, wait a minute. Those cp error are from Monday during our downtime. Also the log file doesn't appear to have rotated or logged new errors since then. |
| 14:51 |
Dyrcona |
Hmm.. looks like messages are only going to syslog, now.... |
| 14:52 |
Dyrcona |
OK. I'm confusing myself. Ignore me. |
| 14:57 |
Dyrcona |
So, resetting Chrome to factory default settings apparently clears out cookies as well. I didn't expect that. |
| 15:13 |
JBoyer |
Dyrcona, looks like you've figured out what's going on there, but re: wal and streaming, wal shipping isn't strictly necessary with streaming, *so long as the replica can keep up* If it gets behind at all it will use the archive retrieve command to grab the ones it's missing until it can reconnect. |
| 15:32 |
Dyrcona |
JBoyer: Yep. Also, it looks like with syslog enabled only some things go to the main log, and we haven't had any of those things since Monday, amazingly enough. |
| 15:50 |
JBoyer |
Yeah, I've noticed that too. I think subprocesses (like the archive commands) always end up in a main file log, even if Pg is syslogging everything else. Makes sense I suppose. |
| 17:13 |
|
jvwoolf1 left #evergreen |
| 17:18 |
|
mmorgan left #evergreen |
| 18:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
