| Time |
Nick |
Message |
| 03:22 |
|
jaswinder joined #evergreen |
| 06:18 |
|
Dyrcona joined #evergreen |
| 06:30 |
pinesol_green |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:02 |
|
agoben joined #evergreen |
| 07:14 |
|
rjackson_isl joined #evergreen |
| 08:12 |
|
collum joined #evergreen |
| 08:25 |
|
Dyrcona joined #evergreen |
| 08:27 |
|
dwgreen joined #evergreen |
| 08:44 |
miker |
comcast-- |
| 08:44 |
miker |
comcast-- |
| 08:45 |
miker |
twice, because two links are down at the same time |
| 08:45 |
miker |
ITS COMCASTIC |
| 08:45 |
Dyrcona |
heh. |
| 08:45 |
Dyrcona |
My day is not off to such a great start, either. |
| 08:45 |
Dyrcona |
I think the new replication server crashed during pg_basebackup |
| 08:46 |
|
mmorgan joined #evergreen |
| 09:04 |
csharp |
comcast-- |
| 09:04 |
csharp |
@karma |
| 09:04 |
pinesol_green |
csharp: Highest karma: "gmcharlt" (282), "berick" (197), "kmlussier" (185), "Dyrcona" (157), and "dbwells" (139). Lowest karma: "comcast" (-34), "systemd" (-14), "^" (-9), "oracle" (-8), and "typos" (-7). You (csharp) are ranked 6 out of 268. |
| 09:04 |
csharp |
comcast-- |
| 09:06 |
mmorgan |
^++ :) |
| 09:13 |
Dyrcona |
@karma ^ |
| 09:13 |
pinesol_green |
Dyrcona: Karma for "^" has been increased 1 time and decreased 9 times for a total karma of -8. |
| 09:13 |
Dyrcona |
:) |
| 09:13 |
Dyrcona |
Yeah, the new replication server crashed. No idea why from the logs. |
| 09:15 |
|
stephengwills joined #evergreen |
| 09:19 |
|
yboston joined #evergreen |
| 09:20 |
|
jaswinder joined #evergreen |
| 09:20 |
|
jaswinder joined #evergreen |
| 09:26 |
|
littlet joined #evergreen |
| 09:28 |
miker |
Dyrcona: I guess one way to look at my morning is that, if I had a failing replica db, I wouldn't even have to see it... so, there that? silver linings |
| 09:32 |
JBoyer |
If a tree falls in the forest and there's no internet connection to the webcam, are there still forests? |
| 09:34 |
miker |
what's a tree? |
| 09:35 |
Dyrcona |
heh |
| 09:35 |
Dyrcona |
Trees are those things that fall and knock out your Internet. |
| 09:36 |
Dyrcona |
I think my problem was caused by the wrong zfs driver installed. |
| 09:37 |
Dyrcona |
zfs-dkms was installed, but not needed. |
| 09:47 |
JBoyer |
Dyrcona, but a crash though? is that driver kind of sketchy or was there some kind of conflict? |
| 09:48 |
Dyrcona |
JBoyer: I really don't know. I just know it was installed on the replication machine but not on the main db server. I have no log information to indicate what caused the crash, so... just making a slightly informed guess. |
| 09:49 |
Dyrcona |
If it crashes again, then it must have been something else. :) |
| 09:50 |
Dyrcona |
I also got some error messages when updating kernels that seemed to be related to it, so there's another data point. |
| 09:50 |
JBoyer |
Ah, with no logs you do kind of have to feel your way around a little. Hopefully that's it. |
| 09:51 |
Dyrcona |
Someone went to the colocation facility, but said nothing came up on the kvm, so no screen shots either. |
| 09:54 |
JBoyer |
If there's a next time: If there's a machine health display on there (don't know the proper name...) it may have some LEDs lit in red to point out hardware faults, and the iLO or DiRAC logs may have something useful if you've got the tools installed to read them. (depending on vendor and etc.) |
| 09:55 |
|
jvwoolf joined #evergreen |
| 10:16 |
|
glen_ joined #evergreen |
| 10:17 |
|
Christineb joined #evergreen |
| 10:24 |
Dyrcona |
So far, looks like zfs-dkms was the problem. The basebackup has made it to 72% without incident. |
| 10:24 |
JBoyer |
++ |
| 10:29 |
csharp |
@quote add < JBoyer> If a tree falls in the forest and there's no internet connection to the webcam, are there still forests? |
| 10:29 |
pinesol_green |
csharp: The operation succeeded. Quote #186 added. |
| 10:31 |
* csharp |
watches the web console with interest while trying to load a 10,000+ list of barcodes into item status |
| 10:32 |
csharp |
looks like the websocket closed before the file loaded - oh well, I didn't expect it to work :-) |
| 10:34 |
berick |
it should reconnect |
| 10:37 |
Dyrcona |
Ninety percent and still going. :) |
| 10:48 |
Dyrcona |
I started postgres; did a ps ax | grep postgres, and now, it looks like it crashed again. :( |
| 10:51 |
JBoyer |
Hardware crash, or just Pg? |
| 10:51 |
JBoyer |
And you said new server, what Pg version on either side? |
| 11:05 |
|
khuckins joined #evergreen |
| 11:10 |
Dyrcona |
JBoyer: Pg version is the same. The hardware is practically the same. |
| 11:11 |
Dyrcona |
The latter was the point of the new server. |
| 11:11 |
Dyrcona |
It's a hardware crash. |
| 11:11 |
Dyrcona |
I lose my ssh connection and host is unreachable. |
| 11:11 |
Dyrcona |
I suspect one or both of the NVMe drives is not properly seated. |
| 11:12 |
Dyrcona |
But, I'm over 50 miles from the server and have to rely on someone else to touch it today. |
| 11:15 |
|
terran joined #evergreen |
| 11:16 |
Dyrcona |
So O/S crash if not hardware, but I suspect hardware since there's nothing in kern.log about a crash. |
| 11:22 |
|
beanjammin joined #evergreen |
| 11:25 |
|
yboston joined #evergreen |
| 11:48 |
|
ngf42 joined #evergreen |
| 12:02 |
|
jihpringle joined #evergreen |
| 12:11 |
Bmagic |
this is weird |
| 12:11 |
Bmagic |
Event reacting failed with Undefined subroutine &Encode::find_mime_encoding called at /usr/local/lib/x86_64-linux-gnu/perl/5.22.1/Encode/MIME/Header.pm line 241. |
| 12:13 |
Bmagic |
The AT doesn't have MIME specified... Is that required? |
| 12:17 |
mmorgan |
Bmagic: sounds like it might be related to lp 1532236 ? |
| 12:17 |
pinesol_green |
Launchpad bug 1532236 in Evergreen "Evergreen should allow for HTML formatted action trigger emails" [Wishlist,Confirmed] https://launchpad.net/bugs/1532236 |
| 12:19 |
Bmagic |
ha! That one is mine and that did come to mind. That patch isn't installed for this example |
| 12:20 |
mmorgan |
Came to my mind, too :) |
| 12:20 |
Bmagic |
Therefore, this code might be responsible for the error? $email->header_set($hfield => map { encode("MIME-Header", $_) } @headers) if ($headers[0]); |
| 12:36 |
|
jvwoolf joined #evergreen |
| 13:11 |
Bmagic |
I restarted all services on the Utility server and the issue seems to have gone away but replaced with a new issue: SendEmail Reactor: unable to send email: $VAR1 = bless( {'string' => 'Can\'t call method "address" on an undefined value at /usr/local/share/perl/5.22.1/Email/Send/SMTP.pm line 29. |
| 13:12 |
Dyrcona |
That's actually funny. :) |
| 13:12 |
Bmagic |
oh? |
| 13:13 |
Dyrcona |
I mean amusing. Not often you bless a string in Perl (though you can) and even rarer to try and bless an error string. |
| 13:14 |
Dyrcona |
Though that may be blessing a hash. |
| 13:14 |
Bmagic |
strange indeed. I wonder if there is a conflicting perl module? |
| 13:14 |
Dyrcona |
Anyway, it made me chuckle. |
| 13:14 |
Dyrcona |
Could be something out of date with the rest of your installation. |
| 13:14 |
Bmagic |
:) |
| 13:21 |
csharp |
Bmagic: if you're using the default action_trigger_filters.json (which we are too) that means that it processed a notice for someone without an email address on their account |
| 13:22 |
csharp |
I've been meaning to noodle out the JSON query syntax to exclude users with out email addresses for that filter |
| 13:22 |
Bmagic |
csharp: the output template shows the headers are correct. complete with from and to addresses |
| 13:22 |
csharp |
s/with out/without/ |
| 13:22 |
csharp |
oh - hmm |
| 13:28 |
jeff |
Bmagic: error in question is for a different template output? *shrug* |
| 13:29 |
Bmagic |
I've got the logs directly connected to action_trigger.event.id |
| 13:29 |
Bmagic |
and the template_output for that event is showing a sound message |
| 13:35 |
Bmagic |
That error is happening hundreds of times. I don't think its a template issue |
| 13:35 |
Bmagic |
The execution is SendEmail.pm $stat = $sender->send($email); (I think) |
| 13:38 |
jeff |
Does the From: line in the output look invalid? can you post a redacted (but not the From: header) version of the template output? |
| 13:39 |
jeff |
your "line 29" error above suggests that Email::Address is failing to parse the From: header: https://metacpan.org/source/RJBS/Email-Send-2.201/lib/Email/Send/SMTP.pm#L29 |
| 13:42 |
Bmagic |
found the issue maybe.. I dumped the variables to the log |
| 13:43 |
Bmagic |
'From';$VAR4 = '=?UTF-8?B?ZW1haWxub3RpY2VAbWNkb3dlbGxwdWJsaWNsaWJyYXJ5Lm9yZw==?='; |
| 13:43 |
Bmagic |
All of the headers are getting hashed for some reason |
| 13:43 |
Bmagic |
$VAR9 = 'Subject';$VAR10 = '=?UTF-8?B?SG9sZCBBdmFpbGFibGUgTm90aWZpY2F0aW9u?=' |
| 13:44 |
Dyrcona |
Well, that is mime_encoded.... |
| 13:44 |
Bmagic |
it's suppose to look like that? |
| 13:44 |
Dyrcona |
UTF-8 and base64 encoded. |
| 13:44 |
Dyrcona |
Dunno for sure. Most mailers can handle it. |
| 13:44 |
jeff |
but Email::Address cannot. |
| 13:44 |
Dyrcona |
Sound like it can't. :) |
| 13:45 |
jeff |
it's unable to parse it, so you have no envelope from to pass |
| 13:46 |
Bmagic |
$Email::Address::VERSION = '1.908'; |
| 13:47 |
jeff |
the changelog for 1.909 is: add some docs saying "don't use this, it can be busted" |
| 13:47 |
jeff |
(laugh/cry here) |
| 13:47 |
Bmagic |
haha |
| 13:47 |
Dyrcona |
Sounds 'bout right. :) |
| 13:48 |
Bmagic |
perhaps the perl code is not supposed to be calling Email::Address ? It's getting called from Email::Send::SMTP.... maybe I have the wrong perl module Email::Send::SMTP ? |
| 13:48 |
jeff |
but I'm more interested in where those encoded strings are turning up. |
| 13:48 |
Bmagic |
looking at the file Email::Send::SMTP - it is calling for "use Email::Address 1.80;" |
| 13:49 |
Bmagic |
I think they are getting encoded by Evergreen in SendEmail.pm |
| 13:49 |
Bmagic |
$email->header_set($hfield => map { encode("MIME-Header", $_) } @headers) if ($headers[0]); |
| 13:53 |
* jeff |
eyes Email::Simple |
| 13:53 |
jeff |
Bmagic: do you have a working and a non-working system? |
| 13:53 |
jeff |
i.e., are you seeing this with a new distro, or on a fresh install on a previously-okay distro, or? |
| 13:53 |
Bmagic |
somehow, this issue came about on the production utility server a couple of days ago |
| 13:54 |
Bmagic |
I have been thinking about moving the action_trigger stuff to a different server to see if it would work elsewhere |
| 13:55 |
Bmagic |
I have confirmed that /usr/local/share/perl/5.22.1/Email/Send/SMTP.pm is no different anywhere else |
| 13:56 |
jeff |
what version of Email::Simple are you working with? packaged as libemail-simple-perl often. |
| 13:57 |
Bmagic |
$Email::Simple::VERSION = '2.214'; |
| 13:58 |
jeff |
are you running Ubuntu 18.04? |
| 13:58 |
|
jvwoolf joined #evergreen |
| 13:58 |
Bmagic |
it seems to be in two places /usr/local/share/perl/5.22.1/Email/Simple.pm and /usr/share/perl5/Email/Simple.pm |
| 13:59 |
Bmagic |
jeff: 16.04 |
| 13:59 |
jeff |
what version is /usr/share/perl5/Email/Simple.pm? |
| 14:01 |
Bmagic |
$Email::Simple::VERSION = '2.208'; |
| 14:01 |
Bmagic |
Something is fishy here for sure. Other installations of 16.04 don't have the module in both places |
| 14:02 |
dbwells |
Sorry to butt in, but wanted to remind everyone that point releases are scheduled for tomorrow. Please take some time for bug review if you are able. We particularly need a volunteer to push bug #1735539 over the finish line. Thanks! |
| 14:02 |
pinesol_green |
Launchpad bug 1735539 in Evergreen 3.1 "web client Cannot delete multiple Items from Item Status Screen" [High,Confirmed] https://launchpad.net/bugs/1735539 |
| 14:02 |
jeff |
have you installed something from CPAN on this host which might have downloaded the new version of Email::Simple and potentially other things? |
| 14:03 |
Bmagic |
it's possible for sure |
| 14:03 |
Bmagic |
can I safely just delete /usr/local/share/perl/5.22.1/Email/Simple.pm ? |
| 14:04 |
jeff |
I can't answer that. |
| 14:04 |
Bmagic |
:) |
| 14:04 |
Bmagic |
I'll move it and see |
| 14:04 |
Bmagic |
ll |
| 14:04 |
jeff |
I'm also not certain that Email::Simple is to blame, but so far it's a promising suspect. |
| 14:04 |
Bmagic |
yeah, considering that it's the thing that is different |
| 14:05 |
Bmagic |
both /usr/share/perl5 and /usr/local/share/perl/5.22.1 are in the path for SendEmail.pm for AT execution (I am guessing) |
| 14:21 |
Dyrcona |
Both of those are typically in the Perl lib path regardless. Not sure which take precedence if something is in both, but my hunch says the latter. |
| 14:43 |
* jeff |
pokes and prods at threads |
| 14:44 |
jeff |
perlbrew++ |
| 14:47 |
Dyrcona |
Don't try Perl threads with Encode.pm. Unless things have changed, it ain't gonna work. |
| 14:48 |
jeff |
i realized the potential for confusion as I was saying that, but in this case we're talking about metaphorical threads. :-) |
| 15:14 |
|
jvwoolf1 joined #evergreen |
| 15:17 |
pinesol_green |
[evergreen|Jane Sandberg] Docs: starting release notes for 3.0.7 - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=e9a32b0> |
| 15:17 |
pinesol_green |
[evergreen|Jane Sandberg] Docs: starting release notes for 3.1.1 - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=1b30c31> |
| 15:44 |
|
khuckins joined #evergreen |
| 15:51 |
jeff |
The behavior of encode("MIME-Header", "foo") is different between Perl 5.22.4 + Encode 2.72_01 and Perl 5.24.4 + Encode 2.80_01 |
| 15:56 |
|
jvwoolf1 left #evergreen |
| 16:04 |
jeff |
but I'm pretty sure we're broken in trying to encode the entire to and/or from header body. |
| 16:06 |
jeff |
Bmagic: did you also end up with a new version of Encode? |
| 16:07 |
Bmagic |
I don't think so, but I am just abandoning the server and making a new one |
| 16:08 |
jeff |
okay. I'll create a bug so we can fix the problem before it bites more folk. :-) |
| 16:14 |
Bmagic |
I'm sure it's additional perl modules that I installed that made a conflict |
| 16:27 |
jeff |
we're improperly attempting to encode the entire field body for structured fields, which is... not right. |
| 16:41 |
jeff |
(I suppose "improperly" already covered the "not right" part) |
| 17:03 |
|
mmorgan left #evergreen |
| 17:14 |
jeff |
Based on this, I'm guessing that nobody here is running Stretch or Bionic in production? |
| 17:14 |
jeff |
(or if they are, they've patched or not noticed that their e-mail notifications are broken?) |
| 18:03 |
|
jaswinder joined #evergreen |
| 18:06 |
|
jaswinder joined #evergreen |
| 18:30 |
pinesol_green |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:37 |
jeffdavis |
On a 3.1.0 test server, workstation reg fails to load, console log shows "Uncaught (in promise) DOMException: Quota exceeded." errors referencing upup.sw.min.js and then some uncaught event errors apparently related to a missing IndexedDB database - anyone seen this? |
| 18:37 |
jeffdavis |
web client workstation registration, that is |
| 18:40 |
jeffdavis |
actually I'm seeing it on the login page before workstation registration too |
| 19:20 |
dbwells |
jeffdavis: from the release tarball, or git branch? |
| 19:22 |
dbwells |
jeffdavis: Actually, I was thinking of something else, so that might be interesting, but won't change my answer for now :) |
| 19:23 |
dbwells |
Yes, we have seen that, but haven't isolated it yet. It is browser/workstation specific (that is, it is only happening for one staff member here). |
| 19:24 |
dbwells |
It happens only in Chrome for her, and comes and goes without any clear reason yet. |
| 19:25 |
dbwells |
Two other notes: |
| 19:26 |
dbwells |
1) Our ticket log says when this happens, the browser still works in Incognito mode (not a long term solution, but a clue). |
| 19:27 |
dbwells |
2) The workstation was very low on disk space, and simply freeing space and doing nothing else cleared this up the last time it happened (since we don't know the cause, this might be totally unrelated). |
| 19:28 |
dbwells |
Until now, I figured this was something broken on this particular PC. |
| 19:28 |
* dbwells |
really is going away now |
| 21:08 |
|
jaswinder joined #evergreen |
| 21:48 |
jeff |
jeffdavis: what OS and what browser/version? If Chrome, what does DevTools -> Application -> Clear storage show for "used" and "storage quota"? |
| 21:54 |
jeffdavis |
This is on my dev laptop, Ubuntu 16.04, Chromium and Chrome 65. A coworker using Chrome on Windows has not been affected. Disk space on my laptop is a little low (though not extremely so), just clearing that up and retesting. |
| 21:56 |
jeffdavis |
(I suppose I don't really need 350GB of music on this thing.) |
| 21:57 |
jeff |
before you started freeing space, what was the available disk space on the filesystem where the chrome profile (likely /home) was stored? |
| 21:59 |
jeffdavis |
4GB. I doubt that's the issue. |
| 22:01 |
jeff |
Chrome 65.what? |
| 22:02 |
jeff |
and was 4 GB the actual available-to-your-user storage, or is that not counting the typical "reserved for uid 0" space? |
| 22:03 |
jeff |
Did you encounter it with both Chromium and Chrome? I'm not familiar enough with Chromium to know if this is possible, but were you using the same user profile, or was it a problem in both with independent user profiles? |
| 22:19 |
|
jaswinder joined #evergreen |
| 23:39 |
jeffdavis |
huh, goes to show what I know ... if I clear up enough disk space the problem goes away, but if I bring my free disk space back down under 4GB or so the problem returns. |
| 23:42 |
jeffdavis |
Chrome/Chromium 65.0.3325.181 (same version for both). They are using different profiles/data directories AFAIK. |
