| Time |
Nick |
Message |
| 02:08 |
|
jeffdavis_ joined #evergreen |
| 02:08 |
|
ejk_ joined #evergreen |
| 02:09 |
|
bshum_ joined #evergreen |
| 05:01 |
pinesol |
News from qatests: Failed Running perl live tests <http://testing.evergreen-ils.org/~live/test.49.html#2019-06-24T04:59:37,643832565-0400 -0> |
| 06:54 |
|
JBoyer joined #evergreen |
| 07:11 |
|
rjackson_isl joined #evergreen |
| 07:56 |
|
Dyrcona joined #evergreen |
| 08:12 |
|
bos20k joined #evergreen |
| 08:45 |
|
mmorgan joined #evergreen |
| 08:56 |
|
jvwoolf joined #evergreen |
| 09:09 |
|
yboston joined #evergreen |
| 09:24 |
|
nfBurton joined #evergreen |
| 09:24 |
nfBurton |
Woah |
| 09:24 |
nfBurton |
This got a facelift |
| 09:38 |
Dyrcona |
This? |
| 09:39 |
nfBurton |
IRC, they made Kiwi the default |
| 09:40 |
Dyrcona |
You mean the web interface? 'Cause nothing has changed for me. |
| 09:42 |
rhamby |
yeah, things look the same from the quassel irc client on mac |
| 09:47 |
nfBurton |
Yeah, the web interface. They did an update over the weekend. Much nicer |
| 09:51 |
|
mmorgan joined #evergreen |
| 10:00 |
|
guest joined #evergreen |
| 10:02 |
|
guest left #evergreen |
| 10:37 |
Dyrcona |
Does anyone know what might be broken when search results look like this: https://drive.google.com/file/d/1WaVXJwoAbvm8Ms6J2Gg7OyoJxZxJ8QMw/view?usp=sharing |
| 10:38 |
|
Christineb joined #evergreen |
| 10:40 |
mmorgan |
Dyrcona: Issue with display fields? Just a guess. |
| 10:42 |
JBoyer |
I'd lean that way too. Though if you haven't intentionally moved all of the My Account stuff over to the left I'd wonder if there is additional strangeness in processing templates. |
| 10:42 |
Bmagic |
Dyrcona: Can you get that result 100% of the time when searching the same way? |
| 10:42 |
Dyrcona |
JBoyer: That is deliberate. |
| 10:43 |
JBoyer |
+1 |
| 10:43 |
Bmagic |
Dyrcona: If so, I would config the server without any custom templates and see if you can get the same issue |
| 10:44 |
Dyrcona |
Well, it's mostly identical to production. I'm trying a different title with the same options per Bmagic's suggestion. |
| 10:45 |
Dyrcona |
Same thing, but it does something stupid and title: gets duplicated if you just edit the search box. |
| 10:46 |
Bmagic |
You can get it to have this issue with stock Evergreen tt2 templates? |
| 10:46 |
Dyrcona |
Bmagic: I haven't tried. |
| 10:46 |
Bmagic |
That'd be my next step |
| 10:47 |
Bmagic |
The database is clearly returning rsults. 28 results. The data is coming back, getting formatted for the page through the templates, and somehow the templates are introducing this issue. |
| 10:50 |
JBoyer |
If you right click on a row and choose Inspect Element is there any identifying anything in there? |
| 10:50 |
Dyrcona |
Doesn't happen in production. |
| 10:51 |
Bmagic |
lol, I suppose then we have to ask the question: what's the difference between production and this machine? |
| 10:53 |
Dyrcona |
Some security patches on both custom and standard templates. The data is from a production dump from Sunday night. This machine is also a lot slower, and the database can be slower, too. |
| 10:53 |
Bmagic |
I'd still remove the custom templates to get a known good starting point |
| 10:54 |
Bmagic |
comment out the line(s) PerlAddVar OILSWebTemplatePath XXXXX |
| 10:55 |
Bmagic |
leaving the stock line |
| 10:55 |
Dyrcona |
Bmagic: I know how to do that, thanks. |
| 10:55 |
Bmagic |
Once it starts working with the stock templates, start adding in one file at a time from the custom folder structure until it breaks |
| 10:55 |
Dyrcona |
And, that won't work in our next upgrade, 'cause I'm ditching the custom template directory. It's too cumbersome to work with. |
| 10:56 |
Dyrcona |
I suspect something in the security patches. I don't remember the last time it was working properly, but it was working in the past two weeks. |
| 11:03 |
Dyrcona |
Still busted after removing the custom templates from the Apache configuration. |
| 11:04 |
Bmagic |
very interesting |
| 11:04 |
Dyrcona |
I just did a make install and restart to see if that makes a difference. I'm inclined to blame the database. |
| 11:04 |
Dyrcona |
Yeah, still busted. It could be something in the security patches I installed. |
| 11:04 |
Dyrcona |
Nothing jumps out as an obvious problem, though. |
| 11:07 |
Dyrcona |
I'm switching to our production templates. |
| 11:07 |
Bmagic |
an odd problem indeed |
| 11:11 |
Dyrcona |
Not happening on training, so if it isn't the data, then it's something that I've installed on the test vm but not on training, yet. |
| 11:20 |
csharp |
Dyrcona: the only time I've seen empty result sets like that was fixed by a reingest but if it's a dump from prod, that doesn't sound like the same thing |
| 11:20 |
Dyrcona |
Still busted. Must be the database. I don't have time to wait for a pingest, so I'm giving up for now. |
| 11:20 |
csharp |
maybe do the searches from srfsh and see what returns? |
| 11:20 |
Dyrcona |
csharp: The dump could have been made in the middle of something going on. |
| 11:20 |
csharp |
ah |
| 11:21 |
Dyrcona |
I could try switching to one of the other copies of the dump. |
| 11:21 |
|
khuckins joined #evergreen |
| 11:22 |
Dyrcona |
I expect the same result, but worth a shot. |
| 11:24 |
Dyrcona |
Hm.. the sed command didn't work. |
| 11:28 |
Dyrcona |
Yeahp. Definitely the dump. |
| 11:51 |
Bmagic |
horay! |
| 12:09 |
|
jihpringle joined #evergreen |
| 12:40 |
Dyrcona |
Is it "normal" for websocketd to have 17 threads and there being 228 osrf-websocket-stdio processes? I assume that there's not a 1:1 relationship. |
| 12:40 |
mmorgan |
Any newish to the web client sites having trouble with patron hold notification preferences?/ |
| 12:40 |
mmorgan |
We're getting lots of reports of no notifications for holds. |
| 12:42 |
mmorgan |
I've looked at lp 1361258, but that's long fixed. |
| 12:43 |
pinesol |
Launchpad bug 1361258 in Evergreen 3.0 "Patron accounts losing notification preferences" [High,Fix released] https://launchpad.net/bugs/1361258 |
| 12:43 |
mmorgan |
We' |
| 12:43 |
mmorgan |
re on 3.2.4 |
| 12:56 |
JBoyer |
Dyrcona, on one of our machines there's 1 websocketd instance with 13 threads and 10 separate osrf-websocket-stdio processes. |
| 12:57 |
JBoyer |
I'm not sure what the threads are for, because as I understand it, the incoming connections are handed to the ...-stdio processes and then websocketd never sees them again. |
| 12:57 |
Dyrcona |
JBoyer: You use the nginx proxy, right? |
| 12:57 |
JBoyer |
We do |
| 12:57 |
JBoyer |
I'm curious what your timeouts are with a count that high |
| 12:59 |
Dyrcona |
Well, I'm not using a nginx proxy for 1 thing. |
| 12:59 |
JBoyer |
Ah, so websocketd is handling SSL and direct incoming connections? |
| 13:00 |
Dyrcona |
Yes. I'm starting to think that could be a mistake. |
| 13:01 |
Dyrcona |
It does appear that osrf-websocket-stdio processes don't hang around forever. A couple of my bricks are now showing fewer than they did earlier. |
| 13:02 |
JBoyer |
It looks like there's no way to timeout websocketd connections that are direct. They will close after the other end goes away, but with nothing to make the other end give up they can stick around for a very long time. |
| 13:03 |
JBoyer |
Nginx is enforcing a 60 second timeout on those connections here. Default might be as high as 5 min. |
| 13:04 |
Dyrcona |
Well, something additional to consider. |
| 13:24 |
Dyrcona |
Well, nginx just failed to install for me on Ubuntu 16.04.... |
| 13:25 |
Dyrcona |
Oh, no. It installed. Failed to start, 'cause I haven't edited apache2, yet..... |
| 13:43 |
berick |
testing w/ a 1 minute timeout locally as well |
| 13:43 |
Dyrcona |
Anyone using HAProxy? |
| 13:46 |
Dyrcona |
The nginx steps in the README are missing some detail when it comes to opensrf_ws.js |
| 13:46 |
Dyrcona |
Looks like there are two copies and the path is not complete. |
| 13:59 |
jihpringle |
mmorgan: https://bugs.launchpad.net/evergreen/+bug/1774268 |
| 13:59 |
pinesol |
Launchpad bug 1774268 in Evergreen "webstaff UX: default hold notification preferences for patrons confusingly presented" [Medium,Confirmed] |
| 13:59 |
mmorgan |
jihpringle: Thanks, I'll take a look! |
| 14:00 |
jihpringle |
there's a change of behaviour between xul and the webclient |
| 14:01 |
|
jihpringle joined #evergreen |
| 14:02 |
jihpringle |
in the web client if you save the patron's account with nothing filled in as the hold default no boxes are checked by default when the patron goes to place a hold |
| 14:03 |
jihpringle |
in xul if nothing was selected as the default hold notification in the patron account email/phone were still checked by default when the patron went to place a hold |
| 14:05 |
Dyrcona |
Does websocketd have to do ssl when nginx is running? Because it looks like I'm getting SSL errors for that. |
| 14:06 |
Dyrcona |
jihpringle: That's a known bug, but I can't think of the bug #. |
| 14:06 |
berick |
Dyrcona: no, but you have to config nginx to talk HTTP to the websocket backend |
| 14:06 |
berick |
Dyrcona: e.g. proxy_pass http://127.0.0.1:7682; |
| 14:06 |
Dyrcona |
berick: That's not already in the osrf-... file? |
| 14:07 |
berick |
Dyrcona: the sample osrf file is straddling 2 configs.. apache2-websockets and websocketd |
| 14:07 |
berick |
one needs ssl, one doesn't |
| 14:07 |
berick |
Dyrcona: i can share my nginx config if it would be helpful |
| 14:07 |
Dyrcona |
So, I change the https to http.. |
| 14:08 |
berick |
yeah |
| 14:08 |
Dyrcona |
Thank. I think I see what I need to do. |
| 14:08 |
Dyrcona |
Looks like there's a little more to change. |
| 14:08 |
* Dyrcona |
considers trying haproxy, too. |
| 14:09 |
Dyrcona |
It's working. |
| 14:11 |
JBoyer |
Given the number of sites using websocketd in production and the state of apache2-websockets, is it time to just say "you're going to use websocketd, the only choice you have to make is whether to use nginix with it?" |
| 14:13 |
berick |
JBoyer: +1 from me |
| 14:13 |
berick |
especially given https://bugs.launchpad.net/opensrf/+bug/1774703/comments/11 |
| 14:13 |
Dyrcona |
+1 |
| 14:13 |
pinesol |
Launchpad bug 1774703 in OpenSRF "Websockets processes locked at 100% CPU" [Undecided,Won't fix] |
| 14:13 |
JBoyer |
Oh, yeah, I forgot about that one. I just knew it was essentially abandonware. |
| 14:14 |
JBoyer |
I suppose if I'm going to open my mouth I'd better at least put in the LP, even if I don't have time to devote to it right away. |
| 14:15 |
* JBoyer |
gathers supporting documentations |
| 14:15 |
berick |
JBoyer++ |
| 14:15 |
berick |
i can help some w/ the branch |
| 14:16 |
JBoyer |
berick++ |
| 14:16 |
berick |
apache2-websockets no longer sparks joy -- it's gotta go |
| 14:19 |
csharp |
berick++ # Kon-Mari-ing Evergreen |
| 14:27 |
Dyrcona |
So, I could replace ldirectord with haproxy on the load balancer, but not sure if that will make much difference. Looks like I'd lose the nginx timeout and having nginx handle ssl instead of websocketd. |
| 14:28 |
Dyrcona |
Oh, wait... haproxy has a timeout. I should experiment with that with a pair of vms. |
| 14:28 |
JBoyer |
Dyrcona++ # do science to it |
| 14:37 |
|
egbuilder joined #evergreen |
| 14:37 |
|
drigney joined #evergreen |
| 14:37 |
|
akilsdonk joined #evergreen |
| 14:37 |
|
jyorio joined #evergreen |
| 14:37 |
|
miker joined #evergreen |
| 14:38 |
|
ericar joined #evergreen |
| 14:38 |
|
abneiman joined #evergreen |
| 14:38 |
|
rhamby joined #evergreen |
| 14:38 |
|
phasefx_ joined #evergreen |
| 14:38 |
|
felicia joined #evergreen |
| 14:38 |
|
jgoodson joined #evergreen |
| 14:38 |
jeff |
*blip* |
| 14:40 |
|
jweston joined #evergreen |
| 14:56 |
mmorgan |
jihpringle++ |
| 14:56 |
mmorgan |
That's exactly what we're seeing |
| 14:58 |
|
khuckins joined #evergreen |
| 15:05 |
Dyrcona |
jihpringle mmorgan: bug 1774268 |
| 15:05 |
pinesol |
Launchpad bug 1774268 in Evergreen "webstaff UX: default hold notification preferences for patrons confusingly presented" [Medium,Confirmed] https://launchpad.net/bugs/1774268 |
| 15:07 |
mmorgan |
Dyrcona: thanks, am digesting that now. |
| 15:13 |
|
sandbergja joined #evergreen |
| 15:15 |
Dyrcona |
JBoyer: I did the following to see how long some of my osrf-websocket-stdio processes have been going: ps -C osrf-websocket-stdio -o pid,etime |
| 15:16 |
csharp |
@band add Osrf Websocket Studio |
| 15:16 |
pinesol |
csharp: Band 'Osrf Websocket Studio' added to list |
| 15:16 |
Dyrcona |
The longest ones have been running for over 6 hours. |
| 15:17 |
JBoyer |
Yowza. |
| 15:17 |
JBoyer |
The longest on that machine I was looking at earlier is still under 7 minutes |
| 15:18 |
Dyrcona |
Looks like we could definitely benefit from a proxy with a timeout. |
| 15:19 |
JBoyer |
Couldn't hurt. I'm not sure how much damage they do just sitting, but there's almost certainly no benefit to it eihter. |
| 15:27 |
Dyrcona |
I'm going to try haproxy on a couple of vms here. |
| 15:45 |
Dyrcona |
Well, I'm taking off early. Good bye for now! |
| 16:06 |
|
gsams joined #evergreen |
| 16:41 |
|
jvwoolf left #evergreen |
| 16:58 |
|
sandbergja joined #evergreen |
| 17:01 |
pinesol |
News from qatests: Failed Running perl live tests <http://testing.evergreen-ils.org/~live/test.49.html#2019-06-24T16:58:43,335152394-0400 -0> |
| 17:07 |
|
mmorgan left #evergreen |
| 19:22 |
|
sandbergja joined #evergreen |
| 22:50 |
|
sandbergja joined #evergreen |
| 23:08 |
|
JasonEDN joined #evergreen |
| 23:37 |
|
sandbergja joined #evergreen |
