| Time |
Nick |
Message |
| 01:02 |
|
eby joined #evergreen |
| 01:03 |
|
sard joined #evergreen |
| 01:03 |
|
ejk joined #evergreen |
| 02:14 |
|
dbwells_ joined #evergreen |
| 02:15 |
|
tsbere_ joined #evergreen |
| 02:15 |
|
remingtron_ joined #evergreen |
| 02:15 |
|
rlefaive_ joined #evergreen |
| 04:31 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 05:02 |
|
Bmagic joined #evergreen |
| 07:05 |
|
JBoyer joined #evergreen |
| 07:19 |
|
rjackson_isl joined #evergreen |
| 08:17 |
|
collum joined #evergreen |
| 08:18 |
|
agoben joined #evergreen |
| 08:48 |
|
bos20k joined #evergreen |
| 08:48 |
|
mmorgan joined #evergreen |
| 08:48 |
|
Dyrcona joined #evergreen |
| 09:05 |
|
mdriscoll joined #evergreen |
| 09:06 |
|
Jillianne joined #evergreen |
| 09:06 |
|
yboston joined #evergreen |
| 09:23 |
|
kmlussier joined #evergreen |
| 09:30 |
kmlussier |
@coffee |
| 09:30 |
* pinesol_green |
brews and pours a cup of Hacienda La Esmeralda, and sends it sliding down the bar to kmlussier |
| 09:31 |
mmorgan |
@coffee kmlussier |
| 09:31 |
* pinesol_green |
brews and pours a cup of Kenya Giakanja, and sends it sliding down the bar to kmlussier |
| 09:31 |
mmorgan |
You'll likely need another :) |
| 09:37 |
kmlussier |
mmorgan: Yes, indeed! Thanks! |
| 09:51 |
|
JBoyer_alt joined #evergreen |
| 09:55 |
remingtron |
I have today as a Web Client Docs Day on my calendar...anybody working on docs today? |
| 09:56 |
remingtron |
If you're interested, https://wiki.evergreen-ils.org/doku.php?id=evergreen-docs:webclient |
| 10:03 |
kmlussier |
remingtron: Yes, it is! I can send out a reminder on the list. |
| 10:15 |
remingtron |
kmlussier: sounds good |
| 10:28 |
|
JBoyer joined #evergreen |
| 10:35 |
kmlussier |
Hmmm...sent the email about 25 minutes ago, but it looks like it hasn't come through yet. Maybe somebody else should send a reminder so that people see it. :) |
| 10:41 |
JBoyer |
kmlussier, can you send a reminder to have someone remind the remainder about it? |
| 10:41 |
JBoyer |
;) |
| 10:41 |
kmlussier |
JBoyer: Yeah, actually, I just did something like that. |
| 10:41 |
* kmlussier |
needs more coffee |
| 10:49 |
Dyrcona |
@coffe kmlussier |
| 10:49 |
pinesol_green |
Dyrcona: I'm sorry, Dave. I'm afraid I can't do that. |
| 10:49 |
Dyrcona |
:) |
| 10:49 |
Dyrcona |
@coffee kmlussier |
| 10:49 |
* pinesol_green |
brews and pours a cup of Kenyan AA, and sends it sliding down the bar to kmlussier |
| 10:49 |
kmlussier |
ha ha. You need coffee to be able to spell coffee. |
| 10:50 |
Dyrcona |
@tea |
| 10:50 |
* pinesol_green |
brews and pours a pot of Hu-Kwa Tea, and sends it sliding down the bar to Dyrcona (http://ratetea.com/tea/mark-t-wendell/hu-kwa-tea/4467/) |
| 10:53 |
* berick |
sees kmlussier's email now |
| 10:56 |
|
terran joined #evergreen |
| 10:58 |
|
jvwoolf joined #evergreen |
| 11:02 |
dbs |
So does anyone who has nginx-proxied see occasional 503 errors (presumably when an unproxied request would have just timed out due to the apache process getting killed off or whatever)? |
| 11:04 |
* dbs |
is wondering a) about making the raw 503 error look better and b) if there's a way to avoid the root cause - maybe max_requests 0 and hope that the memory leaks are gone? |
| 11:04 |
* dbs |
is nervous about the mem leaks though |
| 11:05 |
berick |
yeah, memory leaks aren't gone |
| 11:05 |
dbs |
:/ |
| 11:06 |
berick |
not sure why an apache child recycling would cause that, though |
| 11:06 |
berick |
apache is smarter than that |
| 11:06 |
berick |
it doesn't kill active children or send reuqests to recently dead children |
| 11:07 |
dbs |
We've had those kinds of time-out responses for seemingly innocuous requests forever |
| 11:07 |
dbs |
Intermittent of course. The best kind of bug. |
| 11:07 |
berick |
hm. i'd be very suprised if that had anything to do w/ max_request |
| 11:08 |
jeff |
good for wipers, bad for bugs. |
| 11:09 |
dbs |
berick: might not be that (where "that" is MaxRequestsPerChild or MaxSpareServers for the IRC logs) |
| 11:10 |
berick |
*nod* could be MaxRequestWorkers -- just running out of backends |
| 11:18 |
dbs |
Could be but we're at MaxRequestWorkers 75 and nginx is caching requests for CSS, images, and JS so the load on Apache should be way lighter than in our (woeful initial) 2.10 days |
| 11:19 |
Dyrcona |
Well, you can always count the number of apache processes currently running. |
| 11:19 |
Dyrcona |
pgrep -cf apache2 |
| 11:20 |
Dyrcona |
I think Apache might also throw messages in the error log when it hits maxrequestworkers and wants to spawn new ones. |
| 11:20 |
Dyrcona |
So, I mean, you don't have to speculate. You can check. |
| 11:21 |
JBoyer |
There's also the server status module, that can be handy. (best only accessible to localhost though, for monitoring or w3m-ing. |
| 11:22 |
jeff |
dbs: you mentioned it was intermittent -- presumably that means it's rarely observed live, and only noticed in logs later, and you don't have much in the way of reliable steps to reproduce the issue? |
| 11:23 |
dbs |
jeff: nothing reliable, no |
| 11:23 |
dbs |
Dyrcona: yeah, I haven't had to do this sort of analysis since we upgraded to 2.10 last year, so the skills are dull. |
| 11:24 |
dbs |
JBoyer: right |
| 11:24 |
Dyrcona |
I don't do the nginx proxy, but what I generally find when I investigate apache starvation issues is that the same query was 100 or so times in a matter of seconds to the same brick. |
| 11:25 |
Dyrcona |
Looks like stuck enter key or something dropped on it. |
| 11:25 |
Dyrcona |
I would imagine there is a way to catch that in a proxy server and drop the extra requests. |
| 11:30 |
dbs |
I'll keep my eyes open next time a 503 comes up |
| 11:32 |
|
jihpringle joined #evergreen |
| 11:32 |
Dyrcona |
It is hard tracking this stuff down if you don't see it at the time. |
| 11:38 |
dbs |
Dyrcona: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_lock is "on" which enables "Other requests of the same cache element will either wait for a response to appear in the cache or the cache lock for this element to be released" |
| 11:38 |
Dyrcona |
dbs: OK. I need that feature. :) |
| 11:39 |
* dbs |
revisits his nginx config and notes that "proxy_cache_key $scheme$http_host$request_uri;" isn't restricting based on MIME type at all, it's just doing what Apache tells it it can cache, which keeps things relatively simple |
| 11:57 |
kmlussier |
On current master, I'm having trouble scanning any barcodes into item status in the web client. Can anyone confirm this problem? |
| 11:58 |
Dyrcona |
Scanning them or entering them by hand? |
| 12:00 |
kmlussier |
Pasting some, manually entering others. |
| 12:00 |
kmlussier |
I hit enter and nothing happens. |
| 12:03 |
miker |
kmlussier: anything in the js console? |
| 12:05 |
|
collum_ joined #evergreen |
| 12:12 |
kmlussier |
miker: https://pastebin.com/auBznJfz |
| 12:13 |
miker |
kmlussier: interesting. there should be an error in the opensrf logs, and possibly in the postgres logs |
| 12:14 |
kmlussier |
Yeah, I think I found something in the openserf log. Hold on... |
| 12:16 |
kmlussier |
Actually, it may be something with my test system. I'll investigate more and let you know if it continues to be a problem. |
| 12:25 |
pinesol_green |
[evergreen|Bill Erickson] LP#1694577 Report templates search API & paging - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=675fd78> |
| 12:25 |
pinesol_green |
[evergreen|Bill Erickson] LP#1694577 Reports template searching and paging - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=3380dac> |
| 12:25 |
pinesol_green |
[evergreen|Bill Erickson] LP#1694577 Release notes for report template search - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=40f80bd> |
| 12:25 |
pinesol_green |
[evergreen|Galen Charlton] LP#1694577: tweak searching for words in report templates - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=3fa2705> |
| 12:25 |
pinesol_green |
[evergreen|Galen Charlton] LP#1694577: exclude empheral search results folder from selector - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=71871c2> |
| 12:26 |
kmlussier |
OK, looks like I missed an upgrade script. It's working for me now. Sorry for the noise. |
| 12:33 |
pinesol_green |
[evergreen|Dan Scott] LP#1710747: Absolute URL for media_prefix in templates - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=3725e10> |
| 12:53 |
akilsdonk |
remingtron: I should have documentation for the serials module in the web client soon. Should I just replace the existing serials docs in the 2.12 docs? |
| 12:57 |
remingtron |
akilsdonk: yes, that's what I recall DIG deciding, that 2.12+ docs should only be about the web client. |
| 12:58 |
akilsdonk |
remingtron: thanks! |
| 13:00 |
|
sandbergja joined #evergreen |
| 13:07 |
jeffdavis |
dbs: I'd be curious to know more about your nginx caching setup. |
| 13:09 |
Dyrcona |
Nifty. Never seen opensrf-c invoke oom killer before. |
| 13:12 |
berick |
c'mon LP, you can do it buddy.. |
| 13:13 |
dbs |
jeffdavis: I'll try to write up a blog post, as it would undoubtedly be helpful for future-dbs who again tries to remember wth he was thinking |
| 13:14 |
berick |
dbs++ |
| 13:14 |
berick |
well, pending ++ |
| 13:18 |
jeffdavis |
dbs++ |
| 13:20 |
abneiman |
kmlussier: miker: noting for another data point that I am not seeing the item status issue on webby.evergreencatalog.com, which is on master |
| 13:21 |
kmlussier |
abneiman: yeah, I discovered the problem. It was me, not Evergreen. :) |
| 13:22 |
abneiman |
kmlussier: that's usually how my problems turn out as well :) :) |
| 13:29 |
dbs |
éwin 19 |
| 13:29 |
dbs |
oh alternate keyboards you are so much fun |
| 13:29 |
* dbs |
hits Super+space to switch back to en-us |
| 13:45 |
pinesol_green |
[evergreen|Jason Etheridge] lp1709966 webstaff: Hold Verify prompt - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=dc64505> |
| 13:48 |
JBoyer |
I think en-US key layouts would be improved with a Super key. As is the most impressive we have is Escape. |
| 13:49 |
dbs |
s/Super/Windows logo/ |
| 13:49 |
Dyrcona |
dbs: Mine has an Ubuntu log. |
| 13:50 |
Dyrcona |
logo |
| 13:50 |
Dyrcona |
I turn the Menu key into the Compose key. |
| 13:50 |
dbs |
Nice |
| 13:52 |
kmlussier |
I'm going to start pulling together point release notes for 2.12.5 and 2.11.8. If anyone is planning to merge any last-minute bug fixes, let me know so that I can include them! |
| 13:53 |
Dyrcona |
If I know the pid of an opensrf-c process, how would I find messages about it in the logs? |
| 13:54 |
berick |
Dyrcona: grep for :pid: |
| 13:54 |
berick |
will show any logs that pid created |
| 13:55 |
berick |
if you grep for just the pid (no ":") you'll likely also see other processes talking about sending jabber message to the pid (since it's part of the jabber id) |
| 13:58 |
Dyrcona |
Yeah, looks like _{PID} on the drone name. |
| 13:59 |
Dyrcona |
I don't think this was drone, though. |
| 13:59 |
Dyrcona |
In kern.log drones seem to appear as OpenSRF Drone{ |
| 13:59 |
Dyrcona |
ooops. OpenSRF Drone [ |
| 13:59 |
Dyrcona |
The names are cutoff just at the good part. |
| 14:00 |
Dyrcona |
This one's name is opensrf-c |
| 14:00 |
Dyrcona |
And grep for _PID turns up nothing. |
| 14:01 |
jeffdavis |
kmlussier: I'll add a 2.12 target for bug 1699566 - there's a fix that would be good for 2.12.5 if we can get it tested/signed off. |
| 14:01 |
pinesol_green |
Launchpad bug 1699566 in Evergreen "Barcode completion does not work in web client" [High,Confirmed] https://launchpad.net/bugs/1699566 |
| 14:02 |
Dyrcona |
Last message for that PID was a pcrud message duration, 28 minutes before the oom killer event. |
| 14:02 |
Dyrcona |
If my understanding of the oom killer output is correct, it was using 5.2 GB of RAM. |
| 14:03 |
Dyrcona |
One point 3 million total_vm pages. |
| 14:03 |
berick |
Dyrcona: fun, what pcrud request? |
| 14:04 |
berick |
"give me one of everything" |
| 14:04 |
Dyrcona |
open-ils.pcrud.search.au.atomic "960b53bb5ad697f649684d212b0335e4",{"id":{"!=":null}},{} |
| 14:04 |
berick |
hah, called it |
| 14:04 |
jeffdavis |
yikes |
| 14:04 |
Dyrcona |
But, that's 25 minutes before the crash. |
| 14:04 |
Dyrcona |
Oh! au. not aou! |
| 14:05 |
Dyrcona |
Yeah. |
| 14:05 |
Dyrcona |
That'll do it. |
| 14:05 |
jeff |
depending on how old that crash is, that may still be a valid auth token. you might want to invalidate it if there's any risk. |
| 14:05 |
jeff |
(any risk of it being valid -- i'd nuke it on principle) |
| 14:06 |
jeff |
rather, blindly attempt to nuke it and not care if it was there or not to begin with. :-) |
| 14:07 |
jeff |
(which you may have already done -- i'll stop talking about it now) |
| 14:07 |
berick |
@monologue |
| 14:07 |
pinesol_green |
berick: Your current monologue is at least 1 line long. |
| 14:07 |
berick |
doh |
| 14:08 |
Dyrcona |
@monologue jeff |
| 14:08 |
pinesol_green |
Dyrcona: (monologue [<channel>]) -- Returns the number of consecutive lines you've sent in <channel> without being interrupted by someone else (i.e. how long your current 'monologue' is). <channel> is only necessary if the message isn't sent in the channel itself. |
| 14:08 |
Dyrcona |
jeff: Nuke it for me. :) |
| 14:14 |
|
rlefaive_ joined #evergreen |
| 14:15 |
kmlussier |
jeffdavis: Thanks! I'll take a look at it. |
| 14:34 |
Dyrcona |
This gets weird: That pcrud search is sandwiched between to issuance searches in the gateway log, and I talked to the user. He was working in serials at the time and doesn't know what he did that could have triggered that au search. |
| 14:34 |
Dyrcona |
s/to/two/ |
| 14:34 |
gmcharlt |
updating a routing list? |
| 14:35 |
berick |
yeah, something with a typeahead maybe |
| 14:38 |
Dyrcona |
No, he said he didn't do anything with routing. |
| 14:38 |
Dyrcona |
He did say there was a filter that wasn't working right. I suppose I should find out exactly which one. |
| 14:44 |
Dyrcona |
OK. I think it might have the Creator filter in issuances. |
| 14:44 |
Dyrcona |
have been. |
| 14:44 |
Dyrcona |
I'll have to look into it some more tomorrow. |
| 14:46 |
Dyrcona |
I have a vm where it won't big deal if I crash it. :) |
| 14:46 |
Dyrcona |
I words.... :) |
| 15:11 |
gmcharlt |
kmlussier: bug 1708485 is now worth another look |
| 15:11 |
pinesol_green |
Launchpad bug 1708485 in Evergreen 2.12 "Web Staff Client: Missing courier code on transit slip print templates" [High,New] https://launchpad.net/bugs/1708485 |
| 15:13 |
kmlussier |
gmcharlt++ Thanks! |
| 15:29 |
pinesol_green |
[evergreen|Jeff Davis] LP#1699566: item barcode completion in web client - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=b5a9d11> |
| 15:56 |
pinesol_green |
[evergreen|Chris Sharp] LP#1708485 - Add courier code variable to transit slips. - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=bf7cd14> |
| 15:56 |
pinesol_green |
[evergreen|Galen Charlton] LP#1708485: (follow-up) improve dest_courier_code support - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=0e58c1d> |
| 15:56 |
kmlussier |
OK, I'm really done merging bug fixes now. |
| 15:57 |
gmcharlt |
I expect to merge the stripe one shortly |
| 16:06 |
pinesol_green |
[evergreen|Michele Morgan] LP 1708728 - Fix Stripe penny errors introduced by perl int() function - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=b0f7187> |
| 16:06 |
kmlussier |
gmcharlt++ |
| 16:14 |
|
_bott_ joined #evergreen |
| 16:27 |
|
_bott_ joined #evergreen |
| 16:31 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 16:55 |
|
yboston joined #evergreen |
| 17:02 |
|
mmorgan left #evergreen |
| 17:06 |
pinesol_green |
[evergreen|Kathy Lussier] Docs: Release notes for the 2.11.8 point release - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=fa26ff3> |
| 17:06 |
pinesol_green |
[evergreen|Kathy Lussier] Docs: Release notes for 2.12.5 maintenance release - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=ed74d3a> |
| 17:06 |
kmlussier |
dbwells / Bmagic: The release notes are finally done. Sorry for the delay. If it's too late in the day to build the releases, I would support moving the release to tomorrow. |
| 17:07 |
dbwells |
kmlussier: yes, I will plan on tomorrow. Thank you for your work, as always. |
| 17:08 |
Bmagic |
kmlussier++ |
| 17:11 |
kmlussier |
dbwells++ Bmagic++ Thanks! |
| 18:20 |
jeffdavis |
miker: long shot and probably wrong, but any possible relation between bug 1704396 and bug 1697029? If it matters we've applied berick's patch from https://bugs.launchpad.net/opensrf/+bug/1697029/comments/15 |
| 18:20 |
pinesol_green |
Launchpad bug 1697029 in OpenSRF "Service Listeners Crash when Using an Undefined Value for Syscalls" [High,New] |
| 18:20 |
pinesol_green |
Launchpad bug 1704396 in Evergreen "Slowness for metecord and one-hit searches in 2.12" [High,New] https://launchpad.net/bugs/1704396 |
| 18:21 |
miker |
jeffdavis: I thought it could be that, too, but IIUC that did not help nobel (and maybe Dyrcona?) |
| 18:22 |
miker |
but that should be applied in any case, as should the s/ALARM/ALRM/ patch |
| 18:25 |
jeffdavis |
I did end up applying the ALRM one. I'll apply your socket print error patch tonight. |
| 18:46 |
miker |
jeffdavis: thanks ... informational only, but if you're seeing that regularly... |
| 19:07 |
|
book`_ joined #evergreen |
| 22:10 |
|
Jillianne2 joined #evergreen |
