| Time |
Nick |
Message |
| 00:57 |
|
abowling joined #evergreen |
| 05:48 |
|
dbwells joined #evergreen |
| 07:18 |
|
rjackson_isl joined #evergreen |
| 08:01 |
|
collum joined #evergreen |
| 08:07 |
|
tspindler joined #evergreen |
| 08:30 |
|
remingtron joined #evergreen |
| 08:31 |
csharp |
hmm, I'm trying to generate the log warnings for bug 1624491, but I'm not able to and I'm not seeing the prox_cache uninitialization warnings on PINES production or test servers :-/ |
| 08:32 |
pinesol_green |
csharp: Error: Could not gather data from Launchpad for bug #1624491 (https://launchpad.net/bugs/1624491). The error has been logged |
| 08:32 |
csharp |
pinesol_green: why not? |
| 08:32 |
pinesol_green |
Factoid 'why not?' not found |
| 08:32 |
pinesol_green |
csharp: You probably want hard-boiled eggs. |
| 08:33 |
csharp |
@reload |
| 08:33 |
pinesol_green |
csharp: (reload <plugin>) -- Unloads and subsequently reloads the plugin by name; use the 'list' command to see a list of the currently loaded plugins. |
| 08:34 |
|
rlefaive joined #evergreen |
| 08:34 |
csharp |
bug 1624491 |
| 08:34 |
pinesol_green |
Launchpad bug 1624491 in Evergreen master "Avoid uninit warning for prox_cache in open-ils.circ.hold.is_possible" [Undecided,New] https://launchpad.net/bugs/1624491 |
| 08:35 |
csharp |
okay - just needed a @reload of the bugtracker plugin |
| 08:35 |
csharp |
it was gonna be a long day without that :-) |
| 08:37 |
csharp |
dbs: I'd love to sign off on your bug, but I'm not able to replicate the original problem right now - I know I've seen those in our logs before though :-/ |
| 08:38 |
|
remingtron_ joined #evergreen |
| 08:38 |
|
dbwells_ joined #evergreen |
| 08:38 |
csharp |
and there are a couple of other uninitialized warnings generated by open-ils.circ.title_hold.is_possible |
| 08:38 |
csharp |
Use of uninitialized value $depth in numeric eq (==) at /usr/local/share/perl/5.18.2/OpenILS/Application/Circ/Holds.pm line 2555. (for instance) |
| 08:44 |
|
mmorgan joined #evergreen |
| 08:49 |
rhamby |
graced: incoming |
| 08:58 |
|
mmorgan1 joined #evergreen |
| 09:02 |
|
mmorgan2 joined #evergreen |
| 09:03 |
|
terran joined #evergreen |
| 09:05 |
|
mdriscoll joined #evergreen |
| 09:19 |
|
jvwoolf joined #evergreen |
| 09:24 |
csharp |
StomproJosh: I'm looking at your fix for bug 1616220. With your fixes applied, I'm still seeing many of the same sorts of CSS errors - are you also seeing a lot of them even after your fix, or might I be doing something wrong? :-) |
| 09:24 |
pinesol_green |
Launchpad bug 1616220 in Evergreen "XUL Staff Client CSS Fixes" [Undecided,New] https://launchpad.net/bugs/1616220 |
| 09:26 |
|
yboston joined #evergreen |
| 09:27 |
|
mmorgan joined #evergreen |
| 09:33 |
|
mdriscoll_ joined #evergreen |
| 09:33 |
|
kmlussier joined #evergreen |
| 09:35 |
|
Dyrcona joined #evergreen |
| 09:40 |
|
kbutler joined #evergreen |
| 09:41 |
|
jlundgren joined #evergreen |
| 09:43 |
|
krvmga joined #evergreen |
| 09:45 |
dbs |
csharp: thanks! it's a trickier one to replicate due to the caching variance, I think. |
| 09:48 |
kmlussier |
Happy Bug Squashing Day! |
| 09:48 |
csharp |
kmlussier: and also with you |
| 09:49 |
* dbs |
wonders if anyone keeps MaxRequestWorkers below 500 for mpm_prefork.conf |
| 09:49 |
dbs |
(sez the guy whose server ate through 10GB of free RAM overnight and OOMed again) |
| 09:51 |
csharp |
dbs: our 6-apache-server setup is set to 150 for each one |
| 09:56 |
dbs |
150! |
| 09:56 |
dbs |
Oh, duh, sorry - I meant MaxConnectionsPerChild |
| 09:57 |
csharp |
oh, we have 10000 for that |
| 09:57 |
berick |
5k here |
| 09:57 |
dbs |
we're having massive memory leaks in the apache children |
| 09:57 |
jeff |
we're running with... 0? that's surprising to me. |
| 09:58 |
dbs |
The only two modules that I can think of that we run that most sites probably don't run are open-ils.resolver and open-ils.auth_proxy |
| 10:00 |
dbs |
Not sure if there's an easy way to figure out where a memory leak is occurring in apache with a bunch of c and perl mods |
| 10:00 |
jeff |
in test systems earlier this year, i could trigger high memory utilization in apache children. setting MaxConnectionsPerChild seemed to be a solid workaround, and i did have to set it pretty low. |
| 10:01 |
dbs |
jeff: interesting - but you don't need that workaround any more? |
| 10:01 |
jeff |
i'd have to look at notes, but first thing i'd try would be comparing things handled by EGCatLoader with something like static files. |
| 10:02 |
|
jlundgren joined #evergreen |
| 10:02 |
jeff |
dbs: we don't, but we also don't have much touching apache other than staff clients, some small controlled scraping, and things like jacket images. |
| 10:02 |
jeff |
dbs: in particular, we have no non-staff search traffic hitting apache. |
| 10:03 |
jeff |
so no patrons and no search engine traffic on results / record pages. |
| 10:03 |
dbs |
jeff: z39.50? |
| 10:03 |
jeff |
i expect running with MaxConnectionsPerChild 0 would be causing us more problems otherwise. |
| 10:04 |
jeff |
average VM with apache and opensrf services running is using 13G of memory |
| 10:05 |
dbs |
Yeah, it's been very weird going from 2.7 "let every bot in the world scrape at any pace" to 2.10 "aggressively deny bots and take other measures to limit traffic" |
| 10:06 |
dbs |
Never ran into an OOM on 2.7. But that was also Ubuntu 12.04 so there's 3 evergreen versions + major OS changes at play. |
| 10:06 |
dbs |
(Apache 2.2->2.4, Perl, etc) |
| 10:06 |
jeff |
what MaxConnectionsPerChild are you running at currently? |
| 10:06 |
dbs |
500 |
| 10:06 |
jeff |
and what value for MaxKeepAliveRequests? |
| 10:07 |
dbs |
500 |
| 10:07 |
berick |
keepalive is 1 second? |
| 10:07 |
dbs |
keepalive is 5 seconds |
| 10:07 |
jeff |
MaxKeepAliveRequests 100; KeepAliveTimeout 5; MaxConnectionsPerChild 0 |
| 10:08 |
berick |
dbs: i assume it was 5 seconds before? |
| 10:08 |
* berick |
has always run w/ a 1 second timeout to keep apache counts under control |
| 10:09 |
jeff |
dbs: when you're experiencing low memory conditions, is it a handful of apache processes with high memory utilization, or is it lots of apache instances with low/medium utilization? |
| 10:09 |
berick |
of course, if the number of apache procs is not high, then the timeout is not the issue |
| 10:09 |
dbs |
It was 1 on the old system. We initially matched the old system, and saw very slow response times, so we tried bringing that up and saw better response times |
| 10:11 |
dbs |
jeff: we get a medium number of apache instances with high memory utilization (top ones over 3% RAM usage on a 16GB machine, all the way down to the newer apache instances with about 1% of RAM) |
| 10:12 |
dbs |
Once you tip over 40 apache processes with an average of > 2% RAM, you're in trouble |
| 10:12 |
jeff |
my largest apache2 process in terms of memory has 356M VIRT / 110M RES |
| 10:12 |
* dbs |
will happily drop MaxKeepAliveRequests back to a lower number |
| 10:13 |
dbs |
after our restart about 1/2 an hour ago, our top one is 422692 / 178280 |
| 10:13 |
dbs |
but over time, the RES will grow to enormous amounts. I've seen > 500M |
| 10:14 |
jeff |
my oldest apache child is... minutes old. |
| 10:14 |
* jeff |
re-checks configs |
| 10:16 |
jeff |
dbs: how old is your largest apache child? |
| 10:18 |
* csharp |
reads jeff's last comment out of context and chuckles |
| 10:18 |
berick |
hah |
| 10:19 |
|
dteston joined #evergreen |
| 10:19 |
jeff |
yeah, my largest apache worker process was also my oldest, but had a lifetime of only 5 minutes or so. |
| 10:19 |
dbs |
If 'ps -p "2646" -o etime=' is to be believed, 46 minutes. |
| 10:20 |
Dyrcona |
So, I created a copy by inserting the bib record, copy and call number in the database. |
| 10:21 |
Dyrcona |
When I try to retrieve the copy information with a SIP message 17, it times. |
| 10:21 |
Dyrcona |
out. |
| 10:21 |
Dyrcona |
When I look at osrfsys.log, I don't seen any red flags. |
| 10:21 |
Dyrcona |
Anyone know what I'm missing off the top of their head? |
| 10:22 |
Dyrcona |
If you have to look, don't bother, please. |
| 10:23 |
jeff |
Dyrcona: master or your branch? |
| 10:23 |
* dbs |
goes with 'ps -C /usr/sbin/apache2 -o pid,etime' and confirms his old apache children |
| 10:23 |
jeff |
Dyrcona: have you tried running sipserver in the foreground so you don't miss warnings/errors? |
| 10:23 |
Dyrcona |
jeff: 2.9.8 and SIPServer master no patches. Ubuntu 14.04, perl 5.18? |
| 10:24 |
Dyrcona |
I'll try that. I don't see much oils_sip.xml and the osrfsys.log has millisecond response times. |
| 10:24 |
jeff |
prefork or multiplex? |
| 10:24 |
Dyrcona |
perfork. |
| 10:24 |
jeff |
(for sipserver) |
| 10:24 |
Dyrcona |
prefork. :) |
| 10:25 |
berick |
there's also a log file option to oils_ctl.sh that will capture all the stderr output |
| 10:25 |
berick |
very useful for sip deaths |
| 10:25 |
csharp |
@decide sip deaths or apache children |
| 10:25 |
pinesol_green |
csharp: go with apache children |
| 10:25 |
jeff |
in theory i'm not doing anything special to ensure that apache processes don't stick around for long. |
| 10:26 |
jeff |
in practice, i don't have any that are more than a minute or two old. |
| 10:27 |
jeff |
StartServers 5; MinSpareServers 5; MaxSpareServers 10; MaxRequestWorkers 150; MaxConnectionsPerChild 0 |
| 10:35 |
Dyrcona |
jeff: No messages from SIPServer. :( pysip2 times out. |
| 10:36 |
jeff |
Dyrcona: tried by hand? examine packet trace? see what sipserver logs in terms of the input and output messages? |
| 10:36 |
jeff |
Dyrcona: if you care to share the sql you used to create the bre/acn/acp i'd be interested in trying to reproduce. |
| 10:36 |
jeff |
if nothing else, in the interest of improving logging / error handling / avoiding this myself. ;-) |
| 10:36 |
Dyrcona |
If I use a barcode from a copy other than the ones I added, I get a response in less than 1 second. I think I'm missing something in the database. |
| 10:37 |
jeff |
yeah. |
| 10:37 |
Dyrcona |
jeff: I can paste it. I was thinking of adding these to concerto for testing purposes in the future. |
| 10:37 |
Dyrcona |
I'll bet there is some setting not switched on in concerto that I'm used to having on and that's my issue. |
| 10:38 |
dbs |
Dyrcona: any chance the bib / volume / copy ingest triggers are failing or aren't running? |
| 10:38 |
jeff |
ideally data that postgres allows you to insert wouldn't cause SIP to fail like that, but maybe the marcxml or something is to blame. |
| 10:38 |
* dbs |
is wondering if acpl is populated for example |
| 10:38 |
Dyrcona |
dbs: I'll look at that. |
| 10:38 |
* dbs |
is on the same page as jeff |
| 10:39 |
dbs |
also wonders if the evergreen.is_valid_marcxml() function + trigger needs to be resurrected for this kind of case :) |
| 10:40 |
jeff |
evergreen.is_marcxml_sufficiently_non_brief_as_to_meet_requirements_of_mods_and_sipserver() |
| 10:40 |
Dyrcona |
dbs | jeff: http://pastebin.com/J49h8GrY |
| 10:41 |
Dyrcona |
The marcxml was copied from C/W MARS production with the 901 tag and subfields with code 0 removed. |
| 10:41 |
jeff |
Dyrcona: and you're inserting these into a db with concerto applied? |
| 10:42 |
Dyrcona |
Yes. |
| 10:42 |
Dyrcona |
Since I discovered that SIPServer won't start on Ubuntu 16.04, I'm going to move on to a branch for that and then come back to this after that. |
| 10:42 |
jeff |
isn't this expected to fail on recent Encode.pm? |
| 10:43 |
Dyrcona |
jeff: Yes. That was what I was testing, but this is not a recent Encode AFAIK. |
| 10:43 |
jeff |
ah. |
| 10:43 |
Dyrcona |
It failed on 16.04, and I'm testing 14.04 for duplication, etc. |
| 10:44 |
jeff |
my guess right now is that you're running into the bug where sipserver doesn't properly flush the buffer on the socket. |
| 10:44 |
jeff |
because it's not counting bytes. |
| 10:44 |
Dyrcona |
Encode is 2.49... |
| 10:44 |
jeff |
if you check your network packets you'll probably see that the end of message isn't being sent. |
| 10:44 |
jeff |
so pysip2 hangs. |
| 10:45 |
jeff |
some sip clients will hang, some will read the message as sent so far (i think). |
| 10:45 |
Dyrcona |
I ran a loop of all the barcodes at BR1 yesterday and that worked fine. It's only failing on these new copies. |
| 10:46 |
jeff |
ah, maybe not then -- unless they have something unique about them. |
| 10:46 |
Dyrcona |
I'll get back to it in a bit. I want to fix the UNIVERSAL does not export anything message on Ubuntu 16.04. |
| 10:47 |
dbs |
the bibliotheca client barfed on the "unexpected Control character" in the title of item info until I removed clean_text for that; maybe pysip2 is doing the same? |
| 10:48 |
Dyrcona |
dbs: I doubt it, but I'll look at that. |
| 10:49 |
* Dyrcona |
now wishes he had more RAM on his laptop, so he could run two VMs at the same time. |
| 10:49 |
csharp |
@praise RAM |
| 10:49 |
* pinesol_green |
the upgrade came off brilliantly, and it's all because of RAM |
| 10:50 |
Dyrcona |
Oh, neat. Even though it works with a concerto barcode, I get this from SIPServer: Use of uninitialized value in subroutine entry at /usr/lib/perl/5.18/Encode.pm line 204, <STDIN> chunk 2. |
| 10:50 |
Dyrcona |
It's the Italian record from yesterday afternoon. |
| 10:50 |
dbs |
oldest apache process is now 1h 17minutes, and is up to 224MB / 1.3% RAM |
| 10:50 |
kmlussier |
Lots of quiet typing at NOBLE, where we have 8 people sitting around the table working on Bug Squashing Day. |
| 10:51 |
kmlussier |
Lots of coffee too. :) |
| 10:51 |
Dyrcona |
Human language is hard. Why can't we all speak Perl? |
| 10:51 |
dbs |
and we've grown from 36 apache processes to 42, whee |
| 10:51 |
dbs |
NOBLE++ |
| 10:51 |
dbs |
kmlussier++ |
| 10:51 |
dbs |
coffee++ |
| 10:51 |
|
Christineb joined #evergreen |
| 10:52 |
csharp |
NOBLE++ |
| 10:55 |
Dyrcona |
I'm going to test and push Bmagic's branch on Lp 1613326. (I think I promised to do that last month, and never found my tuit.) |
| 10:55 |
pinesol_green |
Launchpad bug 1613326 in SIPServer "SIPServer UNIVERSAL removed" [Undecided,New] https://launchpad.net/bugs/1613326 |
| 10:56 |
berick |
noble++ kmlussier++ |
| 10:56 |
jeff |
Dyrcona: pretty sure user/jeff/fix_universal_can will fix your "I want to fix the UNIVERSAL does not export anything message on Ubuntu 16.04" issue |
| 10:57 |
Dyrcona |
jeff: Bmagic also has a branch. :) |
| 10:57 |
jeff |
(if you forgive the change in style as well as the removal of the now-fatal import) |
| 10:58 |
Dyrcona |
jeff: That's not in the working SIPServer repo. |
| 10:59 |
* jeff |
raises eyebrow, checks |
| 10:59 |
jeff |
gitweb says otherwise: 4 weeks agouser/jeff/fix_universal_can |
| 11:00 |
jeff |
http://git.evergreen-ils.org/?p=working/SIPServer.git;a=shortlog;h=refs/heads/user/jeff/fix_universal_can |
| 11:00 |
jeff |
the "4 weeks ago" being a bit misleading, of course. |
| 11:02 |
* dbs |
"discovers" mod_status and http://localhost/server_status |
| 11:02 |
Dyrcona |
jeff: you just pushed it. 'Cause I checked out about an hour ago. I'm going with Bmagic's code. :) |
| 11:02 |
jeff |
correct. it did not exist there before i pointed you at it. |
| 11:03 |
terran |
kmlussier: NOBLE++ for all the Bug Squashing Day participants! |
| 11:04 |
jeff |
Dyrcona: fair enough, but please don't merge that branch as-is without correcting the misleading/incorrect commit message. |
| 11:05 |
jeff |
Dyrcona: do you object to the style change, or did you just look at Bmagic's branch first and have therefore already applied it locally, therefore: inertia? |
| 11:06 |
Dyrcona |
jeff: I'll go with yours. :) |
| 11:07 |
jeff |
hah. okay, in that case same request applies: please don't merge mine with its current WIP commit message. ;-) |
| 11:07 |
kmlussier |
To clarify, NOBLE is hosting. We also have representation from C/W MARS and Rodgers Memorial Library. |
| 11:07 |
Dyrcona |
I looked at Bmagic's first, but giving it some thought, the style change from UNIVERSAL::can to just $obj->can is a nice addition. |
| 11:08 |
Dyrcona |
I'll add Bmagic's bug number and give him some credit in commit message. How's that? |
| 11:08 |
jeff |
i'll push a new branch with a reasonable commit message and comment on the bug. |
| 11:10 |
Dyrcona |
OK. I'll wait. |
| 11:13 |
Dyrcona |
dbs | jeff: I'm convinced my solution to the Encode issues is not optimal. I haven't even really tested it yet, either. |
| 11:23 |
|
brahmina joined #evergreen |
| 11:29 |
Dyrcona |
jeff++ |
| 11:29 |
Bmagic |
jeff++ |
| 11:30 |
jeff |
updated bug 1613326 |
| 11:30 |
pinesol_green |
Launchpad bug 1613326 in SIPServer "Fix deprecated (now fatal) UNIVERSAL import" [Undecided,New] https://launchpad.net/bugs/1613326 |
| 11:30 |
jeff |
Bmagic++ thanks again for getting that rolling. i had noted the deprecation log entries but failed to open a bug. :-) |
| 11:30 |
pinesol_green |
[sipserver|Jeff Godin] LP#1613326 change UNIVERSAL::can import, style - <http://git.evergreen-ils.org/?p=SIPServer.git;a=commit;h=bb0bdfe> |
| 11:31 |
Bmagic |
I love Evergreen |
| 11:32 |
Bmagic |
and Dyrcona is a spy |
| 11:32 |
Dyrcona |
Even when I'm not. :) |
| 11:33 |
|
jlundgren joined #evergreen |
| 11:34 |
csharp |
viva la RESISTANCE |
| 11:34 |
csharp |
Bmagic: make sure to bring that to the hackaway! |
| 11:34 |
* Dyrcona |
has to listen to The Doors, now. |
| 11:34 |
Bmagic |
oh it goes without saying! |
| 11:34 |
csharp |
:-D |
| 11:35 |
Bmagic |
csharp: as long as you bring your ukulele |
| 11:35 |
Dyrcona |
:) |
| 11:36 |
berick |
i bought the game, but haven't had a chance to play yet. looking forward to playing at the hackaway. |
| 11:36 |
Bmagic |
berick++ # supporting the resistance |
| 11:36 |
Dyrcona |
Now, to figure out why my copies show up int he staff client, but don't show up in SIP2 requests.... |
| 11:47 |
jeff |
Dyrcona: on that, i still suspect that a packet trace would help you determine if the server is sending a partial response (lacking at least the end of message character) due to failing to properly flush the buffer. |
| 11:47 |
* jeff |
opens a bug for that issue, seeing none |
| 11:49 |
|
jihpringle joined #evergreen |
| 11:52 |
* Dyrcona |
fires up tcpdump |
| 12:01 |
Dyrcona |
Noice! I seem to be getting jibberish back. |
| 12:03 |
Dyrcona |
Hrm. No. I get the 18 response and then the jibberish. Maybe dbs is right and pysip2 is choking. |
| 12:03 |
jeff |
do you have the message terminator (CR, 0x0d)? |
| 12:04 |
jeff |
can you compare with a successful message pair and a problematic one? |
| 12:06 |
* berick |
confirms no CR means pysip2 will continue waiting |
| 12:06 |
Dyrcona |
I might be able to do that. |
| 12:07 |
Dyrcona |
I'll have tcpdump log to a file. |
| 12:08 |
jeff |
if you're using tcpdump, depending on your version you'll want to ensure a useful snaplen. |
| 12:08 |
jeff |
-s0 often does it |
| 12:09 |
jeff |
though checking my advice shows that's the same as the default at least as far as Debian wheezy and jessie are concerned. |
| 12:10 |
jeff |
and if your SIP2 packets are larger than 65535 bytes, we probably don't care what the 65536th byte looks like. |
| 12:17 |
berick |
Dyrcona: something like this might help too: https://gist.github.com/berick/4bf7f1593cecb4ed07667ce0fd036c10 |
| 12:17 |
berick |
when I add that, I can see the \r's in the recv buffer |
| 12:19 |
Dyrcona |
Y'know. I may just try putting clean_text() on autoload Item fields and see what that does. |
| 12:20 |
Dyrcona |
But first, I'll finish lunch. |
| 12:20 |
jeff |
title/author with mangled characters were the most common trigger for us. i think that's due to a different issue in how MODS transforms are being done. |
| 12:25 |
terran |
Bug Squashing Day Update: 6 patches have been tested & signed off on, and 3 new patches have been submitted! |
| 12:34 |
jeff |
Dyrcona: bug 1628995 |
| 12:34 |
pinesol_green |
Launchpad bug 1628995 in SIPServer "SIPServer can incorrectly calculate output message length, which can lead to clients hanging" [Medium,New] https://launchpad.net/bugs/1628995 - Assigned to Jeff Godin (jgodin) |
| 12:37 |
dbs |
jeff++ # different fun with encodings (bytes vs. length), heh |
| 12:38 |
dbs |
err, bytes vs. chars for length |
| 12:38 |
jeff |
yup! |
| 12:40 |
kmlussier |
terran++ |
| 12:41 |
dbs |
jeff: interesting that used to be just 'print "$msg\r";' before the multiplex merge |
| 12:41 |
dbs |
too clever by half? |
| 12:44 |
Dyrcona |
I thought we fixed the bytes vs. chars thing at one point. |
| 12:44 |
|
bmills joined #evergreen |
| 12:44 |
jeff |
Dyrcona: potentially in a error detection context? |
| 12:44 |
dbs |
we did, but that was before multiplex. or at least one time that we fixed it--yes, checksum |
| 12:45 |
dbs |
perldoc -f length suggests "length(Encode::encode_utf8(EXPR))" to calculate bytes :) |
| 12:45 |
dbs |
DRAGONS! THERE BE DRAGONS ALL AROUND! |
| 12:45 |
* jeff |
nods |
| 12:45 |
Dyrcona |
OIC. (finally reading jeff's bug.) |
| 12:45 |
Dyrcona |
dbs++ |
| 12:49 |
dbs |
My longest-lived apache process is now 1h 57m old, has served 394 requests, and is 240M / 1.4% memory. For those who care about such things :) |
| 12:50 |
Dyrcona |
dbs: I recall seeing Apache processes hit 700+MB at one point. |
| 12:51 |
dbs |
Dyrcona: Oh yeah, we have been there before too; 600MB at least |
| 12:53 |
dbs |
Not sure how apache routes requests to children, but most of the other apache processes are much younger (and slimmer), suggesting that this one is getting saved for special requests? Or is just being shunned because it's bigger and slower :) |
| 12:54 |
* Dyrcona |
adds use bytes; before POSIX::write to see if that makes a difference. |
| 12:55 |
Dyrcona |
Bingo! We have a winner! |
| 12:56 |
Dyrcona |
pysip2 doesn't choke on the multi-byte call number. |
| 12:56 |
csharp |
@blame testing the wrong git branch |
| 12:56 |
pinesol_green |
csharp: testing the wrong git branch WILL PERISH UNDER MAXIMUM DELETION! DELETE. DELETE. DELETE! |
| 12:58 |
Dyrcona |
I'm inclined to fix the length calculation as its own bug and not mix it up with the Encode changes. |
| 12:58 |
jeff |
i don't know that that's the best / long term fix, but i'll amend my commit message and push that branch. we've been using it in production for a while. |
| 12:59 |
Dyrcona |
Where'd you put the use bytes; out of curiosity. |
| 12:59 |
Dyrcona |
? |
| 13:06 |
dbs |
makes sense to keep it locally scoped to the length() call, to avoid having any other side effects |
| 13:07 |
* dbs |
greps to see 15 length() calls to audit to see whether use bytes would be warranted or not |
| 13:07 |
Dyrcona |
yeah, in my test, I did the same. |
| 13:09 |
dbs |
+1 to fixing this one little bug first |
| 13:10 |
* miker |
thinks `require bytes; ... bytes::length()` may be better? |
| 13:10 |
Dyrcona |
miker: It may be better. |
| 13:10 |
miker |
or, at least, it causes the least amount of changes, I think, after reading the perldoc for bytes |
| 13:10 |
Dyrcona |
Looking at the other lenght calls, the on in SIPServer.pm on the input might need the same treatment. |
| 13:11 |
Dyrcona |
s/on/one/ |
| 13:11 |
jeff |
added working branch. |
| 13:11 |
jeff |
we've been running with that change since march or may. |
| 13:11 |
* dbs |
chuckles at "perldoc bytes" which says "bytes::length() is admittedly handy if you need to know the byte length of a Perl scalar. But a more modern way is: length(encode('UTF-8', $scalar))" |
| 13:12 |
jeff |
saw the alternate approach on the bytes docs. looking again. i don't remember if i considered that or not. |
| 13:12 |
jeff |
of course, there's also "Use of this module for anything other than debugging purposes is strongly discouraged." |
| 13:13 |
dbs |
in bold, even |
| 13:13 |
jeff |
maybe that's why this branch was flagged WIP here. :-) |
| 13:13 |
* miker |
doesn't care about modern, personally. bytes::length() seems so much more readable |
| 13:13 |
miker |
and that's saying something, in perl |
| 13:15 |
Dyrcona |
:) |
| 13:18 |
dbs |
I'm worried about adding another way of working around the same basic problem--this one where the docs explicitly say "don't use this!"--so much cognitive overload for someone coming into the code to process |
| 13:18 |
* jeff |
nods |
| 13:19 |
Dyrcona |
Well, I'll go along with the consensus. It would be good to wrap the encoding/bytes issues up at once with something comprehensive. |
| 13:20 |
jeff |
i'm going to revise my working branch to use the bytes::length() approach (thanks, miker!), but i'm not going to be disappointed if the entire thing is obsoleted by something more comprehensive. |
| 13:21 |
dbs |
But maybe short-term fix is to use bytes locally or bytes::length() to fix the immediate problem and close the bug, with a reference to a new bug that says "don't use bytes / bytes::length, use Encode" |
| 13:21 |
jeff |
i think the bit about it being more modern to spell that ``length(encode('UTF-8', $scalar))'' raises the issue of "we don't know how things are encoded at this point, which is part of the underlying issue". |
| 13:21 |
dbs |
so that at least we get working code in master |
| 13:21 |
jeff |
right. |
| 13:21 |
dbs |
jeff++ |
| 13:22 |
Dyrcona |
OK. |
| 13:22 |
dbs |
also a comment in the code and in the commit log about the bad code smell of "use bytes" to help any future inquires as to "why the hell was this being used?" :) |
| 13:22 |
Dyrcona |
I think a couple of other calls in Sip.pm could use that same treatment. |
| 13:22 |
jeff |
i was going to continue to say that this fix may be immediately useful to all users of sipserver who may encounter this problem, while a more comprehensive fix that obsoletes this approach may require sipserver + evergreen changes, be part of a larger upgrade/feature/fix, etc. :-) |
| 13:23 |
Dyrcona |
lines 95, and maybe 179 and 199 |
| 13:24 |
Dyrcona |
I misspoke about SIPServer.pm abov. |
| 13:24 |
Dyrcona |
above. |
| 13:29 |
abowling |
i've tackled this before, but i've slept and forgotten. trying to get a test server up and running to get rid of these darned bugs, and getting: /usr/bin/ld: cannot find -ldbdpgsql |
| 13:29 |
abowling |
anyone have any suggestions/memories? |
| 13:29 |
abowling |
i have installed 9.3 |
| 13:36 |
berick |
abowling: what's the output of: cat /etc/ld.so.conf.d/evergreen.ld.conf |
| 13:39 |
abowling |
berick: might be part of my probelm |
| 13:39 |
abowling |
cat: /etc/ld.so.conf.d/evergreen.ld.conf: No such file or directory |
| 13:40 |
jeff |
miker, others: opinions on "require bytes" being in the same lexical scope as the POSIX::write / bytes::length call? I'm tempted to put it in the Sip package itself, since 1) "require bytes" shouldn't have any side effects and 2) there is a slight possibility we may find other calls to length() that should be bytes::length() before we're done. |
| 13:40 |
abowling |
berick: root evergreen-2-10-6-test:/etc/ld.so.conf.d# cat /etc/ld.so.conf.d/eg.conf |
| 13:40 |
abowling |
/usr/local/lib/dbd |
| 13:41 |
berick |
abowling: is there stuff in /usr/local/lib/dbd/ ? |
| 13:42 |
abowling |
berick: there is, but nothing for psql |
| 13:42 |
berick |
k, you should see libdbdpgsql.so |
| 13:42 |
berick |
and .la |
| 13:42 |
abowling |
berick: python2.7 python3.4 |
| 13:43 |
berick |
abowling: either libdbdpgsql.so is somewhere else (try 'find'ing it) or it was not installed. |
| 13:43 |
abowling |
berick: grrr. found it! |
| 13:43 |
berick |
abowling: fwiw, on ubuntu 16.04 it lives in /usr/lib/x86_64-linux-gnu/dbd |
| 13:43 |
abowling |
/usr/lib/i386-linux-gnu/dbd/libdbdpgsql.la |
| 13:43 |
Dyrcona |
berick: It works without any extra configuration on Ubuntu since 12.04, IIRC. |
| 13:43 |
abowling |
what's interesting is it's 14.04. i wonder if it's hyper-v shenanigans |
| 13:43 |
jeff |
Dyrcona: some of those calls to length() might benefit, but some may be limited to "do we have double-encoded characters in a patron barcode received as input?" |
| 13:44 |
berick |
abowling: so add /usr/lib/i386-linux-gnu/dbd/ to eg.conf and 'sudo ldconfig' |
| 13:44 |
abowling |
berick: got it. thanks! |
| 13:44 |
Dyrcona |
jeff: I said "maybe" :) |
| 13:45 |
jeff |
Dyrcona: since we haven't hit any problems in there, I'm also tempted to say that they've of such low priority that they don't bear investigation, focus on the more comprehensive fix that obsoletes them all. ;-) |
| 13:45 |
Dyrcona |
jeff: Fair enough. |
| 13:46 |
miker |
jeff: I'm for putting the require at the top of whatever file uses it |
| 13:46 |
* Dyrcona |
agrees with miker. |
| 13:46 |
berick |
abowling: oh, you may also need to add /usr/lib/i386-linux-gnu/ |
| 13:55 |
Dyrcona |
I dropped the pullrequest tag on Lp 1463943 |
| 13:55 |
pinesol_green |
Launchpad bug 1463943 in SIPServer "Non-ascii Unicode characters in messages cause client problems" [Undecided,New] https://launchpad.net/bugs/1463943 - Assigned to Jeff Godin (jgodin) |
| 14:13 |
jeff |
Dyrcona: still working on it this week, or holding off? |
| 14:17 |
Dyrcona |
I might hold off. I've got a bandaid in place in production. |
| 14:17 |
Dyrcona |
If you want to tackle it, feel free. |
| 14:21 |
jeff |
Dyrcona: |
| 14:21 |
jeff |
heh. |
| 14:27 |
Dyrcona |
:) |
| 14:30 |
Dyrcona |
miker: Is it all right to mess with bug targets for 2.11? We don't have milestones for anything beyond 2.11-rc. |
| 14:34 |
mmorgan |
Since it's bug squashing day, any thoughts on lp 1482757? |
| 14:34 |
pinesol_green |
Launchpad bug 1482757 in Evergreen "Loading records with located URIs should not delete and recreate call_numbers" [Undecided,Confirmed] https://launchpad.net/bugs/1482757 |
| 14:38 |
miker |
Dyrcona: mess away |
| 14:39 |
Dyrcona |
miker: Will do, as bug manager. |
| 15:04 |
mmorgan |
dbwells: I'm testing lp 1422379 and see mention of a less onerous upgrade script than originally proposed https://bugs.launchpad.net/evergreen/+bug/1422379/comments/13 |
| 15:04 |
pinesol_green |
Launchpad bug 1422379 in Evergreen "Move money.billing timestamps back to moment of fine" [Medium,Triaged] - Assigned to Michele Morgan (mmorgan) |
| 15:05 |
mmorgan |
, but don't see the alternate upgrade script, but agree that it sounds much less onerous. |
| 15:06 |
kmlussier |
@praise bug squashers |
| 15:06 |
* pinesol_green |
bug squashers goes to eleven |
| 15:06 |
kmlussier |
Bad grammar. |
| 15:06 |
mmorgan |
I've done a lot of testing, generating a lot of fines for hourly as well as daily loan periods and finesk and it looks good so far. |
| 15:08 |
mmorgan |
Did I really say that??? Been a long day... |
| 15:11 |
|
ethomsen joined #evergreen |
| 15:22 |
jeff |
mmorgan: were you able to do any testing on a system with billings dating back a long ways, to Evergreen 1.2.x.x and such? |
| 15:23 |
mmorgan |
jeff: no, I don't have access to such a system. |
| 15:24 |
mmorgan |
Is the only concern there the upgrade script for such systems? |
| 15:26 |
jeff |
i wouldn't say it's my *only* concern, no. :-) |
| 15:28 |
mmorgan |
jeff: What are some of your other concerns? |
| 15:31 |
jeff |
i should look at the upgrade script to see if they're still valid, but i do have concerns about changing existing billing timestamps. |
| 15:31 |
jeff |
it's possible that the latest doesn't do that. |
| 15:32 |
mmorgan |
The latest upgrade script I see does change the billing timestamps. If they shouldn't/don't need to be changed, is an upgrade script necessary? |
| 15:33 |
|
ethomsen left #evergreen |
| 15:33 |
mmorgan |
jeff: What are your concerns about changing existing billing timestamps? |
| 15:39 |
Bmagic |
Anyone have some insight on this error when updating the xul client? "Update XML file malformed (200)" |
| 15:45 |
jeff |
mmorgan: numerous. not sure i can get into them all now, but there's a good bit of conversation here: http://irc.evergreen-ils.org/evergreen/2014-01-08#i_57827 |
| 15:45 |
jeff |
mmorgan: and probably at least one other batch later, though i've not found it in a quick search. |
| 15:45 |
mmorgan |
jeff: Ok, thanks, will take a look. |
| 15:45 |
* jeff |
looks at the bug again, having not done so in a while |
| 15:54 |
jeff |
also, my data doesn't seem to match the statements in comment 6: https://bugs.launchpad.net/evergreen/+bug/1422379/comments/6 |
| 15:54 |
pinesol_green |
Launchpad bug 1422379 in Evergreen "Move money.billing timestamps back to moment of fine" [Medium,Triaged] - Assigned to Michele Morgan (mmorgan) |
| 15:55 |
jeff |
either that, or i'm misinterpreting what dbwells is saying, or he misunderstood my concern. :-) |
| 15:55 |
* jeff |
makes note to revisit and comment on that bug |
| 15:56 |
* dbs |
retrieves a patron in the 2.10.7 webby client and sees 685 "Error: [$interpolate:interr]" console messages -- but it works ;) |
| 15:58 |
jeff |
oh hey, with most (all?) omnibus-like webstaff branches currently merged to master, now would be the perfect time to rename the controllers! :-) |
| 15:58 |
dbs |
definitely beta |
| 15:58 |
jeff |
and take care of the flash of untemplated content. |
| 15:58 |
jeff |
more bugs to create. |
| 15:59 |
dbs |
yup yup yup! |
| 15:59 |
berick |
jeff: rename the controllers? |
| 16:00 |
berick |
dbs: not seeing those errors in master, fortunately |
| 16:00 |
jeff |
berick: i'm joking, but only half. currently the controllers are all FooBarCtrl, and FooBarController is the "preferred" pattern. |
| 16:01 |
berick |
ah |
| 16:01 |
jeff |
berick: at least one google-recommended (authored? can't recall) tool throws errors/warnings on every single one. |
| 16:02 |
miker |
dbs / berick: if that's the error I'm thinking of, I addressed that recently by pushing a scope.apply() call into a timeout -- it was being called during a digest |
| 16:03 |
dbs |
miker: yeah, not seeing it on the demo webby. yayz! |
| 16:03 |
dbs |
and 2.10 was explicitly called out as not yet production ready; 2.11 too right? So it wasn't bothering me, just amusing me :) |
| 16:03 |
dbs |
miker++ |
| 16:03 |
dbs |
fix0rz |
| 16:04 |
|
tspindler left #evergreen |
| 16:05 |
miker |
dbs: yes on 2.10 ... and I wouldn't replace your staff clients with 2.11 ;) |
| 16:06 |
|
jlundgren joined #evergreen |
| 16:07 |
dbs |
yeah, although it feels like our XUL clients are going to explode. might be our general lagginess/performance issues, hopefully tonight's reboot with much lower maxkeepalivetimeouts / keepalive / startservers / spareservers will address that a bit |
| 16:08 |
* miker |
crosses fingers for dbs |
| 16:10 |
Bmagic |
I'm working on setting up some new app servers with 2.11. It would be great if the staff client from the old installation would upgrade itself. I have got this working before, but for some reason it's not working this time. |
| 16:10 |
Bmagic |
dbs: we recently tweaked those settings and it made a big difference |
| 16:12 |
Bmagic |
What is the magic glue for xulrunner? Which file is it reading on the server when it complains about malformed xml? |
| 16:15 |
dbs |
Bmagic: cool, it keeps my hope alive :) |
| 16:15 |
terran |
Bug Squashing Day Update: 8 new patches submitted, 13 patches signed off on, and 5 patches committed or released! |
| 16:23 |
Bmagic |
terran++ |
| 16:24 |
Dyrcona |
terran: are you counting the bug status changes made by the bugmaster account earlier today? |
| 16:36 |
jeff |
miker: if you're still around, do you remember what brought POSIX::write into the picture around the time of SIP multiplex, over print? Difference in how the file handle is passed? |
| 16:36 |
miker |
jeff: IIRC, because net::server does terrible things to print, et al, and the way to bypass that is to use the POSIX module |
| 16:37 |
Dyrcona |
jeff: I was thinking if we do the encoding in write_msg as in the branch that I just pulled pullrequest from we could do this "the right way." |
| 16:42 |
Dyrcona |
Do you want me to do an example branch? |
| 16:42 |
* Dyrcona |
can't English so good. |
| 16:43 |
miker |
jeff: there may have been some "sip is byte-wise, and so are posix::read and ::write" thinking as well... |
| 16:43 |
Dyrcona |
Hmm... If the message is encoded in the target encoding before length is called, does length get the correct value? |
| 16:43 |
Dyrcona |
Sometimes, Perl gets in the way.... :) |
| 16:43 |
miker |
we could use pack and upack instead, I guess ... :) |
| 16:43 |
Dyrcona |
Yeah, that's the ticket.... :) |
| 16:43 |
* Dyrcona |
gives Sip::Checksum::checksum the hairy eyeball. |
| 16:43 |
abowling |
berick: sorry to be a pest, but this is racking my brain |
| 16:43 |
abowling |
i have this in /etc/ld.so.conf.d/eg.conf |
| 16:44 |
abowling |
/usr/local/lib/dbd |
| 16:44 |
abowling |
/usr/lib/i386-linux-gnu |
| 16:44 |
abowling |
/usr/lib/i386-linux-gnu/dbd |
| 16:44 |
Dyrcona |
abowling did you install libdbpgsql from source and not from a a package? |
| 16:44 |
Dyrcona |
If you're on recent Ubuntu or even Debian, the package just works. |
| 16:45 |
miker |
and, just to confirm the obvious, you ran ldconfig after adjusting ld.so.conf (and friends)? |
| 16:45 |
abowling |
miker: sure did |
| 16:45 |
abowling |
ran from package |
| 16:46 |
abowling |
damnedest thing. i've only done this about 984 times before :) |
| 16:49 |
abowling |
these are present: |
| 16:49 |
abowling |
/usr/lib/i386-linux-gnu/dbd/libdbdpgsql.la |
| 16:49 |
abowling |
/usr/lib/i386-linux-gnu/dbd/libdbdpgsql.so |
| 16:49 |
Dyrcona |
Well, good luck. I've not had that issue with the packages, so I have no suggestions other than those that have already been offered. |
| 16:49 |
* Dyrcona |
calls it a day. |
| 16:50 |
|
serflog joined #evergreen |
| 16:50 |
|
Topic for #evergreen is now Welcome to the #evergreen library system channel! | We are publicly logged: http://irc.evergreen-ils.org/evergreen | Large pastes at http://paste.evergreen-ils.org |
| 17:01 |
terran |
Dyrcona: Yes, I counted the bugmaster changes that I saw |
| 17:01 |
|
jvwoolf left #evergreen |
| 17:09 |
dbwells |
mmorgan: Thanks for testing that branch! I pushed a rebased version with the reworked upgrade script just now: http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbwells/lp1422379_move_billing_timestamps_rebase_v3 |
| 17:16 |
abowling |
berick++ |
| 17:16 |
abowling |
berick: you led me down the trail. the fix is this: cp -R /usr/lib/i386-linux-gnu/* /usr/lib/x86_64-linux-gnu/ |
| 17:16 |
abowling |
after that, make runs fine |
| 17:36 |
pinesol_green |
[evergreen|Remington Steed] LP#802700 Sort funds by code and year - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=a2b809e> |
| 19:50 |
|
dcook joined #evergreen |
| 21:40 |
dbs |
Dear Apache: when I say "MaxConnectionsPerChild 500", you're not supposed to show children with 579 accesses in server_status, right? THOSE CHILDREN ARE SUPPOSED TO DIE. |
| 23:29 |
pinesol_green |
[evergreen|Bill Erickson] LP#1618992 Work log checkin/user sanity checks - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=56099a4> |
| 23:29 |
pinesol_green |
[evergreen|Bill Erickson] LP#1618992 Webstaff checkin error handler repairs - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=bd60cd1> |
| 23:29 |
pinesol_green |
[evergreen|Bill Erickson] LP#1618992 Webstaff checkin UI bib title link repair - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=4b71761> |
| 23:38 |
|
kmlussier joined #evergreen |
| 23:39 |
* kmlussier |
will be playing in the role of bshum this evening. |
| 23:47 |
pinesol_green |
[evergreen|Jim Keenan] LP#1629029: Fixed missing space in line 11 - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=7f06a7b> |
| 23:59 |
pinesol_green |
[evergreen|Bill Erickson] LP#1565009 Webstaff patron search progress bar - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=cfb3755> |
