| Time |
Nick |
Message |
| 01:36 |
|
mrisher_ joined #evergreen |
| 01:41 |
|
mrisher joined #evergreen |
| 06:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 06:55 |
|
rjackson_isl_hom joined #evergreen |
| 08:00 |
|
mmorgan joined #evergreen |
| 08:00 |
|
alynn26 joined #evergreen |
| 08:01 |
|
mantis1 joined #evergreen |
| 08:49 |
|
Dyrcona joined #evergreen |
| 08:50 |
|
rfrasur joined #evergreen |
| 09:33 |
mmorgan |
Are compiled Angular files portable? That is, can the compiled files be copied from one server to another? |
| 09:36 |
|
dbwells joined #evergreen |
| 09:42 |
Dyrcona |
mmorgan: They should be since they're still just JavaScript that runs in a browser. |
| 09:48 |
|
jvwoolf joined #evergreen |
| 09:51 |
berick |
mmorgan: yes, you can copy them around |
| 09:52 |
mmorgan |
berick++ Dyrcona++ |
| 09:52 |
mmorgan |
Thanks! That's helpful! |
| 10:11 |
csharp |
mmorgan: I usually do ng build --prod and rsync them across other servers |
| 10:12 |
csharp |
there are other approaches, but that seems the best way to ensure it's literally the same files everywhere |
| 10:13 |
berick |
assuming identical git checkouts, they will be the same (in case there's any concern there) |
| 10:13 |
csharp |
(for the logs, the directory to sync is /openils/var/web/eg2/) |
| 10:13 |
csharp |
right |
| 10:15 |
csharp |
berick: after 24 hours of running all three patches for the batch actions fixes, I can vouch for all three |
| 10:15 |
mmorgan |
csharp++ |
| 10:15 |
csharp |
do you want me to go ahead and sign off on them separately, or do you want to combine them, or wait for more potential batch things? |
| 10:16 |
* csharp |
would love to hear what jeffdavis thinks about that |
| 10:16 |
berick |
csharp++ # nice |
| 10:18 |
berick |
csharp: maybe note in the LP that they're working for you and ask jeffdavis to further clarify before our imminent merge |
| 10:18 |
csharp |
can do |
| 10:28 |
|
rlefaive joined #evergreen |
| 10:52 |
pastebot |
"Bmagic" at 168.25.130.30 pasted "Query to find mis-matched EDI invoices to POs" (12 lines) at http://paste.evergreen-ils.org/10067 |
| 10:53 |
Bmagic |
Still pounding away at the EDI INVOIC conundrum, It appears that Evergreen quite frequently will assign invoice_entries to PO's that aren't the same PO as the one it matched in a previous step (edi_message). Does that query return anything for anyone? |
| 10:55 |
Bmagic |
adjust date span to taste |
| 10:58 |
tlittle |
Bmagic I got results. I'm looking at it in the UI right now to see what it looks like |
| 10:58 |
Bmagic |
ty! |
| 11:07 |
tlittle |
Bmagic Bear with me, I'm wrapping my brain around it. So is the idea that the PO attached to the EDI message is different than the one attached to the invoice_entry? Cause you can have multiple INVOIC messages in one EDI message |
| 11:31 |
Bmagic |
tlittle: right, but a single incoming file does make multile edi_message rows, one for each PO |
| 11:40 |
Dyrcona |
Invoices could mix and match items from different POs, no? |
| 11:41 |
Dyrcona |
I guess the question that needs answering is where does acq.invoice_entry get the PO number? If its from the INVOIC message, then its probably a data error on the part of the vendor. |
| 11:42 |
Dyrcona |
Just spitballing. |
| 11:46 |
Bmagic |
as far as I can discern, it looks like the code splits the incoming EDI message into individual edi_message rows. One for each detected PO. The purchase_order column is populated accordingly |
| 11:48 |
Bmagic |
which is why it makes since for it to refer back to the row ID number when creating the coorisponding acq.invoice row. It makes a note in the acq.invoice.note column referring to the coorisponding acq.edi_message row ID |
| 11:48 |
Bmagic |
/since/sense |
| 11:49 |
Bmagic |
the purchase_order column located in acq.invoice_entry is redundant because it also links to a lineitem (acq.lineitem) which, in turn links back to a PO again. Those do agree with eachother in my examples |
| 11:51 |
Bmagic |
I'm sure I am just missing something here considering we have examples of the EDI PO mismatch going back many years |
| 11:52 |
Bmagic |
but I do have one example where the EDI message was for PO X (in the EDI message itself) and Evergreen plopped the invoice records onto an ancient PO from 2013, matching lineitems and everything from 2013 |
| 11:53 |
Bmagic |
the part that is funny is that the PO that was recorded onto the row in acq.edi_message IS CORRECT |
| 11:54 |
|
dbwells joined #evergreen |
| 12:05 |
|
jihpringle joined #evergreen |
| 12:37 |
|
rlefaive joined #evergreen |
| 12:50 |
tlittle |
Bmagic I'm wondering if it's taking the first PO ID that it runs across in the EDI message and then stamping that as the PO ID for all rows attached to that message. Here's my query output, slightly modified: https://pastebin.com/NDnjPCei |
| 12:50 |
Bmagic |
tlittle++ # thanks for spending so much time on this! |
| 12:50 |
tlittle |
The first PO ID present in that EDI message in the RFF+LI is 2942, and even though there are subsequent PO IDs in subsequent RFF+LI's, all the rows attached to that edi message have the same PO ID |
| 12:51 |
tlittle |
Np, I like puzzles :) |
| 12:51 |
tlittle |
Also, I stepped away from it for awhile and came back to it |
| 12:52 |
Bmagic |
I expanded my query a little to look for invoices in these categories that were attached to PO's that had lineitems attached to Funds where the YEAR of the fund was more than 2 years different than the invoice creation time. I found none! |
| 12:52 |
pastebot |
"Bmagic" at 168.25.130.30 pasted "Query to find mis-matched EDI invoices to POs more than two years different" (18 lines) at http://paste.evergreen-ils.org/10068 |
| 12:52 |
Bmagic |
change date range to taste |
| 12:54 |
tlittle |
Looks like none here either |
| 12:54 |
Bmagic |
so, I think you might be onto something with the "first PO number stamped for all rows" |
| 12:56 |
Bmagic |
There is logic elsewhere that does the matching invoice->PO->lineitem. Which is where it broke down in my example and attached the invoice to an oustanding PO from 2013. I subsequently modified the column "state" for the 2013 lineitem_detail's to be "received" instead of "on-order" - which should* prevent future processes from turning them up in search results |
| 12:57 |
tlittle |
So it matches the PO as the middle piece? I always think of the lineitem as the connector piece |
| 12:58 |
Bmagic |
invoice_entry points to a PO as well, but that indication seems redundant because invoice_entry->lineitem->purchase_order |
| 12:59 |
tlittle |
Yeah agree |
| 12:59 |
Bmagic |
I did find that in the interface, when viewing a PO, the "View Invoice(s) (X)" button will only turn up related invoices when lineitems are linked to a PO. I made the mistake (when fixing this issue) of only changing acq.invoice_entry.purchase_order to point to the correct PO and didn't update the lineitem columns to match the PO lineitems |
| 13:00 |
Bmagic |
A second step was required to update the lineitems for the mistaken invoice_entry rows -> lineitems of the target PO |
| 13:02 |
tlittle |
Ahh gotcha. Always fun thinking of all the connect-y pieces. :) We have to restore fund debits a decent amount, and so wiring those back up is always fun |
| 13:03 |
|
mantis1 joined #evergreen |
| 13:03 |
tlittle |
Since those come from line item details, which are connected to lineitems, that's probably why I think of line items first and not purchase orders |
| 13:07 |
mantis1 |
I got a library asking for help with putting an image in their receipt template. It shows up in the template preview but not the dialog print preview dialog in Chrome. I was told the image did print out one time today in a checkout receipt though. Has anyone else ran into this problem? |
| 13:09 |
Bmagic |
mantis1: does the template have the entire image embedded using base64 or something like that? Or is a simple <img src="somelink"> ? |
| 13:12 |
Bmagic |
example of an embedded image encoded - http://jsfiddle.net/hpP45/ |
| 13:18 |
mantis1 |
It's <img src="link"> |
| 13:21 |
Bmagic |
try base64 embed method. I think you'll have better results |
| 13:25 |
mantis1 |
Thank you@ |
| 13:25 |
mantis1 |
I'll give it a shot |
| 13:26 |
mantis1 |
Bmagic ++ |
| 13:26 |
Bmagic |
worked? |
| 13:37 |
|
jihpringle14 joined #evergreen |
| 13:46 |
mantis1 |
Bmagic: can you not put a full URL into the string? |
| 13:47 |
|
jvwoolf joined #evergreen |
| 13:50 |
Dyrcona |
Bmagic | tlittle: That sounds like a bug to me if only the first PO id is being used. |
| 13:51 |
|
mrisher joined #evergreen |
| 13:52 |
mantis1 |
Bmagic: I got it! |
| 13:52 |
mantis1 |
Thank you! |
| 14:06 |
|
Christineb joined #evergreen |
| 14:17 |
JBoyer |
mantis1, Another thing to check when a receipt won't show an image like that is that it has to be https and a full link, not a relative link. Relative links will probably work in the preview but not in the real thing. |
| 14:25 |
|
mantis2 joined #evergreen |
| 14:31 |
|
sandbergja joined #evergreen |
| 14:34 |
Dyrcona |
Would anyone be interested in changes to allow Evergreen's Apache to run on plain HTTP and let the proxy server handle the HTTPS? This came up briefly in some discussion at the hack-away. |
| 14:37 |
JBoyer |
Dyrcona Very +1 |
| 14:38 |
JBoyer |
I have a very preliminary patch that I've been meaning to throw up onto working. It's not finished because I want to use straight port 80 on localhost, but it suffices to stick it behind nginx without using SSL between the two. |
| 14:40 |
rjackson_isl_hom |
trying to get reboots of servers happy after our 3.4 upgrade and am a bit lost (as per usual) |
| 14:41 |
rjackson_isl_hom |
need to get websocketd running on reboot |
| 14:41 |
rjackson_isl_hom |
--> /usr/local/bin/websocketd --port 7682 /openils/bin/osrf-websocket-stdio & |
| 14:41 |
rjackson_isl_hom |
does it need a --maxfork=250 (for example) added to the stsrtup? |
| 14:41 |
rjackson_isl_hom |
or leave that param out of the mix? |
| 14:44 |
berick |
rjackson_isl_hom: --maxforks is not required. |
| 14:44 |
berick |
generally good to have, though, since otherwise it will fork until it runs out of memory |
| 14:44 |
berick |
if there were an abusive client |
| 14:44 |
berick |
or just broken |
| 14:45 |
rjackson_isl_hom |
berick++ |
| 14:52 |
Bmagic |
rjackson_isl_hom: In our case, we use ansible on boot to setup the brick. And (in case this applies to you) - that command in particular never worked wrapped in ansible. The solution (for us) was to wrap that command inside a simple bash script, then execute that with ansible. Works! |
| 14:52 |
nfBurton |
I tried to run SSL through our proxy server and Z39.50 got mad at me |
| 14:52 |
nfBurton |
I would be for it though |
| 14:56 |
Bmagic |
Dyrcona | tlittle: I don't think that Evergreen is stamping all of the edi_message rows with the same PO for any given EDI file. check this query (incoming pastebin) |
| 14:56 |
pastebot |
"Bmagic" at 168.25.130.30 pasted "Query to find mis-matched EDI invoices to POs" (7 lines) at http://paste.evergreen-ils.org/10069 |
| 14:57 |
Bmagic |
in my case, for any file, I have exactly one row (count(*)) per PO |
| 14:58 |
Bmagic |
if Evergreen were stamping them all with the same (the first) PO number, then I would expect that query to return > 1 count sometimes |
| 14:58 |
tlittle |
I have a decent amount of them that are >1 |
| 15:00 |
Dyrcona |
We have mismatched numbers according to Bmagic's query, but I don't recall any complaints from libraries about it. |
| 15:00 |
rjackson_isl_hom |
bmagic - definately out of my wheel house - but locally we are going with configuring systemd for the reboot/restart options |
| 15:01 |
Bmagic |
I'm chalking it up to a one-off for now, but if we find another invoice attached to the wrong PO, I'll need to investigate "harder" |
| 15:01 |
Bmagic |
lol |
| 15:01 |
tlittle |
I don't really know what the purchase_order column on the edi_message table "does", in particular. Since lineitem is the thing that connects invoice entries to PO's, I don't know if that's referenced by anything else |
| 15:02 |
csharp |
rjackson_isl_hom: docs on getting websocketd on systemd set up (https://wiki.evergreen-ils.org/doku.php?id=dev:websockets:gateway:websocketd#optional_systemd_setup) |
| 15:02 |
berick |
the systemd setup works well for me |
| 15:03 |
csharp |
same here - with it in place and enabled, we don't have to worry about it |
| 15:03 |
csharp |
(as much) |
| 15:03 |
rjackson_isl_hom |
csharp++ - our local IT professional (not me) found that little tid bit of knowledge) |
| 15:03 |
Dyrcona |
I use a bash script to start websocketd. It basically just runs the websocketd command line in the background. |
| 15:03 |
csharp |
great |
| 15:07 |
rjackson_isl_hom |
initially ran a test (in production) without the websocketd to reboot all application servers and that went over like a lead balloon |
| 15:08 |
Dyrcona |
I'm not a huge fans of systemd, but I may go that route in the future. |
| 15:08 |
Dyrcona |
Well, the OPAC should work just fine without websocketd. |
| 15:08 |
Dyrcona |
The web staff client needs it. |
| 15:09 |
rjackson_isl_hom |
it did - but the staff at the 100+ libraries weren't that impressed to say the least |
| 15:10 |
rjackson_isl_hom |
thank God it is almost quitting time and my ability to cause grief likewise ends |
| 15:10 |
Dyrcona |
websocketd doesn't have to be started in any particular order with the rest of Evergreen, i.e. you start websocketd first, last, or in the middle. |
| 15:10 |
Dyrcona |
:) |
| 15:10 |
rjackson_isl_hom |
Dyrcona++ |
| 15:11 |
Dyrcona |
Something that may not be clear, and this is for the logs, you should always restart Apache after restarting OSRF services, too. |
| 15:12 |
Dyrcona |
But, you pretty much never have to touch websocketd as long as it is running and working. |
| 15:14 |
jeffdavis |
What's the advantage of using HTTP instead of HTTPS behind the proxy? |
| 15:18 |
berick |
jeffdavis: unnecessary encryption setup and encrypt/decrypt time on the cpu |
| 15:18 |
berick |
s/setup/handshakes/ |
| 15:18 |
Dyrcona |
Also, only 1 set of TLS settings to keep up to date. |
| 15:18 |
berick |
indeed |
| 15:19 |
Dyrcona |
If you're running the proxy and Apache on the same machine, then there's almost 0 risk of snooping. |
| 15:20 |
Dyrcona |
If you're running the proxy on a front end and passing traffic over the LAN, then it's as safe as your LAN, so you may want to keep that encrypted. Depends on your local security and level of paranoia. |
| 15:20 |
Dyrcona |
I actually think that haproxy would be usable for if it was not encrypting the connections to Apache. |
| 15:20 |
JBoyer |
Makes local testing way easier too, since Chrome won't complain about self-signed certs |
| 15:20 |
Dyrcona |
s/(for)/\1 us/ |
| 15:22 |
Dyrcona |
Guess I'll file a Lp bug about it...Maybe today, maybe tomorrow. |
| 15:23 |
JBoyer |
I filed one at some point, but LP gonna LP. |
| 15:23 |
Dyrcona |
The trick will be making it optional. Maybe two sets of Apache configs. |
| 15:23 |
Dyrcona |
Ah, you did? I'll see if I can find it. |
| 15:26 |
Dyrcona |
JBoyer: bug 1882967 ? |
| 15:26 |
pinesol |
Launchpad bug 1882967 in Evergreen "SSL / TLS changes" [Wishlist,New] https://launchpad.net/bugs/1882967 |
| 15:26 |
JBoyer |
Dyrcona++ |
| 15:26 |
JBoyer |
That's it, though it may be more vague than I thought. |
| 15:29 |
Dyrcona |
This reminds me that I should continue looking into why I am not getting remote IPs in my Apache logs. |
| 15:29 |
Dyrcona |
Could be TLS? |
| 15:31 |
JBoyer |
Shouldn't be since that's all passed in headers, but it's hard to say. |
| 15:32 |
Dyrcona |
Yeah. I've ruled out typos, unless there's a non-printable space somewhere in the configuration. |
| 15:34 |
jeffdavis |
Dyrcona: In our case I had to ensure our load balancer was passing along the client IP ("option forwardfor" with HAProxy), then add set_real_ip_from and real_ip_header in our nginx config on each brick, in addition to using the standard nginx config for EG and a2enmod remoteip + enabling RemoteIPInternalProxy/RemoteIPHeader in eg_vhost.conf |
| 15:35 |
jeffdavis |
(our setup is a load balancer with HAProxy in front, then nginx + Apache on each brick) |
| 15:35 |
jeffdavis |
set_real_ip_from in nginx.conf was the part that tripped me up when I was trying to figure that out |
| 15:35 |
Dyrcona |
jeffdavis: We're using ldirectord with nginx and apache on each brick. |
| 15:36 |
Dyrcona |
With haproxy in front, you shouldn't need nginx. (I tried that but found it killed our performance.) |
| 15:36 |
Dyrcona |
Maybe ldirectord is breaking things.... |
| 15:37 |
Dyrcona |
I did set up a test set of VMs with haproxy in front on a a pair of load balancers and just apache on the brick vms, and that worked fine. But when I put it in production, things were noticeably slower. |
| 15:38 |
Dyrcona |
Anyway....I'll do some more looking unless something comes up tomorrow. |
| 15:40 |
|
Cocopuff2018 joined #evergreen |
| 15:40 |
Dyrcona |
Hmm... set_real_ip_from. That may be what's missing. |
| 15:40 |
Dyrcona |
jeffdavis++ # suggestions. |
| 15:41 |
jeffdavis |
Hope it helps! |
| 15:45 |
Dyrcona |
I'll give it a shot after perusing the documentation. I don't have a lot going on today. |
| 15:45 |
Dyrcona |
If it works, we may want to update the example file in OpenSRF. |
| 15:47 |
jeffdavis |
FWIW I think we don't need nginx in the sense that I could move websocket proxying to the load balancer, but the load balancer is also used by non-EG services and it's just been easier in our environment to manage the proxying locally. I haven't noticed a performance hit although it would be interesting to compare. Plus I had hopes of using nginx for other stuff, a la |
| 15:47 |
jeffdavis |
https://coffeecode.net/our-nginx-caching-proxy-setup-for-evergreen.html |
| 15:47 |
JBoyer |
Oh! Something else that may help Dyrcona, if nginx isn't running on the same machine (or even if it is, depending on what ips are in use) The RemoteIP* stuff in Apache determine what IPs are trusted to send that x-Forward-For IP header, and the stock config is only localhost. If nginx is on another machine you'll need to change those also. |
| 15:55 |
Dyrcona |
Nope: asontest root: 127.0.0.1 - - [16/Dec/2020:15:54:47 -0500] "GET /eg/opac/browse HTTP/1.0" 200 122726 |
| 15:55 |
Dyrcona |
JBoyer: nginx is on the same host. |
| 15:57 |
JBoyer |
Ok, not as easy as I'd hoped. |
| 15:57 |
Dyrcona |
I'm using RemoteIPInternalProxy. Is that the correct one? |
| 15:58 |
JBoyer |
I believe so. |
| 15:59 |
Dyrcona |
Well, this is just broken for me. |
| 16:01 |
Dyrcona |
I never get anything other than 127.0.0.1 in the remote addr of the log lines. Only time the remote IP shows up is in some errors: jasontest root: [Wed Dec 16 15:54:21.869314 2020] [core:info] [pid 27059] [client 192.168.100.101:52512] AH00128: File does not exist: /openils/var/web/js/dojo/dojo/openils_dojo.js, referer: https://jasontest.cwmars.org/eg/opac/home |
| 16:01 |
Dyrcona |
Do I need to change the log format? |
| 16:02 |
JBoyer |
I seem to remember you posting a log format change in here when you initially figured it out. I don't recall it anymore and I think it's been a year or three since I thought it happened, but I do remember a format change. |
| 16:05 |
Dyrcona |
Yeah, that sounds familiar. |
| 16:10 |
|
jihpringle joined #evergreen |
| 16:11 |
JBoyer |
And to your earlier point, once that's located again we should probably add it to stock! :) |
| 16:12 |
jeffdavis |
csharp: do you happen to have a test plan for reproducing https://bugs.launchpad.net/evergreen/+bug/1896285/comments/7 ? Not quite sure how to test the patch for that one. |
| 16:12 |
pinesol |
Launchpad bug 1896285 in Evergreen "Use batch methods for multi-row grid actions" [Medium,Confirmed] |
| 16:12 |
Dyrcona |
Well, I've added a log format with %{X-Forwarded-For}i, and it looks like my problem is nginx not setting X-Forwarded-For. |
| 16:15 |
Dyrcona |
Too many moving parts.... |
| 16:15 |
Dyrcona |
Yeah, so logging the X-Forwarded-For header in Apache gives me a dash, meaning it's empty or not set. |
| 16:18 |
jeffdavis |
Dyrcona: try putting "real_ip_header X-Forwarded-For;" after set_real_ip_from in nginx config? |
| 16:18 |
Dyrcona |
Ok. If I remove the RemoteIPHeader entry in eg_vhost.conf and then long X-Forwarded-For, I get the remote address in the log, so I am getting X-Forwarded-For. |
| 16:19 |
JBoyer |
huh, I wonder if the RemoteIPHeader is removing it and putting it in another variable or something like that. |
| 16:19 |
Dyrcona |
jeffdavis: I might try that in a bit. I removed the set_real_ip sutff. |
| 16:19 |
Dyrcona |
JBoyer: according to the docs, it should be in remote_addr. |
| 16:20 |
Dyrcona |
Typos don't help. :) |
| 16:27 |
Dyrcona |
Bingo! We have a winner! |
| 16:27 |
Dyrcona |
So, the default of %h is supposed to put in the remote hostname, but only prints 127.0.0.1 for me. |
| 16:27 |
|
mantis2 left #evergreen |
| 16:27 |
Dyrcona |
I use a custom format with %a instead, and I get the actual remote IP address. |
| 16:28 |
Dyrcona |
https://httpd.apache.org/docs/2.4/mod/mod_log_config.html |
| 16:28 |
Dyrcona |
JBoyer++ |
| 16:28 |
JBoyer |
Dyrcona++ |
| 16:29 |
JBoyer |
Now that's ringing a bell, though I thought it was something like %r or simliarly mnemonic. (docs say definitely not. :) ) |
| 16:29 |
Dyrcona |
Right. I was thinking the same, but %r is first line of the request. |
| 16:31 |
Dyrcona |
So, with the new log format, it looks like my original config is working. |
| 16:31 |
Dyrcona |
I.E. nginx was configured correctly. |
| 16:33 |
Dyrcona |
I found it in my local IRC. |
| 16:34 |
Dyrcona |
http://irc.evergreen-ils.org/evergreen/2019-09-23#i_420392 |
| 16:35 |
Dyrcona |
And, somewhere between then and now, I lost that configuration change. It must not have been put in git or got rebased away in a conflict resolution. |
| 16:38 |
Dyrcona |
Likely in the switch from Apache 2.2 to Apache 2.4. |
| 16:38 |
Dyrcona |
But, we should have been on 2.4 in 2019. |
| 16:45 |
Dyrcona |
So, think I'll go with common log format with the %h switched to %a. I don't think I care about referrers or useragent strings. |
| 16:45 |
Dyrcona |
"%a %l %u %t \"%r\" %>s %b" |
| 16:58 |
Dyrcona |
Too many git branches..... |
| 17:00 |
Dyrcona |
Y'know, I think I lost it when I made a "temporary" switch to combined log format to log useragents. |
| 17:00 |
Dyrcona |
Anyway, time to go! |
| 17:04 |
|
sandbergja joined #evergreen |
| 17:04 |
|
mmorgan left #evergreen |
| 18:01 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:22 |
|
rjackson_isl_hom joined #evergreen |
| 20:30 |
|
ejk_ joined #evergreen |
| 21:13 |
|
alynn26 joined #evergreen |
| 22:09 |
|
sandbergja joined #evergreen |
