Evergreen ILS Website

IRC log for #evergreen, 2020-12-16

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
01:36 mrisher_ joined #evergreen
01:41 mrisher joined #evergreen
06:01 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
06:55 rjackson_isl_hom joined #evergreen
08:00 mmorgan joined #evergreen
08:00 alynn26 joined #evergreen
08:01 mantis1 joined #evergreen
08:49 Dyrcona joined #evergreen
08:50 rfrasur joined #evergreen
09:33 mmorgan Are compiled Angular files portable? That is, can the compiled files be copied from one server to another?
09:36 dbwells joined #evergreen
09:42 Dyrcona mmorgan: They should be since they're still just JavaScript that runs in a browser.
09:48 jvwoolf joined #evergreen
09:51 berick mmorgan: yes, you can copy them around
09:52 mmorgan berick++ Dyrcona++
09:52 mmorgan Thanks! That's helpful!
10:11 csharp mmorgan: I usually do ng build --prod and rsync them across other servers
10:12 csharp there are other approaches, but that seems the best way to ensure it's literally the same files everywhere
10:13 berick assuming identical git checkouts, they will be the same (in case there's any concern there)
10:13 csharp (for the logs, the directory to sync is /openils/var/web/eg2/)
10:13 csharp right
10:15 csharp berick: after 24 hours of running all three patches for the batch actions fixes, I can vouch for all three
10:15 mmorgan csharp++
10:15 csharp do you want me to go ahead and sign off on them separately, or do you want to combine them, or wait for more potential batch things?
10:16 * csharp would love to hear what jeffdavis thinks about that
10:16 berick csharp++ # nice
10:18 berick csharp: maybe note in the LP that they're working for you and ask jeffdavis to further clarify before our imminent merge
10:18 csharp can do
10:28 rlefaive joined #evergreen
10:52 pastebot "Bmagic" at 168.25.130.30 pasted "Query to find mis-matched EDI invoices to POs" (12 lines) at http://paste.evergreen-ils.org/10067
10:53 Bmagic Still pounding away at the EDI INVOIC conundrum, It appears that Evergreen quite frequently will assign invoice_entries to PO's that aren't the same PO as the one it matched in a previous step (edi_message). Does that query return anything for anyone?
10:55 Bmagic adjust date span to taste
10:58 tlittle Bmagic I got results. I'm looking at it in the UI right now to see what it looks like
10:58 Bmagic ty!
11:07 tlittle Bmagic Bear with me, I'm wrapping my brain around it. So is the idea that the PO attached to the EDI message is different than the one attached to the invoice_entry? Cause you can have multiple INVOIC messages in one EDI message
11:31 Bmagic tlittle: right, but a single incoming file does make multile edi_message rows, one for each PO
11:40 Dyrcona Invoices could mix and match items from different POs, no?
11:41 Dyrcona I guess the question that needs answering is where does acq.invoice_entry get the PO number? If its from the INVOIC message, then its probably a data error on the part of the vendor.
11:42 Dyrcona Just spitballing.
11:46 Bmagic as far as I can discern, it looks like the code splits the incoming EDI message into individual edi_message rows. One for each detected PO. The purchase_order column is populated accordingly
11:48 Bmagic which is why it makes since for it to refer back to the row ID number when creating the coorisponding acq.invoice row. It makes a note in the acq.invoice.note column referring to the coorisponding acq.edi_message row ID
11:48 Bmagic /since/sense
11:49 Bmagic the purchase_order column located in acq.invoice_entry is redundant because it also links to a lineitem (acq.lineitem) which, in turn links back to a PO again. Those do agree with eachother in my examples
11:51 Bmagic I'm sure I am just missing something here considering we have examples of the EDI PO mismatch going back many years
11:52 Bmagic but I do have one example where the EDI message was for PO X (in the EDI message itself) and Evergreen plopped the invoice records onto an ancient PO from 2013, matching lineitems and everything from 2013
11:53 Bmagic the part that is funny is that the PO that was recorded onto the row in acq.edi_message IS CORRECT
11:54 dbwells joined #evergreen
12:05 jihpringle joined #evergreen
12:37 rlefaive joined #evergreen
12:50 tlittle Bmagic I'm wondering if it's taking the first PO ID that it runs across in the EDI message and then stamping that as the PO ID for all rows attached to that message. Here's my query output, slightly modified: https://pastebin.com/NDnjPCei
12:50 Bmagic tlittle++ # thanks for spending so much time on this!
12:50 tlittle The first PO ID present in that EDI message in the RFF+LI is 2942, and even though there are subsequent PO IDs in subsequent RFF+LI's, all the rows attached to that edi message have the same PO ID
12:51 tlittle Np, I like puzzles :)
12:51 tlittle Also, I stepped away from it for awhile and came back to it
12:52 Bmagic I expanded my query a little to look for invoices in these categories that were attached to PO's that had lineitems attached to Funds where the YEAR of the fund was more than 2 years different than the invoice creation time. I found none!
12:52 pastebot "Bmagic" at 168.25.130.30 pasted "Query to find mis-matched EDI invoices to POs more than two years different" (18 lines) at http://paste.evergreen-ils.org/10068
12:52 Bmagic change date range to taste
12:54 tlittle Looks like none here either
12:54 Bmagic so, I think you might be onto something with the "first PO number stamped for all rows"
12:56 Bmagic There is logic elsewhere that does the matching invoice->PO->lineitem. Which is where it broke down in my example and attached the invoice to an oustanding PO from 2013. I subsequently modified the column "state" for the 2013 lineitem_detail's to be "received" instead of "on-order" - which should* prevent future processes from turning them up in search results
12:57 tlittle So it matches the PO as the middle piece? I always think of the lineitem as the connector piece
12:58 Bmagic invoice_entry points to a PO as well, but that indication seems redundant because invoice_entry->lineitem->purchase_order
12:59 tlittle Yeah agree
12:59 Bmagic I did find that in the interface, when viewing a PO, the "View Invoice(s) (X)" button will only turn up related invoices when lineitems are linked to a PO. I made the mistake (when fixing this issue) of only changing acq.invoice_entry.purchase_order to point to the correct PO and didn't update the lineitem columns to match the PO lineitems
13:00 Bmagic A second step was required to update the lineitems for the mistaken invoice_entry rows -> lineitems of the target PO
13:02 tlittle Ahh gotcha. Always fun thinking of all the connect-y pieces. :)  We have to restore fund debits a decent amount, and so wiring those back up is always fun
13:03 mantis1 joined #evergreen
13:03 tlittle Since those come from line item details, which are connected to lineitems, that's probably why I think of line items first and not purchase orders
13:07 mantis1 I got a library asking for help with putting an image in their receipt template.  It shows up in the template preview but not the dialog print preview dialog in Chrome.  I was told the image did print out one time today in a checkout receipt though.  Has anyone else ran into this problem?
13:09 Bmagic mantis1: does the template have the entire image embedded using base64 or something like that? Or is a simple <img src="somelink"> ?
13:12 Bmagic example of an embedded image encoded - http://jsfiddle.net/hpP45/
13:18 mantis1 It's <img src="link">
13:21 Bmagic try base64 embed method. I think you'll have better results
13:25 mantis1 Thank you@
13:25 mantis1 I'll give it a shot
13:26 mantis1 Bmagic ++
13:26 Bmagic worked?
13:37 jihpringle14 joined #evergreen
13:46 mantis1 Bmagic: can you not put a full URL into the string?
13:47 jvwoolf joined #evergreen
13:50 Dyrcona Bmagic | tlittle: That sounds like a bug to me if only the first PO id is being used.
13:51 mrisher joined #evergreen
13:52 mantis1 Bmagic: I got it!
13:52 mantis1 Thank you!
14:06 Christineb joined #evergreen
14:17 JBoyer mantis1, Another thing to check when a receipt won't show an image like that is that it has to be https and a full link, not a relative link. Relative links will probably work in the preview but not in the real thing.
14:25 mantis2 joined #evergreen
14:31 sandbergja joined #evergreen
14:34 Dyrcona Would anyone be interested in changes to allow Evergreen's Apache to run on plain HTTP and let the proxy server handle the HTTPS? This came up briefly in some discussion at the hack-away.
14:37 JBoyer Dyrcona Very +1
14:38 JBoyer I have a very preliminary patch that I've been meaning to throw up onto working. It's not finished because I want to use straight port 80 on localhost, but it suffices to stick it behind nginx without using SSL between the two.
14:40 rjackson_isl_hom trying to get reboots of servers happy after our 3.4 upgrade and am a bit lost (as per usual)
14:41 rjackson_isl_hom need to get websocketd running on reboot
14:41 rjackson_isl_hom --> /usr/local/bin/websocketd --port 7682 /openils/bin/osrf-websocket-stdio &
14:41 rjackson_isl_hom does it need a --maxfork=250 (for example) added to the stsrtup?
14:41 rjackson_isl_hom or leave that param out of the mix?
14:44 berick rjackson_isl_hom: --maxforks is not required.
14:44 berick generally good to have, though, since otherwise it will fork until it runs out of memory
14:44 berick if there were an abusive client
14:44 berick or just broken
14:45 rjackson_isl_hom berick++
14:52 Bmagic rjackson_isl_hom: In our case, we use ansible on boot to setup the brick. And (in case this applies to you) - that command in particular never worked wrapped in ansible. The solution (for us) was to wrap that command inside a simple bash script, then execute that with ansible. Works!
14:52 nfBurton I tried to run SSL through our proxy server and Z39.50 got mad at me
14:52 nfBurton I would be for it though
14:56 Bmagic Dyrcona | tlittle: I don't think that Evergreen is stamping all of the edi_message rows with the same PO for any given EDI file. check this query (incoming pastebin)
14:56 pastebot "Bmagic" at 168.25.130.30 pasted "Query to find mis-matched EDI invoices to POs" (7 lines) at http://paste.evergreen-ils.org/10069
14:57 Bmagic in my case, for any file, I have exactly one row (count(*)) per PO
14:58 Bmagic if Evergreen were stamping them all with the same (the first) PO number, then I would expect that query to return > 1 count sometimes
14:58 tlittle I have a decent amount of them that are >1
15:00 Dyrcona We have mismatched numbers according to Bmagic's query, but I don't recall any complaints from libraries about it.
15:00 rjackson_isl_hom bmagic - definately out of my wheel house - but locally we are going with configuring systemd for the reboot/restart options
15:01 Bmagic I'm chalking it up to a one-off for now, but if we find another invoice attached to the wrong PO, I'll need to investigate "harder"
15:01 Bmagic lol
15:01 tlittle I don't really know what the purchase_order column on the edi_message table "does", in particular. Since lineitem is the thing that connects invoice entries to PO's, I don't know if that's referenced by anything else
15:02 csharp rjackson_isl_hom: docs on getting websocketd on systemd set up (https://wiki.evergreen-ils.org/doku.php?id=dev:web​sockets:gateway:websocketd#optional_systemd_setup)
15:02 berick the systemd setup works well for me
15:03 csharp same here - with it in place and enabled, we don't have to worry about it
15:03 csharp (as much)
15:03 rjackson_isl_hom csharp++ - our local IT professional (not me) found that little tid bit of knowledge)
15:03 Dyrcona I use a bash script to start websocketd. It basically just runs the websocketd command line in the background.
15:03 csharp great
15:07 rjackson_isl_hom initially ran a test (in production) without the websocketd to reboot all application servers and that went over like a lead balloon
15:08 Dyrcona I'm not a huge fans of systemd, but I may go that route in the future.
15:08 Dyrcona Well, the OPAC should work just fine without websocketd.
15:08 Dyrcona The web staff client needs it.
15:09 rjackson_isl_hom it did - but the staff at the 100+ libraries weren't that impressed to say the least
15:10 rjackson_isl_hom thank God it is almost quitting time and my ability to cause grief likewise ends
15:10 Dyrcona websocketd doesn't have to be started in any particular order with the rest of Evergreen, i.e. you start websocketd first, last, or in the middle.
15:10 Dyrcona :)
15:10 rjackson_isl_hom Dyrcona++
15:11 Dyrcona Something that may not be clear, and this is for the logs, you should always restart Apache after restarting OSRF services, too.
15:12 Dyrcona But, you pretty much never have to touch websocketd as long as it is running and working.
15:14 jeffdavis What's the advantage of using HTTP instead of HTTPS behind the proxy?
15:18 berick jeffdavis: unnecessary encryption setup and encrypt/decrypt time on the cpu
15:18 berick s/setup/handshakes/
15:18 Dyrcona Also, only 1 set of TLS settings to keep up to date.
15:18 berick indeed
15:19 Dyrcona If you're running the proxy and Apache on the same machine, then there's almost 0 risk of snooping.
15:20 Dyrcona If you're running the proxy on a front end and passing traffic over the LAN, then it's as safe as your LAN, so you may want to keep that encrypted. Depends on your local security and level of paranoia.
15:20 Dyrcona I actually think that haproxy would be usable for if it was not encrypting the connections to Apache.
15:20 JBoyer Makes local testing way easier too, since Chrome won't complain about self-signed certs
15:20 Dyrcona s/(for)/\1 us/
15:22 Dyrcona Guess I'll file a Lp bug about it...Maybe today, maybe tomorrow.
15:23 JBoyer I filed one at some point, but LP gonna LP.
15:23 Dyrcona The trick will be making it optional. Maybe two sets of Apache configs.
15:23 Dyrcona Ah, you did? I'll see if I can find it.
15:26 Dyrcona JBoyer: bug 1882967 ?
15:26 pinesol Launchpad bug 1882967 in Evergreen "SSL / TLS changes" [Wishlist,New] https://launchpad.net/bugs/1882967
15:26 JBoyer Dyrcona++
15:26 JBoyer That's it, though it may be more vague than I thought.
15:29 Dyrcona This reminds me that I should continue looking into why I am not getting remote IPs in my Apache logs.
15:29 Dyrcona Could be TLS?
15:31 JBoyer Shouldn't be since that's all passed in headers, but it's hard to say.
15:32 Dyrcona Yeah. I've ruled out typos, unless there's a non-printable space somewhere in the configuration.
15:34 jeffdavis Dyrcona: In our case I had to ensure our load balancer was passing along the client IP ("option forwardfor" with HAProxy), then add set_real_ip_from and real_ip_header in our nginx config on each brick, in addition to using the standard nginx config for EG and a2enmod remoteip + enabling RemoteIPInternalProxy/RemoteIPHeader in eg_vhost.conf
15:35 jeffdavis (our setup is a load balancer with HAProxy in front, then nginx + Apache on each brick)
15:35 jeffdavis set_real_ip_from in nginx.conf was the part that tripped me up when I was trying to figure that out
15:35 Dyrcona jeffdavis: We're using ldirectord with nginx and apache on each brick.
15:36 Dyrcona With haproxy in front, you shouldn't need nginx. (I tried that but found it killed our performance.)
15:36 Dyrcona Maybe ldirectord is breaking things....
15:37 Dyrcona I did set up a test set of VMs with haproxy in front on a a pair of load balancers and just apache on the brick vms, and that worked fine. But when I put it in production, things were noticeably slower.
15:38 Dyrcona Anyway....I'll do some more looking unless something comes up tomorrow.
15:40 Cocopuff2018 joined #evergreen
15:40 Dyrcona Hmm... set_real_ip_from. That may be what's missing.
15:40 Dyrcona jeffdavis++ # suggestions.
15:41 jeffdavis Hope it helps!
15:45 Dyrcona I'll give it a shot after perusing the documentation. I don't have a lot going on today.
15:45 Dyrcona If it works, we may want to update the example file in OpenSRF.
15:47 jeffdavis FWIW I think we don't need nginx in the sense that I could move websocket proxying to the load balancer, but the load balancer is also used by non-EG services and it's just been easier in our environment to manage the proxying locally. I haven't noticed a performance hit although it would be interesting to compare. Plus I had hopes of using nginx for other stuff, a la
15:47 jeffdavis https://coffeecode.net/our-nginx-cac​hing-proxy-setup-for-evergreen.html
15:47 JBoyer Oh! Something else that may help Dyrcona, if nginx isn't running on the same machine (or even if it is, depending on what ips are in use) The RemoteIP* stuff in Apache determine what IPs are trusted to send that x-Forward-For IP header, and the stock config is only localhost. If nginx is on another machine you'll need to change those also.
15:55 Dyrcona Nope: asontest root: 127.0.0.1 - - [16/Dec/2020:15:54:47 -0500] "GET /eg/opac/browse HTTP/1.0" 200 122726
15:55 Dyrcona JBoyer: nginx is on the same host.
15:57 JBoyer Ok, not as easy as I'd hoped.
15:57 Dyrcona I'm using RemoteIPInternalProxy. Is that the correct one?
15:58 JBoyer I believe so.
15:59 Dyrcona Well, this is just broken for me.
16:01 Dyrcona I never get anything other than 127.0.0.1 in the remote addr of the log lines. Only time the remote IP shows up is in some errors: jasontest root: [Wed Dec 16 15:54:21.869314 2020] [core:info] [pid 27059] [client 192.168.100.101:52512] AH00128: File does not exist: /openils/var/web/js/dojo/dojo/openils_dojo.js, referer: https://jasontest.cwmars.org/eg/opac/home
16:01 Dyrcona Do I need to change the log format?
16:02 JBoyer I seem to remember you posting a log format change in here when you initially figured it out. I don't recall it anymore and I think it's been a year or three since I thought it happened, but I do remember a format change.
16:05 Dyrcona Yeah, that sounds familiar.
16:10 jihpringle joined #evergreen
16:11 JBoyer And to your earlier point, once that's located again we should probably add it to stock! :)
16:12 jeffdavis csharp: do you happen to have a test plan for reproducing https://bugs.launchpad.net/eve​rgreen/+bug/1896285/comments/7 ? Not quite sure how to test the patch for that one.
16:12 pinesol Launchpad bug 1896285 in Evergreen "Use batch methods for multi-row grid actions" [Medium,Confirmed]
16:12 Dyrcona Well, I've added a log format with %{X-Forwarded-For}i, and it looks like my problem is nginx not setting X-Forwarded-For.
16:15 Dyrcona Too many moving parts....
16:15 Dyrcona Yeah, so logging the X-Forwarded-For header in Apache gives me a dash, meaning it's empty or not set.
16:18 jeffdavis Dyrcona: try putting "real_ip_header X-Forwarded-For;" after set_real_ip_from in nginx config?
16:18 Dyrcona Ok. If I remove the RemoteIPHeader entry in eg_vhost.conf and then long X-Forwarded-For, I get the remote address in the log, so I am getting X-Forwarded-For.
16:19 JBoyer huh, I wonder if the RemoteIPHeader is removing it and putting it in another variable or something like that.
16:19 Dyrcona jeffdavis: I might try that in a bit. I removed the set_real_ip sutff.
16:19 Dyrcona JBoyer: according to the docs, it should be in remote_addr.
16:20 Dyrcona Typos don't help. :)
16:27 Dyrcona Bingo! We have a winner!
16:27 Dyrcona So, the default of %h is supposed to put in the remote hostname, but only prints 127.0.0.1 for me.
16:27 mantis2 left #evergreen
16:27 Dyrcona I use a custom format with %a instead, and I get the actual remote IP address.
16:28 Dyrcona https://httpd.apache.org/docs​/2.4/mod/mod_log_config.html
16:28 Dyrcona JBoyer++
16:28 JBoyer Dyrcona++
16:29 JBoyer Now that's ringing a bell, though I thought it was something like %r or simliarly mnemonic. (docs say definitely not. :) )
16:29 Dyrcona Right. I was thinking the same, but %r is first line of the request.
16:31 Dyrcona So, with the new log format, it looks like my original config is working.
16:31 Dyrcona I.E. nginx was configured correctly.
16:33 Dyrcona I found it in my local IRC.
16:34 Dyrcona http://irc.evergreen-ils.org/​evergreen/2019-09-23#i_420392
16:35 Dyrcona And, somewhere between then and now, I lost that configuration change. It must not have been put in git or got rebased away in a conflict resolution.
16:38 Dyrcona Likely in the switch from Apache 2.2 to Apache 2.4.
16:38 Dyrcona But, we should have been on 2.4 in 2019.
16:45 Dyrcona So, think I'll go with common log format with the %h switched to %a. I don't think I care about referrers or useragent strings.
16:45 Dyrcona "%a %l %u %t \"%r\" %>s %b"
16:58 Dyrcona Too many git branches.....
17:00 Dyrcona Y'know, I think I lost it when I made a "temporary" switch to combined log format to log useragents.
17:00 Dyrcona Anyway, time to go!
17:04 sandbergja joined #evergreen
17:04 mmorgan left #evergreen
18:01 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
18:22 rjackson_isl_hom joined #evergreen
20:30 ejk_ joined #evergreen
21:13 alynn26 joined #evergreen
22:09 sandbergja joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat