| Time |
Nick |
Message |
| 07:48 |
|
RFrasur joined #evergreen |
| 08:07 |
|
Dyrcona joined #evergreen |
| 08:07 |
|
mantis1 joined #evergreen |
| 09:24 |
Dyrcona |
miker: Do you want more eyes on bug 1931737 or would my sign off be enough to push it? |
| 09:24 |
pinesol |
Launchpad bug 1931737 in Evergreen 3.8 "Did you mean breaks parallel reingest and causes deadlocks when loading/overlaying bib records in the client" [High,Confirmed] https://launchpad.net/bugs/1931737 |
| 10:53 |
berick |
anyone having luck with the B&T FTP issues? still seeing PUT errors |
| 10:54 |
miker |
csharp_: bah, I raccoon'd at some shiny thing on friday and didn't come back to the YAOUS editor bug. I'll look for that this afternoon. apologies |
| 10:55 |
miker |
Dyrcona: IMO, since you know pingest better than anyone, and we haven't gotten more eyes on it yet, I think your confirmation is plenty, and thanks |
| 11:00 |
miker |
berick: no luck, we're seeing the same FTP issue |
| 11:00 |
berick |
thanks miker |
| 11:10 |
|
kmlussier joined #evergreen |
| 11:13 |
Bmagic |
berick: miker: yep, we've been having issues for more than a week. They know there is a problem. They suggested that we use Active mode. That does work, but, you know, firewall.... So, we're pushing back to get PASV mode working again |
| 11:13 |
berick |
thanks Bmagic |
| 11:14 |
Bmagic |
B&T had a "meeting" yesterday. Still no worky today |
| 11:29 |
Dyrcona |
B&T seems to be working with PASV today, but it is taking two minutes per connection. I have logs that look we're getting things via the fetcher. I've not checked the pusher, yet. |
| 11:30 |
Dyrcona |
We're looking at configuring our firewall for active FTP. |
| 11:31 |
Dyrcona |
miker: Cool! I was thinking more eyes would be good, but I'm not sure who else would get to testing it nor when they could make time. |
| 11:31 |
Dyrcona |
miker: I'm testing with production data upgraded to rel_3_9 latest today. |
| 11:56 |
stompro__ |
berick, we added an ftp proxy helper to our firewall and switched to Active mode for B&T ftp last week and that has been working. We noticed that on the 14th, they broke their own invoice uploading to the ftp server so invoices that should have been received that day are missing. |
| 12:03 |
stompro__ |
berick, I just tested passive mode and it does seem to be working now, at least more than it was. The initial file list works now. |
| 12:04 |
berick |
thanks stompro__ |
| 12:05 |
stompro__ |
I'm not seeing the 2 minute delay that Dyrcona mentioned... but maybe that is for actual uploads? |
| 12:06 |
stompro__ |
I tried downloading a response and it seemed to work fine... but I'm just testing with filezilla on my workstation. |
| 12:12 |
Dyrcona |
The 2-minute delay seems to happen when we get 0 files. |
| 12:12 |
|
jihpringle joined #evergreen |
| 12:20 |
miker |
Dyrcona++ |
| 12:23 |
miker |
csharp_: so, are you using a (locally merged, perhaps) version of the angular library settings editor, perchance? (your fix of trimming quotes is correct, btw) |
| 12:47 |
Dyrcona |
A closer look at the fetcher logs indicates that we're not getting files from B&T. The files are coming from other vendors. |
| 12:51 |
Dyrcona |
The pusher has had no purchase orders to process today, so I can't tell if uploads to B&T are working. |
| 12:55 |
Dyrcona |
So, yeah. It doesn't look like B&T is working for us after all. |
| 12:56 |
stompro__ |
I checked our logs... and I'm seeing some edi_fetcher errors... but the path to the file is wrong (in the error message), so I can see why it would fail. I must have a bad config somewhere. |
| 13:03 |
Dyrcona |
Except for 1 login failure, all of my entries for B&T look pretty much like this: |
| 13:03 |
Dyrcona |
Jun 21 12:42:44 util2 /openils/bin/edi_fetcher.pl: [INFO:6822:EDI.pm:92:] EDI check for vendor 143 of 148: ftp://ftp.baker-taylor.com |
| 13:03 |
Dyrcona |
Jun 21 12:44:44 util2 /openils/bin/edi_fetcher.pl: [INFO:6822:EDI.pm:117:] 0 of 0 files at ftp://ftp.baker-taylor.com/3456218.Out |
| 13:07 |
stompro__ |
Ah, the log isn't outputting the actual full file path... just the ftp server / filename. "my $description = sprintf "%s/%s", $account->host, $remote_file;" |
| 13:08 |
stompro__ |
Leaves out the in_dir info. |
| 13:26 |
Dyrcona |
Yeah. That's probably worthy of a bug. |
| 13:50 |
miker |
csharp_: I ask because the bug I'm thinking of has to do with the angular port ... so, it may be unrelated to your issue |
| 14:19 |
|
kmlussier joined #evergreen |
| 14:35 |
Dyrcona |
Active mode FTP is working for us with all vendors that use FTP, so we're switching to that. |
| 15:10 |
Bmagic |
Dyrcona: it seems that in order for that to work in a SNAT/DNAT situation, we have to open all ports from 49152-65535 and "teach" the ftp client what the external IP is |
| 15:10 |
Dyrcona |
Bmagic: That depends on your firewall. Some have proxy modules for FTP that you can enable. |
| 15:11 |
Bmagic |
GCP firewall doesn't support dynamic port as far as I can tell |
| 15:11 |
|
Christineb joined #evergreen |
| 15:11 |
Dyrcona |
Yeah, you may be stuck on Google, but I'd be surprised if you can't do it at all. |
| 15:12 |
Bmagic |
well, I can, just with that gaping hole in the firewall and I'm still googling on how to make the ftp client understand the external IP |
| 15:12 |
Dyrcona |
But, plain FTP is so 30 years ago... so maybe they don't support it at all. |
| 15:13 |
Dyrcona |
It's that last part about the external IP that usually requires a proxy. Our utility server already has its own external IP, so all we had to do was all access to the port range. |
| 15:13 |
Dyrcona |
s/all/allow/ |
| 15:17 |
Bmagic |
I'm still holding out that they will fix PASV on their end. The issue comes down to a single FTP command on their end. When we ask for PASV mode, they reply with their internal IP and not their external IP, which of course, doesn't respond to the LIST command |
| 15:17 |
Dyrcona |
Yeah, it should be a five-second fix to the configuration..... |
| 15:17 |
Bmagic |
exactly |
| 15:18 |
Bmagic |
c'mon people |
| 15:18 |
miker |
"hey GCP, 1998 called and wants to share a web log with you" |
| 15:18 |
Dyrcona |
:) |
| 15:18 |
Bmagic |
miker++ |
| 15:18 |
* RFrasur |
might be ready to throw down with B&T. |
| 15:19 |
miker |
"hi google, lemme google that for you" but also, yeah, B&T, fix your stuff |
| 15:19 |
Dyrcona |
Maybe we should drop FTP? I wonder if that would cause any vendors to add other connection methods, like STFP or SCP? |
| 15:19 |
RFrasur |
Just read up to the Bmagic "B&T had a "meeting" yesterday... |
| 15:20 |
Bmagic |
STFP ? is that like new GOPHER? |
| 15:20 |
Dyrcona |
SFTP. That's like a typo. |
| 15:20 |
Bmagic |
lol, I know dude, you ruined the joke |
| 15:21 |
Dyrcona |
I'm good at that...:) |
| 15:21 |
* Dyrcona |
holds out for Archie support in Evergreen search. :) |
| 15:21 |
Dyrcona |
And...WAIS.... |
| 15:22 |
Bmagic |
RFrasur: meetings are a good start eh? hilarious |
| 15:23 |
RFrasur |
Yeah. Super funny. Kinda seems like maybe the meeting was a few weeks late? Not sure. Might just be me. Let's definitely deploy a thing with zero notice to libraries, and then decide to have a meeting when we discover we broke everything. |
| 15:23 |
Dyrcona |
"We've been broken for a week and customers can't send us orders or get their invoices. Maybe we should have a meeting?" |
| 15:23 |
RFrasur |
Dyrcona++ |
| 15:23 |
Bmagic |
lol, exactly RFrasu++ Dyrcona++ |
| 15:23 |
Bmagic |
RFrasur++ |
| 15:25 |
Bmagic |
https://serverfault.com/questions/863334/cannot-establish-an-external-ftp-server-connection-in-active-mode-from-google |
| 15:26 |
Dyrcona |
Bmagic: Well, that's not good. |
| 15:27 |
Bmagic |
right, lol. Looks like I'm gonna have to edit the ftp client binary |
| 15:27 |
Dyrcona |
RFrasur++ Bmagic++ miker++ |
| 15:28 |
Bmagic |
they just need to fix their thing. It's been working just fine in PASV mode for over 15 years.... so like, people, let's not make it worse. FTP sucks as it is |
| 15:28 |
Dyrcona |
There are better ways to transfer files these days. It's not 1992. :) |
| 15:29 |
miker |
they do, I agree, but how can GCP not support the conntrack iptables module? |
| 15:29 |
Bmagic |
miker: so far, I've not found supporting documentation on GCP on the subject |
| 15:30 |
Bmagic |
It's not a problem on the OS. it's a problem on the GCP FW layer outside of the instance |
| 15:30 |
Dyrcona |
Is GCP using iptables? |
| 15:30 |
Bmagic |
I haven't a clue what GCP uses outside of my environment. It's magic |
| 15:30 |
RFrasur |
I wonder if they have a plan to compensate the organizations that are putting in workarounds while they can't get their act together. "Here's cash to make up for the energy you expended on our behalf." |
| 15:30 |
miker |
Dyrcona: agreed, for sure (re B&T supporting something other than ftp) |
| 15:31 |
Dyrcona |
RFrasur: "Good luck with that!" |
| 15:31 |
RFrasur |
Yeah. That's alternate reality stuff. |
| 15:31 |
Dyrcona |
Looks like all of our vendors are configured for FTP, but I wonder that they actually support. |
| 15:32 |
RFrasur |
Well, they're in a weird place too. B&T...having just been divorced from Follett. Hard to know who got the silverware. B&T didn't get the servers, but they might have gotten a couch or something. |
| 15:33 |
Dyrcona |
No, I don't want to schedule a demo to get a basic question about your EDI support answered.... |
| 15:34 |
Dyrcona |
Dyrcona: "Good luck with that!" |
| 15:34 |
RFrasur |
lol |
| 15:35 |
Dyrcona |
Maybe B&T is in an environment where they can't control the networking, either. |
| 15:40 |
Bmagic |
yeah, I get the impression they they migrated the FTP server somewhere else... AWS maybe |
| 15:40 |
Bmagic |
kept the same dns name |
| 15:41 |
Bmagic |
correction: Azure |
| 15:41 |
Bmagic |
A Windows FTP server even. Sweet |
| 15:41 |
Dyrcona |
Yeah, I was gonna say earlier that it's a Microsoft FTP server. |
| 15:41 |
Dyrcona |
Doesn't mean that they have to be on Azure for that. |
| 15:42 |
Bmagic |
agreed, but whether or not they can edit the PORT reply to tell our server their external IP, might be a fun trick on Windows + Azure |
| 15:43 |
RFrasur |
Oh good lord. |
| 15:43 |
Bmagic |
Everything's in the registry right? heh, hardcode that DString value to be the external IP and let's close this ticket |
| 15:46 |
Bmagic |
I was exploring the man page for FTP hoping there was an undocumented ENV setting that could mess with the PORT command in Active mode. No dice, but I think I might peek at the code, https://linux.die.net/man/1/ftp |
| 15:48 |
csharp_ |
lotsa chat today |
| 15:50 |
csharp_ |
miker: we're still on the old school YAOUS editor, so yeah - I'll just declare victory :-) |
| 15:52 |
Dyrcona |
Bmagic: We're using a Perl FTP module if you're using the Perl Net::FTP module. See OpenILS/Utils/RemoteAccount.pm |
| 15:52 |
Dyrcona |
Bleh... I should have reread that. |
| 15:52 |
Dyrcona |
The EDI fetcher and pusher use the Perl module. |
| 15:52 |
Bmagic |
roger that |
| 15:54 |
Bmagic |
the perl module looks to be a bit more flexible |
| 15:55 |
RFrasur |
1 fraction of a point in favor of B&T. I received not only a response to my support ticket but also a phone call. From two different support people. Completely unaware of each other, but that's...a thing. But so much communication (that amounted to nothing). |
| 15:55 |
Bmagic |
if I'm reading this right, https://perldoc.perl.org/Net::FTP port([$port]) is what I'm looking for (if I wanted to deal with Active connections). Please, lets hope for PASV |
| 16:00 |
Dyrcona |
Bmagic Looks like you can set a LocalAddr in the constructor to your external IP. That might help with active FTP working. |
| 16:04 |
Bmagic |
somewhere in edi_pusher.pl |
| 16:06 |
Bmagic |
or deeper maybe, OpenILS::Application::Acq::EDI->send_core |
| 16:13 |
Dyrcona |
Maybe in the latter. You'd also need to make the change for the fetcher as well. Doing something to look up your external ip and adding that as LocalAddr=>$external_IP to the ftp options would work. It could also be done around line 551 of OpenILS::Utils::RemoteAccount, but doing it elsewhere might be better from a design perspective. |
| 16:16 |
Dyrcona |
Net::FTP also appears to understand the old .netrc file format, so if your external IP is static, you might be able to put the LocalAddr entry in $HOME/.netrc and not need to make any code changes. |
| 16:16 |
Bmagic |
i really don't want to do that |
| 16:16 |
Dyrcona |
Yeah, I don't blame you. It's a bit of a hack to modify the code for this. |
| 16:17 |
Dyrcona |
But, see my remark about .netrc. |
| 16:17 |
Bmagic |
yeah, that's a good idea on the .netrc |
| 16:17 |
Bmagic |
worse comes to worse, I'd try that first |
| 16:17 |
Dyrcona |
If you have a dynamic IP, you could run something to periodically update the .netrc. |
| 16:18 |
Bmagic |
it's static on the util server |
| 16:18 |
Bmagic |
but if it were dynamic, that's still doable like you're suggesting |
| 16:20 |
Dyrcona |
For the .netrc to be read you'll have to set HOME in the crontab. |
| 16:26 |
Dyrcona |
Bmagic: https://metacpan.org/pod/Net::Netrc |
| 16:28 |
Dyrcona |
.netrc may not work for LocalAddr after all. :( |
| 16:29 |
Dyrcona |
might be worth a try, though. |
| 16:40 |
Bmagic |
Dyrcona++ |
| 18:57 |
|
mantis1 joined #evergreen |
| 19:07 |
jmurray-isl |
win 2 |
