| Time |
Nick |
Message |
| 07:53 |
|
collum joined #evergreen |
| 07:58 |
|
BDorsey joined #evergreen |
| 08:36 |
|
Dyrcona joined #evergreen |
| 08:42 |
Dyrcona |
Guess, I'll undo my testing changes on the test database and try the fetcher with the production settings and grab the files from Ingram's server. Maybe there's something that my test changes are masking. |
| 08:43 |
Dyrcona |
I did add vsftp on my sftp host, and configured one account to get files with ftp and others with sftp and that did not seem to cause files to be downloaded more than once in my test environment. I thought that using ftp and sftp with basically the same account would do it, but it didn't. |
| 08:46 |
Dyrcona |
I have also verified that FTP is working. |
| 09:03 |
Dyrcona |
Feels like there should be a way to do this with pg_dump/pg_restore, but so far no dice. I think I'll try a data-only dump with --disable-triggers and a truncate on the tables after. |
| 09:11 |
Dyrcona |
pg_dump: warning: there are circular foreign-key constraints among these tables: |
| 09:11 |
Dyrcona |
pg_dump: detail: edi_account |
| 09:11 |
Dyrcona |
pg_dump: detail: provider |
| 09:11 |
Dyrcona |
pg_dump: hint: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints. |
| 09:11 |
Dyrcona |
pg_dump: hint: Consider using a full dump instead of a --data-only dump to avoid this problem. |
| 09:11 |
Dyrcona |
Cute.... |
| 09:11 |
Dyrcona |
A full dump didn't work either. |
| 09:14 |
Dyrcona |
Fkeys are still getting in the way.... |
| 09:14 |
Dyrcona |
I don't want to wait 2 hours for a full pg_restore, but I guess that's it, then. |
| 09:34 |
Dyrcona |
Now that I've started the full pg_restore of the database from an old dump, I realize I could have probably set the session to replication in the SQL generated by pg_dump.... |
| 09:49 |
Dyrcona |
Blah. "set session_replication_role = replica;" isn't helping either..... Guess I'll just have to wait for the restore to finish in the one database. |
| 09:56 |
Dyrcona |
Maybe I'll ask over in #postgresql later. |
| 09:58 |
|
mmorgan joined #evergreen |
| 10:00 |
|
sleary joined #evergreen |
| 10:02 |
Dyrcona |
ooh.. nifty! It worked with "DELETE FROM table;" rather than TRUNCATE. |
| 10:07 |
Rogan |
Happy Friday Evergreeners |
| 10:09 |
Dyrcona |
Happy Friday, Rogan! |
| 10:11 |
Dyrcona |
berick: I think there's a case to be made for deleting files after they're downloaded via EDI. I'm looking at B&T account that has 754 files sitting there at the moment.\ |
| 10:12 |
berick |
Dyrcona: oh I agree completely, just not as part of the SFTP fix |
| 10:12 |
berick |
been meaning to move in that direction myself |
| 10:13 |
Dyrcona |
OK. I guess we agree more than I thought yesterday. |
| 10:13 |
berick |
also considering keeping local copies of the plain files -- before they go into the DB -- as a secondary backup for debugging, etc. |
| 10:13 |
Dyrcona |
I still haven't been able to replicate what I saw in production on my test environment, so I'm trying to restore the original settings. I'm running the fetcher to get caught up. |
| 10:14 |
berick |
possibly a lighter way to say "we already have this" than querying the DB |
| 10:14 |
Dyrcona |
Yeah, that is worth exploring. |
| 10:15 |
Dyrcona |
Back to the issue, I thought it must have to do with 1 account having sftp in the host and the others having ftp with the same credentials and directory, but even when I did that on my test setup, it didn't pull the files twice. |
| 10:16 |
Dyrcona |
This other account says X of 1722 targets.... |
| 10:34 |
|
jvwoolf joined #evergreen |
| 10:45 |
Dyrcona |
Ok. I've got only the Ingram accounts set to active. I did a run to get caught up and grabbed 63 files. The second run grabbed 0. Now, I'm going to set 1 account to sftp in the host and see what happens. There are other accounts with the same credentials using FTP. This should replicate what we saw in production. |
| 10:48 |
|
sandbergja joined #evergreen |
| 10:54 |
Dyrcona |
Files retrieved: 275 |
| 10:55 |
Dyrcona |
Wow! I think it is the host part of the duplicate check in retrieve_core after all. Dunno why it didn't trigger on my local test server, though. |
| 10:56 |
Dyrcona |
berick: I think we should regexp_replace('^[a-z]+://', host, '') in the host check, or we just need to update all of the same accounts at once? |
| 10:57 |
Dyrcona |
or, maybe [^:]+ instead of [a-z]+? |
| 10:58 |
Dyrcona |
I'm running the same again to see what happens. I suspect it should grab 275 again unless a new file has gone up. |
| 11:05 |
|
jvwoolf joined #evergreen |
| 11:10 |
jvwoolf |
OK northeastern Evergreeners, who felt the earthquake |
| 11:10 |
jvwoolf |
@who felt the earthquake |
| 11:10 |
pinesol |
Bmagic felt the earthquake. |
| 11:10 |
csharp_ |
saw the headline - too far south to feel anything |
| 11:11 |
jvwoolf |
<mean girls quote>Bmagic doesn't even go here!</mean girls quote> |
| 11:11 |
csharp_ |
on Fridays we have earthquakes |
| 11:11 |
jvwoolf |
csharp_++ |
| 11:18 |
Dyrcona |
csharp_++ |
| 11:18 |
Dyrcona |
I didn't feel the earthquake. |
| 11:19 |
Dyrcona |
Also, it retrieved 169 files on the second run. |
| 11:19 |
Dyrcona |
I'm going to set all of this library's Ingram accounts to sftp and see what happenes. |
| 11:19 |
berick |
stop trying to make earthquakes happen |
| 11:20 |
Dyrcona |
Sorry...can't help it. It's one of the side effects of SFTP. :) |
| 11:20 |
berick |
hah |
| 11:20 |
Dyrcona |
It's the Seimic File Transfer Protocol don'tcha know... |
| 11:21 |
Dyrcona |
Seismic that is... |
| 11:25 |
Dyrcona |
Yeah, sftp is definitely grabbing files for the other accounts. |
| 11:27 |
Dyrcona |
Wait a minute... That's not what it looks like in the database exactly. |
| 11:28 |
Dyrcona |
This duplicate message has the same account every time, and it's sftp. |
| 11:28 |
Dyrcona |
Well, the sftp accont now. I should tee the output to a file to see what goes on. |
| 11:29 |
Dyrcona |
Then, I'll update the accounts to sftp and do it again. |
| 11:33 |
Dyrcona |
heh. I hope Ingram doesn't block us. :) |
| 11:41 |
csharp_ |
anyone out there working on any sort of dashboard type thing to replace the need for 3rd party vendors who do collections/usage analysis? |
| 11:45 |
Dyrcona |
csharp_: We had something local that was written in PHP. Now, we're using Metabase. |
| 11:45 |
Dyrcona |
It's not built into Evergreen, though you could add links. |
| 11:46 |
Dyrcona |
So, setting all of the accounts with the same username and password to use sftp resolves the multiple files retrieval, as I thought it might. |
| 11:48 |
Dyrcona |
Moral of the story: Don't just set 1 account to SFTP to "test" it. It's all or nothing. |
| 11:51 |
jeff |
and take steps to ensure that nothing tries to fetch files while you make the config changes to all the accounts in question? |
| 11:52 |
Dyrcona |
jeff: A SQL update should be fast enough and much faster than doing it in through the client. |
| 11:54 |
Dyrcona |
I may suggest a small path to O::A::Acq::EDI.pm to remove the scheme from the URL stored in host field of acq.edi_account. Doing a schemeless comparison would also resolve the issue. |
| 11:56 |
Dyrcona |
We could also just ignore the account in acq.edi_message, but that's dangerous because it's possible two different vendors could generate the same file name. |
| 12:03 |
Dyrcona |
Also, I never quite understood the need for having multiple acq.edi_account entries with the same host, login credentials, and directory. Seems to me there could be a way to share 1 such entry among providers. Probably by deleting the provider column in the account and adding an account column to the provider. If there's other provider-specific information i the edi_account table, then it should probably be moved as well. |
| 12:04 |
|
jihpringle joined #evergreen |
| 12:07 |
csharp_ |
Dyrcona: did you add some sort of aggregating tables to make Metabase work? |
| 12:08 |
csharp_ |
just playing around with the sample data right now |
| 12:08 |
jeffdavis |
terranm++ # continual heroic efforts in launchpad |
| 12:08 |
Dyrcona |
John Amundson set most of it up. You could email him. |
| 12:10 |
csharp_ |
Dyrcona: thanks! |
| 12:10 |
|
smayo joined #evergreen |
| 12:27 |
Dyrcona |
Well, this is weird. Accounts using sftp://ftp1.ingrambook.com blow up when the edi_fetcher is run. They work with ftp, and they work if I manually connect with sftp.... |
| 12:28 |
Dyrcona |
Can't use an undefined value as an ARRAY reference at /usr/local/share/perl/5.34.0/OpenILS/Utils/RemoteAccount.pm line 441. |
| 12:29 |
|
dguarrac joined #evergreen |
| 12:30 |
Dyrcona |
Which is actually line 441 in main. |
| 12:30 |
Dyrcona |
442, sorry. I'd be dangerous if my fingers would cooperate with my brain. |
| 12:31 |
Dyrcona |
It's acting like the directory is empty, but if I sftp from the same machine, I see the files. |
| 12:32 |
* Dyrcona |
switches it to ftp.ingramcontent.com (the same server, AFAICT) and sees what happens. |
| 12:32 |
Dyrcona |
Ooh... it blew up again... |
| 12:38 |
|
BDorsey_ joined #evergreen |
| 12:38 |
|
sandbergja joined #evergreen |
| 12:41 |
Dyrcona |
So, I think something might be going with ls_sftp.... |
| 12:41 |
Dyrcona |
going on with... |
| 12:44 |
Dyrcona |
Ok. One of the bad accounts hasn't been used in 12 years, and when I connect manually, there's no home directory. |
| 12:45 |
Dyrcona |
I think we need to deactivate that one. |
| 12:49 |
|
BDorsey__ joined #evergreen |
| 12:51 |
Dyrcona |
Oh! That's interesting. The other account works after I deactivate the first one.... |
| 12:51 |
Dyrcona |
Did I just miss that it was blowing up on the one the whole time? Probably. |
| 13:05 |
|
BDorsey joined #evergreen |
| 13:24 |
Dyrcona |
I'm finding a number of providers and accounts that have various issues after switching to SFTP: Bad credentials, missing directories on the remote.... Looks like they have not been used before, or not been used for many years. |
| 13:24 |
Dyrcona |
Providers are still active, and ftp apparently recovers better with these issues. |
| 13:27 |
Dyrcona |
Nifty! Provider owned by library A, account owned by library B, not even branches of the same system. |
| 13:28 |
Dyrcona |
And the password doesn't work. |
| 13:31 |
* sleary |
always finds stray changes in the diff *after* pushing the branch... argh |
| 13:43 |
Dyrcona |
Nothing like adopting new stuff to reveal old problems. |
| 13:45 |
sleary |
that sums up my entire week... or year so far... |
| 13:46 |
sleary |
@smayo this just turned up in one of my newsletters. Happy Friday! https://web.dev/articles/light-dark |
| 13:46 |
pinesol |
sleary: Thank you sleary! But our princess is in another castle! |
| 13:46 |
Dyrcona |
sleary++ # It's Friday. |
| 13:46 |
sleary |
dangit, IRC. smayo ^^ |
| 13:47 |
Dyrcona |
@blame Twitter |
| 13:47 |
pinesol |
Dyrcona: Twitter caused the white screen of death! |
| 13:47 |
smayo |
Ooooh, colors |
| 13:49 |
sleary |
Twitter, Slack, Teams, Discord, Discourse... everything built after 2005 basically |
| 13:49 |
Dyrcona |
Does anyone know if Baker & Taylor does SFTP? |
| 13:54 |
Dyrcona |
"Access denied." Maybe they require keys? |
| 14:00 |
* Dyrcona |
quotes Green Day out of conext, "Wake me up when September ends...." |
| 14:00 |
Dyrcona |
context, even. |
| 14:02 |
|
adam_reid joined #evergreen |
| 14:12 |
|
jvwoolf joined #evergreen |
| 14:50 |
|
jvwoolf joined #evergreen |
| 15:28 |
Dyrcona |
OK, so we're rolling out SFTP for all of our Ingram accounts for Monday. if anything else turns up, I'll let you all know. |
| 15:32 |
Bmagic |
Dyrcona++ |
| 15:33 |
Bmagic |
StomproJ: RE general list, I do recall that you made some progress with Evergreen <-> Twillio ? Or was I smokin? |
| 15:33 |
Dyrcona |
In addition to the code the recently went in, we're also using the patch for Lp 2060153. |
| 15:33 |
pinesol |
Launchpad bug 2060153 in Evergreen 3.12 "EDI SFTP Records wrong remote file when sending purchase order " [Undecided,New] https://launchpad.net/bugs/2060153 |
| 15:34 |
StomproJ |
I think Jeff G uses Twillio, we use Flowroute for our SMS and Phone notices. |
| 15:35 |
StomproJ |
I do use Twilio for our Hotspot automation... just to deal with the required SMS 2fa though. |
| 15:36 |
Dyrcona |
We're looking at MessageBee, fwiw. |
| 15:36 |
Bmagic |
StomproJ: Got a branch? |
| 15:36 |
Dyrcona |
And, yeah, I recall jeff or bott having something for Twilio that's "not quite ready for prime time." |
| 15:37 |
Dyrcona |
Bmagic: There's a Lp for flowroute, IIRC. |
| 15:37 |
Bmagic |
looking |
| 15:37 |
Bmagic |
bug 1667080 |
| 15:37 |
pinesol |
Launchpad bug 1667080 in Evergreen "Hold SMS - New Reactor - flowroute" [Undecided,New] https://launchpad.net/bugs/1667080 |
| 15:39 |
Bmagic |
StomproJ: this code is installed on your production environment? |
| 15:40 |
StomproJ |
Yes, for a number of years. |
| 15:40 |
Bmagic |
StomproJ++ |
| 15:40 |
Bmagic |
we gotta get this merged then |
| 15:41 |
StomproJ |
Works for SMS call number also, and for the SMS test message.... although there is a bug there I need to document. |
| 15:42 |
Bmagic |
I'm reading https://gitlab.com/LARL/evergreen-larl/-/commit/73b424779e2a07e351cc5cd78c8c2ee32b12fd6e |
| 15:43 |
StomproJ |
Testing is probably hard since you need a flowroute account setup with real money involved. Plus all the SMS Campaign registration stuff. |
| 15:44 |
Bmagic |
right, so, it'll never get merged.... how do we "trump" that? |
| 15:44 |
Dyrcona |
From what I understand, the campaign registration is pretty much required if you want your messages to get through, even with the commercial gateway. |
| 15:44 |
StomproJ |
Oh, I need to update it for the latest version of their API also, which improves the delivery report data. |
| 15:45 |
Dyrcona |
Bmagic: You've got the power.... |
| 15:45 |
Bmagic |
the branch needs documentation and probably a pg test and a perl test |
| 15:46 |
Dyrcona |
EDI is also hard to test. |
| 15:47 |
StomproJ |
I could probably provide api keys for someone to test... I'll have to think about how to do that. |
| 15:48 |
Bmagic |
StomproJ++ |
| 15:48 |
Dyrcona |
I wouldn't want anyone to get in trouble for sharing test API keys. |
| 15:48 |
Bmagic |
maybe we could create a trial account for testing |
| 15:49 |
StomproJ |
I'll reach out to Flowroute and see if they can provide trial accounts. |
| 15:50 |
Bmagic |
Flowroute might get an increase in business, which makes it feel "icky" to only support them, but we gotta start somewhere |
| 15:50 |
Dyrcona |
There's precedent with stripe. Someone with an account has to test it. |
| 15:51 |
Dyrcona |
Well, that's the code that there is, supplied by a community member. I don't see anyone else sharing anything. (Says the guy who's not a fan of commercial solutions in general, but in this case, what can you do?) |
| 15:55 |
Bmagic |
I prefer Evergreen to talk to a commercial SMS API, rather than generate XML and ship that to MessageBee (for example) |
| 15:56 |
Bmagic |
Shipping XML and maintaining a bunch of AT template toolkit's is more annoying IMO |
| 15:56 |
Dyrcona |
I pretty much agree. I'd like a (meta)Api that sits on top and then specific plugins provide support for different commercial vendors, but that requires more knowledge than I have. |
| 15:57 |
StomproJ |
What I really want to figure out is how to include a callback endpoint in Evergreen to handle the replies in a way that the message gets back to a users pickup or home library. |
| 15:57 |
Dyrcona |
I think the people that pay my salary would prefer whatever works (now) and/or costs less. |
| 15:58 |
StomproJ |
And so staff can reply back to questions that come in via SMS. |
| 15:58 |
Dyrcona |
StomproJ: If there |
| 15:58 |
Dyrcona |
oops... |
| 15:59 |
Dyrcona |
I don't know how that works on Flowroute's side, but if they can stuff a message that can get posted somewhere, there should be a way to do it. |
| 15:59 |
StomproJ |
I'm using a php script to just send them to my email now. You just tell flowroute your callback url. |
| 16:00 |
Dyrcona |
"Click here to spam the world" |
| 16:00 |
Bmagic |
What's the next step to get this Flowroute code merged? We have a bug, and the code, though the code isn't in the "normal" form/branch method? A signoff maybe? It could use the documentation pieces (release notes, and proper doc injections). I'm sure everyone would welcome such an addition, and the SMS fire has been burning for a long time now |
| 16:01 |
Dyrcona |
Bmagic: You just said it. It needs signoff and release notes minimum. Tests and documentation would be most welcome. |
| 16:02 |
Bmagic |
Seems pretty doable to get it over the finish line IMO. StomproJ: maybe we can divide the tasks? |
| 16:04 |
Bmagic |
I think it would be "easier" (for me) if it were in the working repo |
| 16:04 |
StomproJ |
Sure, I think just having someone else interested in it helps... Probably docs on setting up a flowroute account also... |
| 16:04 |
Dyrcona |
Bmagic: You can always check it out from gitlab and push it to our working repo. |
| 16:05 |
Bmagic |
Dyrcona: I thought about that, would that keep the committer pieces? |
| 16:05 |
StomproJ |
I haven't had much motivation to do much more since it works great for us.... and no one else has asked for help getting it setup yet. |
| 16:06 |
Dyrcona |
Author would remain the same. Committer might change. You'd also have to use one of your own branches, probably collab/blake.... |
| 16:06 |
Bmagic |
it was the author I was concerned about. OK, I'll try it |
| 16:07 |
Dyrcona |
StomproJ: It's just that one commit? |
| 16:07 |
Bmagic |
It's three from what I can tell |
| 16:07 |
Bmagic |
https://gitlab.com/LARL/evergreen-larl/-/commits/rel_3_11_0-larl?search=flowroute |
| 16:08 |
StomproJ |
Oh, and it is kind of annoying that the SMS Carrier is required for holds... that could use some more though on how to best get around that. |
| 16:09 |
Dyrcona |
Thanks, Bmagic. I think it can be cherry-picked without adding the remote. |
| 16:09 |
Bmagic |
interesting.... not sure how though |
| 16:09 |
Bmagic |
I just did this: git remote add -f larl https://gitlab.com/LARL/evergreen-larl |
| 16:09 |
Bmagic |
git liked that |
| 16:09 |
Dyrcona |
might not work with cherry-pick after all. I had a trick to get commits out of a repo without adding it. |
| 16:11 |
Dyrcona |
git show probably works, but that would lose the author and commit message. probably better to just add the repo. |
| 16:11 |
Bmagic |
merge conflict on the first commit, resolving.. |
| 16:13 |
Bmagic |
ok, I thihnk I got it worked out. Pushing to collab |
| 16:14 |
Bmagic |
check me: https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/blake/lp1667080_flowroute_support |
| 16:16 |
Dyrcona |
Bmagic: It looks good to me. I see where the conflict was and how you resolved it. |
| 16:17 |
Bmagic |
sweet, StomproJ: you good with rolling with the collab branch? |
| 16:17 |
StomproJ |
Sure, sounds good. |
| 16:17 |
Dyrcona |
Bmagic++ StomproJ++ |
| 16:18 |
Bmagic |
Sounded like you were interested in the documentation additions, and release notes? I can try my hand a tthe tests |
| 16:18 |
StomproJ |
Here are all the flowroute related commits that I have in our custom production repo. https://gitlab.com/LARL/evergreen-larl/-/commits/rel_3_11_1-larl?search=flowroute |
| 16:19 |
Bmagic |
StomproJ: I think/hope those are the exact commits I just put onto our collab branch |
| 16:19 |
Dyrcona |
https://gitlab.com/LARL/evergreen-larl/-/commit/3d4cdbc7d6260640e5f086c2965dccc3797e27d1 isn't in the collab branch, and I wonder if it is necessary. |
| 16:19 |
StomproJ |
The "Patron Registration Customization" one you should probably leave out. |
| 16:19 |
Dyrcona |
Ha! |
| 16:20 |
Bmagic |
I left that one out, yes |
| 16:20 |
Dyrcona |
StomproJ++ |
| 16:20 |
StomproJ |
Great minds and all that. |
| 16:20 |
Bmagic |
It's the top three from your search results URL, rolling with that on the collab branch |
| 16:20 |
StomproJ |
I can add in examples of the other event definitions that we have setup also. |
| 16:22 |
* Dyrcona |
sprouts wings and prepares to fly. |
| 16:22 |
Dyrcona |
Catch you all later. |
| 16:22 |
Bmagic |
have a nice weekend |
| 16:22 |
Dyrcona |
You also! |
| 17:00 |
|
mmorgan left #evergreen |
| 17:02 |
|
jvwoolf left #evergreen |
| 17:21 |
|
jihpringle joined #evergreen |
