| Time |
Nick |
Message |
| 00:36 |
|
degraafk joined #evergreen |
| 08:33 |
|
mmorgan joined #evergreen |
| 08:53 |
|
Dyrcona joined #evergreen |
| 08:57 |
|
smayo joined #evergreen |
| 09:53 |
|
mmorgan1 joined #evergreen |
| 10:28 |
|
redavis joined #evergreen |
| 10:31 |
|
sandbergja joined #evergreen |
| 11:07 |
|
Christineb joined #evergreen |
| 11:49 |
Dyrcona |
Nifty! I can find the difference between two vectors in Rust with about the same amount of code as it takes in Common Lisp. It's just about as fast, too. |
| 11:59 |
|
jihpringle joined #evergreen |
| 13:26 |
|
mantis1 joined #evergreen |
| 13:28 |
mantis1 |
one question for those who use EDI - does anyone know how to reactiviate a PO? |
| 13:32 |
|
jihpringle joined #evergreen |
| 13:41 |
mantis1 |
we're also running into a problem where a file name is not producing, and based on the error message, this seems to be stemming from Evergreen and not the account setup or connection. Does anyone know how the filename is produced? That might give us a good start on where to look. |
| 14:12 |
Dyrcona |
Well, related to acquisitions, I want to ask if anyone is using the Angular acq and having trouble running out of acq drones? We've had 1 person doing some basic stuff in acq on our test system, and we've gotten no children available with 35 drones. |
| 14:13 |
Dyrcona |
mantis1: I can't answer your questions off the top of my head, but what does the error say? |
| 14:17 |
sleary |
Dyrcona bug 2003973 maybe? |
| 14:17 |
pinesol |
Launchpad bug 2003973 in Evergreen "Embedded copy attrs eats acq and actor drones" [High,New] https://launchpad.net/bugs/2003973 |
| 14:21 |
mantis1 |
Dyrcona: put FAILED: put to ftp.brodart.com failed with error: Can't call method 'put' on an undefined value at /usr/local/share/perl/5.34.0/OpenILS/Utils/RemoteAccount.pm line 694 |
| 14:26 |
|
jihpringle joined #evergreen |
| 14:29 |
csharp_ |
mantis1: there should be more messages around that error for context |
| 14:29 |
csharp_ |
grep for "pusher" |
| 14:38 |
csharp_ |
@decide drug pusher or acq pusher |
| 14:38 |
pinesol |
csharp_: go with drug pusher |
| 14:39 |
csharp_ |
Dyrcona: re: the bug sleary linked to, I haven't tracked it closely since we moved to RediSRF, but yeah, we saw all acq drones saturated all the time |
| 14:39 |
csharp_ |
I haven't heard complaints from the field lately of problems with acq |
| 14:42 |
Bmagic |
mantis1: you can issue a command on the command line on your utility server to debug a certain message or resend a certain message. I'll get the incantations, just a minute |
| 14:42 |
Bmagic |
edi_order_pusher.pl --teset-mode --po-id <POID> |
| 14:43 |
Bmagic |
uhhg |
| 14:43 |
Bmagic |
edi_order_pusher.pl --test-mode --po-id <POID> |
| 14:43 |
csharp_ |
not teset mode! that's the worst mode! |
| 14:43 |
Bmagic |
:) |
| 14:43 |
Bmagic |
you can ask it to be more verbose to help diagnose: |
| 14:43 |
Bmagic |
edi_order_pusher.pl --teset-mode --po-id <POID> --verbose |
| 14:44 |
Bmagic |
you can update the acq.edi_message row, and change the status to "retry" |
| 14:46 |
Bmagic |
first, you need to know the ID of the edi_message. SELECT * from acq.edi_message where purchase_order=<POID> AND message_type='ORDERS'; |
| 14:46 |
Bmagic |
then you can update the status like this: UPDATE acq.edi_message set status='retry' where id=<DISCOVERED ID FROM STEP ONE> ; |
| 14:47 |
Bmagic |
then you can run the order pusher again to resend just that one: edi_order_pusher.pl --po-id <POID> --verbose |
| 14:49 |
Bmagic |
it goes without saying: be careful with update queries on production. It's a good idea to use psql command line, and put the update inside of a transaction by using BEGIN; first |
| 14:50 |
mantis1 |
Bmagic: ran the test mode pusher and got back a crazy looking string |
| 14:50 |
Bmagic |
lol, I see that teset poisoned my paste a bunch. |
| 14:50 |
csharp_ |
mantis1: can you paste it somewhere like pastebin.com? |
| 14:51 |
Bmagic |
mantis1: that's the EDI message. If you were a computer it would make sense :) Suffice it to say:t that's expected and probably what the vendor needs in order to get the order started |
| 14:51 |
mantis1 |
https://pastebin.com/8BdGz0zF |
| 14:51 |
Bmagic |
yep, that's the EDI message |
| 14:51 |
mantis1 |
okie doke |
| 14:51 |
Bmagic |
So, all is going well so far |
| 14:52 |
mantis1 |
I was able to look up the message in the db |
| 14:52 |
Bmagic |
That's the stuff that is put into a random filename and FTPed to the vendor |
| 14:53 |
Bmagic |
it sounds like the issue is a transfer issue? In other words: The message is getting created properly, and saved to the server, but it can't transfer it? There's an (S)FTP error of some kind? |
| 14:54 |
mantis1 |
we did talk to our vendor about this and can confirm that there isn't an issue with our firewall etc |
| 14:54 |
mantis1 |
and we figured it was a file issue because the filename isn't producing |
| 14:55 |
mantis1 |
we can confirm the EDI profile is set up correctly |
| 14:55 |
csharp_ |
mantis1: the error you shared seems to imply that the FTP connection isn't actually working |
| 14:55 |
csharp_ |
Can't call method 'put' on an undefined value at /usr/local/share/perl/5.34.0/OpenILS/Utils/RemoteAccount.pm line 694 |
| 14:55 |
mmorgan |
mantis1: Maybe check the EDI Account in Evergreen and make sure the Path and Incoming Directory are what the vendor expects? |
| 14:55 |
Bmagic |
try putting this at the beginning of your command: export FTP_PASSIVE=1 && |
| 14:56 |
mantis1 |
mmorgan: we were and confirmed it was what they need |
| 14:56 |
csharp_ |
the line is eval { $filename = $self->_ftp->put(@{$self->{put_args}}) }; |
| 14:56 |
mantis1 |
Bmagic: for the test mode again? |
| 14:56 |
csharp_ |
so at that point $self->_ftp is undefined |
| 14:57 |
Bmagic |
the whole thing would be "export FTP_PASSIVE=1 && edi_order_pusher.pl --po-id 67414" |
| 14:57 |
Bmagic |
It's not going to retry until you update the row in the DB as described above |
| 14:58 |
Bmagic |
mantis1: what's the hostname for the vendor's FTP server? Does it start with ftp:// ? |
| 14:58 |
csharp_ |
put FAILED: put to ftp.brodart.com |
| 14:58 |
Bmagic |
csharp_: I saw that but I'm wondering if the logs remove the prefix |
| 14:58 |
csharp_ |
oh - hmm |
| 14:58 |
mantis1 |
ftp://ftp.brodart.com |
| 14:58 |
mantis1 |
I also tried the command it just returned empty |
| 14:59 |
csharp_ |
"put to", $self->remote_host, "failed with error: $@" |
| 14:59 |
Bmagic |
ok, that's a good hostname (it has the required prefix to teach Evergreen which protocol to use) |
| 15:00 |
Bmagic |
in theory, you can use your own FTP client on your computer like Filezilla or LeechFTP and connect to the vendor's FTP server with the credentials that you have configured for Evergreen |
| 15:00 |
Bmagic |
and you can see if the file made it's way over |
| 15:00 |
Bmagic |
mantis1: the comment about it running without returning anything: did you first update the row in the DB? |
| 15:01 |
mantis1 |
our cataloger was able to connect with Filezilla |
| 15:01 |
Bmagic |
and, do they see a file in the "incoming" folder? |
| 15:02 |
|
mmorgan1 joined #evergreen |
| 15:02 |
mantis1 |
they did |
| 15:03 |
mantis1 |
and yes, I ran it after updating the status on the message |
| 15:03 |
Bmagic |
maybe it worked then |
| 15:03 |
Bmagic |
the next step would be to get confirmation from the vendor |
| 15:03 |
mantis1 |
I'm still seeing an error |
| 15:03 |
mantis1 |
and the date updated to today |
| 15:03 |
Bmagic |
It sounds like you need to edit your cron jobs and make sure export FTP_PASSIVE=1 && is prefixed on your fetcher/pusher command |
| 15:04 |
mantis1 |
we also don't have a filename producing in the GUI either |
| 15:04 |
Bmagic |
still seeing an error? In the logs? Was that error generated for your manual test? |
| 15:04 |
mantis1 |
I'm seeing the error within the GUI |
| 15:04 |
mantis1 |
under EDI Messages |
| 15:05 |
mantis1 |
it's updated with the retry status though |
| 15:05 |
Bmagic |
That's a reflection of SELECT * from acq.edi_message WHERE purchase_order=67414 AND message_type='ORDERS'; |
| 15:06 |
Bmagic |
if the status is currently "retry" then the order pusher code hasn't done it's thing |
| 15:06 |
mantis1 |
ok I see a new one |
| 15:07 |
mantis1 |
sorry it doesn't look like it went through |
| 15:10 |
|
jihpringle joined #evergreen |
| 15:18 |
Bmagic |
mantis1: hopefully there's a log message to help us out |
| 15:18 |
csharp_ |
also, add "--verbose" next time you run the pusher |
| 15:20 |
mantis1 |
./edi_pusher.pl: [ERR :3734632:EDI.pm:394:] EDI Translator json2edi failed: RPC::XML::Client::send_request: HTTP server error: Can't connect to localhost:9191 (Connection refused) |
| 15:20 |
Bmagic |
wrong script |
| 15:20 |
Bmagic |
edi_order_pusher |
| 15:20 |
mantis1 |
ok |
| 15:20 |
csharp_ |
we should really remove the webrick one |
| 15:20 |
Bmagic |
unless you're using the old Ruby EDI stuff |
| 15:21 |
mantis1 |
ah ok |
| 15:21 |
mantis1 |
failed login to ftp.brodart.com w/ args(e3360334,SuHhEIITXOYFFG,065407) : |
| 15:21 |
Bmagic |
with the FTP_PASSIVE flag set? |
| 15:22 |
mantis1 |
there is also this one but it's pretty similar: (S)FTP put to EDI send to edi_account 102 (ftp://ftp.brodart.com) FAILED: put to ftp.brodart.com failed with error: Can't call method "put" on an undefined value at /usr/local/share/perl/5.34.0/OpenILS/Utils/RemoteAccount.pm line 694. |
| 15:22 |
csharp_ |
mantis1: note that's an actual password in the args |
| 15:22 |
mantis1 |
sorry I'm not used to these logs |
| 15:22 |
mantis1 |
with verbose, it says FTP_PASSIVE is OFF |
| 15:22 |
mantis1 |
along with the message and the status |
| 15:23 |
csharp_ |
probably not an issue, but you might consider asking the vendor for a new password for that account |
| 15:23 |
mantis1 |
chsarp++ |
| 15:23 |
Bmagic |
try again, and make sure FTP_PASSIVE is turned on |
| 15:23 |
Bmagic |
(and double check that the SQL row still has "retry" for the status) |
| 15:23 |
mantis1 |
ok same thing but it just says passive is on |
| 15:24 |
mantis1 |
still has the status retry in the db |
| 15:24 |
Bmagic |
mantis1: are there any other rows in acq.edi_message other than the ORDERS row? In other words, what's the answer to this query: select count(*) from acq.edi_message where purchase_order=67414; |
| 15:25 |
mantis1 |
looks like 3 |
| 15:26 |
Bmagic |
that's odd, what are the message_types? (switch the count(*) to * ) |
| 15:27 |
mantis1 |
looks like my error - it seems like I pushed these without changing the status of the original |
| 15:27 |
mantis1 |
so that's what the other two are |
| 15:27 |
Bmagic |
so all three rows are "ORDERS" ? |
| 15:28 |
mantis1 |
yes |
| 15:28 |
csharp_ |
mantis1: try removing the 065407 from the Account field |
| 15:28 |
Bmagic |
that's, uh, unusual. I'm not sure how the code would have allowed that. Oh, maybe using the old edi_pusher code |
| 15:28 |
csharp_ |
in the EDI account setup |
| 15:29 |
Bmagic |
csharp_++ |
| 15:29 |
Bmagic |
yeah, that's closer to the FTP issue I think |
| 15:29 |
csharp_ |
I think that's it - we've seen that before |
| 15:31 |
Bmagic |
three arguments seems wrong for FTP |
| 15:31 |
csharp_ |
I think "account" is an old thing kept around for backwards compatibility |
| 15:32 |
Bmagic |
I thought it was used for EDI pieces |
| 15:32 |
csharp_ |
there's another vendor account field that is used |
| 15:32 |
csharp_ |
it's stupidly confusing |
| 15:32 |
Bmagic |
agreed |
| 15:33 |
Dyrcona |
I missed a lot of conversation. Had my head down in some code. |
| 15:33 |
Bmagic |
Dyrcona: it'll be a fun read for ya :) |
| 15:33 |
mantis1 |
yayyyyy |
| 15:33 |
csharp_ |
with credentials and everything! |
| 15:33 |
mantis1 |
status complete |
| 15:33 |
Bmagic |
!!! |
| 15:33 |
csharp_ |
mantis1: nice! |
| 15:33 |
Dyrcona |
Mostly caught up already. |
| 15:33 |
mantis1 |
thank you so much |
| 15:33 |
Bmagic |
csharp_++ FTW |
| 15:33 |
mantis1 |
I never would have figured this out |
| 15:34 |
mmorgan1 |
mantis1++ |
| 15:34 |
mantis1 |
csharp++ |
| 15:34 |
mantis1 |
Bmagic++ |
| 15:34 |
mmorgan1 |
Bmagic++ |
| 15:34 |
mmorgan1 |
csharp_++ |
| 15:34 |
csharp_ |
mantis1: I had to go down the exact same road several years ago |
| 15:34 |
Dyrcona |
csharp_++ mantis1++ Bmagic++ |
| 15:34 |
mantis1 |
when should you use that Account field? |
| 15:34 |
Bmagic |
computers are dumb |
| 15:34 |
csharp_ |
mantis1: only if the vendor says it's required, but probably never |
| 15:34 |
Dyrcona |
Back to my acq thing: It's acq drones that are pegged. Dunno why. I haven't really tried to look. |
| 15:35 |
Dyrcona |
"account" is a really old FTP thing, and I don't think anyone really every used it. Usually, the username and password are enough. |
| 15:35 |
csharp_ |
mantis1: it's an argument that can be omitted when connecting via Perl Net::FTP, but if it's there, it tries using it |
| 15:35 |
Bmagic |
Dyrcona: not sure on that one, I've not seen the same behavior, granted we only have one system on 3.15+ |
| 15:36 |
csharp_ |
might be a good bug to include documentation about this somewhere |
| 15:36 |
Dyrcona |
I was told that the user batch updated a fund used in a purchase order's line items. |
| 15:37 |
csharp_ |
(or to remove "account" altogether) |
| 15:37 |
Dyrcona |
Might be a good bug to drop that column if we're talking about "account." |
| 15:37 |
Bmagic |
account != account != account |
| 15:37 |
csharp_ |
so many overloaded terms |
| 15:38 |
Bmagic |
the FTP protocol has a third credential called account? I gotta look this up |
| 15:38 |
Dyrcona |
Bmagic: I believe it did. |
| 15:38 |
csharp_ |
Bmagic: https://perldoc.perl.org/Net::FTP |
| 15:38 |
csharp_ |
login([$login[, $password[, $account]]]) |
| 15:39 |
Bmagic |
https://en.wikipedia.org/wiki/List_of_FTP_commands |
| 15:39 |
Bmagic |
ACCT |
| 15:40 |
csharp_ |
Bmagic: bless you |
| 15:40 |
Bmagic |
generally, you'd issue "USER" |
| 15:40 |
Bmagic |
followed by "PASS" |
| 15:41 |
Bmagic |
I'm still not clear on when you issue "ACCT" |
| 15:42 |
Dyrcona |
It's not the same thing, and I've never had FTP access where I needed it. |
| 15:42 |
Bmagic |
me either |
| 15:42 |
csharp_ |
FTP should die anyway |
| 15:42 |
Bmagic |
we decided to implement it because it's supported by Net::FTP I guess? |
| 15:42 |
csharp_ |
probably |
| 15:42 |
csharp_ |
theoretically possible someone somewhere will need it someday |
| 15:43 |
Dyrcona |
Well, yes, though FTP/S might still be OK. :) SFTP is not the same thing at all. |
| 15:43 |
Bmagic |
We'll be using FTP well into the 2100's |
| 15:43 |
csharp_ |
yeah FTP/S is fine |
| 15:43 |
Dyrcona |
Some EDI vendors only have FTP not SFTP or FTP/S, IIRC. |
| 15:43 |
jeff |
thisisfine.gif |
| 15:44 |
jeff |
ftpsisfine.gif? |
| 15:44 |
Bmagic |
lol |
| 15:44 |
Dyrcona |
panic!("gif not found"); |
| 15:46 |
Bmagic |
mantis1: be sure and get the vendor to change your FTP account password whenever they can |
| 15:47 |
Bmagic |
The creds are now public :( but on the other hand FTP isn't encrypted so the creds are kind of public anyways |
| 15:47 |
|
mantis1 left #evergreen |
| 15:49 |
Dyrcona |
I would say that we should implement our EDI stuff in Rust, but I'll be berick has already done that... :) |
| 15:49 |
Dyrcona |
s/be/bet/ |
| 15:49 |
Bmagic |
yeah probably :) |
| 15:50 |
Dyrcona |
My program to make database upgrade scripts mostly works. I want to get the -r (review in your editor) feature working and clean up the code, add more doc comments, and a license/readme before I post it somewhere. |
| 15:51 |
Bmagic |
Dyrcona++ # rust or bust |
| 15:52 |
Dyrcona |
Well, I'm starting to like Rust. It's boring, which is good. It's verbose, which is less good. The compiler tells you exactly what's wrong. |
| 15:53 |
Dyrcona |
I'm learning to take advantage of more of the language's features. |
| 15:53 |
Bmagic |
With as much news that I get about it, I'm starting to get the impression that it's good :) |
| 15:55 |
Bmagic |
I tend to respect Linus Torvalds. His vulgar misgivings not withstanding |
| 15:55 |
Dyrcona |
Back to the acq thing: We also have open-ils.actor drones pegged. |
| 15:55 |
csharp_ |
probably a million perm checks on things like Item Status |
| 15:56 |
Dyrcona |
Linus is frank. That's part of the reason that I respect his opinions. I don't always agree, but I still see where he's coming from. |
| 15:56 |
csharp_ |
https://i.makeagif.com/media/7-11-2017/1aqoM5.gif |
| 15:56 |
Dyrcona |
csharp_: all coming from the client most likely. I think we do too much in the client since it has moved to the web. |
| 15:56 |
Bmagic |
And you can make it happen easily? A repeatable method? Can we get it to happen on a vanilla EG machine? |
| 15:56 |
csharp_ |
yeap |
| 15:57 |
Dyrcona |
I have no idea what made it happen, but what csharp_ said sounds about right. |
| 15:57 |
csharp_ |
you can see it happen in the activity logs - just grep open-ils.actor |
| 15:58 |
Dyrcona |
IMHO the client should be the presentation layer and not much more. The work should happen in the middle ware. |
| 15:58 |
Bmagic |
Dyrcona: I completely agree |
| 15:59 |
Bmagic |
you mean once it happens, the server can't recover? |
| 15:59 |
Dyrcona |
osrfsys.log is 2.2GB. Do we ever rotate it? |
| 16:00 |
csharp_ |
eewww |
| 16:00 |
csharp_ |
we use the log-breaker-outer config for rsyslog |
| 16:01 |
Bmagic |
Dyrcona: yes, 2.2GB is from today only |
| 16:01 |
Dyrcona |
Nothing bad seems to happen so far, except Zabbix blaring at me and the logs getting spammed with 'No children available'. No one has complained about things not working, but it's a dev machine barely used by a handful of people. |
| 16:01 |
csharp_ |
https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/examples/evergreen-rsyslog.conf;h=ba25cea727c4aadead5635197f86040392c5ecf2;hb=HEAD |
| 16:02 |
Bmagic |
The log is getting rotated yall, daily. It could go hourly if you want. But is that a problem that needs solved? |
| 16:03 |
Dyrcona |
No, the log is fine. It's just a slog to work with one that big. |
| 16:07 |
Bmagic |
one thing I've been tweaking lately is max_requests. Years ago, I set that to 1000, and I've learned that means that the drone won't rotate until it's had that number of requests. Which makes for higher memory usage because each Drone lives longer. The OpenSRF code doesn't have the best garbage collection |
| 16:10 |
Bmagic |
which is (probably) why I was having issues with Redis. Unconfirmed though |
| 16:12 |
Dyrcona |
The OpenSRF code doesn't have any garbage collection. It's C and Perl. |
| 16:12 |
Dyrcona |
We have confirmed leaks in the C code. I've opened a bug about 1. |
| 16:16 |
Bmagic |
csharp_: open-ils.actor.user.has_work_perm_at.batch is the winner with 1784 requests |
| 16:17 |
Bmagic |
grep open-ils.actor osrfsys.log|grep -v cstore|grep -v auth|awk '{print $7}'|sort|uniq -c|sort -r|less |
| 16:18 |
csharp_ |
yeppers - that sounds right |
| 16:20 |
Bmagic |
seems low to me for 150 drones to be maxed |
| 16:21 |
Bmagic |
Drycona: maybe lower the max_requests to 200 from 1000 on open-ils.actor, so they rotate quicker? |
| 16:22 |
Dyrcona |
Dunno if that would stop us from hitting max children, but it would likely save a little memory. |
| 16:23 |
Bmagic |
I was thinking that the total number of Drones should fall sooner rather than later once the spike in activity finishes |
| 16:24 |
Dyrcona |
Guess it depends on how evenly the requests get spread out. |
| 16:24 |
Bmagic |
I'm thinking that the shipped default max_children (100) is probably good. csharp_: do you tweak that number? |
| 16:25 |
Bmagic |
sorry, max_requests not max_children |
| 16:25 |
Dyrcona |
maybe I'll give it a whirl tomorro. |
| 16:25 |
Dyrcona |
Yes, I understood max_requests. |
| 16:25 |
Dyrcona |
max_children stays the same. lower max_requests |
| 16:26 |
Bmagic |
right, for you. I was asking csharp_ a question |
| 16:37 |
* Dyrcona |
bugs out for the day. |
| 17:10 |
|
sandbergja joined #evergreen |
| 17:15 |
|
mmorgan left #evergreen |
| 17:28 |
|
jihpringle joined #evergreen |
| 18:19 |
|
jihpringle joined #evergreen |
| 18:28 |
csharp_ |
Bmagic: sorry - missed your question - we have ours set to 192, which was necessary under ejabberd/opensrf but we don't come anywhere near that now under redis |
| 18:42 |
|
jihpringle joined #evergreen |
