| Time |
Nick |
Message |
| 06:30 |
pinesol_green |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:13 |
|
Dyrcona joined #evergreen |
| 07:18 |
|
jboyer joined #evergreen |
| 07:19 |
|
agoben joined #evergreen |
| 07:32 |
|
Dyrcona joined #evergreen |
| 07:35 |
Dyrcona |
@blame The heat |
| 07:35 |
pinesol_green |
Dyrcona: The heat crafted the perfect SHA-1 collision, breaking Git |
| 07:37 |
|
JBoyer joined #evergreen |
| 07:37 |
Dyrcona |
@weather |
| 07:37 |
pinesol_green |
Dyrcona: Methuen, MA :: Clear :: 77F/25C | Tuesday: Scattered clouds with the possibility of an isolated thunderstorm developing this afternoon. High 96F. Winds WSW at 5 to 10 mph. Chance of rain 30%. Tuesday Night: Widely scattered showers or a thunderstorm this evening. Then partly cloudy. Low 72F. Winds light and variable. Chance of rain 30%. |
| 07:37 |
Dyrcona |
JBoyer: You have trouble with Freenode, too? |
| 07:37 |
JBoyer |
Yeah, DNS lookups were only returning IPv6 addresses for a while. |
| 07:38 |
* JBoyer |
was >< this close to assuming that the state had blocked all of the ports and whatnot. :/ |
| 07:38 |
Dyrcona |
I switched to irc.freenode.net and that worked, then I switched back to chat.freenode.net. |
| 07:39 |
Dyrcona |
By the time I tried a DNS lookup, I guess chat was returning IPv4 addresses again. |
| 07:39 |
Dyrcona |
@blame The State |
| 07:39 |
pinesol_green |
Dyrcona: The State is NOT CONNECTED TO THE NETWORK!!! |
| 07:39 |
JBoyer |
Both were failing for me earlier. Networking is hard, so I hear. |
| 07:40 |
Dyrcona |
heh. |
| 07:40 |
Dyrcona |
Maybe I caught right on the end of it being broken. |
| 07:41 |
Dyrcona |
Something similar happened with Vandelay this morning, too. |
| 07:42 |
Dyrcona |
Someone opened a ticket at 6:40 am saying the batch record import interface would not load for them on xul or in the web staff client. |
| 07:42 |
Dyrcona |
By the time I saw the ticket and looked into it at 7:20, it worked for me. |
| 07:42 |
Dyrcona |
Gonna be one of those days, I think. |
| 07:44 |
agoben |
I say we all sign out and go for an early holiday! :) |
| 07:44 |
Dyrcona |
+1 |
| 07:44 |
Dyrcona |
agoben++ |
| 07:44 |
JBoyer |
\o/ |
| 07:47 |
|
collum joined #evergreen |
| 07:53 |
rhamby |
agoben++ |
| 07:53 |
|
rlefaive joined #evergreen |
| 07:54 |
agoben |
:D ....now to persuade the bosses. |
| 07:55 |
Dyrcona |
:) |
| 07:56 |
|
rlefaive joined #evergreen |
| 07:59 |
|
rjackson_isl joined #evergreen |
| 08:04 |
|
stephengwills joined #evergreen |
| 08:08 |
Dyrcona |
Figured out the Vandelay problem: router bh2.private/open-ils.cstore IS NOT CONNECTED TO THE NETWORK!!! |
| 08:08 |
Dyrcona |
Guess pinesol_green was right. |
| 08:20 |
csharp |
so right now, the script to run notices is no longer running, but the drone is still going - all events are sitting in "Collected" state - nothing in the error or warn logs showing a failure |
| 08:21 |
Dyrcona |
Well, that one event def had 63,000+ entries. |
| 08:21 |
csharp |
not sure whether to kill off the drone (which is apparently still updating events) or let it finish |
| 08:21 |
csharp |
yeah - that's high for us but not insane |
| 08:22 |
Dyrcona |
Looks like a custom event. |
| 08:22 |
csharp |
I'm concerned that the script is dead with the process still running though |
| 08:22 |
csharp |
yeah - all of ours have been customized |
| 08:22 |
Dyrcona |
Script timed out, but the drone keeps going. |
| 08:22 |
csharp |
shouldn't there be a log message saying that somewhere? |
| 08:23 |
Dyrcona |
I.E. the requestor timed out waiting on a response from the drone. |
| 08:23 |
Dyrcona |
Should be, yeah. |
| 08:23 |
Dyrcona |
Not sure it's easy to find. |
| 08:26 |
Dyrcona |
I'm having a hard time finding an example. My logs this morning are dominated by what I shared earlier. |
| 08:26 |
Dyrcona |
Though, I did find a typo that someone made in their EDI configuration. |
| 08:27 |
csharp |
yeah... acq is definitely something where the fewer hands on it the better |
| 08:27 |
Dyrcona |
They typoed a hostname, it looks like. |
| 08:27 |
csharp |
yeah - I've seen that |
| 08:28 |
csharp |
another place where failure can become catastrophic from small things |
| 08:28 |
Dyrcona |
"Name or service not known" |
| 08:32 |
Dyrcona |
Um... No. |
| 08:32 |
Dyrcona |
The hostname resolves. |
| 08:33 |
csharp |
did they leave off the protocol? |
| 08:35 |
csharp |
jeez - debug makes the log files almost too large to be manageable |
| 08:35 |
Dyrcona |
csharp: It does, and you'll fill your disks in a hurry. |
| 08:35 |
Dyrcona |
Doesn't look like they did. |
| 08:36 |
Dyrcona |
2018-07-03 08:30:03 util2 ./edi_fetcher.pl: [ERR :3230:RemoteAccount.pm:553:] OpenILS::Utils::RemoteAccount : new Net::FTP('ftp1.ingrambook.com , ...) FAILED: Name or service not known |
| 08:36 |
Dyrcona |
That's the error. |
| 08:36 |
csharp |
yep' |
| 08:37 |
Dyrcona |
There's 47 of those in the database and non have a stray ' in them. |
| 08:37 |
csharp |
weird |
| 08:38 |
Dyrcona |
But, one has an extra space on the end. :) |
| 08:39 |
Dyrcona |
Maybe the code should strip trailing spaces, but I'll update this one. |
| 08:54 |
JBoyer |
btrim all the things! |
| 08:55 |
Dyrcona |
This was the only one with a space. |
| 08:55 |
Dyrcona |
But yeah. |
| 08:56 |
Dyrcona |
:) |
| 09:04 |
Dyrcona |
Well, kernel update on the laptop. BBIAB! |
| 09:06 |
|
bos20k joined #evergreen |
| 09:07 |
|
Dyrcona joined #evergreen |
| 09:35 |
|
yboston joined #evergreen |
| 09:48 |
csharp |
okay - short-term, looks like we need a custom hook that excludes circs for patrons without email accounts - has anyone already worked out the JSON SQL for that kind of thing? |
| 09:48 |
* csharp |
goes crosseyed looking at JSON query |
| 09:51 |
berick |
i have something that should work... |
| 09:52 |
berick |
csharp: http://libmail.georgialibraries.org/pipermail/open-ils-general/2017-April/013808.html |
| 09:52 |
berick |
add email: {"!=":null} to the "where" clause toward the bottom |
| 09:52 |
berick |
(and remove the profile stuff, presumably) |
| 09:52 |
berick |
well, that's just an example -- but you get the idea |
| 09:56 |
csharp |
berick++ # excellent - thanks! |
| 10:00 |
|
terran joined #evergreen |
| 10:08 |
|
sandbergja joined #evergreen |
| 10:09 |
Dyrcona |
Bleh... |
| 10:09 |
Dyrcona |
postgresql-- |
| 10:10 |
Dyrcona |
Because creating a table with a variable for the name behaves differently if you use \set or -v. |
| 10:11 |
Dyrcona |
With -v, you syntax error at or near ":" |
| 10:11 |
Dyrcona |
With \set, it works. |
| 10:12 |
Dyrcona |
At least interactively. I got errors non-interactively, IIRC. |
| 10:12 |
* Dyrcona |
double checks. |
| 10:18 |
Dyrcona |
Doesn't seem to work with -c... |
| 10:19 |
Dyrcona |
Fails silently. |
| 10:19 |
JBoyer |
Dyrcona, does it work as expected if you use :"blah" than just plain :blah ? I've never liked having to specify the correct quoting and whatnot when using \set or -v. |
| 10:19 |
Dyrcona |
JBoyer: I've tried quoting variations, but I'll try that one. |
| 10:20 |
Dyrcona |
It can be fun figuring out the variable quoting. :) |
| 10:21 |
Dyrcona |
So, :"vartable" does the same as :vartable with -c, i.e. nothing. |
| 10:22 |
* Dyrcona |
tries that quoting with -v |
| 10:23 |
Dyrcona |
When I quoted like so, ":vartable" a table named :vartable was created. |
| 10:23 |
Dyrcona |
I half-expected that. |
| 10:23 |
|
Christineb joined #evergreen |
| 10:23 |
Dyrcona |
Maybe I should ask in #postgresql. |
| 10:25 |
Dyrcona |
nano messes me up. I start typing vi commands and it puts them in the buffer. I should change my remote editor to emacs or vi. |
| 10:26 |
Dyrcona |
With -v, I still get the syntax error. :( |
| 10:27 |
Dyrcona |
Grr. psql-- |
| 10:27 |
Dyrcona |
Inconsistent behavior. |
| 10:27 |
Dyrcona |
The \set works interactively. |
| 10:28 |
* Dyrcona |
tries it in a script, but that was what brought me to experiment. I believe it failed in a script. |
| 10:30 |
Dyrcona |
Well, it worked from a script. |
| 10:30 |
Dyrcona |
with \set |
| 10:32 |
Dyrcona |
So, -c behaves differently from -f |
| 10:34 |
Dyrcona |
Seems to work with -v and -f.... |
| 10:34 |
* Dyrcona |
shrugs |
| 10:36 |
jeff |
what does your command line look like? |
| 10:37 |
Dyrcona |
psql -f ~/create-test.sql -v vartable=test_table |
| 10:37 |
Dyrcona |
That worked. |
| 10:37 |
Dyrcona |
I'll paste two that didn't. |
| 10:41 |
|
stephengwills joined #evergreen |
| 10:41 |
Dyrcona |
https://pastebin.com/Bi8Hh0s5 |
| 10:42 |
JBoyer |
Dyrcona, with quotes the : still goes on the outside. :"vartable" |
| 10:43 |
Dyrcona |
JBoyer, I now that, thanks. :) |
| 10:43 |
Dyrcona |
s/now/know/ |
| 10:43 |
JBoyer |
Ah, couldn't tell from your previous message. |
| 10:43 |
Dyrcona |
That fails. |
| 10:45 |
JBoyer |
Oh, hey. This is kind of important: (man psql, -c... ) command must be either a command string that is completely parsable by the server (i.e., it contains no psql-specific features), |
| 10:45 |
|
jvwoolf joined #evergreen |
| 10:45 |
JBoyer |
:vars are very much psql-specific |
| 10:46 |
JBoyer |
(the rest: or a single backslash command_ |
| 10:47 |
* jeff |
nods |
| 10:47 |
jeff |
that's why the CREATE TABLE is never executed. |
| 10:47 |
JBoyer |
Looks like -f or echo '...' | psql is the way to fo |
| 10:47 |
JBoyer |
fo? go. pogo. |
| 10:48 |
jeff |
and also why even if you turn it into two -c arguments, the variable isn't parsed in the second. |
| 10:51 |
Dyrcona |
I know what I need to know, now. |
| 10:51 |
Dyrcona |
Doesn't help that the last hour here has been a little chaotic. |
| 10:51 |
JBoyer |
:( |
| 10:52 |
Dyrcona |
I'm running scripts using -f. |
| 10:52 |
Dyrcona |
These are scripts from the esi migration_tools repo. |
| 10:52 |
Dyrcona |
One of them sets a table name via variable and that par failed. |
| 10:52 |
Dyrcona |
s/par/part/ |
| 10:53 |
Dyrcona |
That was yesterday. |
| 10:55 |
csharp |
okay - I also upped the number of concurrent collect and react processes, which is the kind of tuning that solved our problems like this in the past - gonna work on the new hook with the email filter next |
| 10:56 |
* csharp |
kind of thinks the increase in parallel procs might Just Work™ |
| 11:08 |
|
sandbergja joined #evergreen |
| 11:12 |
Dyrcona |
phasefx++ |
| 11:12 |
Dyrcona |
javascript-- |
| 11:20 |
JBoyer |
Dyrcona, joke's on you, 156 lines up 'javascript' was coerced into a string! ;) |
| 11:22 |
Dyrcona |
:) |
| 11:28 |
|
jvwoolf1 joined #evergreen |
| 11:29 |
Dyrcona |
@weather |
| 11:29 |
pinesol_green |
Dyrcona: Methuen, MA :: Clear :: 91F/33C | Heat Index: 99F/37C | Tuesday: Scattered clouds with the possibility of an isolated thunderstorm developing this afternoon. High near 95F. Winds WSW at 5 to 10 mph. Chance of rain 30%. Tuesday Night: Some clouds. A stray shower or thunderstorm is possible. Low 72F. Winds light and variable. |
| 11:37 |
|
khuckins joined #evergreen |
| 11:49 |
Dyrcona |
Hm... |
| 11:50 |
Dyrcona |
So, the query with the table created via a variable still doesn't work, but I don't think the variable is the problem. |
| 11:50 |
Dyrcona |
Ah ha! |
| 11:51 |
Dyrcona |
There's a failure of a different query in the transaction that creates the table. |
| 11:51 |
Dyrcona |
Fkey constraint. |
| 11:56 |
Dyrcona |
Call numbers owned by 1 lib but copies owned by another. |
| 11:59 |
Dyrcona |
Real life "data." |
| 11:59 |
|
dwgreen joined #evergreen |
| 12:00 |
JBoyer |
Uh, that's pretty common; what's that script doing? |
| 12:01 |
JBoyer |
Not important I suppose; just seems an odd thing to get hung up on. |
| 12:10 |
|
yboston joined #evergreen |
| 12:14 |
|
beanjammin joined #evergreen |
| 12:23 |
|
jihpringle joined #evergreen |
| 12:26 |
Dyrcona |
JBoyer: http://git.esilibrary.com/?p=migration-tools.git;a=blob;f=remove_ou_data/08_remove_volumes.sql;h=819ac7530bfc270966e12dac1bfc98333adbf553;hb=dbc902eb8bf45fc6c6287dba5d9242923caaa18d |
| 12:28 |
Dyrcona |
In my case, 3 call numbers owned by the ou that I'm trying to delete were on copies owned by a state-wide virtual catalog. (The old one, not the current one.) |
| 12:28 |
Dyrcona |
So, I changed the owning_lib on the call numbers and ran the script again. |
| 12:29 |
Dyrcona |
It's sitting on commit; so I think that's a good sign. |
| 12:51 |
|
Dyrcona joined #evergreen |
| 13:11 |
|
yboston joined #evergreen |
| 13:36 |
csharp |
nope - still broken |
| 13:36 |
csharp |
this is serious |
| 13:38 |
JBoyer |
csharp, you could also reset about 8-10K of them, process them, etc. to bring down the backlog to see if that helps or if there's some bad data in there somewhere. (Not a long term solution, but at least you might be able to not work all day tomorrow) |
| 13:40 |
csharp |
yeah :-( |
| 13:42 |
JBoyer |
I did some looking to try to determine the point at which running action_triger_runner.pl actually creates an SQL statement, and it's pretty deep. :/ |
| 13:46 |
|
Dyrcona joined #evergreen |
| 13:47 |
Dyrcona |
JBoyer: Networking is hard. |
| 13:47 |
csharp |
yeah - this entire infrastructure is pretty hard to understand and administer - I'm going to propose re-architecting the entire notifications mechanism |
| 13:47 |
JBoyer |
And a potentially useful Q: are these huge counts on consortium-wide event defs, or are there multiple events with the same granularity piling up? (i.e. can you separate them out into smaller granularity chunks?) |
| 13:47 |
JBoyer |
Dyrcona++ |
| 13:47 |
JBoyer |
I heard that somewhere. |
| 13:48 |
Dyrcona |
My wifi went out, then I plugged in the wired connection wouldn't work. |
| 13:48 |
csharp |
they are as granular as they can get - they are consortium-wide |
| 13:48 |
Dyrcona |
I blame recent kernels for that, but could not really troubleshoot. |
| 13:48 |
JBoyer |
bummer. |
| 13:49 |
csharp |
because of the structure of A/T, we have to repeat the process per notification method |
| 13:49 |
JBoyer |
Dyrcona, do you run an LTS on your laptop or the most recent annual? |
| 13:49 |
Dyrcona |
I usually run most recent. |
| 13:49 |
csharp |
so run all of them for emails, all of them for sms, all of them for patron message center... |
| 13:49 |
Dyrcona |
JBoyer: I'm presently on 18.04. |
| 13:50 |
Dyrcona |
The wired connection worked a week or two ago. I can check the date. |
| 13:50 |
csharp |
the emails are the most important and for whatever reason, they succeeded today on the first run |
| 13:50 |
Dyrcona |
Last time I did a full backup. |
| 13:50 |
csharp |
I'm currently suspecting bad data, but since logging for this sucks and debug creates unmanageable log files, I'm starting to despair a bit |
| 13:52 |
* csharp |
accepts his fate and changes job title to "Action Trigger Runner" |
| 13:53 |
csharp |
saw a couple of db deadlocks, so I reduced the number of parallel procs back to 3 |
| 13:53 |
csharp |
(hadn't seen those before making the change) |
| 13:53 |
JBoyer |
csharp, I suspect the way to address that is a combination multi-validator/multi-reactor setup, which trades many events for very complex events. Not sure that's a win. |
| 13:53 |
Dyrcona |
Actually, I'm running the kernel that was current then, and the networking is still busted. |
| 13:53 |
Dyrcona |
I'm on wifi now. |
| 13:53 |
csharp |
yeah, that's why I think the whole thing needs to be reconsidered |
| 13:54 |
JBoyer |
Capping the # of events and returning a "there are more events pending" type of message so the runner (or something else?) knows to do the whole thing again until the call returns with 0 events remaining. |
| 13:54 |
JBoyer |
Dyrcona, I don't want to be That Guy, but that's one of the reasons I've never been into Linux on laptops. ;) |
| 13:54 |
csharp |
really, if OpenSRF could be taught not to just keep trying to RAM things through to a dead drone would help too |
| 13:54 |
csharp |
s/RAM/ram/ |
| 13:55 |
Dyrcona |
JBoyer: I know, I know...I should switch to FreeBSD! :P |
| 13:55 |
csharp |
apparently I have muscle memory for RAM as well as things like PINES |
| 13:55 |
JBoyer |
(That may well be more robust. ;p ) |
| 13:56 |
JBoyer |
I've never cared for the SHOUTY SERVICE NAMES, even though we have one ourselves. (Nobody wanted my opinion 20 years ago...) |
| 13:57 |
Dyrcona |
JBoyer: Maybe I need to install Silverlight? :P |
| 13:57 |
JBoyer |
That wasn't me trying to snark on Linux, I think BSD code is very well written. And even MS came to their senses re: Silverlight. |
| 13:58 |
* bshum |
instantly felt wrong considering giving Microsoft positive karma... and refrains for now |
| 13:59 |
Dyrcona |
heh. |
| 13:59 |
Dyrcona |
Well, that script is still on commit. |
| 13:59 |
Dyrcona |
tmux++ |
| 14:01 |
JBoyer |
csharp, what kind of resources does that machine have available to it? |
| 14:05 |
csharp |
32GB RAM, 16 cores, SSDs |
| 14:05 |
csharp |
it's a VM, so it can be upped, but the machine does not appear to be under any sort of duress when this happens |
| 14:06 |
miker |
csharp: fwiw, I've been looking for support for a multi-reactor enhancement since, well, after the "it does enough, we don't want to pay for more work" initial version </rant> |
| 14:07 |
csharp |
miker: look for someone willing to pay in the near future - we're really concerned over here |
| 14:07 |
csharp |
as an admin, I need it to be more configurable and less prone to random breakage |
| 14:08 |
* csharp |
is happy to help make that happen |
| 14:09 |
miker |
csharp: noted. fwiw, it'f it's just the timeout at the at-runner level that's causing your pain, look at line ~25 of action_trigger_runner.pl ... consider adding a 0 |
| 14:10 |
csharp |
miker: is that ms? |
| 14:10 |
miker |
no, seconds |
| 14:10 |
csharp |
oh - hmm |
| 14:15 |
csharp |
does anything log if it times out? I'm currently at a loss for the cause of this |
| 14:15 |
csharp |
it goes from fine to NOT CONNECTED TO THE NETWORK!!! |
| 14:17 |
miker |
csharp: can you (or have you) paste the whole not-connected log line? I suspect that's the server saying it can't send a result because the client that made the request is no longer there ... which is what happens when the client times out in ->receive() and goes away |
| 14:18 |
csharp |
2018-07-03 13:16:06 utility03 open-ils.trigger: [ERR :3443:EX.pm:66:] Exception: OpenSRF::EX::Session 2018-07-03T17:16:06 OpenSRF::Transport /usr/local/share/perl/5.22.1/OpenSRF/Transport.pm:83 Session Error: opensrfprivate.utility03.gapines.org/client_at_utility03.gapines.org_3526 IS NOT CONNECTED TO THE NETWORK!!! |
| 14:18 |
miker |
yeah, note: client_at_utility03.gapines.org_3526 |
| 14:19 |
miker |
that's what's not connected. not a drone |
| 14:19 |
csharp |
so client would be the A/T runner script? |
| 14:19 |
miker |
the trigger service is saying "I sent something to client_at_utility03.gapines.org_3526 but it bounced because that jid isn't there" |
| 14:19 |
miker |
yes |
| 14:21 |
miker |
csharp: and if you run at-runner in verbose mode, cron will send you an email of what it does / how far it gets. there's a debug option for even more info |
| 14:22 |
csharp |
ok - thanks for those pointers |
| 14:24 |
JBoyer |
Maybe add 2 0's to kick that can on down the line a little harder? :) |
| 14:24 |
miker |
heh |
| 14:24 |
miker |
300 hours :) |
| 14:24 |
JBoyer |
(only mostly kidding) |
| 14:30 |
csharp |
miker++ # helpin' |
| 14:30 |
csharp |
JBoyer++ # too |
| 14:31 |
JBoyer |
csharp++ # perseverance. |
| 14:34 |
Dyrcona |
JBoyer: Logs are pointing to it being a systemd problem. |
| 14:35 |
JBoyer |
Well that's going to be buckets of no fun. |
| 14:39 |
Dyrcona |
Yeahp. Looks kind of like bug 1654878 |
| 14:39 |
pinesol_green |
Launchpad bug 1654878 in systemd (Ubuntu) "segmentation fault in systemd-resolve" [Undecided,Expired] https://launchpad.net/bugs/1654878 |
| 14:42 |
JBoyer |
Not very reassuring. :/ |
| 15:03 |
Dyrcona |
I'm done with work for the day, so I'm signing out and putting the laptop to sleep. |
| 15:17 |
|
rlefaive joined #evergreen |
| 15:29 |
JBoyer |
Yay bug 1779920 |
| 15:29 |
pinesol_green |
Launchpad bug 1779920 in Evergreen "wishlist: Add Auto Renewal functionality" [Wishlist,New] https://launchpad.net/bugs/1779920 |
| 15:30 |
* JBoyer |
is looking forward to it. |
| 15:43 |
Bmagic |
what's the thought behind having two different indexes on action.circulation for (usr)? Save disk? |
| 15:43 |
Bmagic |
It seems that it would be faster if the index were on all rows for (usr) in order to make the join from the action.all_circulation view faster ? |
| 15:44 |
Bmagic |
https://explain.depesz.com/s/WKRu |
| 15:46 |
jeff |
Bmagic: context? |
| 15:46 |
Bmagic |
diving down the rabbit hole trying to figure out why the item status screen isn't showing previous circs. I am assuming it's a timeout issue |
| 15:47 |
Bmagic |
I run the query that (I think) the server is running to come up with the data. Which, correct me if I am wrong, is on action.all_circulation |
| 15:55 |
Bmagic |
weird, it's only on the xul runner staff client |
| 15:56 |
|
yboston joined #evergreen |
| 16:29 |
berick |
Bmagic: the browser client uses the new 'aacs' class action.all_circulation_slim which is leaner/faster than all_circulation |
| 16:29 |
Bmagic |
aha! |
| 16:29 |
berick |
don't think the xul client was given that ability |
| 18:15 |
|
jvwoolf1 left #evergreen |
| 18:30 |
pinesol_green |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 21:59 |
|
sandbergja joined #evergreen |
