| Time |
Nick |
Message |
| 06:17 |
|
eady joined #evergreen |
| 06:30 |
pinesol_green |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 06:41 |
|
JBoyer joined #evergreen |
| 07:03 |
|
agoben joined #evergreen |
| 07:10 |
|
rjackson_isl joined #evergreen |
| 07:12 |
|
rjackson_isl_ joined #evergreen |
| 07:32 |
|
stephengwills joined #evergreen |
| 07:33 |
JBoyer |
csharp, re: the A/T processing dying and leaving a bunch of notices on the floor, do they usually have a start time set? You could set up a nagios/icinga/etc. check for events whose state != |
| 07:33 |
JBoyer |
pending, invalid, or error, and whose start_time is more than some time in the past. (an hour or two should do, subject to some guestimate + verification) |
| 07:35 |
JBoyer |
I suppose you could use also run_time if start_time isn't set. |
| 08:07 |
|
Dyrcona joined #evergreen |
| 08:08 |
JBoyer |
csharp, uh, if you do take that advice, you'll obviously want to include complete in that list of statuses or that will be one whiney check. |
| 08:44 |
csharp |
JBoyer: thanks - I was pondering something similar :-) |
| 08:44 |
|
sandbergja joined #evergreen |
| 08:44 |
JBoyer |
It would be nice to track down the issue though, so robustification can ensue. |
| 08:45 |
csharp |
JBoyer: one thing I'm doing (that's been a to-do for years) is to re-tool the a/t filter so that it excludes patrons with no email address/sms notify set |
| 08:46 |
csharp |
since it's coinciding with higher circ loads, I'm wondering if it's a "too many to process" problem |
| 08:46 |
csharp |
consistently dying on the grouping step though, that I know |
| 08:48 |
csharp |
the other change this year is that we now have Savannah & surrounding areas in PINES - might be pushing our limits even more |
| 08:48 |
JBoyer |
It sounds like it needs to limit and loop rather than just run everything at once. |
| 08:48 |
csharp |
yeah |
| 08:48 |
JBoyer |
(though there is real benefit to ignoring email/sms events with no contact info) |
| 08:48 |
csharp |
I definitely think there's room for optimization there |
| 08:49 |
* JBoyer |
stares sadly at the empty tuits shelf in the cupboard. |
| 09:03 |
|
bos20k joined #evergreen |
| 09:12 |
|
collum joined #evergreen |
| 09:25 |
* csharp |
just realized that the Hack-A-Way is scheduled over Election Day |
| 09:25 |
csharp |
so all y'all need to vote early :-) |
| 09:30 |
|
yboston joined #evergreen |
| 09:41 |
|
jvwoolf joined #evergreen |
| 09:46 |
Dyrcona |
csharp: And often... :) |
| 09:47 |
|
yboston joined #evergreen |
| 10:06 |
JBoyer |
I'm planning to vote early as long as I'm living where I am because at least then there's paper involved.// |
| 10:28 |
|
khuckins joined #evergreen |
| 10:34 |
|
terran joined #evergreen |
| 10:49 |
Bmagic |
When doing year end rollover, the system will create new funds from all of the previous funds (that have rollover=true) and increment the year value. This is performed without the checkbox close-out. Once that is done, is it normal to come in weeks later and do it again with the close-out checked? |
| 10:59 |
berick |
Bmagic: i think so, yes, so you can start using the new funds before the old ones are done. |
| 11:00 |
Bmagic |
I have a problem because the second time that this UI was used, it created 2020 funds... not sure what happend but something wasn't flipped or checked correctly |
| 11:02 |
* csharp |
looks around for tlittle who would be able to answer for PINES' process |
| 11:03 |
csharp |
hmm - looks like at least part of my issue is that open-ils.trigger drones were exhausted |
| 11:03 |
csharp |
upped the max_children from 15 to 25 - we'll see what happens |
| 11:04 |
Bmagic |
when in doubt: MOAR DRONES |
| 11:05 |
csharp |
osrf_control --diagnostic FTW |
| 11:05 |
Bmagic |
berick: it doesn't appear that the system knows anything about fiscal year dates, therefore, I am confused how it knows which set of funds to close-out during the second stroke. I do see two tables: acq.fiscal_calendar(1 row) and acq.fiscal_year(0 rows) |
| 11:06 |
csharp |
you have to add the year manually (as far as I know) |
| 11:07 |
csharp |
manually = no UI, must be done directly in the DB |
| 11:07 |
Bmagic |
the year-end UI seemed to create the year (or at least assign the auto-created funds a year in acq.fund.year which is a ::TEXT column) |
| 11:08 |
csharp |
Bmagic: for comparison, we have a fiscal_calendar named "Default" and have created years for each FY since 2016 |
| 11:08 |
Bmagic |
you have rows in acq.fiscal_year ? |
| 11:09 |
pastebot |
"csharp" at 64.57.241.14 pasted "for Bmagic" (15 lines) at http://paste.evergreen-ils.org/13578 |
| 11:09 |
csharp |
Bmagic: I have created each of those in advance of each fiscal year |
| 11:10 |
Bmagic |
ah! Cool - you have to manually insert into acq.fiscal_year? |
| 11:10 |
csharp |
yeah |
| 11:10 |
csharp |
seems like a good UI to create with new angular :-) |
| 11:12 |
Bmagic |
confusing... acq.fund.year is not a foriegn key to fiscal_year |
| 11:13 |
Bmagic |
Any advice on how to fix the issue with having 2019 AND 2020? The best outcome would be to remove all 202 funds and bring 2019 back to life |
| 11:13 |
Bmagic |
202/2020 |
| 11:14 |
|
beanjammin joined #evergreen |
| 11:21 |
Bmagic |
safe to "delete from acq.fund where year=2020" ? |
| 11:21 |
Bmagic |
none of the funds have allocations |
| 11:25 |
|
stephengwills joined #evergreen |
| 11:27 |
Bmagic |
well - I did that and it seems to be fine..... |
| 11:28 |
Dyrcona |
Yeah, it should be OK. |
| 11:29 |
Bmagic |
I suppose my next question is: "What does it mean to close-out"? Does that simply flip the active column from true to false for acq.fund? |
| 11:35 |
Bmagic |
The text on the UI says that it will move the encumbrances and rollover unspent money (unless the library setting says otherwise) - I am going to flip 2019 to active and hope for the best |
| 11:50 |
|
idjit joined #evergreen |
| 12:29 |
csharp |
look like upping the drones has solved the immediate issue, which means the futzing with JSON SQL to exclude users without emails goes back on the back burner :-) |
| 12:29 |
csharp |
(referring to my A/T issue from the weekend) |
| 12:33 |
Dyrcona |
For me, it was cstore drones last Thursday/Friday. |
| 12:56 |
|
sandbergja joined #evergreen |
| 13:14 |
|
bwicksall joined #evergreen |
| 15:14 |
csharp |
well, no, in fact, that didn't fix the A/T issue |
| 15:19 |
JBoyer |
:/ |
| 15:19 |
JBoyer |
How many events do you have in various states (that should run today?) |
| 15:22 |
pastebot |
"csharp" at 64.57.241.14 pasted "A/T snapshot" (47 lines) at http://paste.evergreen-ils.org/13579 |
| 15:22 |
csharp |
JBoyer: ^^ |
| 15:23 |
csharp |
the most frustrating part of this is that our A/T has been purring along with *no* issues since January |
| 15:24 |
JBoyer |
Wow there's a big ol' chunk of things in there. :/ |
| 15:24 |
csharp |
and I'm about to be off for a few days, but since this is happening, not sure that's going to actually be time off |
| 15:24 |
csharp |
yeah :-/ |
| 15:24 |
JBoyer |
Savannah has been on for a while, yes? So this shouldn't necessarily be because they've pushed you over some thresshhold? |
| 15:24 |
JBoyer |
(recently, that is) |
| 15:25 |
csharp |
well, this is the first summer with them on board |
| 15:25 |
JBoyer |
Oh, makes sense. |
| 15:26 |
csharp |
my only working theory at this point is that there are too many to group |
| 15:26 |
JBoyer |
Logs have nothing to say I suppose? |
| 15:26 |
csharp |
but no evidence from logs |
| 15:26 |
csharp |
just about to say that |
| 15:26 |
JBoyer |
(including DB logs, i.e. killing a query because of oom, etc.?) |
| 15:26 |
csharp |
it just goes from fine to NOT CONNECTED TO THE NETWORK!!! |
| 15:26 |
csharp |
yeah, nothing like that |
| 15:26 |
csharp |
DB is totally fine |
| 15:26 |
JBoyer |
cstore or trigger or ? |
| 15:27 |
csharp |
trigger is where the bottleneck was earlier |
| 15:27 |
JBoyer |
And it's the service giving the NOT CONNECTED... errors? |
| 15:27 |
Dyrcona |
Any "Could not launch a child" messages? |
| 15:30 |
Dyrcona |
csharp: FYI, we're running max of 20 open-ils.trigger and 60 open-ils.cstore on our utility server at CW MARS. |
| 15:31 |
Dyrcona |
Don't recall every having max number of trigger drones, but have hit the wall with cstore. |
| 15:31 |
Dyrcona |
s/every/ever/ |
| 15:32 |
Dyrcona |
No, mistyped, it's 70 for cstore. |
| 15:33 |
csharp |
Dyrcona: thanks |
| 15:33 |
Dyrcona |
If DB logs are OK, maybe some query is taking too long and cstore or trigger is timing out? |
| 15:35 |
csharp |
maybe :-( |
| 15:35 |
csharp |
@hate action_triggers |
| 15:35 |
pinesol_green |
csharp: But csharp already hates action_triggers! |
| 15:35 |
csharp |
@hate action_triggers more |
| 15:35 |
pinesol_green |
csharp: The operation succeeded. csharp hates action_triggers more. |
| 15:39 |
JBoyer |
I looked at some of our numbers and for today we had 15K day-of emails, and 28K 3 day pre-emails and 28K 3 day pre-sms messages. everything else is sub-2K. |
| 15:41 |
JBoyer |
Our limits are a little absurd (250 cstore, 120 trigger) but icinga's graphs show that we don't use more than 50/25. |
| 15:41 |
JBoyer |
The utility server does hit a lot of load though, I don't knkow what would happen if it was thrown 40-50K things to do. |
| 15:48 |
Dyrcona |
csharp: What SQL did you use to generate the pasted output? |
| 15:50 |
Dyrcona |
I did a simple sum using event.run_time > '2018-07-02 00:00' and my numbers are not even close to yours. |
| 15:51 |
pastebot |
"csharp" at 64.57.241.14 pasted "SQL for Dyrcona" (10 lines) at http://paste.evergreen-ils.org/13580 |
| 15:52 |
csharp |
originally generated by a reports template we get daily for the previous day's events |
| 15:52 |
Dyrcona |
Yeah, I gathered its from a report. |
| 15:53 |
csharp |
I think the "fix the JSON query" approach to this might be back on the table |
| 15:53 |
Dyrcona |
That produces pretty much the same output as what I ran for me. |
| 15:53 |
csharp |
ok |
| 15:53 |
Dyrcona |
My biggest number is 5,271. |
| 15:53 |
Dyrcona |
That's for SMS hold notifications with a state of invalid. |
| 15:55 |
Dyrcona |
So, my numbers are a lot smaller than yours. |
| 16:23 |
|
stephengwills joined #evergreen |
| 16:48 |
|
jvwoolf left #evergreen |
| 17:51 |
|
jboyer-isl joined #evergreen |
| 18:30 |
pinesol_green |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:41 |
|
sandbergja joined #evergreen |
| 20:07 |
|
sandbergja joined #evergreen |
| 21:29 |
csharp |
action_triggers-- |
| 21:29 |
csharp |
action_triggers-- |
| 21:29 |
csharp |
action_triggers-- |
| 21:36 |
* csharp |
enables debugging logs in hopes of finding where the failure happens |
| 23:27 |
|
beanjammin joined #evergreen |
