| Time |
Nick |
Message |
| 04:31 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 07:18 |
|
rjackson_isl joined #evergreen |
| 08:35 |
|
Dyrcona joined #evergreen |
| 08:44 |
|
mmorgan joined #evergreen |
| 08:46 |
|
bos20k joined #evergreen |
| 09:09 |
|
bos20k joined #evergreen |
| 09:22 |
|
Jillianne joined #evergreen |
| 09:45 |
|
jvwoolf joined #evergreen |
| 11:07 |
jeff |
@coffee #evergreen |
| 11:07 |
* pinesol_green |
brews and pours a cup of Rusty's Hawaiian Red Caturra Natural, and sends it sliding down the bar to #evergreen |
| 11:08 |
Dyrcona |
@tea #evergreen |
| 11:08 |
* pinesol_green |
brews and pours a pot of English Breakfast, and sends it sliding down the bar to #evergreen (http://ratetea.com/tea/harney/english-breakfast/3525/) |
| 11:11 |
mmorgan |
@quote random |
| 11:11 |
pinesol_green |
mmorgan: Quote #65: "< paxed> #evergreen: wrong questions answered correctly." (added by csharp at 02:27 PM, August 13, 2013) |
| 11:15 |
|
Freddy_Enrique joined #evergreen |
| 11:16 |
Freddy_Enrique |
Buenos dias Folks :) |
| 11:18 |
csharp |
@hate action_triggers |
| 11:18 |
pinesol_green |
csharp: The operation succeeded. csharp hates action_triggers. |
| 11:18 |
bshum |
@hates action_triggers |
| 11:18 |
pinesol_green |
bshum: action_triggers doesn't seem to hate anything. |
| 11:18 |
bshum |
:) |
| 11:18 |
jeff |
hah |
| 11:18 |
jeff |
@whocares action_triggers |
| 11:18 |
pinesol_green |
csharp hates action_triggers |
| 11:18 |
jeff |
there you go. |
| 11:18 |
bshum |
No I did mine on purpose |
| 11:19 |
jeff |
ah. |
| 11:19 |
bshum |
We hate action_triggers, it doesn't hate us |
| 11:19 |
bshum |
:D |
| 11:19 |
jeff |
well, now that you've said that... |
| 11:19 |
jeff |
:P |
| 11:19 |
bshum |
Heh |
| 11:24 |
mmorgan |
action_triggers just likes to toy with us ;-) |
| 11:29 |
csharp |
when they work, they're fine, but when they don't they fail spectacularly |
| 11:29 |
csharp |
and there's not a dependable way to monitor them in real time (that I'm aware of) |
| 11:29 |
csharp |
so you just find out later that they failed |
| 11:29 |
mmorgan |
Indeed. |
| 11:30 |
csharp |
I might just hate OpenSRF though :-) |
| 11:30 |
Dyrcona |
csharp: You're not sending the output to /dev/null are you? |
| 11:30 |
Dyrcona |
Of course, the email is still after the fact, but.... |
| 11:30 |
Dyrcona |
I love/hate computers. :) |
| 11:31 |
csharp |
no, I get notified |
| 11:31 |
csharp |
and we can monitor the lock file age or whatever, but that's not great either |
| 11:31 |
Dyrcona |
I was just adding something to our cron, and most of our local scripts go to /dev/null, so thought I'd ask. |
| 11:31 |
mmorgan |
computers also like to toy with us ;-) |
| 11:32 |
Dyrcona |
mmorgan: Not yet, but soon. :) |
| 11:33 |
berick |
csharp: i've been pondering an action_trigger.event watcher script that looks for events stuck in 'collected', etc. |
| 11:34 |
mmorgan |
csharp: When I'm running a trigger by hand, I monitor by querying the state, count(*) of the relevant rows in action_trigger.event. |
| 11:34 |
jeff |
while we're on the subject, what's the best thing to read to bring myself up to speed with A/T filters? |
| 11:34 |
mmorgan |
Not sure how that might apply to those run by cron. |
| 11:34 |
jeff |
i've been meaning to dig into docs/code/commits/launchpad/list-archives/slides to seek that out. |
| 11:35 |
Dyrcona |
And, I typoed my crontab entry. :P |
| 11:36 |
berick |
jeff: what do you mean by A/T filters.. validators? |
| 11:36 |
csharp |
berick: I would support that - if you want a tester/extra eyes/help, let me know |
| 11:37 |
jeff |
berick: A/T filters as in Open-ILS/examples/action_trigger_filters.json.example |
| 11:37 |
bshum |
Maybe he means granularities? |
| 11:37 |
bshum |
Oh those, heh |
| 11:37 |
berick |
ah |
| 11:38 |
jeff |
Something along the lines of using them to exclude patron groups from certain notification methods. I think someone mentioned something along those lines recently and I was curious to know more, including knowing if that was just a clever hack or a core purpose of filters. |
| 11:38 |
berick |
jeff: in short, the 'filter' component of a filter is a cstore query, run from the context of the target (hook core_type) search call. (though maybe that much was obvious) |
| 11:39 |
berick |
i think that was me :) |
| 11:40 |
csharp |
the other thing I want is a "nofitication method filter" that checks that the user in question actually has a valid email/sms entered before processing |
| 11:41 |
csharp |
during the gathering phase so circs for users without those entered aren't even considered |
| 11:41 |
jeff |
heh. state general mailing list for libraries often has "we're getting rid of STUFF, does anyone want it?" -- often for things that are still recent, but a library is weeding for their own local reasons, etc. |
| 11:41 |
mmorgan |
csharp++ |
| 11:41 |
jeff |
today's "anyone want these?" is regarding a large quantity of periodical backissues, and includes a spreadsheet with a column labeled "Approximate Linear Feet" :-) |
| 11:42 |
berick |
csharp: i think that would be possible using a similar trick to what jeff is referring to.. |
| 11:42 |
* berick |
finds the email |
| 11:42 |
miker |
filters are there to avoid having huge piles of state=invalid events generated by the validator. though, since validators are kinda underutilized (originally meant for folks to create their own -- they'll load from outside the EG modules following a simple set of rules), filters have become more important |
| 11:44 |
mmorgan |
jeff: berick: http://markmail.org/message/gqoltyr64hms3wmy |
| 11:44 |
Dyrcona |
I don't see any reason you couldn't filter with patron groups. |
| 11:44 |
berick |
csharp: http://libmail.georgialibraries.org/pipermail/open-ils-general/2017-April/013808.html -- replace the "profile" check with regex.. |
| 11:44 |
berick |
er, email regex |
| 11:44 |
berick |
or simple <> null check |
| 11:44 |
berick |
oh heh thanks mmorgan |
| 11:45 |
* mmorgan |
also had plans for delving into filters more. :) |
| 11:45 |
jeff |
that's what i was thinking of! |
| 11:45 |
jeff |
berick++ mmorgan++ miker++ |
| 11:48 |
csharp |
berick++ # that looks promising |
| 11:49 |
* Dyrcona |
fell behind in the conversation reviewing the filters used here. :) |
| 11:49 |
berick |
I think A/T deaths would be much less painful if it didn't have to collect everything up front, i.e. process groups start to finish before moving on to the next group. I wonder how hard it would be to teach the event group code to collect the groups in the initial query instead of loading everything to manually group. |
| 11:49 |
* csharp |
really needs to get more confortable with json queries and their perl dialects |
| 11:50 |
csharp |
berick: I would totally love that |
| 11:50 |
csharp |
we're dealing with 75K notices daily sometimes and that's a lot to check |
| 11:51 |
csharp |
the suggested A/T filter may mitigate some of that though |
| 11:51 |
berick |
yeah, and one hiccup means starting over from nothing |
| 11:51 |
csharp |
right |
| 11:51 |
* csharp |
prays that the re-do doesn't spill over into July 4th |
| 11:51 |
csharp |
I didn't get a Memorial Day holiday either for similar reasons |
| 11:52 |
Dyrcona |
Holiday? What's that? :) |
| 11:52 |
Dyrcona |
Oh, yeah, upgrade/maintenance days. :/ |
| 11:53 |
miker |
well, you can't process a group untill you've validated all events in the group, and you don't know what's in the group to validate until you've collected all events |
| 11:54 |
csharp |
the other factor that makes us (well, me) miserable is that to do email AND sms AND patron message center means three separate processes over the same data |
| 11:54 |
berick |
right, but the collect stage could (theoretically) grab items in groups before validation |
| 11:55 |
csharp |
and with preminder/courtesy notices that means everyone who doesn't return before the due date gets a message |
| 11:55 |
miker |
berick: sure ... that would leave them in collected, rather than validated, in the case of a death, though, no? |
| 11:56 |
miker |
csharp: I've longed for a chance to create "stacked" reactors. a way to specify more than one reactor to run over the data during the reacting phase. but, tuits |
| 11:57 |
berick |
miker: does it matter? any stuck state means "do over". |
| 11:57 |
miker |
(and, transactionally, it's more complicated that it might seem. what do we do when one of three reactors fails for some reason on an event) |
| 11:57 |
berick |
stacked_reactors++ |
| 11:59 |
miker |
berick: that's my point, really, I guess. if they're going to be left in a do-over state, does it matter which? IOW, what does making the state tree deep (collect-all, validate[0], react[0], validate[1], react[1]) instead of wide (collect-all, validate-all, react-all) buy us? |
| 12:00 |
miker |
it would make FTTB (for lack of a better term) faster, but the total time would be the same, or a bit longer |
| 12:01 |
|
_adb joined #evergreen |
| 12:03 |
miker |
and the logic would be more complicated, but that's not a hard argument against it if lower TTFB (not FTTB, duh) is an improvement |
| 12:04 |
berick |
miker: my point was only this, if we collected events in groups from the DB instead of collecting all events into the ML, then grouping in the ML, then A/T could better recover from a failure. it would avoid cases where we leave thousands of events stuck in collected, etc. |
| 12:04 |
berick |
death of one group would not kill the overall process |
| 12:07 |
miker |
but group_field isn't a value ... there's nothing to group them until you can analyze the full target set. I'm not saying it's impossible, but I think it would add an up-front query that spits out, say, an array of targets for each event, which we then loop over row-wise, using the values in the array as group_field filter values. is that what you're thinking? |
| 12:07 |
berick |
yes, exactly. hadn't thought the implementation that far, but that's idea. |
| 12:08 |
jeff |
it's a bad sign with the header of a CSV file isn't valid CSV. |
| 12:09 |
berick |
hm, could you also just sort on the group field in the initial query, then collect row-wise until it finds a row from the next group, etc. |
| 12:10 |
miker |
in open-ils.trigger.event.find_pending? probably, yeah |
| 12:11 |
berick |
yeah |
| 12:12 |
miker |
though you can't use parallel gather then |
| 12:12 |
miker |
because there's no shared state between gather requests in parallel (MultiSession) mode |
| 12:13 |
berick |
true |
| 12:13 |
miker |
and parallel gather can help a lot |
| 12:15 |
berick |
yeah, it can. just pondering the trade-off there |
| 12:18 |
berick |
e.g. if you replaced parallel=3 with running 3 separate granularity (email, sms, etc.) a/t processors at the same time, would the end result be about the same |
| 12:19 |
csharp |
maybe also figure out a way to configure some hard limits ("don't process more than 30000 events at one time") |
| 12:19 |
berick |
each would take longer, but the whole shebang would take about the same amount of time |
| 12:21 |
miker |
adding a sibling to find_pending that returned a pregrouped result (my thought experiment above) would allow gathering targets in parallel grouped by event. probably with more new code. |
| 12:21 |
miker |
so, we /could/ have both. |
| 12:22 |
miker |
also, having a granularity per notification type doesn't allow stacked reactors, which will cut actual CPU time be a lot by removing all pre-react duplication |
| 12:23 |
miker |
with stacked reactors, we allow more unrelated triggers to be processed at the same time, shortening the whole a/t daily run |
| 12:24 |
csharp |
miker++ # missed your stacked reactors comment |
| 12:27 |
berick |
miker: so collect the array of target values (as you noted above), chop those up and hand them off to be collected via parallel process? If we do that, the parallel process could also just go ahead and react too, i would think |
| 12:31 |
miker |
berick: sure, we could rework open-ils.trigger.event.run_all_pending and open-ils.trigger.event.find_pending_by_group to work a bit differently. probably a whole new code path that smooshes the two together, interleaving the state loops |
| 12:31 |
|
bos20k joined #evergreen |
| 12:32 |
miker |
s/$/ would be best/ |
| 12:32 |
berick |
yeah |
| 12:33 |
miker |
and a new find_pending that pregroups, obv |
| 12:33 |
mmorgan |
berick: miker: So at what point does the validation happen? |
| 12:34 |
miker |
mmorgan: after collect and before react |
| 12:35 |
mmorgan |
ok, gotcha. Thanks. |
| 12:36 |
miker |
the order is, aprox: 'pending','found','collecting','collected','validating',['invalid','valid'],'reacting','reacted','cleaning',['complete','error'] ... with stops at invalid after validating, or error at any point |
| 12:38 |
berick |
the order each event goes throuh would be the same, but each group of events (e.g 5 overdue circs for 1 patron) would go through the whole process ('pending' -> 'complete') before the next event group starts. |
| 12:41 |
mmorgan |
So if it was an email trigger and the 1 patron had no email address the whole group would be invalid right then and move on to the next group. |
| 12:42 |
miker |
or in parallel (to the degree max-parallel is allowed) ... however, there's a race condition here: by the time we start processing the last pre-grouped event set there may be more events that should be added to the group (or some that should have aged out of the group, perhaps). by doing all the collecting and validating up front, we shrink that window ... but by effectively serializing everything up to validation, especially over a large total event |
| 12:42 |
miker |
set, we'll likely have more instances of "missed" or "invalid" events |
| 12:42 |
miker |
now... |
| 12:42 |
miker |
that may not be a problem in practice |
| 12:42 |
miker |
if the large runs are all daily, and off-hours |
| 12:43 |
miker |
but, the race window /is/ wider |
| 12:43 |
miker |
mmorgan: yes. though jeff and csharp were asking about a way to avoid those altogether via filters |
| 12:43 |
miker |
in that specific case |
| 12:44 |
berick |
miker: hm, mabye the pre-grouping lookup code can grab the group value and the target ID. |
| 12:46 |
mmorgan |
berick++ |
| 12:46 |
miker |
berick: oh, i figured that'd be exactly what it does. something like: select $group_field as grouper, target, agg_arry(event) from event where ... group by 1,2; |
| 12:47 |
miker |
but that doesn't close the race condition window. the first event group will be (basically) perfect, and the last will be as out of date as (number_of_groups/parallelism)*avg_pending_to_complete_time |
| 12:48 |
miker |
right now, the window is more like (number_of_events*avg_found_to_validate_time) |
| 12:50 |
miker |
stacked reactors makes the hypothetical new way look better than it would without them, since the number of groups would shrink by a /lot/. by 66% in the case of email+sms+print stacked reactors |
| 12:50 |
* miker |
disappears for lunch |
| 12:51 |
berick |
ah, right, so that almost suggests ignore the targets during pre-lookup, then determine the targets as each group is collected. |
| 12:52 |
berick |
that would slow it down.. but yeah not sure how much it matters in practice |
| 13:38 |
miker |
note: stacked reactors can't just be "split reactor field on commas, run each" because some reactors are really only the start of a larger process, like print notices, and you need to know which reactor created which output. it's a whole infrastructure thing |
| 13:58 |
pinesol_green |
Showing latest 5 of 8 commits to Evergreen... |
| 13:58 |
pinesol_green |
[evergreen|Jason Etheridge] lp1661685 webstaff: add money.grocery to pcrud - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=3630820> |
| 13:58 |
pinesol_green |
[evergreen|Jason Etheridge] lp1661685 fieldmapper label change for circ - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=5a42920> |
| 13:58 |
pinesol_green |
[evergreen|Jason Etheridge] lp1661685 webstaff: Fix Owning Lib in Item Status - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=dada0a5> |
| 13:58 |
pinesol_green |
[evergreen|Jason Etheridge] lp1661685 webstaff: Circ Lib column for Items Out - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=f6a1851> |
| 13:58 |
pinesol_green |
[evergreen|Jason Etheridge] lp1661685 fix circ lib in patron holds list - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=972202a> |
| 15:07 |
|
mmorgan1 joined #evergreen |
| 15:53 |
|
Freddy_Enrique joined #evergreen |
| 16:29 |
|
mmorgan joined #evergreen |
| 16:31 |
pinesol_green |
News from qatests: Test Success <http://testing.evergreen-ils.org/~live> |
| 16:37 |
|
jvwoolf left #evergreen |
| 16:42 |
csharp |
@quote random |
| 16:42 |
pinesol_green |
csharp: Quote #87: "<jeff> responsive XUL isn't really a thing." (added by bshum at 02:49 PM, July 30, 2014) |
| 16:49 |
Freddy_Enrique |
excuse me guys, this is actually gonna be my first time using the mailing lists and dont know how to properly use it |
| 16:50 |
csharp |
Freddy_Enrique: we're all ears ;-) |
| 16:51 |
Freddy_Enrique |
is it necessary to subscribe to each one of them according to my needs or will it be enough with one subscription? |
| 16:51 |
Freddy_Enrique |
I've just subscribed to Evergreen General Discussion List |
| 16:53 |
csharp |
Freddy_Enrique: that's the "main" one, yes - the others are for specific topics (-dev is for general tech topics) |
| 16:54 |
csharp |
big community announcements/discussions come through on General |
| 16:55 |
Freddy_Enrique |
uhm, One of the many question I made was... When exactly and with what EG version will the staff client stop receiving support? |
| 16:55 |
csharp |
Freddy_Enrique: I would probably join General and Dev at a minimum |
| 16:56 |
csharp |
Freddy_Enrique: 3.0 is the target release for deprecating the staff client (meaning that it will be available but not receiving further support/development) |
| 16:56 |
Freddy_Enrique |
was my question appropiate for that list? |
| 16:56 |
csharp |
sure |
| 16:57 |
Bmagic |
don't be shy |
| 16:57 |
csharp |
since Evergreen is software, even the General list trends toward somewhat technical discussions sometimes |
| 16:57 |
Bmagic |
csharp: Evergreen is a way of life |
| 16:57 |
Bmagic |
sometimes softeware though |
| 16:57 |
csharp |
ommmmmmmmmmmm |
| 16:57 |
Bmagic |
sometimes software though |
| 16:58 |
Freddy_Enrique |
oh, but when you say it will not longer receive support/development what comes to my mind is...crashes |
| 16:58 |
Freddy_Enrique |
errors...etc etc |
| 16:58 |
Freddy_Enrique |
So, after installing and manipulating the client for a while I think its time to go web now |
| 16:59 |
Bmagic |
Freddy_Enrique: if you get a version working on a particular environment, it will probably work like that for a long time. It's when you need to upgrade the underlying stuff like the OS, when you run into issues mostly |
| 17:00 |
* csharp |
is kind of amazed that the client works with Windows 10 given how old it is |
| 17:00 |
Freddy_Enrique |
what the....really???? |
| 17:00 |
Bmagic |
That is MS doing the magic mostly |
| 17:00 |
csharp |
I totally expected craziness but it's been rock solid |
| 17:01 |
Freddy_Enrique |
then...as long as the staff client works, not really necessary to go web. |
| 17:01 |
Freddy_Enrique |
to tell you the truth, I tried to install the web But I ended up messing things up |
| 17:02 |
berick |
Freddy_Enrique: it will eventually be removed from EG |
| 17:02 |
berick |
the XUL client |
| 17:02 |
Freddy_Enrique |
better take my precautions |
| 17:03 |
Freddy_Enrique |
first of all, the web installation is not oficial yet, so there are many sections scattered |
| 17:03 |
Bmagic |
It's just a training issue more than anything |
| 17:03 |
Freddy_Enrique |
it starts from the opensrf instalation |
| 17:03 |
Freddy_Enrique |
but continues with the EG instalation |
| 17:04 |
Freddy_Enrique |
and then we have the.... Windows Hatch Installer??? |
| 17:05 |
Freddy_Enrique |
I was told that I can deal with it after I installed the opensrf |
| 17:05 |
|
mmorgan left #evergreen |
| 17:11 |
Freddy_Enrique |
jajajajaja.... csharp. I've just picture Budda |
| 17:11 |
Freddy_Enrique |
csharp++ |
| 18:48 |
|
jvwoolf joined #evergreen |
| 18:48 |
|
jvwoolf left #evergreen |
