| Time |
Nick |
Message |
| 00:08 |
|
sandbergja joined #evergreen |
| 07:02 |
|
agoben joined #evergreen |
| 07:08 |
|
rjackson_isl joined #evergreen |
| 08:01 |
|
Dyrcona joined #evergreen |
| 08:05 |
|
bos20k joined #evergreen |
| 08:06 |
|
rfrasur joined #evergreen |
| 08:47 |
Dyrcona |
Um... I don't think the following is supposed to happen: |
| 08:50 |
Dyrcona |
Using Chromium 77 with a normal tab, I get a blank window when I go to Local Administration with a recent checkout of master. If I reload, I get "Welcome to Webby." The link to go to the splash page seems to do nothing. I tried emptying the cache and hard reload. |
| 08:51 |
Dyrcona |
Using Firefox 69.0.3 and a Private Window to login to the same evergreen instance, I get prompted to register the workstation again when I go to Local Administration. After registering the workstation, clicking Use Now, and logging in again, it seems to work. |
| 08:56 |
Dyrcona |
That's on a test system set up at CW MARS last week with Pg 10 for the database, Ubuntu 16.04 for the brick vms, and haproxy in front of the bricks. |
| 08:57 |
Dyrcona |
I have some VMs that I built on my laptop Sunday that I can also test with, and I will do so in a bit. |
| 09:00 |
Dyrcona |
Trying again with Chromium 77, an Incognito Window, and the default admin, rather than a Global Administrator that I added, seems to just work. |
| 09:00 |
|
yboston joined #evergreen |
| 09:01 |
Dyrcona |
Yeah, I only get the blank screen with my added global administrator account.... |
| 09:02 |
Dyrcona |
Oh, nifty! main.de76656881bc89a98b3d.js:1 ERROR Error: Uncaught (in promise): User does not have staff permissions |
| 09:02 |
Dyrcona |
The Global Administrator profile doesn't give you that? |
| 09:03 |
Dyrcona |
Guess it's a Concerto data bug? |
| 09:05 |
Dyrcona |
Well, maybe not. Global Administrator has perm -1 at depth 0, i.e. "Everything at Everywhere." It also has 1 other permission added, so this looks like a case of a perm check not looking for perm -1. |
| 09:08 |
Dyrcona |
Looks like the bug may be here: open-ils.actor.user.has_work_perm_at.batch |
| 09:19 |
csharp |
are you behind nginx? could it be the IP address forwarding issue we've seen before? |
| 09:26 |
Dyrcona |
csharp: No, it's haproxy on the load balancer, and I have the IP forward headers configured properly. It's only failing with a user who's not a super user and has the Global Administrator profile. I now suspect it's a problem with the depth parameter on the permission look up request, but I'm still looking. |
| 09:28 |
csharp |
gotcha |
| 09:31 |
miker |
Dyrcona: .has_[work_]perm_at is the bane of my existence... |
| 09:32 |
agoben |
We've got a library that's having trouble with offline mode not caching the workstation info. If they log in and *then* go offline, it works fine, but if they haven't logged in on a given day, it's wiped out. Any suggestions? |
| 09:32 |
miker |
we should add a short circuit test for -1 in those |
| 09:34 |
agoben |
Note: the workstation registration isn't also disappearing, just the offline info. |
| 09:36 |
Dyrcona |
miker: It checks for -1 in joins, but it doesn't short circuit the way that super user does. |
| 09:37 |
Dyrcona |
agoben: Do they do something locally to clear the cache? Windows policies, maybe? |
| 09:39 |
Dyrcona |
miker: I'm a bit stumped as to why this isn't work. Looks like it should, but I'll have to find that place where the permissions check is made. |
| 09:40 |
agoben |
Dyrcona: Claiming no. And I would expect a cache-clear to take the whole registration and not just the offline info. |
| 09:41 |
JBoyer |
Dyrcona, does your GA user have a working location set? I wouldn't expect that to matter with a -1 perm, but (not having looked at the queries involved) if there's a query that expects *something* to be returned for working location and there's nothing, that might be an issue. |
| 09:42 |
Dyrcona |
JBoyer: No, but I'll add one and see what happens. |
| 09:42 |
csharp |
agoben: cache clear doesn't affect workstation, just cookies - possibly a misunderstanding between cache/cookies on staff's part? |
| 09:44 |
agoben |
Ok, so offline isn't part of site data? |
| 09:44 |
|
remingtron joined #evergreen |
| 09:45 |
csharp |
offline is stored in IndexedDB (which is visible if you browse storage in the dev tools) |
| 09:45 |
agoben |
This has coincided with an update from their ISP, so that *could* be part of it, but no one else on that ISP (and there are a lot) is reporting the same issue. |
| 09:46 |
csharp |
any caching proxies in the mix? (just throwing things out there) |
| 09:47 |
agoben |
I'll have to ask about their proxy setups |
| 09:47 |
|
Christineb joined #evergreen |
| 09:47 |
agoben |
Thanks for the help! |
| 09:48 |
Dyrcona |
JBoyer++ That fixed it! |
| 09:50 |
miker |
Dyrcona: ah, nm, I was thinking of the angular js wrapper |
| 09:50 |
miker |
JBoyer++ |
| 09:50 |
Dyrcona |
I always forget about setting working locations. I almost never do this sort of thing. |
| 09:51 |
JBoyer |
That's so easy to miss probably because you can't set it when registering the account. One common thing I've seen is running a cron job that just assigns your home lib as a working location if your profile is a child of Staff and none are set. |
| 09:51 |
JBoyer |
There's probably a better way to deal with those. |
| 09:52 |
Dyrcona |
Well, it was quick enough to insert for the two accounts that I made. |
| 09:55 |
|
mdriscoll joined #evergreen |
| 10:04 |
Dyrcona |
We're also using Pg 10 on this test cluster, and so far, no problems. |
| 10:09 |
|
sandbergja joined #evergreen |
| 10:13 |
|
cmalm joined #evergreen |
| 10:58 |
|
sandbergja joined #evergreen |
| 11:03 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 12:01 |
|
jihpringle joined #evergreen |
| 12:08 |
|
sandbergja joined #evergreen |
| 12:11 |
|
aabbee joined #evergreen |
| 12:29 |
|
yboston joined #evergreen |
| 12:49 |
|
khuckins joined #evergreen |
| 13:16 |
|
jwoodard joined #evergreen |
| 13:22 |
|
jwoodard left #evergreen |
| 13:24 |
|
jwoodard joined #evergreen |
| 13:32 |
jeffdavis |
Here's a strange bug. On 3.3, after adding a title to your basket, a subsequent search is giving us an internal server error. Removing the cartcache cookie makes the error go away. We're only seeing this when load-balancing between multiple servers. |
| 13:33 |
jeffdavis |
Call to [open-ils.storage.biblio.multiclass.staged.search_fts.atomic] failed for session [...], thread trace [1]: Can't call method "opac_visible" on an undefined value at /usr/local/share/perl/5.22.1/OpenILS/Application/Storage/Driver/Pg/QueryParser.pm line 1177. |
| 13:35 |
jeffdavis |
hm, wonder if it is a locale thing... |
| 13:49 |
jeffdavis |
looks like default_locale in opensrf.xml was set to en-CA in our multi-server environment but en-US in single-server test environments; changing to en-US seems to be avoiding the issue so far; never had a problem with en-CA there until now though |
| 14:57 |
|
nfBurton joined #evergreen |
| 15:07 |
|
yboston joined #evergreen |
| 15:31 |
jeffdavis |
Definitely seeing much higher open-ils.actor drone counts on 3.3. I increased max_children from 50 to 75 but we're still hitting that new limit (currently hovering just below it on 2 out of 3 production servers). |
| 15:34 |
Dyrcona |
jeffdavis: It's the web staff client. |
| 15:36 |
Dyrcona |
jeffdavis: Ours is 60 per drone, so 120 per brick. |
| 15:37 |
dbs |
jeffdavis / Dyrcona: do you know how many simultaneous users you have, to help figure out a rough drone-per-user rule of thumb? |
| 15:38 |
Dyrcona |
dbs: Nope. I've not bothered trying to figure it out. |
| 15:40 |
jeffdavis |
Hmm, we could try counting authtokens in memcached I guess? |
| 15:41 |
csharp |
jeffdavis: which release did you upgrade from? |
| 15:41 |
jeffdavis |
I wonder if we can reduce the number of open-ils.actor calls though, a la bug 1745499 |
| 15:41 |
pinesol |
Launchpad bug 1745499 in Evergreen 3.0 "Web client: Barcode file upload inflates pcrud drone count" [Medium,Fix released] https://launchpad.net/bugs/1745499 |
| 15:41 |
jeffdavis |
csharp: 3.1.7 -> 3.3.4 |
| 15:41 |
csharp |
yeah, I suspect lots of service calls could be batched somehow |
| 15:42 |
jeffdavis |
we just went live this morning (well, over the weekend but this is most libraries' first day using it), so haven't had a chance to track down specific causes yet |
| 15:42 |
csharp |
jeffdavis: I'm pretty sure we saw significant increases going from 3.0 -> 3.2 and made similar adjustments |
| 15:42 |
csharp |
jeffdavis: congratulations! |
| 15:43 |
jeffdavis |
Thanks! It's early yet but it's been quite smooth so far. |
| 15:43 |
csharp |
yeah - our headaches were mostly invisible to end users - stuff like high drone counts/system load |
| 15:45 |
Dyrcona |
We still run out of some drones. I need to look into that again and up a few counts. |
| 15:47 |
Dyrcona |
Like csharp said, we really noticed it going from 3.0 to 3.2 and staff using the web staff client more. |
| 16:00 |
|
jihpringle joined #evergreen |
| 16:00 |
|
khuckins joined #evergreen |
| 16:17 |
miker |
re actor being more heavily used in web client, it's all because each "page" is stateless. discarding cstore calls, open-ils.actor.settings.retrieve.atomic is consistently the #2 most called method (~6% of all non-cstore calls), behind open-ils.auth.session.retrieve at a full 50%+ of all calls. for those still using xul, it can't even be found in the top 20 (less than 1% of calls) |
| 16:18 |
miker |
in the xul client, you may recall, there was a login-time dance where we fetched, well, TONS of stuff. user and library settings were two sets of data cached for the whole xul session. not so in webclient, we get them on demand. |
| 16:27 |
jeffdavis |
We did see an increase in actor calls when we moved to web client-only with 3.1, but this upgrade to 3.3 is another big step up in actor usage. |
| 16:29 |
jeffdavis |
I wonder if there is a difference between AngJS and Ang too. |
| 16:31 |
jeffdavis |
Should we consider caching settings? |
| 16:31 |
|
jwoodard joined #evergreen |
| 16:39 |
miker |
jeffdavis: we did, at the session level (effectively) in xul, so ... sure? |
| 16:40 |
|
jihpringle joined #evergreen |
| 16:41 |
miker |
of course, one of the remaining xul "bugs" is "make the login process faster" ... only way to do that would be to disable some of those caching calls ;) (can't just delay them in the background, because they'll be needed for the first thing you want to do, very likely) |
| 16:44 |
miker |
one thing we could do is make heavier use of memcache. but that means figuring out proper cache segmentation, and cache invalidation logic, which is ... tricky. or, "push all individual settings into k/v pairs in memcache, and teach $something_in_the_middle_of_everything_maybe_a_trigger to invalidate on value change for a key" |
| 16:51 |
miker |
we did offer something like that, for a time, for org tree and perm lookups. there are vestiges of that in our pgmemcache-*.sql files. csharp may recall those -- I wrote that for PINES many thousands of years ago. I am not personally excited about reviving that :) |
| 17:11 |
* jeffdavis |
contemplates adding a "cacheable" column on config.org_unit_setting_type, then generating a JSON blob for cacheable setting values with autogen |
| 17:19 |
|
jihpringle joined #evergreen |
| 18:08 |
|
sandbergja_ joined #evergreen |
| 19:53 |
|
b_bonner left #evergreen |
| 20:13 |
|
JBoyer_ joined #evergreen |
| 23:03 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 23:52 |
|
sandbergja joined #evergreen |
