| Time |
Nick |
Message |
| 01:00 |
|
JBoyer_ joined #evergreen |
| 01:01 |
|
jweston_ joined #evergreen |
| 01:07 |
|
berick_ joined #evergreen |
| 06:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 08:04 |
|
rjackson_isl_hom joined #evergreen |
| 08:11 |
|
Keith-isl joined #evergreen |
| 08:33 |
|
mmorgan joined #evergreen |
| 08:42 |
|
Keith_isl joined #evergreen |
| 08:52 |
|
awitter joined #evergreen |
| 08:54 |
|
Dyrcona joined #evergreen |
| 09:24 |
|
rfrasur joined #evergreen |
| 09:37 |
csharp_ |
so awitter and jlamos analyzed the packet capture and saw TCP ZeroWindow errors - we've upped net.core.rmem.max by 25% on one of our servers as an experiment and will run tshark again |
| 09:50 |
|
alynn26 joined #evergreen |
| 10:02 |
|
jvwoolf joined #evergreen |
| 10:18 |
Dyrcona |
Grr. Not sure if code is broken or the test environment is too slow. I can see in the logs that my search returned results, but the Angular catalog screen results section is "blank." |
| 10:18 |
Dyrcona |
And, now, a result is there. Just slow, I guess. |
| 10:42 |
Dyrcona |
I can't figure out why the Angular staff catalog patron search freezes for us as soon as it opens. |
| 10:43 |
Dyrcona |
It freezes the browser tab, even the JS console is non-responsive. |
| 10:43 |
Dyrcona |
Nothing in the Evergreen logs looks suspicious. |
| 10:43 |
Dyrcona |
It's consistent on a test vm and on our training server with Chrome Version 97.0.4692.99 (Official Build) (64-bit) |
| 10:45 |
Dyrcona |
There's a Chrome process running at 100.7% CPU. |
| 10:47 |
Dyrcona |
If I kill that Chrome process, I get the "Aw, snap" screen. |
| 11:13 |
|
mantis1 joined #evergreen |
| 11:13 |
alynn26 |
trying to get everything back up and running from replacing our Db server. Got most things up and running except for when we search for titles now results are displayed |
| 11:14 |
alynn26 |
We get the following in the Concole logs Can't use an undefined value as an ARRAY reference at /usr/local/share/perl/5.26.1/OpenILS/Application/Search/Biblio.pm line 3150. |
| 11:17 |
abowling |
Can someone remind me how to reset a password in the db on actor.usr. I *think* there's something special you have to do, but I have since forgotten. |
| 11:17 |
Dyrcona |
alynn26: I can't say what that line is doing because I don't know which release you're on, but those undefined reference errors are mostly just noise. |
| 11:25 |
berick_ |
abowling: see actor.change_password() in the DB |
| 11:26 |
abowling |
berick++ |
| 11:26 |
Dyrcona |
So, I've install a vanilla rel_3_7 connecting to a copy of our production database, and Angular patron search still freezes for me. |
| 11:29 |
alynn26 |
anyone know why unapi.biblio_record_entry_feed( queries would just hang. |
| 11:31 |
Dyrcona |
So, with a Firefox private window, I can't register a workstaion. I get Uncaught TypeError: UpUp is null |
| 11:32 |
Dyrcona |
alynn26: Missing index in the database causing a table scan? just guessing |
| 11:32 |
alynn26 |
how do we get that index back. We just moved the db to a new server |
| 11:34 |
Dyrcona |
I'm just guessing. You'll want to compare tables to the base schema and recreate any missing indexes. I really don't know since I have no visibility into your system. |
| 11:35 |
Dyrcona |
Well, same thing happens with Angular patron search in Firefox. |
| 11:35 |
Dyrcona |
Guess we're hosed/not upgrading. |
| 12:01 |
jeff |
Dyrcona: Firefox + Private Browsing currently disables local storage APIs that Evergreen (currently) depends on. It's not expected to work until/unless we remove that requirement (either by changing something on our end or upgrading UpUp, etc) |
| 12:02 |
jeff |
csharp_: I'm curious to know what the XMPP stream looks like for the timed out / disconnected session. Are there any Jabber errors? |
| 12:04 |
jeff |
(though if one end is setting its receive window to zero as you said, that might result in a timeout with no XMPP-level error stanzas sent, if the end that would generate an error is honoring the window size request) |
| 12:07 |
|
stephengwills joined #evergreen |
| 12:08 |
|
stephengwills left #evergreen |
| 12:20 |
csharp_ |
jeff: from jlamos: 194609187.91856273510.10.10.1010.10.10.10TCP65549[TCP ACKed unseen segment] 51656 → 5222 [ACK] Seq=6431988 Ack=10703840 Win=24567 Len=65483 TSval=3006493276 TSecr=3006493256 [TCP segment of a reassembled PDU] < about 10 of those |
| 12:21 |
jeff |
when you were running tshark, were you writing to a pcap/pcapng file, or were you just dumping to stdout? in other words, do you have the content of the packets or just the summarized display from tshark? |
| 12:22 |
jeff |
(in other other words, did you use the -w argument to tshark to capture to a file?) |
| 12:22 |
jeff |
(or use dumpcap?) |
| 12:23 |
|
jihpringle joined #evergreen |
| 12:26 |
|
Keith__isl joined #evergreen |
| 12:29 |
csharp_ |
we did capture to pcap files so we can see the error - I want to capture a few more and see if there are patterns about the call causing the trouble |
| 12:30 |
csharp_ |
s/error/packets/ |
| 12:31 |
csharp_ |
so far it looks like something that's pulling in detailed/fleshed org data - maybe around messages? |
| 12:32 |
csharp_ |
hmmmmmmm |
| 12:33 |
* berick |
makes popcorn |
| 13:07 |
csharp_ |
theory is that something in the new user message stuff is bringing in all the fleshed orgs - but I need to go talk to some food about this |
| 13:12 |
Dyrcona |
Seems like a decent hypothesis. |
| 13:13 |
|
awitter joined #evergreen |
| 13:14 |
Dyrcona |
As for my problem, it doesn't happen on a clean vm with concerto data, so I'm leaning toward an issue with our data or some crud hanging around in the test environment. I am told the problem happens on 3.5 with the experimental catalog enalbed. |
| 13:24 |
jeff |
csharp_: remind me -- is this 3.7 or 3.8? |
| 13:24 |
jeff |
(3.8 with messages, I think, but I'm being lazy and asking instead of verifying.) |
| 13:36 |
csharp_ |
jeff: 3.8.0 |
| 13:41 |
csharp_ |
we have an example of an open-ils.actor.patron.update call that includes all the fleshed OUs |
| 13:41 |
csharp_ |
it is recorded in activity.log, but never makes it to osrfsys.log |
| 13:42 |
csharp_ |
whether that is the issue itself or not - that can't be right :-/ |
| 13:53 |
|
jihpringle89 joined #evergreen |
| 13:54 |
Dyrcona |
Yeah, that can't be right. |
| 13:55 |
|
Bmagic joined #evergreen |
| 13:55 |
|
akilsdonk joined #evergreen |
| 14:20 |
csharp_ |
I can correlate the events with open-ils.actor.patron.update calls that appear in activity.log but not osrfsys |
| 14:21 |
csharp_ |
so something in the client is loading in all fleshed orgs and adding them all to the update call |
| 14:21 |
* csharp_ |
stares hard at Open-ILS/web/js/ui/default/staff/circ/services/circ.js |
| 14:24 |
jeff |
find the most recent call, look up the user being updated as well as the staff user/workstation associated with the auth token, and make a phone call to the staff/library in question to see if they remember what they were doing to that patron at that time? |
| 14:24 |
jeff |
(as well as chase logs / code, but sometimes a helpful pointer in the right direction from the end user can be a boost) |
| 14:28 |
csharp_ |
good idea |
| 14:37 |
jeff |
can you tell where in the update message the full org path is being included? |
| 14:41 |
jeff |
s/path/tree/ i guess |
| 14:44 |
csharp_ |
it takes so long to paste the message into something that I can prettify to find out that it's going to take a while :-/ |
| 14:48 |
csharp_ |
we're talking tens of thousands of lines |
| 14:49 |
csharp_ |
620416 characters |
| 14:58 |
berick |
egad |
| 15:00 |
jeff |
since open-ils.actor.patron.update is typically **PARAMS REDACTED**, are you getting the payload from your pcap, or are you not redacting that API call? |
| 15:00 |
csharp_ |
yes, that came from the pcap |
| 15:01 |
csharp_ |
(sorry jeff missed your DM) |
| 15:08 |
csharp_ |
redacted JSON up to the point where it starts the humongous org blob: https://pastebin.com/2B0VVJG1 |
| 15:09 |
jeff |
Okay, I can reproduce this on 3.8 by setting a patron note that's at the Everywhere depth, then editing the user. When I save the edits, the resulting API call has a full org tree. |
| 15:10 |
|
jvwoolf left #evergreen |
| 15:11 |
csharp_ |
jeff++ |
| 15:13 |
jeff |
it's the standing_penalties field on the au object that appears to be inappropriately fleshed |
| 15:13 |
jeff |
(excessively fleshed?) |
| 15:13 |
csharp_ |
ewww |
| 15:13 |
* Dyrcona |
wonders how that is possible. |
| 15:14 |
jeff |
the notes field contains the note I added without a full org tree. |
| 15:16 |
jeff |
on the au object, standing_penalties links to ausp and notes links to aum |
| 15:17 |
jeff |
both standing_penalties and notes are in the defaultFleshFields array in the egUser service. |
| 15:17 |
jeff |
defined in Open-ILS/web/js/ui/default/staff/services/user.js |
| 15:21 |
jeff |
on the aum object, org_unit and aum_sending_lib don't get fleshed |
| 15:22 |
jeff |
but on the standing_penalties object the org_unit field is fleshed, and if it's au 1, you get the whole tree. |
| 15:23 |
csharp_ |
jeff: I will buy you a six pack of the beverage of your choice at the next in-person EG Conf :-) |
| 15:23 |
jeff |
(this is just looking at the resulting open-ils.actor.patron.update call) |
| 15:23 |
jeff |
on a 3.8 training system here running concerto |
| 15:25 |
jeff |
more telling might be that OpenILS::Application::Actor::flesh_user has a $fields array that includes standing_penalties but not notes. |
| 15:25 |
jeff |
as well as user_retrieve_fleshed_by_id in that same package |
| 15:27 |
jeff |
new_flesh_user in that same package does some fleshing on standing_penalties that it does not do on notes. |
| 15:29 |
Dyrcona |
This is the part that puzzles me: "if it's au 1, you get the whole tree." |
| 15:29 |
jeffdavis |
csharp_: do a lot of your users have consortium-wide patron notes? |
| 15:30 |
Dyrcona |
Hmm... Is flesh depth 2 and is it also fleshing children? |
| 15:31 |
Dyrcona |
That might explain it..... |
| 15:45 |
jeff |
open-ils.actor.user.fleshed.retrieve issued by the browser doesn't seem to be returning an overly-fleshed object. something else must be fleshing it. looking. |
| 16:03 |
csharp_ |
jeffdavis: oh yes - most are expected to be consortium-level |
| 16:03 |
mmorgan |
csharp_: Do you have action triggers that add patron messages? |
| 16:03 |
csharp_ |
yes |
| 16:05 |
mmorgan |
A lot of them? We have only one or two triggers that add a patron message. |
| 16:26 |
Dyrcona |
Well, my freezing of Angular patron search has to be related to our database. It only happens when Evergreen is talking to a copy of our production data. It's fine on concerto. I built a totally fresh vm to rule out code or config cruft. |
| 16:39 |
jeff |
localFlesh! |
| 16:40 |
jeff |
in web/js/ui/default/staff/services/patron_search.js patronSvc gets a function localFlesh defined, which among other things fleshes org_unit on any standing penalties. |
| 16:42 |
jeff |
been there forever-ish, only a problem now-ish. |
| 16:42 |
Dyrcona |
:) |
| 16:46 |
|
BAMkubasa joined #evergreen |
| 16:46 |
BAMkubasa |
Do the things in eg/staff/circ/holds/shelf live in a specific database table or are they derived from data in other tables? |
| 16:49 |
* jeff |
tests a fix |
| 16:50 |
Dyrcona |
BAMkubasa: I think that data is derived from action.hold_request where the shelf_time is not null and the shelf_lib is equal to the current working location. |
| 16:52 |
BAMkubasa |
awesome |
| 16:52 |
BAMkubasa |
thanks. I just came across shelf_lib and was starting to poke around and see what it contained |
| 17:39 |
|
mmorgan left #evergreen |
| 17:42 |
|
alynn26_ joined #evergreen |
| 18:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:01 |
jeff |
csharp_: one option to fix -- worked in quick testing here: https://git.evergreen-ils.org/?p=working/Evergreen.git;a=commitdiff;h=1d6f2b25a7436060b20326e2b41b3b8ef4c306b0 |
| 18:01 |
jeff |
also, lp 1959461 |
| 18:01 |
pinesol |
Launchpad bug 1959461 in Evergreen "Fleshing org unit on standing penalties should omit org unit children" [Undecided,New] https://launchpad.net/bugs/1959461 - Assigned to Jeff Godin (jgodin) |
| 18:02 |
jeff |
we might not even need to send standing_penalties on the user object when saving, but we probably shouldn't flesh the children either. |
| 18:03 |
jeff |
(and we should more gracefully handle the large object being present, lest we have DoS issues) |
| 18:03 |
jeff |
good luck, and I'm curious to hear if this helps! |
| 18:19 |
|
alynn26_ joined #evergreen |
| 18:23 |
|
alynn26_ joined #evergreen |
| 23:48 |
|
alynn26 joined #evergreen |
