| Time |
Nick |
Message |
| 01:50 |
|
khuckins_ joined #evergreen |
| 02:40 |
|
khuckins joined #evergreen |
| 06:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 06:18 |
|
khuckins joined #evergreen |
| 06:27 |
|
khuckins joined #evergreen |
| 07:17 |
|
Dyrcona joined #evergreen |
| 07:31 |
|
rjackson_isl_hom joined #evergreen |
| 08:04 |
|
collum joined #evergreen |
| 08:12 |
|
Dyrcona joined #evergreen |
| 08:25 |
|
mantis1 joined #evergreen |
| 08:26 |
|
rfrasur joined #evergreen |
| 08:39 |
|
mmorgan joined #evergreen |
| 09:01 |
|
jvwoolf joined #evergreen |
| 09:18 |
Dyrcona |
Grabbing 1248 |
| 09:30 |
pinesol |
[evergreen|Jason Boyer] LP1703658: Convert GIST Indexes to GIN - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=378588d> |
| 09:30 |
pinesol |
[evergreen|Jason Stephenson] Lp 1703658: Repair DB Upgrade - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=23287f0> |
| 09:30 |
pinesol |
[evergreen|Jason Stephenson] Lp 1703658: Stamp DB Upgrade - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=7c4a18e> |
| 09:46 |
Bmagic |
Is there a bug that I can't find in LP about large invoices not loading in the UI completely? |
| 09:48 |
bshum |
It sounds familiar to me, like a result of that chunking change for stanza size for ejabberd |
| 09:49 |
bshum |
Like forever ago, we left the stanzas bigger in ejabberd config to deal with those while they were sorted |
| 09:49 |
bshum |
But maybe it didn't get logged |
| 09:49 |
Dyrcona |
action trigger still has issues with stanza size while we're mentioning it. |
| 09:49 |
bshum |
https://bugs.launchpad.net/opensrf/+bug/1725317 |
| 09:50 |
pinesol |
Launchpad bug 1725317 in OpenSRF ""XML stanza is too big" still possible with chunking and bundling" [Undecided,New] - Assigned to Galen Charlton (gmc) |
| 09:50 |
bshum |
In comment 3, dbwells confirms that saving a large invoice broke |
| 09:50 |
Dyrcona |
bshum++ I was just about to search for that one. |
| 09:50 |
bshum |
For same reasons |
| 09:50 |
Dyrcona |
Bmagic: How big are these invoices, i.e. # of line items? |
| 09:51 |
Bmagic |
60 or 70 lines |
| 09:51 |
Dyrcona |
Oh, and I guess that includes a chunk of MARC for each one, so if the records are detailed..... |
| 09:51 |
bshum |
Yep |
| 09:52 |
Dyrcona |
Assuming 15K per entry, you're looking about 1MB to retrieve all that in one go. |
| 09:53 |
Bmagic |
I solved the stanza size issue with a config tweak to ejabberd. But this other thing about incompletely loading the UI is a different thing. An issue I've seen for many months. Maybe years. But shrugged it off because the interface is still dojo |
| 09:53 |
Dyrcona |
Bmagic: I'd check for XML stanza is too big in the logs, just the same. |
| 09:54 |
Bmagic |
alright |
| 09:54 |
Bmagic |
trouble is, it works for me |
| 09:54 |
Dyrcona |
Just to rule it out. I set max_stanza_size to 10MB on our utility server that handles notices for accounts with more than 100 items checked. |
| 09:56 |
Dyrcona |
You probably have a) a faster connection to server, b) a better computer with faster CPU and more RAM, c) a better supported architecture for the browser with better JIT, d) the advantage of trying it after it failed for someone else and so getting the data from RAM on the db serve, or some combination of the 3. |
| 09:56 |
Dyrcona |
Or it could be something else entirely. |
| 09:57 |
Bmagic |
right, those are some of the similiar things I was thinking |
| 09:57 |
Dyrcona |
Web "development" is "fun." :) |
| 09:57 |
Bmagic |
it's always on larger invoices. Which is a big clue |
| 09:57 |
Dyrcona |
Can you get JavaScript console logs when it fails? |
| 09:57 |
Bmagic |
I was thinking of making a bug report - but since it's dojo.... |
| 09:59 |
Dyrcona |
I think a bug would be worth it if you can add console logs. It could help with the push to eliminate Dojo for acquisitions. |
| 10:00 |
Bmagic |
The user did submit the console log, but there isn't anything wrong that I can see |
| 10:01 |
Bmagic |
Comparing it to my log when I load the page, it has the same number of lines: 76 |
| 10:01 |
Dyrcona |
Is the Angular implementation of acquisitions released, yet? |
| 10:01 |
Bmagic |
I think the log is suspect |
| 10:01 |
Dyrcona |
IDK. I'd have to look at it. |
| 10:04 |
Bmagic |
sure |
| 10:06 |
pastebot |
"Bmagic" at 168.25.130.30 pasted "invoice dojo UI load" (335 lines) at http://paste.evergreen-ils.org/10117 |
| 10:07 |
Dyrcona |
The garbage in teh view.js lines may be significant. Do you see those in your console log? |
| 10:10 |
Bmagic |
I've got another log output to share |
| 10:10 |
Bmagic |
one that has actual errors |
| 10:11 |
pastebot |
"Bmagic" at 168.25.130.30 pasted "invoice dojo UI load2" (1708 lines) at http://paste.evergreen-ils.org/10118 |
| 10:12 |
Dyrcona |
Well, that one has error getting dojo.js, but I think those are "normal" unless you use the custom dojo.tgz from the downloads page/created by make_release. |
| 10:12 |
Bmagic |
Dyrcona: oh, that's a paste/character conversion by the pastebin -> html |
| 10:13 |
Bmagic |
RE garbage ^^ |
| 10:14 |
Dyrcona |
Ok. I gathered that after looking at your second paste. :) |
| 10:14 |
Dyrcona |
Unknown server error isn't very useful. Did you say whether or not they've tried clearing the cache? |
| 10:16 |
Bmagic |
Yeah, I made sure they did a hard refresh and cleared cache and whatnot |
| 10:16 |
Bmagic |
VM4275:174 Uncaught Error: Method error: undefined : An unknown server error occurred - line 174 is this line "addInvoiceEntry(entry);" |
| 10:16 |
Bmagic |
I guess I'll make the bug and go from there |
| 10:17 |
Dyrcona |
Yeah. I don't think I can be much more help right now. |
| 10:25 |
Bmagic |
Drycona: ty for your time |
| 10:25 |
Dyrcona |
yw |
| 10:25 |
Bmagic |
Dyrcona++ |
| 10:25 |
Bmagic |
bug 1917482 |
| 10:25 |
pinesol |
Launchpad bug 1917482 in Evergreen "Invoice UI fails to load completely" [Undecided,New] https://launchpad.net/bugs/1917482 |
| 10:38 |
|
Stompro joined #evergreen |
| 11:00 |
|
stephengwills joined #evergreen |
| 11:09 |
|
alynn26 joined #evergreen |
| 11:14 |
csharp |
Bmagic: see https://bugs.launchpad.net/opensrf/+bug/1912834/comments/5 for what sounds like a similar issue for us after we applied the fix in https://bugs.launchpad.net/opensrf/+bug/1912834/comments/4 (from the last comment on that bug, sounds like you applied it at the end of Jan) |
| 11:14 |
pinesol |
Launchpad bug 1912834 in OpenSRF "Browser client should limit the number of parallel requests" [High,New] |
| 11:15 |
csharp |
as the old saying goes, "what is good for the Angular is bad for the Dojo" |
| 11:16 |
Bmagic |
csharp: that bug is no longer in production |
| 11:16 |
csharp |
ok good - just ruling it out |
| 11:16 |
Bmagic |
that broke tons of acq stuff |
| 11:16 |
csharp |
dojo-- |
| 11:16 |
csharp |
@insult dojo |
| 11:16 |
pinesol |
dojo: You are nothing but a villainous gob of infected squirrel. |
| 11:18 |
Bmagic |
csharp: the solution for us turned out to be nginx limiting. The production system was getting overwhelmed by 3rd parties hitting unapi URL. I supplied an example config in a branch: https://bugs.launchpad.net/opensrf/+bug/1913617 |
| 11:18 |
pinesol |
Launchpad bug 1913617 in OpenSRF "NGINX could use a DOS mitigation example" [Undecided,New] |
| 11:19 |
|
malexander joined #evergreen |
| 11:20 |
berick |
Bmagic: are you saying the unapi/nginx limiting *also* resolved any remaining drone exhaustion (unrelated to unapi) ? |
| 11:20 |
Bmagic |
it resolved the problem, yes. Though I realize that the unapi service is not heavily used by Evergreen |
| 11:21 |
berick |
Bmagic: but did it resolve multiple problems? :) |
| 11:21 |
berick |
there's the unapi thing and then there's drone exhaustion from too many parallel requests |
| 11:21 |
berick |
now that I think about it.. it can't solve the drone issue |
| 11:22 |
Bmagic |
It's hard to say that csharp and I were having the exact same issue. It was just timely that we were having similar issues at the same time. I was thinking that the bricks could* have been overwhelmed for the same reasons. I was willing to try anything* |
| 11:22 |
berick |
since those are Websockets requests, I suspect they are not impacted by nginx limiting, but I could be wrong |
| 11:22 |
berick |
Bmagic: gotcha, ok, so those are really 2 different issues. just clarifying in case we had an option that we didn't realize |
| 11:23 |
Bmagic |
berick: nginx has a special block to handle the websockets requests (and I didn't impose a limit in that block) - so no, nginx is not limiting the websockets requests |
| 11:23 |
berick |
ah, ok, thanks for clarifying |
| 11:23 |
Bmagic |
see branch above |
| 11:23 |
* berick |
nods |
| 11:24 |
Bmagic |
though, with that nginx syntax - it could be imposed on any* URL. Which is nice to have in our tool chest |
| 11:26 |
berick |
yeah, was thinking it may be worth exploring as a solution to drone exhaustion issues. |
| 11:29 |
Bmagic |
it worked wonders, so I think it's worth checking into for other things. NGINX can apply leaky bucket plus delayed combined with allowances for bursting. It's all spelled out here: https://www.nginx.com/blog/rate-limiting-nginx/ |
| 11:29 |
berick |
Bmagic++ |
| 11:32 |
Bmagic |
the first day after updating that config, it mitigated over 90k requests. And Evergreen was unaffected |
| 11:43 |
Dyrcona |
Node.js gives me the heebie-jeebies. |
| 11:43 |
|
khuckins joined #evergreen |
| 11:44 |
Bmagic |
Dyrcona: I think Node.js takes a backseat to this: https://images.app.goo.gl/JzfX6pjf5yDhNv9R9 |
| 11:45 |
Bmagic |
the classic 1980's glow worm |
| 11:51 |
Dyrcona |
While doing npm update: npm WARN bootstrap 4.3.1 requires a peer of popper.js@^1.14.7 but none is installed. You must install peer dependencies yourself. |
| 11:51 |
Dyrcona |
I suppose npm update is a bad idea considering how we pin packages? |
| 12:02 |
berick |
Dyrcona: well, we're not exactly curating the packages, but they are consistent across a swath of time because of the lockfile. |
| 12:02 |
|
jihpringle joined #evergreen |
| 12:02 |
berick |
there's nothing wrong with updating, but you may be moving into new territory |
| 12:02 |
berick |
but you may also be applying useful patches, so... |
| 12:15 |
Bmagic |
here is an interesting one: we had a patron notice that the due date for one of their items shows a different day depending on whether they are looking at their checkouts on their phone or the computer. Both using chrome. It's a day off. I immediately suspected a timezone issue where the computer is correct and the phone is wrong. |
| 12:16 |
Bmagic |
but, the due date on the phone displayed a day before what the computer browser said. And the due time is 11:59:59 on day 9. The phone shows day 8. If it were a timezone issue, I would suspect that it would have rolled over to the next day, not the day before. I've confirmed that the DB says 11:59:59 on day 9 |
| 12:18 |
Bmagic |
I guess the timezone on the phone could have been set to somewhere on the opposite side of Earth, where it's 24 hours off? Seems unlikely. |
| 12:18 |
Bmagic |
we're letting that one go, but I thought it was a "fun" one to talk about here |
| 12:36 |
Dyrcona |
Time is hard. |
| 12:43 |
Dyrcona |
When I add dates to milestones on Launchpad, I sometimes have to change the day because they can be off by a day. |
| 12:46 |
* Dyrcona |
wonders how difficult it would be for "Use Now" on workstation registration to just work without having to log in again. |
| 12:50 |
Dyrcona |
So, I just tried using MARC batch import with master, and I get upload progress 100%, but enqueue progress is 0%. There's also nothing in the /tmp directory AFAICT. |
| 12:51 |
jeff |
have to teach open-ils.auth how to assign a workstation to a workstationless login and change type. Might also violate some other assumptions. |
| 12:51 |
jeff |
Dyrcona: is your web server running with private temp, and you're using /tmp as the queue location? You'll need to either change to a different dir or stop using private tmp for the apache service. |
| 12:52 |
jeff |
(many public services like apache are launched by systemd with private /tmp by default on a lot of systems now) |
| 12:52 |
Dyrcona |
jeff: RE workation: Yeah. I might look into that. RE /tmp. I"m looking. It made a queue. |
| 12:52 |
jeff |
it's been that way for a while on Debian. I don't remember if it recently changed for Ubuntu. |
| 12:53 |
Dyrcona |
Queue is empty. I'm using default configurations with Concerto, so I probably need to change the Apache or opensrf.xml. |
| 12:53 |
Dyrcona |
jeff++ |
| 12:59 |
Dyrcona |
Bingo! I switched from /tmp to /openils/var/tmp in opensrf.xml. I had to mkdir /openils/var/tmp, but that fixed it. |
| 12:59 |
Dyrcona |
Apache can't see /tmp, apparently, but I didn't look too hard at the configuration. |
| 13:00 |
Dyrcona |
Well, no, it can see /tmp..... |
| 13:03 |
jeff |
apache's /tmp is not your /tmp :-) |
| 13:04 |
jeffdavis |
I believe private /tmp is a change between Ubuntu 16.04 and 18.04 fwiw |
| 13:07 |
Dyrcona |
Well, I hadn't noticed because I don't use it much on development, and we've been using /openils/var/tmp mounted via NFS in production for years. |
| 13:08 |
Dyrcona |
So, I'm actually testing a security bug and trying to import a MARC record to trigger it, but the record won't import. |
| 13:08 |
Dyrcona |
When I select it and hit Import Selected Records, the screen goes back to the main Import view, and the record does NOT end up in biblio.record_entry. |
| 13:09 |
Dyrcona |
FWIW, I have no idea what I'm doing in the staff client, particularly the Angular interface. |
| 13:28 |
Dyrcona |
So, maybe Vandelay is broken in master on Ubuntu 20.04? |
| 13:40 |
jeff |
can you manually create a MARC record from a template? can you edit an existing bib record? database or opensrf logs (or browser console) might have more clues for you. |
| 13:40 |
Dyrcona |
I can manually edit bib records. |
| 13:42 |
Dyrcona |
It seem to pick on the flat text editor more than I'd like. If used the flat text editor, switched to enriched editor, then create a new record it seems to go to the flat text editor. I don't recall making flat text editor a default. |
| 13:44 |
Dyrcona |
Creating a new record works. |
| 14:02 |
Dyrcona |
jefdavis++ |
| 14:02 |
Dyrcona |
So, it also appears that my Vandelay record was eventually imported. |
| 14:05 |
Dyrcona |
Oh, no. I realize now, based on the time stamp, that I made that record in the flat text editor via copy and paste..... |
| 14:10 |
Dyrcona |
Vandelay is acting the same on a VM running Ubuntu 18.04, so it's probably me not knowing what I'm doing. |
| 14:40 |
Dyrcona |
jeffdavis++ JBoyer++ I'm going to add my signoff to the security branch. |
| 14:41 |
Dyrcona |
Also, I was doing Vandelay wrong. I got it working. |
| 14:41 |
jeffdavis |
Yay! |
| 14:41 |
JBoyer |
Dyrcona++ |
| 14:41 |
JBoyer |
jeffdavis++ |
| 14:44 |
|
terranm joined #evergreen |
| 14:49 |
Dyrcona |
Also, jamesrf++ |
| 15:03 |
|
jvwoolf joined #evergreen |
| 15:07 |
Dyrcona |
I wonder if XUL really matters for bug 1076582? |
| 15:07 |
pinesol |
Launchpad bug 1076582 in Evergreen "Remove or document references to openils_dojo.js" [Low,Confirmed] https://launchpad.net/bugs/1076582 - Assigned to Jason Stephenson (jstephenson) |
| 15:08 |
Dyrcona |
After applying the branch, I found a couple of active references to openils_dojo.js in the XUL code, and I wonder if building a XUL client and testing it is worth the trouble at this point? |
| 15:13 |
|
sandbergja_ joined #evergreen |
| 15:20 |
|
jihpringle joined #evergreen |
| 15:24 |
JBoyer |
If there's not a reference under Open-ILS/xul/server I don't think it matters whether the client works or not. (I was thinking there *may* be something still using parts of server/* ?) |
| 15:25 |
JBoyer |
Since it can't use any TLS without manual intervention and lacks so much functionality I think it should be removed from 3.7 even if we're not quite ready to rm -rf Open-ILS/xul yet. |
| 15:26 |
|
sandbergja_ joined #evergreen |
| 15:27 |
Dyrcona |
JBoyer: I references to XULG in acquisitions dojo code. |
| 15:28 |
JBoyer |
Ah, that's about where I'd expect it to still be. |
| 15:31 |
Dyrcona |
I am tempted to build openils_dojo.js to see if it really makes much difference these days. |
| 15:34 |
jeff |
I tried to build it a year or two ago. It was not a simple or pleasant task. |
| 15:37 |
jeff |
Without digging for notes, I don't remember if I ended up with something that worked or not. I think it took some effort just to find a version of the tools that was just the right mix of old enough / new enough. |
| 15:37 |
Dyrcona |
Well. I'll have a look. |
| 15:38 |
jeff |
Good luck! |
| 15:38 |
Dyrcona |
Not that I'm going to try building it, but I want to see how hard it is to find the Dojo 1.3.3. src. |
| 15:40 |
Dyrcona |
It's not that hard to find. I'll give the build a whirl just to see what's involved. |
| 15:40 |
Dyrcona |
:) |
| 15:41 |
JBoyer |
Isn't there some weird java-based tool involved? That might be the real tricky bit. |
| 15:41 |
JBoyer |
That said, if Dyrcona does get it built we could just say it's never getting updated again and check it in, heh. |
| 15:42 |
Dyrcona |
Yeah, it requires Rhino, which is gone for the most part. (I used to play with Rhino back in the day.) |
| 15:43 |
Dyrcona |
OK. I take it back. Rhino isn't gone, but modern Java comes with a "subsitute" that's almost as good. ;) |
| 15:49 |
Dyrcona |
If that works, it was pretty easy. ;) |
| 15:49 |
Dyrcona |
IF that that works that is.... (Big IF) :) |
| 15:53 |
Dyrcona |
Seems a bit late to be messing with this. |
| 15:57 |
Dyrcona |
It looks like most of the places where openils_dojo.js is still used are obsoleted by newer interfaces in 3.6. |
| 15:59 |
|
mantis1 left #evergreen |
| 16:01 |
Dyrcona |
Well. It seems to have undone the security patch that I just installed, but I think that's because I switched branches when building the custom dojo. |
| 16:03 |
Dyrcona |
I seem to be getting a lot of uncaught exceptions, too. |
| 16:19 |
Dyrcona |
I should probably wipe /openils and install fresh. |
| 16:27 |
csharp |
so we're running the upgrade script for bug 1893997 against a PINES test server and lordy that's going to take a long time to run |
| 16:27 |
pinesol |
Launchpad bug 1893997 in Evergreen "Wishlist: Did you mean? Single word, single class substitutions" [Wishlist,New] https://launchpad.net/bugs/1893997 |
| 16:27 |
csharp |
followed by a reingest |
| 16:28 |
csharp |
gonna research whether the changes can be applied outside of a downtime window |
| 16:28 |
csharp |
unless miker just knows offhand |
| 16:29 |
Dyrcona |
Well, you can run the reingest while people are using the system. |
| 16:29 |
csharp |
yeah, I just didn't want any changed DB-side functions to lead to unexpected results before the did you mean stuff was populated |
| 16:29 |
csharp |
which is the part that's going to take forever |
| 16:30 |
csharp |
the generated SQL script is 52M individual select calls to a function that populates the tables |
| 16:32 |
Dyrcona |
Well, 52M selects is not necessarily slow. |
| 16:32 |
Dyrcona |
It depends a lot on what they do. |
| 16:33 |
csharp |
I kicked it off this morning at about 10 a.m. and it's still on title keywords |
| 16:33 |
Dyrcona |
Well, OK, then. |
| 16:35 |
csharp |
https://git.evergreen-ils.org/?p=working/Evergreen.git;a=blob;f=Open-ILS/src/sql/Pg/upgrade/XXXX.schema.symspell.sql;h=09cb8281868ecb1b202d956a3a8364dd0491667a;hb=0e82dbb567fc3526027dbea0681d65539ba5ef33#l777 is the script I'm referring to (which is generated by those commands run by the admin) |
| 16:35 |
Dyrcona |
Yeahp. |
| 16:36 |
Dyrcona |
It looks to me like you'll be ok as long as you don't set the opac.did_you_mean.max_suggestions org unit setting. |
| 16:45 |
|
sandbergja_ joined #evergreen |
| 16:57 |
Dyrcona |
Ah, well. Testing will have to continue tomorrow. |
| 16:58 |
csharp |
Dyrcona: thanks for checking on that |
| 16:59 |
Dyrcona |
Well, I'm not 100% certain. I based that assumption on what I see in the upgrade script. |
| 16:59 |
|
jvwoolf left #evergreen |
| 17:01 |
Dyrcona |
I'll be back tomorrow. Have a good rest or your day or night, everyone! |
| 17:12 |
|
stompro_ joined #evergreen |
| 17:24 |
|
mmorgan left #evergreen |
| 18:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:02 |
|
sandbergja_ joined #evergreen |
| 20:39 |
|
sandbergja_ joined #evergreen |
| 20:51 |
|
malexander joined #evergreen |
| 21:15 |
|
sandbergja_ joined #evergreen |
| 23:36 |
|
sandbergja_ joined #evergreen |
