Evergreen ILS Website

IRC log for #evergreen, 2021-03-02

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
01:50 khuckins_ joined #evergreen
02:40 khuckins joined #evergreen
06:00 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
06:18 khuckins joined #evergreen
06:27 khuckins joined #evergreen
07:17 Dyrcona joined #evergreen
07:31 rjackson_isl_hom joined #evergreen
08:04 collum joined #evergreen
08:12 Dyrcona joined #evergreen
08:25 mantis1 joined #evergreen
08:26 rfrasur joined #evergreen
08:39 mmorgan joined #evergreen
09:01 jvwoolf joined #evergreen
09:18 Dyrcona Grabbing 1248
09:30 pinesol [evergreen|Jason Boyer] LP1703658: Convert GIST Indexes to GIN - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=378588d>
09:30 pinesol [evergreen|Jason Stephenson] Lp 1703658: Repair DB Upgrade - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=23287f0>
09:30 pinesol [evergreen|Jason Stephenson] Lp 1703658: Stamp DB Upgrade - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=7c4a18e>
09:46 Bmagic Is there a bug that I can't find in LP about large invoices not loading in the UI completely?
09:48 bshum It sounds familiar to me,  like a result of that chunking change for stanza size for ejabberd
09:49 bshum Like forever ago,  we left the stanzas bigger in ejabberd config to deal with those while they were sorted
09:49 bshum But maybe it didn't get logged
09:49 Dyrcona action trigger still has issues with stanza size while we're mentioning it.
09:49 bshum https://bugs.launchpad.net/opensrf/+bug/1725317
09:50 pinesol Launchpad bug 1725317 in OpenSRF ""XML stanza is too big" still possible with chunking and bundling" [Undecided,New] - Assigned to Galen Charlton (gmc)
09:50 bshum In comment 3, dbwells confirms that saving a large invoice broke
09:50 Dyrcona bshum++ I was just about to search for that one.
09:50 bshum For same reasons
09:50 Dyrcona Bmagic: How big are these invoices, i.e. # of line items?
09:51 Bmagic 60 or 70 lines
09:51 Dyrcona Oh, and I guess that includes a chunk of MARC for each one, so if the records are detailed.....
09:51 bshum Yep
09:52 Dyrcona Assuming 15K per entry, you're looking about 1MB to retrieve all that in one go.
09:53 Bmagic I solved the stanza size issue with a config tweak to ejabberd. But this other thing about incompletely loading the UI is a different thing. An issue I've seen for many months. Maybe years. But shrugged it off because the interface is still dojo
09:53 Dyrcona Bmagic: I'd check for XML stanza is too big in the logs, just the same.
09:54 Bmagic alright
09:54 Bmagic trouble is, it works for me
09:54 Dyrcona Just to rule it out. I set max_stanza_size to 10MB on our utility server that handles notices for accounts with more than 100 items checked.
09:56 Dyrcona You probably have a) a faster connection to server, b) a better computer with faster CPU and more RAM, c) a better supported architecture for the browser with better JIT, d) the advantage of trying it after it failed for someone else and so getting the data from RAM on the db serve, or some combination of the 3.
09:56 Dyrcona Or it could be something else entirely.
09:57 Bmagic right, those are some of the similiar things I was thinking
09:57 Dyrcona Web "development" is "fun." :)
09:57 Bmagic it's always on larger invoices. Which is a big clue
09:57 Dyrcona Can you get JavaScript console logs when it fails?
09:57 Bmagic I was thinking of making a bug report - but since it's dojo....
09:59 Dyrcona I think a bug would be worth it if you can add console logs. It could help with the push to eliminate Dojo for acquisitions.
10:00 Bmagic The user did submit the console log, but there isn't anything wrong that I can see
10:01 Bmagic Comparing it to my log when I load the page, it has the same number of lines: 76
10:01 Dyrcona Is the Angular implementation of acquisitions released, yet?
10:01 Bmagic I think the log is suspect
10:01 Dyrcona IDK. I'd have to look at it.
10:04 Bmagic sure
10:06 pastebot "Bmagic" at 168.25.130.30 pasted "invoice dojo UI load" (335 lines) at http://paste.evergreen-ils.org/10117
10:07 Dyrcona The garbage in teh view.js lines may be significant. Do you see those in your console log?
10:10 Bmagic I've got another log output to share
10:10 Bmagic one that has actual errors
10:11 pastebot "Bmagic" at 168.25.130.30 pasted "invoice dojo UI load2" (1708 lines) at http://paste.evergreen-ils.org/10118
10:12 Dyrcona Well, that one has error getting dojo.js, but I think those are "normal" unless you use the custom dojo.tgz from the downloads page/created by make_release.
10:12 Bmagic Dyrcona: oh, that's a paste/character conversion by the pastebin -> html
10:13 Bmagic RE garbage ^^
10:14 Dyrcona Ok. I gathered that after looking at your second paste. :)
10:14 Dyrcona Unknown server error isn't very useful. Did you say whether or not they've tried clearing the cache?
10:16 Bmagic Yeah, I made sure they did a hard refresh and cleared cache and whatnot
10:16 Bmagic VM4275:174 Uncaught Error: Method error: undefined : An unknown server error occurred - line 174 is this line "addInvoiceEntry(entry);"
10:16 Bmagic I guess I'll make the bug and go from there
10:17 Dyrcona Yeah. I don't think I can be much more help right now.
10:25 Bmagic Drycona: ty for your time
10:25 Dyrcona yw
10:25 Bmagic Dyrcona++
10:25 Bmagic bug 1917482
10:25 pinesol Launchpad bug 1917482 in Evergreen "Invoice UI fails to load completely" [Undecided,New] https://launchpad.net/bugs/1917482
10:38 Stompro joined #evergreen
11:00 stephengwills joined #evergreen
11:09 alynn26 joined #evergreen
11:14 csharp Bmagic: see https://bugs.launchpad.net/op​ensrf/+bug/1912834/comments/5 for what sounds like a similar issue for us after we applied the fix in https://bugs.launchpad.net/op​ensrf/+bug/1912834/comments/4 (from the last comment on that bug, sounds like you applied it at the end of Jan)
11:14 pinesol Launchpad bug 1912834 in OpenSRF "Browser client should limit the number of parallel requests" [High,New]
11:15 csharp as the old saying goes, "what is good for the Angular is bad for the Dojo"
11:16 Bmagic csharp: that bug is no longer in production
11:16 csharp ok good - just ruling it out
11:16 Bmagic that broke tons of acq stuff
11:16 csharp dojo--
11:16 csharp @insult dojo
11:16 pinesol dojo: You are nothing but a villainous gob of infected squirrel.
11:18 Bmagic csharp: the solution for us turned out to be nginx limiting. The production system was getting overwhelmed by 3rd parties hitting unapi URL. I supplied an example config in a branch: https://bugs.launchpad.net/opensrf/+bug/1913617
11:18 pinesol Launchpad bug 1913617 in OpenSRF "NGINX could use a DOS mitigation example" [Undecided,New]
11:19 malexander joined #evergreen
11:20 berick Bmagic: are you saying the unapi/nginx limiting *also* resolved any remaining drone exhaustion (unrelated to unapi) ?
11:20 Bmagic it resolved the problem, yes. Though I realize that the unapi service is not heavily used by Evergreen
11:21 berick Bmagic: but did it resolve multiple problems? :)
11:21 berick there's the unapi thing and then there's drone exhaustion from too many parallel requests
11:21 berick now that I think about it.. it can't solve the drone issue
11:22 Bmagic It's hard to say that csharp and I were having the exact same issue. It was just timely that we were having similar issues at the same time. I was thinking that the bricks could* have been overwhelmed for the same reasons. I was willing to try anything*
11:22 berick since those are Websockets requests, I suspect they are not impacted by nginx limiting, but I could be wrong
11:22 berick Bmagic: gotcha, ok, so those are really 2 different issues.  just clarifying in case we had an option that we didn't realize
11:23 Bmagic berick: nginx has a special block to handle the websockets requests (and I didn't impose a limit in that block) - so no, nginx is not limiting the websockets requests
11:23 berick ah, ok, thanks for clarifying
11:23 Bmagic see branch above
11:23 * berick nods
11:24 Bmagic though, with that nginx syntax - it could be imposed on any* URL. Which is nice to have in our tool chest
11:26 berick yeah, was thinking it may be worth exploring as a solution to drone exhaustion issues.
11:29 Bmagic it worked wonders, so I think it's worth checking into for other things. NGINX can apply leaky bucket plus delayed combined with allowances for bursting. It's all spelled out here: https://www.nginx.com/blog/rate-limiting-nginx/
11:29 berick Bmagic++
11:32 Bmagic the first day after updating that config, it mitigated over 90k requests. And Evergreen was unaffected
11:43 Dyrcona Node.js gives me the heebie-jeebies.
11:43 khuckins joined #evergreen
11:44 Bmagic Dyrcona: I think Node.js takes a backseat to this: https://images.app.goo.gl/JzfX6pjf5yDhNv9R9
11:45 Bmagic the classic 1980's glow worm
11:51 Dyrcona While doing npm update: npm WARN bootstrap@4.3.1 requires a peer of popper.js@^1.14.7 but none is installed. You must install peer dependencies yourself.
11:51 Dyrcona I suppose npm update is a bad idea considering how we pin packages?
12:02 berick Dyrcona: well, we're not exactly curating the packages, but they are consistent across a swath of time because of the lockfile.
12:02 jihpringle joined #evergreen
12:02 berick there's nothing wrong with updating, but you may be moving into new territory
12:02 berick but you may also be applying useful patches, so...
12:15 Bmagic here is an interesting one: we had a patron notice that the due date for one of their items shows a different day depending on whether they are looking at their checkouts on their phone or the computer. Both using chrome. It's a day off. I immediately suspected a timezone issue where the computer is correct and the phone is wrong.
12:16 Bmagic but, the due date on the phone displayed a day before what the computer browser said. And the due time is 11:59:59 on day 9. The phone shows day 8. If it were a timezone issue, I would suspect that it would have rolled over to the next day, not the day before. I've confirmed that the DB says 11:59:59 on day 9
12:18 Bmagic I guess the timezone on the phone could have been set to somewhere on the opposite side of Earth, where it's 24 hours off? Seems unlikely.
12:18 Bmagic we're letting that one go, but I thought it was a "fun" one to talk about here
12:36 Dyrcona Time is hard.
12:43 Dyrcona When I add dates to milestones on Launchpad, I sometimes have to change the day because they can be off by a day.
12:46 * Dyrcona wonders how difficult it would be for "Use Now" on workstation registration to just work without having to log in again.
12:50 Dyrcona So, I just tried using MARC batch import with master, and I get upload progress 100%, but enqueue progress is 0%. There's also nothing in the /tmp directory AFAICT.
12:51 jeff have to teach open-ils.auth how to assign a workstation to a workstationless login and change type. Might also violate some other assumptions.
12:51 jeff Dyrcona: is your web server running with private temp, and you're using /tmp as the queue location? You'll need to either change to a different dir or stop using private tmp for the apache service.
12:52 jeff (many public services like apache are launched by systemd with private /tmp by default on a lot of systems now)
12:52 Dyrcona jeff: RE workation: Yeah. I might look into that. RE /tmp. I"m looking. It made a queue.
12:52 jeff it's been that way for a while on Debian. I don't remember if it recently changed for Ubuntu.
12:53 Dyrcona Queue is empty. I'm using default configurations with Concerto, so I probably need to change the Apache or opensrf.xml.
12:53 Dyrcona jeff++
12:59 Dyrcona Bingo! I switched from /tmp to /openils/var/tmp in opensrf.xml. I had to mkdir /openils/var/tmp, but that fixed it.
12:59 Dyrcona Apache can't see /tmp, apparently, but I didn't look too hard at the configuration.
13:00 Dyrcona Well, no, it can see /tmp.....
13:03 jeff apache's /tmp is not your /tmp :-)
13:04 jeffdavis I believe private /tmp is a change between Ubuntu 16.04 and 18.04 fwiw
13:07 Dyrcona Well, I hadn't noticed because I don't use it much on development, and we've been using /openils/var/tmp mounted via NFS in production for years.
13:08 Dyrcona So, I'm actually testing a security bug and trying to import a MARC record to trigger it, but the record won't import.
13:08 Dyrcona When I select it and hit Import Selected Records, the screen goes back to the main Import view, and the record does NOT end up in biblio.record_entry.
13:09 Dyrcona FWIW, I have no idea what I'm doing in the staff client, particularly the Angular interface.
13:28 Dyrcona So, maybe Vandelay is broken in master on Ubuntu 20.04?
13:40 jeff can you manually create a MARC record from a template? can you edit an existing bib record? database or opensrf logs (or browser console) might have more clues for you.
13:40 Dyrcona I can manually edit bib records.
13:42 Dyrcona It seem to pick on the flat text editor more than I'd like. If used the flat text editor, switched to enriched editor, then create a new record it seems to go to the flat text editor. I don't recall making flat text editor a default.
13:44 Dyrcona Creating a new record works.
14:02 Dyrcona jefdavis++
14:02 Dyrcona So, it also appears that my Vandelay record was eventually imported.
14:05 Dyrcona Oh, no. I realize now, based on the time stamp, that I made that record in the flat text editor via copy and paste.....
14:10 Dyrcona Vandelay is acting the same on a VM running Ubuntu 18.04, so it's probably me not knowing what I'm doing.
14:40 Dyrcona jeffdavis++ JBoyer++ I'm going to add my signoff to the security branch.
14:41 Dyrcona Also, I was doing Vandelay wrong. I got it working.
14:41 jeffdavis Yay!
14:41 JBoyer Dyrcona++
14:41 JBoyer jeffdavis++
14:44 terranm joined #evergreen
14:49 Dyrcona Also, jamesrf++
15:03 jvwoolf joined #evergreen
15:07 Dyrcona I wonder if XUL really matters for bug 1076582?
15:07 pinesol Launchpad bug 1076582 in Evergreen "Remove or document references to openils_dojo.js" [Low,Confirmed] https://launchpad.net/bugs/1076582 - Assigned to Jason Stephenson (jstephenson)
15:08 Dyrcona After applying the branch, I found a couple of active references to openils_dojo.js in the XUL code, and I wonder if building a XUL client and testing it is worth the trouble at this point?
15:13 sandbergja_ joined #evergreen
15:20 jihpringle joined #evergreen
15:24 JBoyer If there's not a reference under Open-ILS/xul/server I don't think it matters whether the client works or not. (I was thinking there *may* be something still using parts of server/* ?)
15:25 JBoyer Since it can't use any TLS without manual intervention and lacks so much functionality I think it should be removed from 3.7 even if we're not quite ready to rm -rf Open-ILS/xul yet.
15:26 sandbergja_ joined #evergreen
15:27 Dyrcona JBoyer: I references to XULG in acquisitions dojo code.
15:28 JBoyer Ah, that's about where I'd expect it to still be.
15:31 Dyrcona I am tempted to build openils_dojo.js to see if it really makes much difference these days.
15:34 jeff I tried to build it a year or two ago. It was not a simple or pleasant task.
15:37 jeff Without digging for notes, I don't remember if I ended up with something that worked or not. I think it took some effort just to find a version of the tools that was just the right mix of old enough / new enough.
15:37 Dyrcona Well. I'll have a look.
15:38 jeff Good luck!
15:38 Dyrcona Not that I'm going to try building it, but I want to see how hard it is to find the Dojo 1.3.3. src.
15:40 Dyrcona It's not that hard to find. I'll give the build a whirl just to see what's involved.
15:40 Dyrcona :)
15:41 JBoyer Isn't there some weird java-based tool involved? That might be the real tricky bit.
15:41 JBoyer That said, if Dyrcona does get it built we could just say it's never getting updated again and check it in, heh.
15:42 Dyrcona Yeah, it requires Rhino, which is gone for the most part. (I used to play with Rhino back in the day.)
15:43 Dyrcona OK. I take it back. Rhino isn't gone, but modern Java comes with a "subsitute" that's almost as good. ;)
15:49 Dyrcona If that works, it was pretty easy. ;)
15:49 Dyrcona IF that that works that is.... (Big IF) :)
15:53 Dyrcona Seems a bit late to be messing with this.
15:57 Dyrcona It looks like most of the places where openils_dojo.js is still used are obsoleted by newer interfaces in 3.6.
15:59 mantis1 left #evergreen
16:01 Dyrcona Well. It seems to have undone the security patch that I just installed, but I think that's because I switched branches when building the custom dojo.
16:03 Dyrcona I seem to be getting a lot of uncaught exceptions, too.
16:19 Dyrcona I should probably wipe /openils and install fresh.
16:27 csharp so we're running the upgrade script for bug 1893997 against a PINES test server and lordy that's going to take a long time to run
16:27 pinesol Launchpad bug 1893997 in Evergreen "Wishlist: Did you mean? Single word, single class substitutions" [Wishlist,New] https://launchpad.net/bugs/1893997
16:27 csharp followed by a reingest
16:28 csharp gonna research whether the changes can be applied outside of a downtime window
16:28 csharp unless miker just knows offhand
16:29 Dyrcona Well, you can run the reingest while people are using the system.
16:29 csharp yeah, I just didn't want any changed DB-side functions to lead to unexpected results before the did you mean stuff was populated
16:29 csharp which is the part that's going to take forever
16:30 csharp the generated SQL script is 52M individual select calls to a function that populates the tables
16:32 Dyrcona Well, 52M selects is not necessarily slow.
16:32 Dyrcona It depends a lot on what they do.
16:33 csharp I kicked it off this morning at about 10 a.m. and it's still on title keywords
16:33 Dyrcona Well, OK, then.
16:35 csharp https://git.evergreen-ils.org/?p=working/E​vergreen.git;a=blob;f=Open-ILS/src/sql/Pg/​upgrade/XXXX.schema.symspell.sql;h=09cb828​1868ecb1b202d956a3a8364dd0491667a;hb=0e82d​bb567fc3526027dbea0681d65539ba5ef33#l777 is the script I'm referring to (which is generated by those commands run by the admin)
16:35 Dyrcona Yeahp.
16:36 Dyrcona It looks to me like you'll be ok as long as you don't set the opac.did_you_mean.max_suggestions org unit setting.
16:45 sandbergja_ joined #evergreen
16:57 Dyrcona Ah, well. Testing will have to continue tomorrow.
16:58 csharp Dyrcona: thanks for checking on that
16:59 Dyrcona Well, I'm not 100% certain. I based that assumption on what I see in the upgrade script.
16:59 jvwoolf left #evergreen
17:01 Dyrcona I'll be back tomorrow. Have a good rest or your day or night, everyone!
17:12 stompro_ joined #evergreen
17:24 mmorgan left #evergreen
18:00 pinesol News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
18:02 sandbergja_ joined #evergreen
20:39 sandbergja_ joined #evergreen
20:51 malexander joined #evergreen
21:15 sandbergja_ joined #evergreen
23:36 sandbergja_ joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat