Evergreen ILS Website

IRC log for #evergreen, 2019-10-24

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
01:19 bwicksall joined #evergreen
06:57 agoben joined #evergreen
07:06 rjackson_isl joined #evergreen
07:24 collum joined #evergreen
08:00 tlittle joined #evergreen
08:20 Dyrcona joined #evergreen
08:21 remingtron Bmagic++
08:21 remingtron devted++
08:24 csharp all_yall++
08:25 agoben Heh, I'm with csharp on that.  Very awesome to see all the work that was accomplished this week!
08:26 csharp agoben++ # especially
08:26 JBoyer agoben++
08:26 tlittle Yes, definitely agoben++
08:27 remingtron agoben++ #great hostess
08:28 agoben Aww, thanks!  Always a pleasure to have everyone here!
08:29 csharp also...
08:29 csharp agobens_mom++
08:30 csharp pretty sure sugarcream pie is going to be on our thanksgiving table :-)
08:31 bos20k joined #evergreen
08:35 csharp oh - nice - FF 70 with Enhanced Tracking Protection - I got a notification that it blocked Facebook trackers on a site I was just on
08:35 csharp firefox++
08:35 csharp mozilla++
08:37 Dyrcona agoben++
08:40 collum Ditto - agoben++
08:41 Dyrcona GB2312_data_that_is_really_KOI8--
08:41 Dyrcona try/except++
08:42 Dyrcona ...since we're doling out the karma. :)
08:42 Dyrcona chardet++
08:42 mmorgan joined #evergreen
08:45 Dyrcona Email can be worse than MARC when it comes to character set issues.
08:48 dbwells agoben++
08:48 * Dyrcona goes back to writing an email to summarize hack-away activity for CW MARS staff.
08:53 * mmorgan would love to see a message summarizing the hackaway go out to the general list.
08:57 dbwells "...It was good, but not as good as when Dyrcona is there..."
08:57 Dyrcona Ha!
09:02 Dyrcona I'm sorry I missed the sugar cream pie.
09:02 Dyrcona :)
09:03 * Dyrcona has had a lot going on lately and didn't really have the time to go this year.
09:04 mdriscoll joined #evergreen
09:04 Dyrcona mmorgan: https://wiki.evergreen-ils.org/doku.ph​p?id=hack-a-way:hack-a-way-2019-agenda
09:15 mmorgan Dyrcona: Thanks, I was watching that. Was just thinking that something put out to the list for general consumption might bring to the attention of the general community how much gets accomplished.
09:18 Dyrcona Well, someone who was there should probably do that one. :)
09:19 mmorgan Right!
09:27 yboston joined #evergreen
09:55 mmorgan left #evergreen
09:57 sandbergja joined #evergreen
10:02 mmorgan joined #evergreen
10:10 nfBurton joined #evergreen
10:27 collum_ joined #evergreen
10:28 collum_ joined #evergreen
10:29 collum_ joined #evergreen
10:30 collum joined #evergreen
10:31 collum joined #evergreen
10:32 collum__ joined #evergreen
10:35 collum_ joined #evergreen
10:44 dbs huh, the links for the qatests failures return a 404, is that a known issue?
10:48 collum joined #evergreen
10:48 sandbergja joined #evergreen
10:50 berick dbs: i think that happens if a test is in progress
10:51 collum joined #evergreen
10:56 dbs berick: ahh
11:00 jvwoolf joined #evergreen
11:01 pinesol News from qatests: Failed Installing AngularJS web client <http://testing.evergreen-ils.org/~li​ve/test.28.html#2019-10-24T11:00:47,985113758-0400 -0>
11:01 pinesol News from qatests: Failed Installing Angular web client <http://testing.evergreen-ils.org/~li​ve/test.29.html#2019-10-24T11:00:48,026755118-0400 -2>
11:01 pinesol News from qatests: Failed Log Output: osrfsys.log <http://testing.evergreen-ils.org/~li​ve/test.76.html#2019-10-24T11:00:48,068169737-0400 -4>
11:01 collum_ joined #evergreen
11:02 collum__ joined #evergreen
11:02 Christineb joined #evergreen
11:02 berick I have a fix for the Angular test failures if someone could S/O and merge...
11:03 berick working/user/berick/lp1830​973-ang8-test-spec-repair
11:03 berick note the LP number references the (now-closed) angular 8 bug
11:19 Bmagic agoben++
11:26 stephengwills I’m converting MARC21 to MARCXML w/ the python script and hitting “UnicodeDecodeError: 'marc8_to_unicode' codec can't decode bytes in position 0-156: invalid multibyte character encoding”. is there a way to tell that script to skip the record it’s barfing on and keep processing the file, or is that a bad idea?
11:26 jeff "the python script"?
11:26 stephengwills http://docs.evergreen-ils.org/3.2/_mig​rating_your_bibliographic_records.html
11:26 jeff pymarc had some annoying charset issues last time I tried to make use of it a few years ago.
11:27 jeff I'm not sure the origins of that script.
11:28 stephengwills I was hoping not to have to hack pymarc :)
11:28 jeff Those docs might need some attention. I'm not sure how close to reality they are.
11:29 Dyrcona abbreviations_in_configuration-- # I'm looking at you, haproxy's option httpchk.
11:32 berick Status 220 HTTP Chicken
11:35 JBoyer berick++
11:35 berick i'm suddenly craving Beef Manhattan
11:36 Dyrcona :)
11:37 Bmagic I am finding that some days SIP is not running. The last log entry is: raw_transport: shutting down: EOF during login
11:37 Dyrcona Well, the httpchk option isn't doing what I want.
11:39 Dyrcona Bmagic: I've not seen that.
11:39 Bmagic strange
11:40 Dyrcona I want haproxy to mark a brick down if OpenSRF services go down and apache starts returning 500 errors. I know there's a way to do this, and I'll figure it out, eventually.
11:41 Dyrcona Bmagic: SIPServer should fork. It might shut the main process down if max_servers is not set or equals 1.
11:42 * dbs was the origin of that script, jeff / stephengwills
11:42 Dyrcona But I can't think of anything else that would do that.
11:43 Bmagic this machine is setup PreFork and max_servers='200'
11:44 dbs stephengwills: pymarc does not fail gracefully when it encounters encoding issues - often either because LDR09 is lying, or because non-UTF8 / non-MARC8 chars (like iso-8859-1) have been added by other less strict tools
11:44 berick Bmagic: could be a client connecting to the socket, but never sending the login message (within the configured timeout).
11:45 berick hm, though, I would also expect to see a timeout log message in that case
11:45 berick or the client opens the socket, then disconnects without ever doing anything.
11:47 Dyrcona Sure, but that shouldn't kill all of the SIPServer processes.
11:48 berick oh, yeah, it certainly should not.  I missed that bit.
11:48 Bmagic maybe they die off one at a time throughout the day?
11:48 Bmagic (and never come back....)
11:49 dbs stephengwills: so as long as your MARC records are perfectly encoded and structured, pymarc should be happy - heh
11:49 stephengwills dbs record.out should hold, up to the record containing the offending char and allow me to correct the offending bib, I assume, and then I rinse and repeat?
11:50 Dyrcona I had SIPServer die mysteriously on one of our hosts recently, but never figured out why.
11:50 stephengwills sorry…up to the record BEFORE the offending record
11:50 * Dyrcona runs out to get some lunch. Back in a few.
11:52 stephengwills seems like the next step is make this script try and catch or something.  I’ll play with it.
11:53 dbs stephengwills: a more forgving MARC21 to MARCXML tool is yaz-marcdump; something like "yaz-marcdump -i marc -o marcxml -f utf8 -t utf8 -l 9=97 inputfile.mrc > outputfile.xml"
11:53 stephengwills ok… will try that first :). thanks
11:54 dbs there was a recent discussion about "Encoding issues of Doom" on the pymarc mailing list, possibly a similar sitation to what you're facing: https://groups.google.com/d/msg/​pymarc/grZpM5Me34s/BTQX6bnFBwAJ
11:54 stephengwills ls
11:55 stephengwills sorry
11:57 stephengwills yaz-marcdump cleared it.  Thanks Dan
11:57 stephengwills I’ll add that to the docs.
11:59 stephengwills now I wonder what Evergreen is going to do with those records when I try and ingest them :)
12:05 dbs stephengwills: yeah, yaz-marcdump might have created some unusual MARC :)
12:07 stephengwills as well as a lot of extra \n I’m needing to remove to get one record per line.  but that just makes it all more fun.
12:09 jihpringle joined #evergreen
12:21 yboston joined #evergreen
12:35 collum joined #evergreen
12:36 * Dyrcona isn't he only one having encoding issues today, I see.
12:38 alynn26 joined #evergreen
12:46 Dyrcona Heh. "Fixed" my httpchk issue, but now they're all down because I forgot that / does a redirect. :)
12:51 khuckins joined #evergreen
12:52 Dyrcona Do websockets return http status codes? (maybe I should look that up.)
12:57 JBoyer Dyrcona, maybe the upgrade response? Not sure if that's the initial response or only in response to certain types of request.
12:58 Dyrcona JBoyer: Thanks, but I found a simpler solution. I can tell haproxy to check a different port or even a different server, so I think if I tell it to check port 80, then websockets is down when http is down, which is good enough.
12:58 * Dyrcona is about to test that.
12:59 JBoyer That's how I've always done it in ldirector, yeah. :)
13:03 Dyrcona Yeah, it's working.
13:03 Dyrcona :)
13:04 jeff Assuming I followed that correctly, do you now run the danger of sending websockets requests to a backend where websockets are dead but HTTP on port 80 is fine?
13:05 Dyrcona jeff: I will check that after I see if things come back automatically, but that said, I've never had websocketd die.
13:07 Dyrcona Restart services and apache2 and the brick is working again.
13:08 Dyrcona stop websocketd and it still shows as up. I really want to check both conditions....
13:13 Dyrcona Hm... Looks like I need to write an external command to do this.
13:14 Dyrcona That's beyond the scope of the remainder of today and tomorrow.
13:16 JBoyer Looks like page 4 of the RFC should help. You send a request with an upgrade header, you should get back a response with a 101 status code.
13:16 JBoyer https://tools.ietf.org/html/rfc6455#section-1.2
13:17 JBoyer For whenever. :)
13:21 Dyrcona JBoyer: Yes. I see that, and I've found some examples. A straightforward "check" works if websocketd is dead. I basically want to check that, plus if services have died, and I think I get the 101 even if services are dead, but I'll write a little script to check that.
13:21 Dyrcona JBoyer++
13:21 Dyrcona jeff++
13:22 Dyrcona complication--
13:22 Dyrcona webosckets--
13:22 jeff habanero_hellfire_chili++
13:22 JBoyer Oh, yeah, websockets will outlive a service problem since it doesn't know what's going on, just where to throw the bytes.
13:23 * JBoyer is almost, but not quite, jealous of jeff's chili.
13:24 * Dyrcona had supermarket pepperoni pizza....is also jealous of jeff's chili.
13:25 Dyrcona JBoyer: Yeah, but I just realized that writing this check script will be easier than I thought, so I might as well do it.
13:25 JBoyer Nice.
13:26 Dyrcona The question is: curl or LWP?
13:26 Dyrcona @decide curl or LWP
13:26 pinesol Dyrcona: go with curl
13:50 remingtron Bmagic: your Antora README says "git clone git://git.evergreen-ils.org/Evergreen.git". Shouldn't that be "/working/Evergreen.git" ?
13:50 remingtron meaning, I think we need "/working" added before "/Evergreen.git"
14:01 Bmagic sure!
14:01 Bmagic Pushing the change now
14:01 Bmagic remingtron ^^
14:04 remingtron Bmagic: cool!
14:05 remingtron I'm about to email the DIG list about our progress, and to invite helpers
14:05 Bmagic remingtron: BTW - I've been pushing lots of changes this morning and last night as late as 1am, lol
14:05 remingtron ha! nice work man
14:06 Bmagic you GOTTA see the UI now
14:07 Bmagic also - I saw you posted on LP about the next/last links. Those are easy to add with the handlebars lingo: https://docs.antora.org/an​tora-ui-default/templates/
14:08 sandbergja joined #evergreen
14:08 Bmagic if we want that - I believe I can put it in there. {{page.next}} {{page.previous}}
14:14 nfBurton How much of the OPAC UI is getting changed with this Angular update and how much is staying tt2?
14:17 JBoyer nfBurton, I think the only Angular OPAC work is staff-only, so nothing changes for the public interface for now.
14:18 nfBurton Oh okay
14:18 nfBurton I was woried my work would be rendered pointless lol
14:19 nfBurton I'm real excited for those speedy searches though!
14:19 remingtron Bmagic++ #finding solution to prev/next links
14:20 Bmagic remingtron: lol, just about done implementing it
14:21 remingtron awesome
14:22 nfBurton The working repo seems to be missing some npm modules? I haven't been able to build for the staff client. Is there a step
14:22 nfBurton I am missing?
14:23 jeff We've been using the Windows build of Hatch 0.3.2 from Comment 1 in bug 1830391. That installs without error on a 32 bit install of Windows, but then you have a Native Messaging Host that doesn't actually work (and "hatch.bat test" fails).
14:23 pinesol Launchpad bug 1830391 in Evergreen "Hatch omnibus circa 3.3 (Java updates and more)" [Undecided,New] https://launchpad.net/bugs/1830391
14:23 jeff I suspect that I can drop a 32 bit build of OpenJDK in place and it'll then work, but I haven't tested that theory.
14:23 Dyrcona Ok. So, I can connect to the websocket port, but it just hangs.  If I do a HEAD request, I get a 405, which may be good enough.
14:38 berick jeff: let me know how it goes...
14:38 berick perhaps we'll need 2 windows installers
14:44 jeff ...and perhaps some installer logic to error out when trying to install 64 bit on 32 bit OS install.
14:45 jeff one installer that's approximately twice the size probably wouldn't be ideal.
14:45 jeff especially since the number of 32 bit installs would be (hopefully) low.
14:47 remingtron nfBurton: what's going wrong? are you seeing errors?
14:51 JBoyer jeff, I think 2 separate installers would be good, and erroring out on the wrong bit-ness is much easier than all of the back and forth we're currently doing.
14:53 nfBurton yeah. most of the modules in .staging just aren't there
14:54 nfBurton Open-ILS/src/eg2/node_modules/.staging
14:54 nfBurton It expects them though :S
14:56 yboston joined #evergreen
14:56 JBoyer Oh, that. nfBurton, you might have to sudo rm -rf /usr/local/lib/node_modules and start again. May also have to clear out /usr/local/bin/ node, npm, ng, and npx
14:57 JBoyer I was hitting that a lot earlier this week and that's what it took to get past it. :/
14:57 nfBurton oh! okay!
14:57 nfBurton JBoyer++
14:58 JBoyer Also, bear in mind that if you run Makefile.install <osname>-developer you'll be getting a lot more packages.
14:59 JBoyer (depending on the branch you
14:59 JBoyer re using)
15:01 Dyrcona Y'know, I usually like chroot, but not so much today.
15:01 mmorgan1 joined #evergreen
15:01 nfBurton yeah im using working and gathering all the dependencies
15:02 Dyrcona osname-developer doesn't install that much. -packager or -translator install a pile of extra stuff.
15:02 nfBurton Also, for anyone using the working branch, does the main My Account page work for you? I seem to get a 500 no matter what, even using the default template.
15:04 nfBurton For my sanity
15:04 jeff which working branch?
15:05 nfBurton the main working branch
15:06 nfBurton It seems the page with payments/charges wont show.
15:08 jeff do you mean master? i'm not understanding "the main working branch", as the way we use working branches there are many... but perhaps confusingly the working repo also has a copy of master...
15:08 jeff nfBurton: what is the commit hash of the tip of the branch you're asking about?
15:08 Dyrcona I was about to ask the very same question.
15:08 nfBurton I just cloned git://git.evergreen-ils.org/working/Evergreen.git
15:08 jihpringle joined #evergreen
15:08 jeff there's a main repo and a working repo, and many "working" branches in the working repo which mostly start with either user/username/ or collab/username
15:09 nfBurton I havent actually pushed anything yet
15:09 nfBurton Kind of a GIT noob.
15:10 jeff got it, then if you just cloned from that URL your git client likely checked out a copy of the master branch from the working/Evergreen.git repo, which is normally automatically sync'd with the master branch of the git://git.evergreen-ils.org/Evergreen.git repo.
15:11 nfBurton oh yes the master branch sorry
15:11 jeff sorry, trying to be precise without sounding picky.
15:12 Dyrcona Details matter. Programmers have to be picky.
15:12 nfBurton No, it's fine. I'm still learning my way around this. And I'm like a day or 2 from pushing LP1778972. I just want to get it right
15:12 jeff but back to my other question... what is the commit hash of the tip of the branch you're asking about? it would be the first line starting with "commit" in the output of "git show" or "git log" or...
15:13 nfBurton Dyrcona++
15:13 nfBurton for sure
15:13 jeff pinesol: bug 1778972
15:13 pinesol jeff: git diff origin/hamster Fleshing children complete
15:13 pinesol Launchpad bug 1778972 in Evergreen "Wishlist - OPAC Redesign" [Wishlist,In progress] https://launchpad.net/bugs/1778972 - Assigned to Christopher Burton (cburton)
15:14 nfBurton 11868e97844a47d57489062197ac6809d9473366
15:14 pinesol nfBurton: [evergreen|Jason Boyer] Stamp upgrade script for LP 1849152 - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=11868e9>
15:20 jeff nfBurton: okay, that matches current master. anything in apache logs when you get the 500 error?
15:23 JBoyer Dyrcona, I'm just noticing your "-developer doesn't install that much..." I added chromium and firefox to that recently, though now that we're discussing it, maybe there should be a -tester target for those... (they're for replacing PhantomJS)
15:24 Dyrcona Yes, there should probably be a -tester or -testing target.
15:24 Dyrcona Or, they could go under -packager, since that's about the only time tests are run other than on the test server.
15:26 JBoyer I suppose we could also expand that target out to include things like pgtap and other things...
15:26 dbs Aren't commit reviewers / signers-off supposed to run tests?
15:27 dbs I guess if we had a travis-ci environment or the like that tested every branch every time they were pushed, that would help alleviate the burden of running tests
15:27 jeff the machine i do dev on isn't always the machine i run tests on.
15:27 jeff but yes, better/easier CI is also a nice goal.
15:28 Dyrcona dbs: I really doubt they do every time, and most new bugs are not caught by the tests. We've also been lax in adding tests.
15:29 Dyrcona gitlab has CI. github has CI.
15:30 Dyrcona I imagine CI for us would be  not so much fun to set up, given how eclectic our codebase is.
15:31 nfBurton @jeff egweb: Context Loader error: Can't locate object method "content" via package "OpenSRF::DomainObject::oilsMethodException"
15:31 pinesol nfBurton: No, you're a puzzleheaded kraken!
15:32 Dyrcona pinesol: Pipe down. It's hard enough to think with this headache.
15:32 pinesol Dyrcona: Have you run autogen.sh?
15:33 Dyrcona nfBurton: Have you run autogen.sh?
15:33 nfBurton lol
15:33 Dyrcona Sometimes the bot has good ideas.
15:33 nfBurton I'm currently rebuilding but will
15:34 Dyrcona I need to sort out what I'm going to do tomorrow since I spent today doing something unplanned and only partly successful.
15:36 sandbergja joined #evergreen
15:39 nfBurton still nothing. It seems to be looking for opac/api/mylist/retrieve.tt2 which doesn't exist
15:42 nfBurton creating the file doesn't help though
15:42 nfBurton Context Loader error: Can't locate object method "content" via package "OpenSRF::DomainObject::oilsMethodException" at /usr/local/share/perl/5.22.1/Ope​nILS/WWW/EGCatLoader/Account.pm line 2081.
15:49 Dyrcona nfBurton: I haven't tried installing master in a few days. I was going to but got sidetracked.
15:58 Bmagic remingtron: We have Previous/Next pushed to the branch
15:59 remingtron Bmagic: sweet!
16:04 JBoyer nfBurton, I'm building another master install real quick to see if I run into the same problem
16:05 nfBurton Thanks
16:14 JBoyer Well, for better or worse it's working fine for me, so I can't see what might be going on here.
16:15 mmorgan joined #evergreen
16:16 nfBurton Oh boy, okay. Thanks for checking.
16:16 JBoyer What version of OpenSRF did you install (or already have in place)?
16:17 nfBurton It even does that for the default template so it's a bit of a head scratcher. Maybe I just can it all and start fresh
16:17 nfBurton 3.2
16:17 khuckins joined #evergreen
16:17 jeff oh, wait. this was running master with non-stock templates you were getting that error?
16:18 JBoyer I will say the /api/ in the url looks odd to me, but I don't know if that means anything other than I haven't paid attention to that yet.
16:18 nfBurton working master branch with default templates. I'm trying to create my non-stock but ran into that issue with last upgrade.
16:18 jeff ah.
16:19 nfBurton I may just trash openils and start fresh
16:19 JBoyer mv /openils /openils.prev maybe? :)
16:20 * JBoyer has forgotten about small things when doing a 'rm -rf /openils' in the past
16:20 nfBurton yeah
16:23 jeff presumably there was a failure in the call to open-ils.cstore.direct.money.open_b​illable_transaction_summary.search -- you might find that error wherever your opensrf logs lie
16:24 jihpringle joined #evergreen
16:24 jeff but ensuring that you have a working IDL in place and all Evergreen opensrf services running successfully and a clean autogen.sh run... those might be good first things to check.
16:25 jeff but also that tt2 file not existing makes me wonder about your apache config.
16:26 jeff @decide more sanity checks or better remote diagnosis tools
16:26 pinesol jeff: That's a tough one...
16:27 JBoyer sometimes the bot really does have the right answer, heh.
16:33 Dyrcona nfBurton: What Linux distro and release are you installing on?
16:35 nfBurton xenial 16.04
16:36 * jeff attempts to think of creative ways to tie a staff search to a staff user from potentially incomplete logs
16:37 JBoyer incomplete how?
16:38 jeff on the test box in question, i'm pretty sure that some things (at least "gateway" logs) are not being saved.
16:38 jeff (an oversight, not intentional)
16:39 Dyrcona nfBurton: Never mind. What I was thinking of only applies to Debian Stretch and Debian Buster.
16:39 JBoyer Ah, I thought you meant something like rate-limited drops or very short retention periods, etc. So long as you have everything normally in osrfsys.* you've got a connection from username to authtoken.
16:40 jeff yeah, I'm just not sure I've a link from "this search" to "this auth token"
16:40 Dyrcona nfBurton: The symptoms would also be that osrf_control --diagnostic [-l] shows none of the C services running: cstore, pcrud, auth, auth_internal....
16:41 JBoyer Oh, right, it's not like the osrfsys files actually have the search terms. Oops.
16:42 yboston joined #evergreen
16:50 mmorgan left #evergreen
16:52 * Dyrcona signs out for today.
18:12 jonadab joined #evergreen
19:39 jvwoolf joined #evergreen
20:11 sandbergja joined #evergreen
22:33 sandbergja joined #evergreen
22:56 jvwoolf joined #evergreen
23:01 pinesol News from qatests: Failed Installing AngularJS web client <http://testing.evergreen-ils.org/~li​ve/test.28.html#2019-10-24T23:00:48,595715594-0400 -0>
23:01 pinesol News from qatests: Failed Installing Angular web client <http://testing.evergreen-ils.org/~li​ve/test.29.html#2019-10-24T23:00:48,642897535-0400 -2>
23:01 pinesol News from qatests: Failed Log Output: osrfsys.log <http://testing.evergreen-ils.org/~li​ve/test.76.html#2019-10-24T23:00:48,690042733-0400 -4>
23:17 sandbergja joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat