Evergreen ILS Website

IRC log for #evergreen, 2018-06-04

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
02:20 beanjammin joined #evergreen
06:13 Bmagic_ joined #evergreen
06:16 jeff_ joined #evergreen
06:16 tsadok joined #evergreen
06:31 pinesol_green News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
07:01 agoben joined #evergreen
07:15 rjackson_isl joined #evergreen
07:49 collum joined #evergreen
07:56 bdljohn joined #evergreen
08:02 _bott_ joined #evergreen
08:27 JBoyer joined #evergreen
08:28 JBoyer Hi! I know I've seen this somewhere but apparently I can't find it now. We're getting this a lot in PG logs: ERROR Unexpected end of string in search.highliht_display_fields(...)
08:29 JBoyer And I can't tell if that's what's constantly killing us or something new, because it is now a Very Bad Day.
08:31 * csharp looks for that in the PINES logs
08:32 JBoyer I'm almost certain I've seen it in LP but you know how that goes.
08:32 dbs JBoyer: we get a ton of those as well
08:32 JBoyer Shoot. Then that's probably not the issue. :/
08:33 csharp I'm not seeing it as of the last couple of hours
08:33 dbs we just upgraded to 3.1.2, seemed like it was worse before the bib reingest completed, but I'm seeing it with records that do have metabib.display_entry rows now
08:33 csharp JBoyer: what are the other symptoms?
08:35 dbs the fields in the records in question are not highlighted in search results / record display
08:36 JBoyer I haven't looked at hte logs enough to find much else to go on, but searches are failing immediately for one thing
08:37 csharp I see
08:37 * csharp just tried a couple of searches that should definitely return results
08:40 JBoyer The large number of "no connection to the server" db errors would be the most serious issue I suppose..
08:40 csharp JBoyer: we had to bump up cstore and pcrud to larger max_children
08:41 csharp JBoyer: osrf_control --diagnostic should show what might be up
08:41 JBoyer Ours were already up there a bit, but I'll check the numbers out
08:41 csharp also, tailing the osrfwarn log which is where you'll see the max_children error messages
08:42 * csharp assume you prolly already know that
08:42 csharp *assumes
08:42 dbs this is day one of people actually using our 3.1.2 system since the upgrade, so I'll be keeping an eye on actual db connection death as well
08:43 dbs we bumped up open-ils.fielder as well per BC Libraries Co-op & PINES suggestions
08:43 csharp JBoyer: for reference, our max_children for pcrud is at 72
08:43 csharp fielder is also at 72
08:43 JBoyer Ah. I believe this would be the root of the issue: 0 SSL SYSCALL error: EOF detected
08:44 JBoyer That's cstore dying possibly from the postmaster killing it or similar.
08:44 JBoyer (The calls were just the usual "what's the value of this OUS?
08:45 JBoyer Annnddd my display fields only pingest keeps telling me that the db is in recovery mode. That's pretty much the worst thing that can happen there.
08:46 JBoyer So I know what's going on, just not why.
08:47 csharp JBoyer: check disk usage on the DB server - possible that it has nowhere to write xlog?
08:48 csharp scary as hell, but postgres almost always does the Right Thing™ in my experience
08:49 JBoyer Running out of space is what was going on the other day. A pg_dump/restore has over 200GB free now. It's dying on segfaults shortly after leaving recovery mode.
08:49 JBoyer First one in one batch was just a metabib.reingest_metabib_field_entries call
08:50 JBoyer (so recovery, everything dies, metabib.reingest_metabib_field_entries, pid crashes)
08:50 JBoyer And when something crashes like that it "may have corrupted shared memory" and it's all gone.
08:52 csharp maybe high RAM caused by pingest.pl (or too many parallel jobs)?  I've seen that kind of thing happen
08:53 JBoyer 482GB free
08:53 csharp well, I mean as a cause of what made the DB enter recovery mode
08:53 csharp maybe not ongoing
08:54 mmorgan joined #evergreen
08:55 JBoyer rjackson_isl has found this in the kern log on one of the servers: segfault at 20 ip 00007f5f9c6c6bb8 sp 00007ffce9fc2dc0 error 4 in libperl.so.5.18.2 Which is related to what I was seeing. I'll look for any OOM messages justto be sure
08:55 dbs csharp++ # osrf_control --diagnostic !
08:55 dbs didn't know about that before
08:56 csharp dbs: yeah - that's an awesome feature :-)
08:56 csharp berick++
08:57 csharp JBoyer: ewww
08:57 JBoyer And all of those segfaults are completely isolated in syslog, no oom, no anything around them. This is why I Just Love(tm) code running inside databases.
08:58 csharp jeez
08:58 JBoyer There was something somewhere recently about segfaulting perl wasn't there? I don't know if it was related to MARC parsing or what, but that's not an easy combination of things to seach for,
09:00 remingtron one recent seg fault bug: bug 1764542
09:00 pinesol_green Launchpad bug 1764542 in Evergreen "Incorrect format on config.metabib_field insert results in segmentation fault" [High,Confirmed] https://launchpad.net/bugs/1764542 - Assigned to Galen Charlton (gmc)
09:01 csharp remingtron++ # search-fu
09:01 remingtron (searching my email for LP bug updates)
09:01 remingtron :)
09:02 csharp :-)
09:02 csharp another "greybeard" problem :-/
09:03 * csharp stroke his beard and shakes his cane before spitting tobacco juice
09:04 JBoyer What's worse is that I thought I had already made mods33 the default as soon as I saw that bug, but sure enough, nothing like %mods33% in my format anywhere, :(
09:06 bos20k joined #evergreen
09:09 JBoyer HEY YOU GUYS!
09:09 pinesol_green [evergreen|Dan Wells] Forward port 3.0.8 upgrade script - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=a4856e5>
09:09 pinesol_green [evergreen|Dan Wells] Forward port 3.1.2 upgrade script - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=9560b33>
09:09 JBoyer pingest is burning through them now, no errors.
09:10 JBoyer dbwells and remingtron have dinner on me at the hackaway or the conference, whenever I see either of you next.
09:10 rjackson_isl Jboyer++ for deciding not to sleep after frustrating upgrade
09:10 JBoyer Our bacon is saved, etc.
09:11 rjackson_isl JBoyer++ even
09:11 JBoyer remingtron++
09:12 csharp JBoyer++
09:12 JBoyer One small downside, this display fields reingest is taking WAY longer now. ;)
09:12 rjackson_isl remingtron++
09:13 Dyrcona joined #evergreen
09:13 jonadab joined #evergreen
09:15 lsach joined #evergreen
09:25 jvwoolf joined #evergreen
09:30 yboston joined #evergreen
09:46 kmlussier joined #evergreen
10:05 mmorgan1 joined #evergreen
10:10 beanjammin joined #evergreen
10:25 Dyrcona berick: We're going to update OpenSRF and websockets on Wednesday night.
10:25 Dyrcona For Lp 1774703
10:25 pinesol_green Launchpad bug 1774703 in OpenSRF "Websockets processes locked at 100% CPU" [Undecided,Confirmed] https://launchpad.net/bugs/1774703
10:29 csharp Dyrcona++ # pioneering
10:29 berick Dyrcona++
10:38 bshum You all read about Microsoft buying Github?  Hmm
10:39 csharp the ProgrammingHumor subreddit is on fire about it
10:49 mmorgan joined #evergreen
10:58 JBoyer HA! from twitter: "I'd like to add you to my professional network on GitHub."
10:58 Dyrcona :)
10:58 JBoyer Were that the case I'd happily delete my account.
10:59 * Dyrcona never joined LinkedIn.
10:59 * Dyrcona prefers anti-social media, like IRC. :)
10:59 * berick forwards all linkedin email to Dyrcona
10:59 JBoyer Certainly less email.
10:59 Dyrcona :)
11:00 Dyrcona I've seen some tweets about moving private Github repos to Gitlab hosted on AWS or Google....
11:01 Dyrcona People simply aren't thinking clearly if they trust Google and Amazon but not Microsoft. :)
11:02 Dyrcona Google recently dropped the "Don't be evil" thing, but that was always a joke.
11:07 berick i'd love to see the notes from that meeting
11:07 yboston joined #evergreen
11:22 csharp berick: photo from the meeting: https://frinkiac.com/meme/S04E01/827109.j​pg?b64lines=CiBHRU5UTEVNRU4sIFRPIEVWSUwu
11:24 berick csharp++
11:25 berick I see Jimbo is talking, but I hear Kearney's voice
11:29 Dyrcona OK. I'm pre-pared for Wednesday night. All of the branches and files are in place.
11:30 Dyrcona Well, file, not files.... Gonna sed the /etc/apache2-websockets/apache2.conf for trusted origin.
11:33 Dyrcona I'm not planning to do anything about my Github use/account at the moment.
11:52 Jaswinder joined #evergreen
11:54 Jaswinder Hi Guys, I need to get all the records using pcrud call without passing any parameter. Currently, I am using a workaround (see pasteit link). How do I all records using pcrud api without any filter?
11:54 pastebot "Jaswinder" at 64.57.241.14 pasted "Need to get all records using pcrud call" (12 lines) at http://paste.evergreen-ils.org/7143
11:56 khuckins joined #evergreen
11:57 csharp berick: better: https://frinkiac.com/meme/S04E01/827943.j​pg?b64lines=R0VOVExFTUVOLCBUTyBFVklMLg==
11:58 miker Jaswinder: the canonical spelling of that would be: { name => {'!=' => undef} }  ... but you have it right, barring a class with an empty name (which should not happen, ever)
11:59 berick csharp: for reasons I can't explain, that just reminds me of https://frinkiac.com/meme/S08E25/1044192.jp​g?b64lines=VEhBVCBXQVMgU09NRSBHT09EIENPUk4u
11:59 berick (well, i guess because of the corn)
12:00 beanjammin joined #evergreen
12:01 idjit joined #evergreen
12:02 jihpringle joined #evergreen
12:05 csharp berick: ha!
12:10 Jaswinder miker: thanks!
12:15 csharp de867e05
12:15 pinesol_green csharp: [evergreen|miker] Patch from Galen Charlton: - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=de867e0>
12:23 csharp 79c49afe4
12:23 pinesol_green csharp: [evergreen|dbs] Upgrade to MODS 3.3 for ingest and ModsParser.pm; use an explicit format for config.metabib_field in seed data - <http://git.evergreen-ils.org/?p=​Evergreen.git;a=commit;h=79c49af>
12:36 Dyrcona Jaswinder: You don't want to flesh name and label. They are text fields. You only ever want to flesh fields that are foreign keys. You don't want to flesh at all in that example.
12:41 mmorgan1 joined #evergreen
12:45 csharp okay, looking closely at bug 1764542 ...
12:45 pinesol_green Launchpad bug 1764542 in Evergreen "Incorrect format on config.metabib_field insert results in segmentation fault" [High,Confirmed] https://launchpad.net/bugs/1764542 - Assigned to Galen Charlton (gmc)
12:45 sandbergja joined #evergreen
12:46 csharp is it okay practice to have a regexp_replace in a DB upgrade script?  looks like dropping and recreating/repopulating the table (which feels more "correct" to me) would be catastrophic
12:47 csharp i.e., config.metabib_field is referred by everything
12:48 csharp maybe drop/recreate the constraints from all the referring tables?
12:49 csharp or is this one of those that older installations just need to handle "manually" and just move on?
12:52 Jaswinder Dyrcona: Thanks for clarification! How do I only retrieve two fields?
12:52 Dyrcona I don't see anything wrong with the regexp_replace.
12:53 JBoyer just setting format='mods33' where xpath like '%mods33%' and format != 'mods33' is enough for this particular instance, dropping the table would be a huge hassle.
12:54 Dyrcona Jaswinder: You'd have to use a cstore query to get only two fields, but you where you can use cstore is limited.
12:55 JBoyer This is a large enough problem that you're never going to have to make more than 1 correction. (unless we adopt all of the MODS versions between 3.2 and 3.6 at once and for some bizarre reason add field entries for more than one in a single upgrade script)
12:56 Dyrcona Yeah, what JBoyer said. :P
12:56 csharp ok - sounds sane - thanks
12:57 JBoyer It's also been tested just today, if you take my meaning, heh.
12:59 khuckins_ joined #evergreen
13:00 jeffdavis xpath doesn't need to be updated at all so regexp_replace isn't really required, format just needs to be set correctly
13:02 dbs aha, oh yay we have lots of mods32 columns too. Curse our 1.6-ish roots
13:03 Jaswinder Dyrcona: can you share an example to cstore?
13:04 JBoyer I guess I assumed there was some reason that 3.2 was preferred for certain fields. (I can't think of a *good* reason, but still) so a brand new install today is all 33?
13:04 dbs jeffdavis: huh, so it's okay to have format='mods33' and xpath = '//mods32:mods/mods32:subject[local-name(./*[1]) = "topic"] ' for example
13:04 jeffdavis no, format just needs to match the mods version in your xpath
13:04 dbs ahhh, okay
13:04 jeffdavis you'd want mods32 in that example
13:04 dbs important distinction :)
13:04 Dyrcona Jaswinder: There are many in the Evergreen code itself, but have a loo at this: https://wiki.evergreen-ils.org/doku.php​?id=documentation:tutorials:json_query
13:05 dbs all good here then, with our mouldy old mods32
13:06 jeffdavis JBoyer: looks to me like a fresh install contains a mix of mods32 and mods33 cmf entries
13:07 JBoyer Ok. So it is a mix, but I doubt they *need* to be different versions.
13:08 Jaswinder thanks!
13:10 JBoyer (I get it though, any change will need testing and do you try to go all the way to "current" and so on and etc., also tuits...)
13:12 Jaswinder Dyrcona: I can call the db using the cstore but I can't find an example to filter only two columns
13:14 Dyrcona Jaswinder: { "select": { "cmc": [ "name", "label"] "from" : "cmc"} # Off the top of my head.
13:14 Dyrcona oops....
13:14 Dyrcona { "select": { "cmc": [ "name", "label"] }, "from" : "cmc"}
13:15 Dyrcona You might need to convert that to Perl hashref syntax depending on what method you use to run it.
13:20 csharp Jaswinder: dbs: then my branch for bug 1764542 may be off then
13:20 pinesol_green Launchpad bug 1764542 in Evergreen "Incorrect format on config.metabib_field insert results in segmentation fault" [High,Confirmed] https://launchpad.net/bugs/1764542
13:20 csharp I just did _bott_'s regexp_replace and updated the format columns for any with mods32
13:21 csharp (haven't made that change in production, just on a test server)
13:21 Dyrcona jeffdavis: ^^ I think csharp meant to be talking to you. :)
13:21 csharp Dyrcona: yeah - sorry Jaswinder
13:22 bdljohn joined #evergreen
13:24 kmlussier JBoyer: No, it doesn't "need" to be a mix. It's just a mix because the configs that have been around for years have never been updated.
13:25 jeffdavis csharp: I don't think existing cmf entries need to be modified at all, except for the ones added by db update 1100 which may need to be updated to format='mods33' if the default format was something other than that when the update was run
13:26 jeffdavis (also there are other xpath columns in cmf which would potentially need updating - display_xpath etc)
13:27 jeffdavis updating the xpath is probably fine (it worked for bott) but I think it's overkill
13:35 khuckins__ joined #evergreen
13:40 sandbergja I'm trying to wrap my mind around IDL and permacrud, so... checking my understanding:
13:40 sandbergja I can't use open-ils.pcrud or open-ils.fielder to retrieve data from authority.simple_heading, because in the IDL, the controller="open-ils.cstore" (not open-ils.pcrud), right?
13:40 Dyrcona sandbergja: Correct.
13:41 sandbergja Thanks!  And I can't access authority.simple_heading.sort_value through cstore, because the sort_value field is not listed in the IDL?
13:41 Dyrcona That, I'm less sure of. If you pull all of the fields, does it come out?
13:42 Dyrcona But, those are both likely to be bugs, and should be fixed.
13:43 csharp jeffdavis: gotcha - since I'm pre-1100, I'll keep our entries as-is and update my branch
13:43 sandbergja I tried fetching all the fields, and all I get are some foreign keys and the value field
13:44 sandbergja So, I should report a launchpad bug that cstore can't get all those other fields from authority.simple_heading?
13:56 _bott_ overkill++
14:00 csharp ok, I updated my branch (forced) if people are interested
14:02 Dyrcona sandbergja: Yes, I think so.
14:03 sandbergja Dyrcona: thanks, I will.  I appreciate your help!
14:04 csharp ugh - just saw another error :-/
14:14 sandbergja Dyrcona++
14:15 frank_g Hi all, Is in web client any way to could see the patron's photo? And could edit the photo url?
14:16 kmlussier frank_g: There is a field to store photo URLs in the actor,usr table in the database. It will display in the web client, but you can't add the photo via the web client.
14:17 kmlussier It should be the same as the functionality that was available in the xul client.
14:19 frank_g kmlussier: In xul client when I updated new EG versión I added the <tr fmclass='au' fmfield='photo_url'/> in register_table.tt2, Is there any similar modification to could do this in web client?
14:19 Dyrcona Has anyone added a Lp bug to add the ability to upload the patron photo?
14:20 kmlussier yes
14:20 kmlussier bug 1183872
14:20 pinesol_green Launchpad bug 1183872 in Evergreen "Add patron's photo at registry process" [Wishlist,Triaged] https://launchpad.net/bugs/1183872
14:26 Dyrcona frank_g: Probably here Open-ILS/src/templates/staf​f/circ/patron/register.tt2 and here Open-ILS/web/js/ui/default/s​taff/circ/patron/register.js
14:26 Jaswinder Hey, How do I create/insert a record using pcrud/cstore?
14:27 Dyrcona And, if you get that working, adding a branch on the above-mentioned bug would be most helpful for everyone else.
14:37 Dyrcona Jaswinder: You typically need to create the object via Fieldmapper, get a pcrud or cstore transaction and then create the object.
14:38 frank_g What I see is that the photo patron is only visible in web client when the Url contains https instead of http
14:39 Dyrcona Jaswinder: Here's an example function that creates a skeletal MARC record, call number, and copy: http://git.evergreen-ils.org/?p=NCIPServer.git;​a=blob;f=lib/NCIP/ILS/Evergreen.pm;h=3fc3ff1930​9b91c7a8f91acfd9ed6a1cdb268513;hb=master#l2103
14:40 Dyrcona It's in Perl and uses pcrud.
14:40 Dyrcona frank_g: Mixed content warnings/errors?
14:40 miker sandbergja: fwiw, it was intentional that some fields that are only used by db-layer code (stored functions, views) are not exposed in the IDL. but it doesn't hurt to expose them, per se
14:41 miker sandbergja: if it's not, that table should be marked as readonly in the IDL, though. it should only ever be populated or updated by triggers
14:41 frank_g Dyrcona: yes
14:42 Dyrcona frank_g: Such is the modern web browser. You'll need to get the photos via https.
14:43 khuckins joined #evergreen
14:44 Jaswinder Dyrcona: Do I have to use transactions with pcrud?
14:44 sandbergja miker: that makes sense.  My vote would be that the two fields I mentioned be exposed as readonly in the IDL, since -- while they could be generated from other fields -- it's really nice to have them all ready to go.
14:44 Dyrcona Jaswinder: Yes, and with cstore if you want to create, update, or delete.
14:46 Jaswinder So, for both cstore and pcrud, I will have to use transactions?
14:46 Dyrcona Yes.
14:46 Dyrcona Are you writing your code in Perl?
15:54 Jaswinder Hey Guys, I have a table that contains auto increment ids and when I call pcrud to retrieve the id field, I receive undef somehow. However, the rest of the columns are being displayed
15:56 beanjammin joined #evergreen
16:15 khuckins joined #evergreen
16:37 yboston joined #evergreen
16:38 jvwoolf left #evergreen
17:03 csharp bug 1710293 is one of those where I want to help do that kind of work/cleanup, but I'm not exactly sure what to do in a couple of cases - should I just give it a go with the hope that miker or someone will review it and correct me or is it better to just leave it for someone else? :-)
17:03 pinesol_green Launchpad bug 1710293 in Evergreen 3.1 "Remaining chunk/bundle work" [Medium,New] https://launchpad.net/bugs/1710293
17:07 miker csharp: max_chunk_count should def be spelled max_bundle_count, and the stuff in the ->can() tests can go away now (3.0+) ... and max_chunk_size=>0 should also change as described.  I don't think there are any gotchas there (pending testing, obv) if you have the tuits to spare!
17:08 csharp miker: cool - thanks - I'll give it a shot :-)
17:09 mmorgan left #evergreen
17:09 miker csharp++
17:32 gsams I'm excited and hopeful, even though it's like a tiny change, but I just pushed a fix.
17:36 idjit is there a suite of unit tests i should be running prior to submitting pullrequests for things like bug 1724348 or bug 1743801?
17:36 pinesol_green Launchpad bug 1724348 in Evergreen "Web client: set default view not sticky" [Undecided,Confirmed] https://launchpad.net/bugs/1724348
17:36 pinesol_green Launchpad bug 1743801 in Evergreen 3.0 "web client: item status list view display issues" [High,Confirmed] https://launchpad.net/bugs/1743801
18:17 jeffdavis idjit: there are some unit tests in Open-ILS/web/js/ui/default/staff/test but I don't know how well they can catch bugs like those two
18:18 jeffdavis IIRC: `cd Open-ILS/web/js/ui/default/staff && npm run build && npm run test`
18:19 idjit thanks! running now
18:20 idjit well, at least my changes didn't break any of the existing tests. :-)
18:20 idjit jeffdavis++
18:23 idjit hey, while you're here, do you remember bug 1761276? i get a new behavior as of this morning. it pulls up a blank page, with the javascript error "Uncaught TypeError: payload.statusCode is not a function". do you see similar?
18:23 pinesol_green Launchpad bug 1761276 in Evergreen "Odd behaviour when clicking on title hyperlink " [High,Confirmed] https://launchpad.net/bugs/1761276
18:24 idjit (i get this when clicking the title on the "item status" page)
18:28 jeffdavis I haven't seen that one. Is that on master?
18:29 idjit i think so. i'm not sure, was hoping someone else could help confirm.
18:30 beanjammin joined #evergreen
18:31 pinesol_green News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
18:33 jeffdavis I don't have a test env with master handy, will need to set one up first
18:34 idjit ok, nevermind. i must've screwed something up this morning. i'm back on master branch now and i get the reported behavior.
19:09 beanjammin joined #evergreen
19:58 JBoyer joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat