Evergreen ILS Website

IRC log for #evergreen, 2018-06-02

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
06:32 pinesol_green News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
09:06 idjit joined #evergreen
09:29 Dyrcona joined #evergreen
09:30 Dyrcona I know that likely very few people are paying attention, but this is so bizarre I have to share.
09:31 Dyrcona I successfully ran the pgtap tests in t this morning, but I also ran the perl live tests beforehand so the pgtap livet_t test failed.
09:32 Dyrcona Oh... Another case of me being an idiot.....
09:32 Dyrcona I was typing 'So, reload the database....' and realized my problem.
09:32 Dyrcona No tap extension in the db. ;)
09:33 Dyrcona I haz an embarrassed. :)
09:38 dbs don't worry, i was paying attention :)
09:38 Dyrcona hah!
09:39 * dbs upgraded to 3.1.2 last night, still running pingest but most things appear to be in decent shape
09:39 Dyrcona It's funny, 'cause I remembered to create extension pgtap the first time I reloaded the db before running the tests.
09:39 Dyrcona dbs: Cool. Did you apply berick's patch from bug 1774703?
09:39 pinesol_green Launchpad bug 1774703 in OpenSRF "Websockets processes locked at 100% CPU" [Undecided,Confirmed] https://launchpad.net/bugs/1774703
09:40 Dyrcona I'm looking at it this morning and thought I'd run all the tests while I'm at it.
09:42 dbs Dyrcona: I did not, it seemed like too much of a stretch last night
09:42 dbs Trying to be somewhat conservative before taking off for a while
09:43 Dyrcona Makes sense.
09:43 Dyrcona I'm going to put it on a test vm with our custom 3.0 branch and real data and try some things with it.
09:44 Dyrcona If nothing goes horribly wrong, I'll see about installing it in production next week.
09:46 Dyrcona This has been hard to reproduce because it seems to require a load on the server.
09:46 Dyrcona I did find one segmentation fault in the logs on my test vm from Wednesday, 23 May, when we were testing rather heavily for the upgrade.
09:49 Dyrcona S'pose I could write a bot to hammer the OPAC...
09:50 csharp dbs++ # congrats
09:56 Dyrcona csharp: Dunno if you've tried, yet, but the jchampio apache-websocket seems to work. I've applied berick's branch to a master vm, but I'm going to test with real data next.
09:56 dbs Dyrcona++
09:57 dbs right now our only issue seems to be our LDAP authentication, for no apparent reason. *shrug*
09:57 dbs I'm sure that many more will turn up though :)
09:57 dbs berick++
09:59 Dyrcona dbs++
09:59 Dyrcona We've had a lot of tickets on our internal ticket system since the 3.0 upgrade.
09:59 Dyrcona berick++
10:02 dbs well, the load testing at this point is just me (libraries are all closed on the weekend) and even the XUL staff client feels slower as I watch fields populate in the patron screen, etc, so Monday will be the real test
10:03 Dyrcona That could be the ingest.
10:04 Dyrcona I noticed that things seem slower while the authority update happens.
10:04 Dyrcona mixed-tenses--
10:04 dbs Oh I haven't even started the authority update. I fear that is just going to be a terrible mess.
10:05 Dyrcona It took longer than the rest of the upgrade process.
10:05 Dyrcona The pingest finished rather quickly.
10:09 Dyrcona Checked Nagios and we've got 1 apache2-websockets process using 200% cpu on brick 4.
10:09 Dyrcona I'm gonna try to strace it for grins before I kill it.
10:11 Dyrcona So, two in "x32 mode" and 1 waiting on futex. Typical....
10:14 Dyrcona hm... This isn't good. I get a blank login screen on my test vm with real data.
10:16 Dyrcona Ok. It's the origin check.
10:17 Dyrcona At least, I think so.
10:22 Dyrcona Hm... changing ServerName in the configs to the fqdn did not fix it....
10:26 Dyrcona Interesting... I had to add WebSocketOriginCheck Off
10:26 Dyrcona I guess it's using DNS on the server to check the host name. Maybe I should configure dnsmasq on the vm.
10:28 Dyrcona Explains why it just worked on my local vm, though, since DNS lookups work for it.
10:29 dbs yeah, on a freshly dumped & loaded database, pingest took 82 minutes. on the production database, it's been running for about ten hours and has a few more to go
10:29 dbs pretty wild discrepancy :)
10:31 Dyrcona I mainly see a difference with RAM on the server.
10:31 Dyrcona It runs fast on production because that server has enough RAM to load our entire DB.
10:31 Dyrcona It runs slower on the test DB server because it doesn't, and it has more than 1 database.
10:32 dbs .... and I just hit CTRL-C on the pingest process. gah.
10:32 Dyrcona I also find things are faster on the test DB server in databases that have been freshly loaded.
10:32 dbs yeah, our test & prod db servers are identically specced, so it's all about the lack of bloat I think
10:32 Dyrcona Gah! If that was unintentional.
10:33 Dyrcona Yeah.
10:33 dbs yeah, meant CTRL-Shift-C because Google Docs doesn't accept pastes from the middle button clipboard for some reason :/
10:33 dbs but missed the shift
10:34 * dbs runs a few reindexes to reduce index bloat before kicking that off again
10:34 Dyrcona yeah, I've noticed that lack of middle button paste.
10:35 Dyrcona Our test db server used to be a mail server. It has 128Gb of RAM and 6TB of disk space, so a decent choice for storing multiple copies of production data and other, assorted databases.
10:35 Dyrcona Gb should have been GB.
10:36 dbs you could use "GO" for giga-octets
10:37 Dyrcona I wonder if I blow my copy templates our and replace them with someone else's if that will "work" in the web staff client.
10:37 dbs some of these metabib indexes have 1 GB of bloat, heh
10:38 Dyrcona Yes, but the way memory and drive manufacturers count it's probably closer to Gb. :)
10:38 Dyrcona Yeah, I should probably do some maintenance on tables and indexes.
10:43 * dbs peers at pingest -- so are the batches done in purely sequential order, so that I could theoretically pass in a --start-id before the last of the 56 batches out of this batch of 256?
10:44 * dbs happily has an XSL error relating to the reingest of record # 2258047 that would give me a good starting point
10:51 Dyrcona dbs: The batches are in database order unless you pass ids in.
10:52 Dyrcona I don't recall there being an order by on the standard query.
10:53 Dyrcona BTW, bug 1768715 has a branch, now.
10:53 pinesol_green Launchpad bug 1768715 in Evergreen "Add pingest.pl to evergreen" [Wishlist,Confirmed] https://launchpad.net/bugs/1768715
10:54 Dyrcona Oops. I wanted to delete just cat.copy_templates from local storage and ended up deleting all of my local storage for the test vm.
10:54 Dyrcona NBD....I suppose.
10:55 Dyrcona dbs: Turns out there is: ORDER by id ASC
10:55 Dyrcona So, you could just --start-id=.
10:57 * Dyrcona should pay more attention to mouseover. The button I clicked says "Clear All."
11:01 dbs Dyrcona: yeah, I saw the ORDER BY statement, just wasn't 100% sure that the batches were done in parallel sequentially. Makes sense!
11:05 dbs Dyrcona++ # again
11:10 Dyrcona The batches are done sequentially. -- For the logs
11:12 Dyrcona The browse ingest is special and does all the records in a single process since it cannot be run in parallel with itself, so it's an exception, sort of.
11:13 Dyrcona I'm not sure the browse is required for 3.1. I haven't looked and guess it depends where you're starting from.
11:14 Dyrcona It is not required to go from 2.12 to 3.0.
11:14 Dyrcona RE copy templates: I think I'll copy the copy templates from the user who has the most into mine for the sake of testing.
11:27 Dyrcona Hmm... The copy templates were not converted.
11:28 dbs ruh-roh
11:28 * dbs notes that there are Hatch install instructions on Windows and Linux, but nothing for MacOS - has anyone got that particular combo working?
11:29 dbs <-- non-MacOS person, but a few of our libraries are itching for it
11:29 Dyrcona Actually, my copy editor is busted. I'm not getting copy locations, either. I think I shoud have copied templates for the same library, probably.
11:30 * Dyrcona wipes out local and session storage and offline indexed db and starts over. :)
11:31 Dyrcona dbs: I thought I saw some Mac OS instructions, but not sure.
11:31 * Dyrcona may be confusing XUL installation instructions with Hatch.
11:36 dbs yeah, https://evergreen-ils.org/egdownloads/ only mentions Windows and Linux instructions
11:36 Dyrcona Right.
11:37 Dyrcona I guess I am mistaken. I can't find them.
11:38 Dyrcona I assume it is similar to Linux, but adjustments required. :)
11:44 dbs I seem to recall the extension working on Firefox but don't see anything about that either. Hrm.
11:45 dbs https://bugs.launchpad.net/evergreen/+bug/1731922 I guess
11:45 pinesol_green Launchpad bug 1731922 in Evergreen "Firefox add-on for Hatch" [Wishlist,Confirmed]
11:45 Dyrcona Yeah.
11:45 Dyrcona I have not personally tried Hatch much.
11:45 Dyrcona Not lately, at any rate.
11:49 dbs getting a bunch of ""Error with query [SELECT * FROM search.highlight_display_fields( '88110', '''''::HSTORE' ) AS "search.highlight_display_fields" ;]"" errors, I thought they were just for records that had not been reingested but 88110 would have been reingested in the wee hours this morning
11:49 Dyrcona And, copy templates came over this time.
11:50 Dyrcona dbs: How recent is your pingest? Doe it have the display fields ingest?
11:51 dbs hmm
11:51 dbs heh, no
11:52 dbs DAMNIT, that's what I get for just using the version that worked so well last time
11:53 Dyrcona Well, you can run it with --end-id and skip everything but the display fields for speed.
11:53 dbs yeah, that's what I'll do
11:56 dbs Dyrcona++
11:57 Dyrcona YW.
11:57 Dyrcona I wonder if I should be concerned about this: Unable to fopen log file /openils/var/log/gateway.log for writing; logging to standard error
11:57 Dyrcona It appears in my apache2-websockets error.log on my test vm. I think it has to do with a configuration issue, i.e. syslog vs non-syslog.
11:58 Dyrcona Except /openils/var/log/gateway.log is there.
12:00 Dyrcona It appears in most of my logs on this vm.
12:00 * Dyrcona check a production server, but doesn't recall seeing it there.
12:01 Dyrcona Yeah, doesn't happen in production. I'll ignore it. :)
14:11 jeffdavis dbs: we've been seeing a lot of those search.highlight_display_fields errors too but I haven't traced it to an actual display problem so far
14:12 jeffdavis if it worked in testing maybe it's a schema problem with the hstore type?
14:23 dbs jeffdavis: good thought! I guess we'll see :)
14:23 dbs I can see records where the matches are highlighted, and matches where they're not, so it seemed related to the ingest
14:56 dan_learns joined #evergreen
15:12 bshum joined #evergreen
17:00 Dyrcona Interesting... Right click seems to not be working for me. The menu pops up just long enough to see it, but then it does what a left click does.
17:00 Dyrcona This is with Chromium on Ubuntu 18.04.
17:02 Dyrcona That was on a link entry in an egGrid.
17:05 Dyrcona Ah.. That is only happening for me on links. If I click on a non-link field in the grid, then I get the left-click menu.
17:10 Dyrcona Dunno if that is worthy of a bug or not.
17:10 Dyrcona And, no, I haven't spent my whole day on Evergreen. :)
17:10 * Dyrcona signs out.
18:14 Dyrcona joined #evergreen
18:14 Dyrcona OK, not my whole day, just most of it.
18:32 pinesol_green News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
18:41 dbs interesting, a display-only reingest via pingest doesn't populate metabib.display_entry, but a full reingest does
18:41 * dbs adds a custom '''$where .= " AND NOT EXISTS (SELECT source FROM metabib.display_entry WHERE source = biblio.record_entry.id)";
18:42 dbs to reingest only the records with an empty metabib.display_entry (those do seem to be the cause of the search.highlight_display_field errors)
18:46 * Dyrcona just had the funn of oom-killer on the honkin' postgres db server: 768GB of RAM is not enough. :(
18:47 Dyrcona dbs: Hm... Not sure why that is. pingest uses db functions, so must be a bug in the db somewhere.
18:47 Dyrcona or pingest is using the wrong function.
18:54 dbs Maybe it needs to do the browse ingest as part of the ingest?
18:55 * dbs tries this out
18:56 Dyrcona I don't think so. My understanding is display and browse are supposed to be orthogonal to each other, but I trust my understanding less and less these days.
18:59 dbs Just doing a "SELECT metabib.reingest_metabib_field_entries(####)" results in metabib.display_entry getting populated
19:03 dbs wow, oom-killer on a machine like that? wow.
19:03 Dyrcona OK. I've got too much going on. I forgot how that function works.
19:04 Dyrcona I think it was a report, but I don't know which of the 3 that were running at the time, because it looks like the query for the culprit PID was not logged.
19:05 Dyrcona dbs: You're right about metabib.reingest_metabib_field_entries... But, it should still work if you skip everything but display.
19:06 Dyrcona If not, then I'd say that's a bug in the db function.
19:13 dbs yeah. right now if you skip everything but display in pingest, it skips doing anything (need to add && skip_display to the if condition before reingest)
19:13 dbs but even if you add that && skip_display test, so it runs reingest with just display enabled, it goes way too fast :)
19:15 Dyrcona OIC: bug on line 205 of pingest.pl. dbs++
19:16 Dyrcona And on line 274!
19:16 Dyrcona So, guess I'll fix both the github repo and the branch for Lp.
19:18 dbs cool, I was going to push something later but kind of focused right now :)
19:19 dbs SELECT metabib.reingest_metabib_field_entries(1013957, TRUE, FALSE, TRUE, TRUE); -- worked, so *???*
19:21 Dyrcona dbs: I pushed the fix to the github repo.
19:21 dbs Ah, that's what line 274 would have caused. Perfect!
19:21 dbs thank you!
19:21 Dyrcona Thank you!
19:23 Dyrcona I never tested with everything but display skipped, I guess.
19:37 Dyrcona hmm. One drawback of adding pingest.pl to evergreen is the way I've done the branch by copying the latest version of the file.
19:37 Dyrcona It loses the history of commits from berick, Bmagic, and jeff....
19:37 Dyrcona Maybe I should redo it?
19:39 Dyrcona Not so easy moving history from one repo to another...
19:41 Dyrcona Sure, there are one-liners, but not when you want to rename the file...
19:42 Dyrcona Hm... maybe if I rename it in the patch file?
19:45 Dyrcona That just might work!
19:45 Dyrcona :)
19:48 Dyrcona sed to relocate the file on the fly!
19:52 Dyrcona And, of course, 1 patch won't apply because conflicts
19:54 Dyrcona Maybe if I reorder things in the patch files...
19:56 Dyrcona yes,  a move of 1 patch and a small edit to the one that was technically before...
19:57 Dyrcona or, there's an easier solution... jsut edit the second patch a bit.
20:14 * Dyrcona edits several of the commit messages and adds some missing signoffs.
20:44 Dyrcona Think I earned another "dan" in git fu. :)
20:45 Dyrcona Well, good [insert local time of day]!

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat