| Time |
Nick |
Message |
| 06:32 |
pinesol_green |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 09:06 |
|
idjit joined #evergreen |
| 09:29 |
|
Dyrcona joined #evergreen |
| 09:30 |
Dyrcona |
I know that likely very few people are paying attention, but this is so bizarre I have to share. |
| 09:31 |
Dyrcona |
I successfully ran the pgtap tests in t this morning, but I also ran the perl live tests beforehand so the pgtap livet_t test failed. |
| 09:32 |
Dyrcona |
Oh... Another case of me being an idiot..... |
| 09:32 |
Dyrcona |
I was typing 'So, reload the database....' and realized my problem. |
| 09:32 |
Dyrcona |
No tap extension in the db. ;) |
| 09:33 |
Dyrcona |
I haz an embarrassed. :) |
| 09:38 |
dbs |
don't worry, i was paying attention :) |
| 09:38 |
Dyrcona |
hah! |
| 09:39 |
* dbs |
upgraded to 3.1.2 last night, still running pingest but most things appear to be in decent shape |
| 09:39 |
Dyrcona |
It's funny, 'cause I remembered to create extension pgtap the first time I reloaded the db before running the tests. |
| 09:39 |
Dyrcona |
dbs: Cool. Did you apply berick's patch from bug 1774703? |
| 09:39 |
pinesol_green |
Launchpad bug 1774703 in OpenSRF "Websockets processes locked at 100% CPU" [Undecided,Confirmed] https://launchpad.net/bugs/1774703 |
| 09:40 |
Dyrcona |
I'm looking at it this morning and thought I'd run all the tests while I'm at it. |
| 09:42 |
dbs |
Dyrcona: I did not, it seemed like too much of a stretch last night |
| 09:42 |
dbs |
Trying to be somewhat conservative before taking off for a while |
| 09:43 |
Dyrcona |
Makes sense. |
| 09:43 |
Dyrcona |
I'm going to put it on a test vm with our custom 3.0 branch and real data and try some things with it. |
| 09:44 |
Dyrcona |
If nothing goes horribly wrong, I'll see about installing it in production next week. |
| 09:46 |
Dyrcona |
This has been hard to reproduce because it seems to require a load on the server. |
| 09:46 |
Dyrcona |
I did find one segmentation fault in the logs on my test vm from Wednesday, 23 May, when we were testing rather heavily for the upgrade. |
| 09:49 |
Dyrcona |
S'pose I could write a bot to hammer the OPAC... |
| 09:50 |
csharp |
dbs++ # congrats |
| 09:56 |
Dyrcona |
csharp: Dunno if you've tried, yet, but the jchampio apache-websocket seems to work. I've applied berick's branch to a master vm, but I'm going to test with real data next. |
| 09:56 |
dbs |
Dyrcona++ |
| 09:57 |
dbs |
right now our only issue seems to be our LDAP authentication, for no apparent reason. *shrug* |
| 09:57 |
dbs |
I'm sure that many more will turn up though :) |
| 09:57 |
dbs |
berick++ |
| 09:59 |
Dyrcona |
dbs++ |
| 09:59 |
Dyrcona |
We've had a lot of tickets on our internal ticket system since the 3.0 upgrade. |
| 09:59 |
Dyrcona |
berick++ |
| 10:02 |
dbs |
well, the load testing at this point is just me (libraries are all closed on the weekend) and even the XUL staff client feels slower as I watch fields populate in the patron screen, etc, so Monday will be the real test |
| 10:03 |
Dyrcona |
That could be the ingest. |
| 10:04 |
Dyrcona |
I noticed that things seem slower while the authority update happens. |
| 10:04 |
Dyrcona |
mixed-tenses-- |
| 10:04 |
dbs |
Oh I haven't even started the authority update. I fear that is just going to be a terrible mess. |
| 10:05 |
Dyrcona |
It took longer than the rest of the upgrade process. |
| 10:05 |
Dyrcona |
The pingest finished rather quickly. |
| 10:09 |
Dyrcona |
Checked Nagios and we've got 1 apache2-websockets process using 200% cpu on brick 4. |
| 10:09 |
Dyrcona |
I'm gonna try to strace it for grins before I kill it. |
| 10:11 |
Dyrcona |
So, two in "x32 mode" and 1 waiting on futex. Typical.... |
| 10:14 |
Dyrcona |
hm... This isn't good. I get a blank login screen on my test vm with real data. |
| 10:16 |
Dyrcona |
Ok. It's the origin check. |
| 10:17 |
Dyrcona |
At least, I think so. |
| 10:22 |
Dyrcona |
Hm... changing ServerName in the configs to the fqdn did not fix it.... |
| 10:26 |
Dyrcona |
Interesting... I had to add WebSocketOriginCheck Off |
| 10:26 |
Dyrcona |
I guess it's using DNS on the server to check the host name. Maybe I should configure dnsmasq on the vm. |
| 10:28 |
Dyrcona |
Explains why it just worked on my local vm, though, since DNS lookups work for it. |
| 10:29 |
dbs |
yeah, on a freshly dumped & loaded database, pingest took 82 minutes. on the production database, it's been running for about ten hours and has a few more to go |
| 10:29 |
dbs |
pretty wild discrepancy :) |
| 10:31 |
Dyrcona |
I mainly see a difference with RAM on the server. |
| 10:31 |
Dyrcona |
It runs fast on production because that server has enough RAM to load our entire DB. |
| 10:31 |
Dyrcona |
It runs slower on the test DB server because it doesn't, and it has more than 1 database. |
| 10:32 |
dbs |
.... and I just hit CTRL-C on the pingest process. gah. |
| 10:32 |
Dyrcona |
I also find things are faster on the test DB server in databases that have been freshly loaded. |
| 10:32 |
dbs |
yeah, our test & prod db servers are identically specced, so it's all about the lack of bloat I think |
| 10:32 |
Dyrcona |
Gah! If that was unintentional. |
| 10:33 |
Dyrcona |
Yeah. |
| 10:33 |
dbs |
yeah, meant CTRL-Shift-C because Google Docs doesn't accept pastes from the middle button clipboard for some reason :/ |
| 10:33 |
dbs |
but missed the shift |
| 10:34 |
* dbs |
runs a few reindexes to reduce index bloat before kicking that off again |
| 10:34 |
Dyrcona |
yeah, I've noticed that lack of middle button paste. |
| 10:35 |
Dyrcona |
Our test db server used to be a mail server. It has 128Gb of RAM and 6TB of disk space, so a decent choice for storing multiple copies of production data and other, assorted databases. |
| 10:35 |
Dyrcona |
Gb should have been GB. |
| 10:36 |
dbs |
you could use "GO" for giga-octets |
| 10:37 |
Dyrcona |
I wonder if I blow my copy templates our and replace them with someone else's if that will "work" in the web staff client. |
| 10:37 |
dbs |
some of these metabib indexes have 1 GB of bloat, heh |
| 10:38 |
Dyrcona |
Yes, but the way memory and drive manufacturers count it's probably closer to Gb. :) |
| 10:38 |
Dyrcona |
Yeah, I should probably do some maintenance on tables and indexes. |
| 10:43 |
* dbs |
peers at pingest -- so are the batches done in purely sequential order, so that I could theoretically pass in a --start-id before the last of the 56 batches out of this batch of 256? |
| 10:44 |
* dbs |
happily has an XSL error relating to the reingest of record # 2258047 that would give me a good starting point |
| 10:51 |
Dyrcona |
dbs: The batches are in database order unless you pass ids in. |
| 10:52 |
Dyrcona |
I don't recall there being an order by on the standard query. |
| 10:53 |
Dyrcona |
BTW, bug 1768715 has a branch, now. |
| 10:53 |
pinesol_green |
Launchpad bug 1768715 in Evergreen "Add pingest.pl to evergreen" [Wishlist,Confirmed] https://launchpad.net/bugs/1768715 |
| 10:54 |
Dyrcona |
Oops. I wanted to delete just cat.copy_templates from local storage and ended up deleting all of my local storage for the test vm. |
| 10:54 |
Dyrcona |
NBD....I suppose. |
| 10:55 |
Dyrcona |
dbs: Turns out there is: ORDER by id ASC |
| 10:55 |
Dyrcona |
So, you could just --start-id=. |
| 10:57 |
* Dyrcona |
should pay more attention to mouseover. The button I clicked says "Clear All." |
| 11:01 |
dbs |
Dyrcona: yeah, I saw the ORDER BY statement, just wasn't 100% sure that the batches were done in parallel sequentially. Makes sense! |
| 11:05 |
dbs |
Dyrcona++ # again |
| 11:10 |
Dyrcona |
The batches are done sequentially. -- For the logs |
| 11:12 |
Dyrcona |
The browse ingest is special and does all the records in a single process since it cannot be run in parallel with itself, so it's an exception, sort of. |
| 11:13 |
Dyrcona |
I'm not sure the browse is required for 3.1. I haven't looked and guess it depends where you're starting from. |
| 11:14 |
Dyrcona |
It is not required to go from 2.12 to 3.0. |
| 11:14 |
Dyrcona |
RE copy templates: I think I'll copy the copy templates from the user who has the most into mine for the sake of testing. |
| 11:27 |
Dyrcona |
Hmm... The copy templates were not converted. |
| 11:28 |
dbs |
ruh-roh |
| 11:28 |
* dbs |
notes that there are Hatch install instructions on Windows and Linux, but nothing for MacOS - has anyone got that particular combo working? |
| 11:29 |
dbs |
<-- non-MacOS person, but a few of our libraries are itching for it |
| 11:29 |
Dyrcona |
Actually, my copy editor is busted. I'm not getting copy locations, either. I think I shoud have copied templates for the same library, probably. |
| 11:30 |
* Dyrcona |
wipes out local and session storage and offline indexed db and starts over. :) |
| 11:31 |
Dyrcona |
dbs: I thought I saw some Mac OS instructions, but not sure. |
| 11:31 |
* Dyrcona |
may be confusing XUL installation instructions with Hatch. |
| 11:36 |
dbs |
yeah, https://evergreen-ils.org/egdownloads/ only mentions Windows and Linux instructions |
| 11:36 |
Dyrcona |
Right. |
| 11:37 |
Dyrcona |
I guess I am mistaken. I can't find them. |
| 11:38 |
Dyrcona |
I assume it is similar to Linux, but adjustments required. :) |
| 11:44 |
dbs |
I seem to recall the extension working on Firefox but don't see anything about that either. Hrm. |
| 11:45 |
dbs |
https://bugs.launchpad.net/evergreen/+bug/1731922 I guess |
| 11:45 |
pinesol_green |
Launchpad bug 1731922 in Evergreen "Firefox add-on for Hatch" [Wishlist,Confirmed] |
| 11:45 |
Dyrcona |
Yeah. |
| 11:45 |
Dyrcona |
I have not personally tried Hatch much. |
| 11:45 |
Dyrcona |
Not lately, at any rate. |
| 11:49 |
dbs |
getting a bunch of ""Error with query [SELECT * FROM search.highlight_display_fields( '88110', '''''::HSTORE' ) AS "search.highlight_display_fields" ;]"" errors, I thought they were just for records that had not been reingested but 88110 would have been reingested in the wee hours this morning |
| 11:49 |
Dyrcona |
And, copy templates came over this time. |
| 11:50 |
Dyrcona |
dbs: How recent is your pingest? Doe it have the display fields ingest? |
| 11:51 |
dbs |
hmm |
| 11:51 |
dbs |
heh, no |
| 11:52 |
dbs |
DAMNIT, that's what I get for just using the version that worked so well last time |
| 11:53 |
Dyrcona |
Well, you can run it with --end-id and skip everything but the display fields for speed. |
| 11:53 |
dbs |
yeah, that's what I'll do |
| 11:56 |
dbs |
Dyrcona++ |
| 11:57 |
Dyrcona |
YW. |
| 11:57 |
Dyrcona |
I wonder if I should be concerned about this: Unable to fopen log file /openils/var/log/gateway.log for writing; logging to standard error |
| 11:57 |
Dyrcona |
It appears in my apache2-websockets error.log on my test vm. I think it has to do with a configuration issue, i.e. syslog vs non-syslog. |
| 11:58 |
Dyrcona |
Except /openils/var/log/gateway.log is there. |
| 12:00 |
Dyrcona |
It appears in most of my logs on this vm. |
| 12:00 |
* Dyrcona |
check a production server, but doesn't recall seeing it there. |
| 12:01 |
Dyrcona |
Yeah, doesn't happen in production. I'll ignore it. :) |
| 14:11 |
jeffdavis |
dbs: we've been seeing a lot of those search.highlight_display_fields errors too but I haven't traced it to an actual display problem so far |
| 14:12 |
jeffdavis |
if it worked in testing maybe it's a schema problem with the hstore type? |
| 14:23 |
dbs |
jeffdavis: good thought! I guess we'll see :) |
| 14:23 |
dbs |
I can see records where the matches are highlighted, and matches where they're not, so it seemed related to the ingest |
| 14:56 |
|
dan_learns joined #evergreen |
| 15:12 |
|
bshum joined #evergreen |
| 17:00 |
Dyrcona |
Interesting... Right click seems to not be working for me. The menu pops up just long enough to see it, but then it does what a left click does. |
| 17:00 |
Dyrcona |
This is with Chromium on Ubuntu 18.04. |
| 17:02 |
Dyrcona |
That was on a link entry in an egGrid. |
| 17:05 |
Dyrcona |
Ah.. That is only happening for me on links. If I click on a non-link field in the grid, then I get the left-click menu. |
| 17:10 |
Dyrcona |
Dunno if that is worthy of a bug or not. |
| 17:10 |
Dyrcona |
And, no, I haven't spent my whole day on Evergreen. :) |
| 17:10 |
* Dyrcona |
signs out. |
| 18:14 |
|
Dyrcona joined #evergreen |
| 18:14 |
Dyrcona |
OK, not my whole day, just most of it. |
| 18:32 |
pinesol_green |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 18:41 |
dbs |
interesting, a display-only reingest via pingest doesn't populate metabib.display_entry, but a full reingest does |
| 18:41 |
* dbs |
adds a custom '''$where .= " AND NOT EXISTS (SELECT source FROM metabib.display_entry WHERE source = biblio.record_entry.id)"; |
| 18:42 |
dbs |
to reingest only the records with an empty metabib.display_entry (those do seem to be the cause of the search.highlight_display_field errors) |
| 18:46 |
* Dyrcona |
just had the funn of oom-killer on the honkin' postgres db server: 768GB of RAM is not enough. :( |
| 18:47 |
Dyrcona |
dbs: Hm... Not sure why that is. pingest uses db functions, so must be a bug in the db somewhere. |
| 18:47 |
Dyrcona |
or pingest is using the wrong function. |
| 18:54 |
dbs |
Maybe it needs to do the browse ingest as part of the ingest? |
| 18:55 |
* dbs |
tries this out |
| 18:56 |
Dyrcona |
I don't think so. My understanding is display and browse are supposed to be orthogonal to each other, but I trust my understanding less and less these days. |
| 18:59 |
dbs |
Just doing a "SELECT metabib.reingest_metabib_field_entries(####)" results in metabib.display_entry getting populated |
| 19:03 |
dbs |
wow, oom-killer on a machine like that? wow. |
| 19:03 |
Dyrcona |
OK. I've got too much going on. I forgot how that function works. |
| 19:04 |
Dyrcona |
I think it was a report, but I don't know which of the 3 that were running at the time, because it looks like the query for the culprit PID was not logged. |
| 19:05 |
Dyrcona |
dbs: You're right about metabib.reingest_metabib_field_entries... But, it should still work if you skip everything but display. |
| 19:06 |
Dyrcona |
If not, then I'd say that's a bug in the db function. |
| 19:13 |
dbs |
yeah. right now if you skip everything but display in pingest, it skips doing anything (need to add && skip_display to the if condition before reingest) |
| 19:13 |
dbs |
but even if you add that && skip_display test, so it runs reingest with just display enabled, it goes way too fast :) |
| 19:15 |
Dyrcona |
OIC: bug on line 205 of pingest.pl. dbs++ |
| 19:16 |
Dyrcona |
And on line 274! |
| 19:16 |
Dyrcona |
So, guess I'll fix both the github repo and the branch for Lp. |
| 19:18 |
dbs |
cool, I was going to push something later but kind of focused right now :) |
| 19:19 |
dbs |
SELECT metabib.reingest_metabib_field_entries(1013957, TRUE, FALSE, TRUE, TRUE); -- worked, so *???* |
| 19:21 |
Dyrcona |
dbs: I pushed the fix to the github repo. |
| 19:21 |
dbs |
Ah, that's what line 274 would have caused. Perfect! |
| 19:21 |
dbs |
thank you! |
| 19:21 |
Dyrcona |
Thank you! |
| 19:23 |
Dyrcona |
I never tested with everything but display skipped, I guess. |
| 19:37 |
Dyrcona |
hmm. One drawback of adding pingest.pl to evergreen is the way I've done the branch by copying the latest version of the file. |
| 19:37 |
Dyrcona |
It loses the history of commits from berick, Bmagic, and jeff.... |
| 19:37 |
Dyrcona |
Maybe I should redo it? |
| 19:39 |
Dyrcona |
Not so easy moving history from one repo to another... |
| 19:41 |
Dyrcona |
Sure, there are one-liners, but not when you want to rename the file... |
| 19:42 |
Dyrcona |
Hm... maybe if I rename it in the patch file? |
| 19:45 |
Dyrcona |
That just might work! |
| 19:45 |
Dyrcona |
:) |
| 19:48 |
Dyrcona |
sed to relocate the file on the fly! |
| 19:52 |
Dyrcona |
And, of course, 1 patch won't apply because conflicts |
| 19:54 |
Dyrcona |
Maybe if I reorder things in the patch files... |
| 19:56 |
Dyrcona |
yes, a move of 1 patch and a small edit to the one that was technically before... |
| 19:57 |
Dyrcona |
or, there's an easier solution... jsut edit the second patch a bit. |
| 20:14 |
* Dyrcona |
edits several of the commit messages and adds some missing signoffs. |
| 20:44 |
Dyrcona |
Think I earned another "dan" in git fu. :) |
| 20:45 |
Dyrcona |
Well, good [insert local time of day]! |
