Evergreen ILS Website

IRC log for #evergreen, 2014-12-08

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
00:20 nhilton joined #evergreen
00:32 jeff dbs++
04:54 pinesol_green Incoming from qatests: Test Success - http://testing.evergreen-ils.org/~live/test.html <http://testing.evergreen-ils.org/~live/test.html>
06:31 chatley joined #evergreen
07:25 wsmoak joined #evergreen
07:42 TaraC joined #evergreen
07:54 remingtron joined #evergreen
07:54 jboyer-isl joined #evergreen
08:06 rjackson-isl joined #evergreen
08:19 ericar joined #evergreen
08:22 graced joined #evergreen
08:22 mrpeters joined #evergreen
08:35 collum joined #evergreen
08:35 Shae joined #evergreen
08:36 Dyrcona joined #evergreen
08:43 akilsdonk joined #evergreen
09:03 dbs Hey Dyrcona, ever see "Failed to spawn a child at 166" in pingest.pl right after it finishes the first MAXCHILD + 1 processes? I've seen that with maxchild = 8 and maxchild = 4 now.
09:04 Dyrcona No, I have not seen that.
09:05 Dyrcona I've never seen it fail, but I have only run it once or twice since I recently made some changes to it.
09:05 dbs huh. stock ubuntu precise system. oh well, I can always parellelize it in bash if need be :)
09:06 Dyrcona Does that happen every time you run it?
09:06 dbs I've only run it twice, but so far 100% of the time :)
09:06 * Dyrcona has been seeing odd stuff with recent versions of Perl fwiw.
09:07 dbs Cool. I'm going to move from work-at-home to work-at-work mode but will be paying attention soon
09:07 Dyrcona OK.
09:08 * Dyrcona was referring to weird SpamAssassin segfaults with Perl 5.18 on a FreeBSD box.
09:09 Dyrcona I love it when you have a problem, google it, and the only results are someone else asking about the very same problem a month ago with no resolution.
09:09 Dyrcona Really, spamd segfaults on certain messages with a certain option. spamassassin itself has no problem with the messages, but anyway.
09:10 Dyrcona S'pose I give pingest a whirl on my dev vm.
09:21 csharp Dyrcona: http://xkcd.com/979/ is what comes to mind for me
09:22 csharp that's especially true when trying to track down weird kernel errors that only happen on specific model servers, etc.
09:24 csharp I've run pingest.pl recently on a 12.04 server with no issues, fwiw
09:24 maryj joined #evergreen
09:26 Dyrcona csharp++
09:30 tsbere I dunno.....I find that half the time when I look for weird stuff that nobody else seems to see all I find are previous iterations of myself looking for help with the same thing.
09:30 yboston joined #evergreen
09:31 Dyrcona heh.
09:41 csharp tsbere: I've seen that too ;-)
09:50 mtcarlson joined #evergreen
10:19 jeff csharp: the fun ones are the kernel errors on specific kernel + OS + hardware where you find that one of the other people in the ticket is someone you know who works across town
10:20 csharp heh
10:57 dcook__ joined #evergreen
10:59 Dyrcona Whee! Always fun the morning after an upgrade.
11:00 jeff indeed.
11:01 tsbere Fun! https://bugs.launchpad.net/evergreen/+bug/1400376
11:02 pinesol_green Launchpad bug 1400376 in Evergreen "Circ errors due to issue with metabib.record_attr_fla view" (affected: 1, heat: 6) [High,New]
11:02 * tsbere missed the t in flat, but just corrected it on Launchpad. <_< Ooops.
11:06 bshum Are those circ failures coming from use of circ by marc types?
11:07 * bshum is wondering why he hasn't seen those problems before and assumes that aspect might be the difference.
11:07 tsbere bshum: We are seeing them for really brief bib records used for virtual catalog copies
11:08 tsbere Though I can't say much more than that because we were only running the code in production since last night.
11:08 tsbere Oh, and we pushed the new view into production to stop the errors and are waiting to see if that makes new issues or not
11:09 jihpringle joined #evergreen
11:10 Dyrcona MARC types are not involved in the matchpoints that come up.
11:11 Dyrcona Specifically, our matchpoints 2 and 314, not that that tells *you* anything useful. :)
11:11 RoganH joined #evergreen
11:20 bshum Hmm, maybe vr_format?
11:20 bshum Well anyways
11:21 * bshum is looking for these NULLs
11:24 Dyrcona These records are missing lots of data. They're basically a leader, a 245, and an author, and that's about it.
11:25 Dyrcona They are created by issa.pl: https://github.com/Dyrcona/issa
11:27 tsbere bshum: The problem isn't what the matchpoints use, the error happens before it even loads the matchpoint data
11:27 Dyrcona Yep.
11:28 Dyrcona Also, not even an author for these.
11:28 bshum Gotcha... so I'd need to have some truly terrible bib records to try replicating some of that.
11:28 jboyer-isl So is it looking like the lack of 008's + the left joins are causing the issue, or is that still an early assumption to make?
11:28 vlewis joined #evergreen
11:28 Dyrcona jboyer-isl: It looks like it is the left joins, definitely, as for what MARC tags are missing, I can't say.
11:29 dbs Would the alternative to removing the left joins be COALESCE()ing the resulting nulls with some reasonable default?
11:29 csharp sounds like we'll have similar issues then - we have lots of sh*tty data
11:29 Dyrcona The records in question are createed with a leader, 005, 082, and 245 only.
11:29 * dbs is very much in the early stages of the brave new metabib world
11:30 csharp dbs: that sounds like a good approach to me
11:31 RoganH csharp: It's not bad data, it's legacy opportunities for improvement.
11:31 csharp RoganH++
11:31 csharp @quote add < RoganH> It's not bad data, it's legacy opportunities for improvement.
11:31 pinesol_green csharp: The operation succeeded.  Quote #100 added.
11:32 csharp (balloons burst down and fireworks explode upon reaching 100 quote)
11:32 csharp s/100/100th/
11:35 Dyrcona I'm not sure can get a reasonable default. The join criteria is a any from a list, so a reasonable default for one field won't make sense for others.
11:35 eeevil tsbere: I commented on the ticket, but it seems that hstore has no problem storing nulls
11:36 tsbere eeevil: Oh, the problem isn't *storing* nulls. The problem is null *keys*
11:36 dbs PG 9.1 vs. 9.3 or something?
11:38 eeevil tsbere: ah, you have records with no entry in record_attr_vector_list, you mean. that was not at all clear from the ticket, and seems unlikely. they'd at least have a title, right? (for the titlesort uncontrolled attr)
11:39 tsbere eeevil: Also, I inspected various levels of the code (by walking back through to see where things may have gone wrong) and the older version of the view seems to be more in line with my updated version than the "improved" version.
11:41 eeevil tsbere: "more in line" doesn't really provide much information. and why are you dismissing a performance improvement?
11:41 Dyrcona eeevil: I think the question is what is the purpose of the left join in the view?
11:42 tsbere eeevil: I am not dismissing the improvement. If I were I would have reverted to the original. The original joins the vector list with a normal join to a "combine two tables" view, basically. The left joins in the improved one causes possible nulls to show up where the original could not have generated them, as far as I can tell, so removing the left portion brings the output back to what was expected.
11:42 eeevil tsbere: but really, if your records have a 245 but no entry in the vector list table, I'd suggest looking at that. that is definitely not a good position to be in
11:45 eeevil Dyrcona: to get "value, or NULL if not set" for each record attr def out of metabib.record_attr_flat
11:45 tsbere eeevil: The left join doesn't do that. You get null attrs too.
11:46 tsbere attr and value come from the left joined table
11:46 eeevil sure they do
11:47 eeevil so your problem stems from not having a coded_value_map-side value, at all, from those records. is that correct?
11:48 tsbere The problem stems from "we are getting at least one null when the previous, pre-peformance-improvement code would never have spit a null out" regardless of *where* that null comes from. Removing the left joins brings the "faster" (I haven't tested that myself) code back into not outputting those nulls.
11:49 eeevil you're seeing, what, a row for titlesort from uncontrolled, and a second row with nulls for both attr and value. is that right?
11:49 Dyrcona They have mravl entries.
11:50 Dyrcona And, what purpose does returning nulls serve, other than to cause a database error?
11:50 eeevil Dyrcona: thanks. I'm guessing the int array has only negative values in them?
11:50 tsbere eeevil: The ones I am seeing in my own tests have only *positive* values.
11:51 * tsbere isn't going all that deep, mind you, just trying to find ones that output nulls
11:51 eeevil tsbere: so they have no title or author fields?
11:51 mllewellyn joined #evergreen
11:52 eeevil Dyrcona: the left join was key to helping the planner with query optimization in my tests with large data sets
11:52 eeevil when going through multiple views
11:52 tsbere eeevil: Looks like at least one of them has a title.
11:53 eeevil tsbere: then it should have a negative value in the int array -- or the title wasn't found by the attr
11:53 eeevil attr def, that is
11:55 tsbere eeevil: Given that this apparently happens with real world data either the record_attr_flat view needs to never spit out nulls (as it hadn't before) or the next view up the chain needs to not dump the output directly into hstore as keys.
11:56 eeevil tsbere: see my comments on the bug
12:00 tsbere eeevil: I will also point out I am not finding a significant difference in plans between the left join and normal join versions of the view when I run things through explain analyse.
12:01 eeevil tsbere: what data set size are you testing with
12:01 tsbere eeevil: I am poking around in a copy of our production system at the moment.
12:03 tsbere eeevil: Really, the only difference I can see in the plans themselves are "Nested Loop" vs "Nested Loop Left Join"
12:03 eeevil well, you can either believe that I'm not lying, or you can test all code that uses either of those views (and the rec_descriptor view), I guess
12:05 tsbere eeevil: I will point out that I get *very* different plans from the old version of record_attr_flat. I just don't get any improvements when I remove the LEFT from the joins in the current one.
12:06 tsbere eeevil: AKA, I see improvements in the current one, but I don't see the LEFT portion of the joins aiding those improvements.
12:07 eeevil a left join opens up more options for plans depending on the field from the view that is joined in a larger query. that expanded set of plan options was necessary to make certain queries fast. it was months ago that I was working on this, so I don't recall (or have time to dig up) the details right now
12:11 tsbere eeevil: Well, I am not seeing those differences as I go through iterations of queries I can find happening elsewhere as I walk up from one level to the next. Major improvements on not using full_attr_id_map, but nothing on left join vs regular join.
12:15 eeevil tsbere: is the filter in the next view up really so offensive to you that you'd rather try to find and test every possible way the views in question could be used?
12:17 mrpeters Fix for bug 1319964 pushed to http://git.evergreen-ils.org/?p=working/Ev​ergreen.git;a=shortlog;h=refs/heads/user/m​rpeters/lp1319964_fix_summaries_ampersand -- it's an old annoying bug that most people just fixed locally i think, but its been busted in master for a while
12:17 tsbere eeevil: I am using your own "keep things outputting the same thing they did" gig - The version you replaced does not, in any of my tests, output null attr columns. Your replacement does, but is much faster in all my tests. My change to your replacement maintains the same performance boost, from what I can see, but without the null attr columns.
12:17 pinesol_green Launchpad bug 1319964 in Evergreen "double-escaped entity in "Summaries & More" label" (affected: 3, heat: 18) [Low,New] https://launchpad.net/bugs/1319964 - Assigned to Athira S (athirasnamby)
12:18 tsbere eeevil: As such, unless you can point at a good reason for the LEFT JOIN version to remain and spitting out null attrs I am against the view spitting out null attrs.
12:21 phasefx joined #evergreen
12:21 eeevil tsbere: here's my good reason: it was needed to get bshum's server to be able to search in a reasonable amount of time, because the plans became not-insane.  do you believe that I am lying to you? does the solution i've suggested to your problem not work?
12:23 eeevil and I would absolutely love to be able to not change the output of the view. but not at the cost of killing performance to the point of "broken for the user"
12:24 bshum I haven't really tested either of the fixes (I'm not sure I can really test it without crummier bib records) but I think it'll be good to look at what has been suggested thus far.
12:24 tsbere eeevil: I have no problem with changing the output of the view if you can provide information on where, exactly, things go wrong with the output remaining the same. The fact that I get massive performance boosts with the union version, with or without the left joins, says it works. I am arguing that the left join part does *not seem to matter* on that front.
12:24 bshum At a cursory glance, I think eeevil's treatment for the check for not null attr columns looks sane to me.
12:24 tsbere eeevil: If you can dig up evidence that I can't for the left join portion mattering I am willing to look at it.
12:26 eeevil tsbere: I've provided you with a code-local solution, and you're trying to change non-local code. you're the one that needs to provide "proof" that yours is a better solution.
12:26 eeevil and you have not answered either of my questions:  do you believe that I am lying to you?  does the solution i've suggested to your problem not work?
12:27 tsbere eeevil: At this point I will refer you to my comments on the bug.
12:28 tsbere eeevil: As for "are you lying to me": I think you believe the LEFT JOIN matters. I cannot find evidence that it does. Which could be that some other difference between MVLC's data and Bibliomation's data may be causing differences in planning. As I don't have the relevant plans from what you claim to have seen I can only base my opinion on what *I* see at this time.
12:29 tsbere eeevil: And what *I* see is "the view outputs things differently when it doesn't need to"
12:31 eeevil tsbere: you've looked at it for less than an hour
12:32 tsbere eeevil: And you admit it has been a while since you looked at it. As such, from my POV my "less than an hour" is currently more relevant than your recollections from the past, unless you pull relevant data out that you collected in the past on this issue to show me.
12:34 eeevil tsbere: look, you've broken search before with code that I warned would not work in situations beyond your test environment, and I had to revert and rewrite  that.  If I need to revert more of your code and replace it, I'll do so.  you do whatever you feel you need to do for MVLC
12:35 nhilton joined #evergreen
12:35 bshum Alright guys, so in the interest of testing by third-parties, I'd like to request we get some sample bib records (that create the mess) that we can use to replicate the issue initially.
12:35 bshum And then let's see how testing pans out for the solutions proposed.
12:35 tsbere eeevil: And you also yell at people that change the output of existing *anything* in the database, because who knows what else is using it. Your changes change the output, my adjustment to your changes restores the previous output.
12:37 tsbere eeevil: As I can't find a situation where the database plans differently between our two versions, but both versions are a vast (identical, as far as I can see) improvement over the version you replaced, the fact that the output is restored to what used to come out makes sense to me.
12:52 mtcarlson joined #evergreen
12:57 nhilton joined #evergreen
13:05 sandbergja joined #evergreen
13:21 * csharp would be happy to test this issue too, FWIW
13:21 csharp especially since we're moving to 2.7 in about a month ;-)
13:21 RoganH We're a little further than a month out but not much and I'm interested in testing for the same reason.
13:22 csharp we're also bound to have a wide array of poorly cataloged records
13:23 csharp and/or records mangled in multiple ILS migrations over the years, non-MARC-compliant ILSs, etc.
13:23 RoganH csharp: I feel like we could have a contest to see who has imported more bad records into the consortium.  :)
13:23 csharp heh
13:24 csharp our "bad" data is mostly from pre-PINES days, but some came over in PINES phase II
13:24 csharp c. 2001
13:25 RoganH Pre-consortium for us as well but since the oldest was 2009 and some as recent as 2013 a lot still hasn't aged out.
13:27 Dyrcona In our case the records are very brief bibs added by ILL software.
13:27 jeff "aged out." you say that as if bad data grew weaker and expired over time, as opposed to becoming more deeply entrenched and growing stronger with each passing day...
13:27 tsbere RoganH: I can't help but feel that in that contest the winner is the loser.
13:28 RoganH tsbere: that is very, very true
13:35 wjr joined #evergreen
13:44 nhilton_ joined #evergreen
13:48 kmlussier joined #evergreen
14:04 afterl joined #evergreen
14:13 ldwhalen joined #evergreen
14:22 mmorgan joined #evergreen
14:48 * bshum always feels nervous giving SQL instruction on the general list without knowing everything.
14:48 bshum :(
14:49 Dyrcona @praise bshum
14:49 * pinesol_green bshum is the very model of a modern major hacker
14:49 bshum @blame MARC
14:49 pinesol_green bshum: MARC broke Evergreen.
14:54 jboyer-isl MARC breaks everything. It's the anti-ILS whisperer.
14:55 * kmlussier chuckles
14:55 kmlussier If I knew how to add a quote to thebot, I would add that one.
14:56 tsbere kmlussier: er, @quote add "quote" I think
14:56 tsbere assuming you are authed to the bot, anyway
14:56 Dyrcona You have to be authenticated.
14:56 jboyer-isl That's ok, the log never forgets. The lonely, lonely log.
14:56 kmlussier Yeah, I'm lazy. I was hoping my comment would motivate somebody else to do it for me. ;)
14:56 * tsbere is never sure when he is authed to the bot
14:56 * kmlussier always forgets how to authenticate to the bot.
14:56 bshum phasefx++ # my curiosity is sated
14:57 Dyrcona @quote add <jboyer-isl> MARC breaks everything. It's the anti-ILS whisperer.
14:57 pinesol_green Dyrcona: The operation succeeded.  Quote #101 added.
14:57 bshum Permissions are weird.  End of line.
14:57 kmlussier Dyrcona++
14:57 kmlussier phasefx++
14:58 kmlussier @quote random
14:58 pinesol_green kmlussier: Quote #84: "pinesol_green grabs some Cookies and Cream Ice Cream for Mr. Spock: Something fascinating just happened." (added by bshum at 06:11 PM, May 10, 2014)
14:59 bshum That was special.
14:59 Dyrcona I missed that one.
14:59 kmlussier Hey, I made that quote happen.
14:59 Dyrcona @quote get 4
14:59 pinesol_green Dyrcona: Error: There is no Quote with id #4 in my database for #.
14:59 Dyrcona We're missing some numbers. I'm not sure we actually have 100.
15:00 Dyrcona @quote random
15:00 pinesol_green Dyrcona: Quote #29: "<Rogan_> Apparently I'm a trouble maker or busy body or something." (added by gmcharlt at 02:28 PM, July 11, 2012)
15:02 jboyer-isl Hmm.
15:02 jboyer-isl @quote serch @who
15:02 pinesol_green jboyer-isl: Yeah, well, you know, that's just, like, your opinion, man.
15:02 jboyer-isl @quote search @who
15:02 pinesol_green jboyer-isl: No matching quotes were found.
15:02 jboyer-isl Bummer.
15:03 Dyrcona @quote search jboyer-isl
15:03 pinesol_green Dyrcona: 4 found: #101: "<jboyer-isl> MARC breaks everything. It's the...", #76: "<jboyer-isl> Our copy location table is looking...", #83: "< jboyer-isl> PEBCAKEs, while delicious, rarely...", and #89: "<jboyer-isl> Assumption soup isn’t nearly as..."
15:03 Dyrcona @quote search marc
15:03 pinesol_green Dyrcona: 6 found: #101: "<jboyer-isl> MARC breaks everything. It's the...", #38: "<jcamins> At least your MARC frameworks aren't...", #46: "<_bott_> I am not a cataloger, but I speak...", #52: "<dbs> MARC is not machine readable.", #75: "< _bott_> I fear that MARC was a humorous...", and #77: "< Dyrcona> Sure, send someone binary MARC..."
15:03 * bshum laughed out loud at #52 again.
15:04 jboyer-isl I was trying to see if search and who would work together. Either it doesn't process them in the correct order, or you'd have to do it a dozen times to hit one.
15:04 Dyrcona @quote search @who
15:04 pinesol_green Dyrcona: No matching quotes were found.
15:04 Dyrcona It might not process @who in that case.
15:05 Dyrcona @who stole bradl's tux doll
15:05 pinesol_green collum stole bradl's tux doll.
15:05 bshum Or it is and it's just not finding anyone randomly.
15:05 Dyrcona Anyway....
15:05 bshum @quote search @who
15:05 pinesol_green bshum: No matching quotes were found.
15:05 * bshum is curious what the server side log says now
15:06 collum Drat!  Busted again.
15:06 Dyrcona heh
15:06 Dyrcona @who ate the chocolate crinkles
15:06 pinesol_green wjr ate the chocolate crinkles.
15:07 wjr probably true
15:07 bshum Yeah I think it's not picking up the other @who
15:07 bshum Alas.
15:12 b_bonner joined #evergreen
15:13 mtcarlson_away joined #evergreen
15:17 jeff jboyer-isl: you're looking for this:
15:17 jeff @quote search [who]
15:17 pinesol_green jeff: No matching quotes were found.
15:17 jeff but it doesn't tell you what it searched for.
15:17 jeff @quote search [who]
15:17 pinesol_green jeff: No matching quotes were found.
15:17 jeff @quote search [who]
15:17 pinesol_green jeff: No matching quotes were found.
15:17 jeff alas.
15:17 jeff not enough people with quotes :-)
15:17 bshum Hehe
15:17 jeff @quote search [who]
15:17 pinesol_green jeff: No matching quotes were found.
15:17 bshum Again!
15:18 bshum :D
15:18 dbs @blame [who]
15:18 pinesol_green dbs: (who [<channel>] <question>) -- Answers <question> with a random nick from <channel>. <channel> is only necessary if the message isn't sent in the channel itself. wants the TRUTH?! (who [<channel>] <question>) -- Answers <question> with a random nick from <channel>. <channel> is only necessary if the message isn't sent in the channel itself. CAN'T HANDLE THE TRUTH!!
15:18 jeff oh.
15:18 dbs @blame [who [quote]]
15:18 pinesol_green eby Try restarting apache. 's bugfix broke dbs's feature!
15:18 jeff silly me. i'll bet @who requires an argument
15:18 csharp @blame [someone]
15:18 pinesol_green eeevil wants the TRUTH?! eeevil CAN'T HANDLE THE TRUTH!!
15:18 csharp pinesol_green++
15:18 jeff @quote search [someone]
15:18 pinesol_green No matching quotes were found.
15:19 jeff ah well. that's what i should have been smashing, but i'll not generate more noise now (at least, not of that variety)
15:19 jeff @who
15:19 pinesol_green jeff: (who [<channel>] <question>) -- Answers <question> with a random nick from <channel>. <channel> is only necessary if the message isn't sent in the channel itself.
15:19 jeff @someone
15:19 pinesol_green vlewis
15:20 jeff (so of course, @quote [who] was searching for a quote that matched the usage/help message for the who command)
15:20 dbs (yep)
15:24 mtcarlson joined #evergreen
15:30 b_bonner joined #evergreen
15:31 b_bonner left #evergreen
15:35 b_bonner joined #evergreen
15:36 mtcarlsoz joined #evergreen
15:36 nhilton joined #evergreen
15:42 mtcarlson left #evergreen
15:45 jihpringle joined #evergreen
15:47 mtcarlsoz left #evergreen
16:05 kmlussier @dessert
16:05 * pinesol_green grabs some Sweet Potato Pie for kmlussier
16:06 Dyrcona @dessert [someone]
16:06 * pinesol_green grabs some Apple Pie for bradl
16:13 mtcarlson joined #evergreen
16:28 mrpeters joined #evergreen
16:59 afterl left #evergreen
17:09 mmorgan left #evergreen
17:20 RBecker joined #evergreen
17:22 nhilton_ joined #evergreen
18:09 kmlussier left #evergreen
22:52 sarabee joined #evergreen

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat