| 13:29 |
tsbere |
jeff: Yea. Though ensuring you don't hit max too soon might be an issue. |
| 13:30 |
tsbere |
not to mention overnight issues <_< |
| 13:31 |
Dyrcona |
jeff: Sound overly complicated. Are the exclusions based on some rarely changing property of the user, or is it arbitrary for each report? |
| 13:32 |
jeff |
Dyrcona: the idea is "don't count circs by test or otherwise not-real patrons". usually this is just done with an infrequently (never!) changing patron profile on the "patron" account. |
| 13:32 |
jeff |
but right now i'd like (for training and/or for testing-for-real) to have some patrons who are in other groups also be excluded, without excluding that entire group. |
| 13:33 |
jeff |
i don't want to edit every report to "also exclude these patrons" or "also exclude patrons with this attribute", hence the idea that "at this point, if i decide to do this, i want a view to help me" :-) |
| 13:34 |
jeff |
(a reporting view for sql reports, not necessarily a full-on view in the IDL and such. |
| 13:34 |
jeff |
) |
| 13:35 |
jeff |
but another option is to change a test user in the excluded group to a non-excluded group to test "can this user really check out only one chromebook and no other items at all", then return the user to the excluded group -- but that involves: |
| 13:35 |
jeff |
1) hope nobody runs a report that would be affected |
| 13:35 |
jeff |
2) don't forget to change the group back |
| 13:35 |
jeff |
sometimes i think i just worry too much about reporting on "not real" things. |
| 13:36 |
jeff |
other times i think "more test servers!" |
| 13:36 |
* jeff |
shrugs |
| 13:36 |
jeff |
i'm just babbling now |
| 13:37 |
tsbere |
jeff: Make a table of "don't report on these user ids", fill it however you want (perhaps with a DB function), then ensure that reports say "and the user id is not in that table" either directly or via a view? |
| 13:37 |
|
rfrasur joined #evergreen |
| 13:37 |
tsbere |
Would likely be faster than looking up stat cat or user pref entries, though you could do that to populate the table |
| 15:57 |
dbwells |
jtaylorats: Yes. At that point they will display in the OPAC and be editable in the client. If the records contain any kind of individual copy/volume level data, there is no code which will do anything special with that type of data. |
| 15:57 |
* Dyrcona |
decides to program in BrainFuck from now on. |
| 15:58 |
kmlussier |
Dyrcona++ bshum++ # Doing your part to contribute to Evergreen even if you are human and sometimes make mistakes. |
| 15:58 |
jtaylorats |
Thanks. That is the last piece. I've gotten the test migration done for the entire database. |
| 15:58 |
jboyer-isl |
Dyrcona, have you seen Whitespace? http://compsoc.dur.ac.uk/whitespace/ |
| 15:59 |
jtaylorats |
Now to wipe it out and put all the pieces together and do it again :-) |
| 15:59 |
jtaylorats |
Appreciate all the help. |
| 16:04 |
bshum |
jboyer-isl: Heh, well it's fixed for rel_2_5 now too. |
| 16:05 |
bshum |
Every time I think about being a "hero" I think of the movie Sky High. Where there's heros and sidekicks. |
| 16:05 |
bshum |
I always thought I was more of a sidekick. |
| 16:05 |
jboyer-isl |
Huzzah! Thanks. We were just about to cut off changes to a new test environment 10 mins ago :) |
| 16:05 |
jboyer-isl |
I haven't seen that one, but I vaguely remember when it came out. |
| 16:28 |
pinesol_green |
[opensrf|Bill Erickson] osrf_control router-de/re-register and reload commands - <http://git.evergreen-ils.org/?p=OpenSRF.git;a=commit;h=5b3f804> |
| 16:28 |
pinesol_green |
[opensrf|Galen Charlton] typo fix - <http://git.evergreen-ils.org/?p=OpenSRF.git;a=commit;h=0c0b02e> |
| 16:28 |
pinesol_green |
[opensrf|Galen Charlton] x - <http://git.evergreen-ils.org/?p=OpenSRF.git;a=commit;h=d90a415> |
| 16:56 |
* bshum |
watches it explode now |
| 16:57 |
pinesol_green |
[opensrf|Bill Erickson] LP#1066131: srfsh.py should not require opensrf.settings - <http://git.evergreen-ils.org/?p=OpenSRF.git;a=commit;h=2fae356> |
| 16:59 |
gmcharlt |
bshum: while you're in the area, a look at 1155446 would be nice |
| 17:00 |
* bshum |
doesn't really want to test on Lucid |
| 17:00 |
bshum |
But I'll look at the rest of it. |
| 17:00 |
gmcharlt |
well, lucid is less relevant now than it was in March of last year |
| 17:01 |
gmcharlt |
I'll see how wheezy likes it |
| 17:01 |
bshum |
I'm checking precise right now. |
| 17:20 |
bshum |
gmcharlt: Looks okay to me on precise. |
| 17:20 |
bshum |
I'll sign off and push it |
| 17:20 |
gmcharlt |
thanks |
| 17:22 |
pinesol_green |
[opensrf|Galen Charlton] LP#1155446: add Debian/Ubuntu deps required for Python tests to pass - <http://git.evergreen-ils.org/?p=OpenSRF.git;a=commit;h=02a93da> |
| 17:25 |
* jeff |
waits patiently to see if collectionHQ extract will complete |
| 17:26 |
bshum |
egbuilder: status |
| 17:26 |
egbuilder |
evergreen-master-debian-6.00-x86_64: idle, last build 1h30m42s ago: build successful |
| 10:13 |
|
tsbere joined #evergreen |
| 10:14 |
eeevil |
there is that path, yes ;) |
| 10:31 |
|
mcooper joined #evergreen |
| 10:55 |
mceraso |
gmcharlt: I tested the candidate tarball for OpenSRF 2.2.2 using Ubuntu 12.04 LTS and it works well |
| 10:56 |
gmcharlt |
mceraso++ # thanks for testing! |
| 10:56 |
|
mrpeters joined #evergreen |
| 10:56 |
mceraso |
Anytime :) |
| 10:57 |
|
mcooper joined #evergreen |
| 13:39 |
|
schatz joined #evergreen |
| 13:39 |
mrpeters |
we're going to be building out a new installation with 900TB of SSD for their DB's. Should be fun to see how it screams. |
| 13:39 |
mrpeters |
I know PINES has talked about going SSD as well. |
| 13:40 |
bshum |
900TB? Whoa? |
| 13:40 |
bshum |
:) |
| 13:41 |
bshum |
So far, I'm noticing significant gains in search response and hold placement via the catalog on my test server with SSD. |
| 13:41 |
bshum |
Like an example bib where hold placement took 24 seconds in production system took 6 seconds in that test laptop. |
| 13:43 |
* bshum |
should switch out to GIN indexes and see what happens.... |
| 13:44 |
eeevil |
oh, that reminds me. 9.4 looks like it's going to greatly improve GIN speed |
| 13:44 |
bshum |
Yay! :) |
| 13:44 |
* bshum |
loves postgres |
| 14:09 |
|
hbrennan joined #evergreen |
| 14:13 |
|
snowball joined #evergreen |
| 14:26 |
|
akilsdonk joined #evergreen |
| 14:33 |
bshum |
berick: I just setup the prototype code on one of our test servers, we're going to play around. Just curious about one thing though: |
| 14:33 |
bshum |
What might cause this to happen where it seems to get stuck a bit on the patron search and I can't seem to retrieve new result sets. |
| 14:34 |
bshum |
Logging out I could get back to it again and things seemed fine, but on my first go, it found a patron, then I opened their record, then couldn't search for anybody new. |
| 14:34 |
* bshum |
keeps playing around |
| 14:34 |
berick |
bshum: haven't seen that. i'd suggest checking the JS console for errors |
| 14:34 |
berick |
bshum++ # testing |
| 14:34 |
bshum |
berick: Yeah I thought of that after I'd left the client |
| 14:34 |
bshum |
I'll try to replicate it and check the console next time. |
| 14:35 |
bshum |
It's quite fun so far. |
| 15:40 |
sseng |
jeff: gmcharlt: alright, thanks! |
| 15:46 |
Dyrcona |
rfrasur: Turns out, one of our members is also doing a seed lending program. |
| 15:47 |
Dyrcona |
rfrasur: Groton Public Library (gpl.org), and they are reusing the card catalog shelves. :) |
| 16:06 |
sseng |
gmcharlt: can I confirm that from the memory leak test report, I perform the test in the same tab right? that is to say, I press F1, enter the patron barcode, checkout a book, and click "Done", and this represent one iteration? |
| 16:06 |
jeff |
Right now, if you have a standing penalty such as PATRON_EXCEEDS_FINES, you can configure what things that penalty blocks (CIRC, or CIRC|RENEW, etc), but you can't have that penalty block certain things for different groups of patrons. Does anyone else have interest in such functionality? |
| 16:07 |
gmcharlt |
sseng: yes |
| 16:07 |
jeff |
I'm trying to decide between a hack and a wishlist bug. :-) |
| 16:56 |
bshum |
Bah, and now it works again. Might just be some weird browser effect... |
| 16:56 |
bshum |
I'll play more tomorrow. |
| 16:57 |
berick |
bshum: holler when you do. Now that it's browser-based, I can poke too! (if you give me a login :) |
| 16:58 |
bshum |
berick: Unfortunately I rolled it in with my local SSD test server. Which is hidden behind too many layers to unveil. |
| 16:58 |
bshum |
But I'm going to work on installing it to our public demo system soon. |
| 17:00 |
berick |
ah |
| 17:02 |
bshum |
Just have to remind myself how to log into everything... too many passwords. |
| 17:02 |
berick |
feel free to type them in here if you need a backup |
| 17:03 |
bshum |
:) |
| 17:22 |
|
kimo_sabe joined #evergreen |
| 17:31 |
kimo_sabe |
I'm having an odd issue trying to do some testing on 2.5.2 but when I try and clone my production database over to dev to try the upgrade stuff goes missing. Anyone seen that before? I'm at a loss |
| 17:39 |
|
mmorgan left #evergreen |
| 18:10 |
dbwells |
kimo_sabe: There have been a few recent conversations about needing to set the search_path before using pg_restore. |
| 18:11 |
dbwells |
e.g. here: http://evergreen-ils.org/irc_logs/evergreen/2014-01/%23evergreen.22-Wed-2014.log at 12:54:55 |
| 11:51 |
dbs |
yikes. |
| 11:51 |
ldwhalen |
eeevil: I think it leaves out all of the public. normalizers |
| 11:51 |
|
schatz joined #evergreen |
| 11:53 |
eeevil |
looks like it never did ... |
| 11:54 |
eeevil |
ldwhalen: that's purely an oversight, there's no reason to not add them (especially since we attempt to make use of them). I suspect a big fat me-- for not propagating testing env changes to the seed and upgrade scripts |
| 11:55 |
ldwhalen |
eeevil: If I add them is there anything in config.index_normalizer that needs a specific value other than config.index_normalizer.func? |
| 11:56 |
eeevil |
ldwhalen: depends on the function ... if it takes parameters /beyond/ the input text, the number of them must be recorded in param_count |
| 11:56 |
eeevil |
for that one, it's 0 |
| 15:39 |
jeff |
and much credit to berick and eeevil (and others?) for their help on that a few years ago. :-) |
| 15:39 |
jeff |
jboyer-isl: i'll be interested in what your eventual resolution is, if you care to follow up. |
| 15:41 |
* berick |
notes that w/ websockets (and no translator) we can remove the x-domain jabber requirement / brick cross-talk |
| 15:41 |
jboyer-isl |
I can smack it with a hammer whenever, it's just on our test instance. :) Thanks for the pointers, if I can get it going I'll be back to grouse/brag, heh. |
| 15:41 |
eeevil |
ldwhalen: the seed file should really just insert the proc name (no schema), no need to check pg_proc. see related ones for isbn and replace normalizers. the the upgrade file could check pg_proc to avoid double insertion, I suppose |
| 15:44 |
eeevil |
berick: or, at least not depend on it for staff client interaction ... still nice for simple HTTP clients with transactional requirements :) |
| 15:47 |
eeevil |
ldwhalen: or just check that it's not there already with a CTE, or blindly insert outside any explicit transaction |
| 15:49 |
ldwhalen |
eeevil: Is it worth while adding the param_count from pg_proc? |
| 15:50 |
eeevil |
you need the param_count when it's not 0. I'd suggest just setting it to what it should be. I wouldn't involve pg_proc at all. IMO, just look at the definition of the proc, count the params requested, and subtract 1 |
| 15:53 |
jboyer-isl |
jeff: route_subdomains was commented out on both test bricks, but setting it to s2s doesn't seem to have been the whole fix. I could/should check the other service machines, but these bricks should be talking straight to the db without too much bouncing around. |
| 15:58 |
eeevil |
jboyer-isl: can the other domains find each other via dns or /etc/hosts? (IOW, is ejabberd listening on all IPs, and are the foreign IPs named something they would recognized) |
| 16:01 |
tsbere |
jboyer-isl: I would also check for local firewalling - If the server(s) aren't allowing incoming traffic on the S2S port the rest of the proper config won't help....and are your different bricks using different ejabberd domains for that matter? |
| 16:03 |
jboyer-isl |
the machines appear to be able to find each other by name, I don't believe there's a firewall running (they're off on a private 10. network) and they are all using individual private.name.evergreen.lib.in.us and public.evergreen...., and localhost domains. |
| 16:14 |
eeevil |
berick: agreed ... I was really just thinking of xdr JS clients, though I don't know if that's strictly possible right now, and since only older MOZ supports multipart/mixed-replace, it's probably never going to be worth worrying about |
| 16:15 |
* dbs |
does a bit of a double-take at "Bill Erickson moved some example seed data out of the base install and into a sample data set." |
| 16:16 |
* eeevil |
goes to look more closely and make sure he didn't lie... |
| 16:17 |
dbs |
berick does _tons_ of stuff that deserves credit, and I certainly appreciate his testing & signing off |
| 16:17 |
eeevil |
I did lie. dbs, my apologies |
| 16:17 |
* eeevil |
updates |
| 16:21 |
dbs |
thanks eeevil |
| 10:36 |
csharp |
senator: yeah - that's what I'm trying to do now - thanks for the suggestion |
| 10:37 |
senator |
if you can get a log trace of a single hold placement (which as you know invokes the targeter for that specific hold) that's granular enough to show us more precisely where the slow part is |
| 10:37 |
senator |
yeah |
| 10:38 |
csharp |
I've got one with DEBUG enabled on our test server |
| 10:38 |
csharp |
I'm trolling through it |
| 10:45 |
RoganH |
I change windows and all I see is chsarp saying something about trolling. It's going to be that kind of day, eh? |
| 10:54 |
* csharp |
IS IN UR IRC CHANNEL TROLLIN U LOL LOL LOL |
| 10:54 |
jeff |
@decide Totally Awesome District Library or Trolling All Day Long |
| 11:41 |
senator |
csharp: if possible (and i'm sure you'll want to do some redacting of authtokens, barcodes, and IPs) a pastebin of the log would be awesome |
| 11:42 |
csharp |
senator: yeah - I'm trying to make sure I don't paste anything private - looking for authtokens, passwd hashes, etc. |
| 11:45 |
berick |
csharp: you can look for 32-character words (md5 authtokens and password hashes) w/ something like: grep -w '\w\{32\}' filename |
| 11:46 |
csharp |
berick: excellent - thanks |
| 11:47 |
csharp |
http://pastebin.com/6LCW2Luf |
| 11:48 |
csharp |
this is at loglevel 3 on our live server, btw |
| 11:48 |
csharp |
I can do one with DEBUG enabled on our test server if that helps |
| 11:50 |
berick |
wonder what's happening between 11:01:04 and 11:01:13 |
| 11:51 |
|
sylvar joined #evergreen |
| 11:52 |
berick |
that is a lot of OUS calls. just switching to the cstore / stored-proc variant would help a lot |
| 11:54 |
csharp |
I can see how this wouldn't have manifested itself in an instance with fewer Ous |
| 11:54 |
berick |
yeah |
| 11:57 |
csharp |
that gap looks like this on the test server (different hold): http://pastebin.com/YmC7pUNq |
| 11:58 |
csharp |
(between Processing hold blah... and when it starts checking OU settings |
| 11:58 |
csharp |
) |
| 12:00 |
berick |
huh, wonder which of those is taking so long on your prod servers. I take it the prod hold is on a popular record with lots of copies? |
| 12:04 |
csharp |
lemme look |
| 12:06 |
csharp |
227 total copies - 4 current holds |
| 12:07 |
csharp |
http://gapines.org/eg/opac/record/5606034 - "Calculated in Death" - apropos title ;-) |
| 12:07 |
berick |
heh |
| 12:08 |
csharp |
I'll place a hold on it in our test server and see what the log output is there |
| 12:10 |
tsbere |
I wonder if we have similar problems when records have a couple hundred copies. We appear to have less than 300 total records with >100 copies, actually... |
| 12:13 |
csharp |
whoa 22968 lines for that threadtrace |
| 12:13 |
csharp |
but it took 73 seconds to process |
| 12:13 |
tsbere |
Hmmm. 48% of our records have ONE copy. >_> |
| 12:34 |
senator |
csharp: if you're feeling saucy, on your test system, |
| 12:34 |
senator |
you might try working/collab/senator/slow-targeter-try-faster-settings-lookup |
| 12:34 |
* csharp |
is feeling saucy and SASSY |
| 12:34 |
senator |
that has two commits |
| 12:34 |
|
rsinger joined #evergreen |
| 12:51 |
sseng |
dbs: that's sounds like it, now curious to see where in eg code the barcode gets written to that subfield. |
| 12:52 |
sseng |
dbs: thanks a bunch for your help! |
| 12:52 |
|
smyers_ joined #evergreen |
| 12:54 |
csharp |
senator: I saw that speed up from 73 to 61 seconds on the test server after applying the patches and restarting opensrf |
| 12:54 |
csharp |
Method duration for [open-ils.storage.action.hold_request.copy_targeter]: 59.706 |
| 12:55 |
csharp |
Method duration for [open-ils.circ.holds.test_and_create.batch]: 61.294 |
| 12:55 |
csharp |
^^after |
| 12:55 |
csharp |
vv before |
| 12:55 |
csharp |
Method duration for [open-ils.circ.holds.test_and_create.batch]: 71.550 |
| 12:56 |
csharp |
Method duration for [open-ils.storage.action.hold_request.copy_targeter]: 73.370 |
| 12:56 |
csharp |
the frontend still times out, but it's better on the server side |
| 12:58 |
* dbs |
wonders about caching the COUST rather than fetching it fresh every time |
| 12:59 |
eeevil |
sseng: re barcode in 905u, evergreen doesnt write that, it reads it from external tools |
| 13:00 |
* eeevil |
disappears as quickly as he appeared |
| 13:00 |
sseng |
eeevil: ooh i see. so, might not be a good idea to comment that check out, thanks! |
| 13:01 |
|
Dyrcona joined #evergreen |
| 13:02 |
Dyrcona |
More pgtap questions. |
| 13:02 |
Dyrcona |
Should a test file related to a database upgrade script go in t or t/regress? I assume t, but thought I would ask. |
| 13:15 |
|
rsinger joined #evergreen |
| 13:20 |
|
rfrasur joined #evergreen |
| 13:25 |
|
RoganH joined #evergreen |
| 13:27 |
Dyrcona |
Hmm. Also, what about a branch with two upgrade scripts? So far I'm putting the tests in a single .pg file. |
| 13:28 |
Dyrcona |
The scripts could be merged into one upgrade script, I guess. |
| 13:31 |
|
ericar joined #evergreen |
| 13:54 |
tsbere |
senator / csharp: Thinking about holds and the stuff being done, wouldn't most of the lookups in the targeter not affect placement at all? respond_complete happens *before* the hold creation process fires off a blind request to the copy targeter after all... |
| 13:58 |
* csharp |
has no idea... |
| 13:59 |
* senator |
thinks and looks |
| 13:59 |
senator |
csharp: also, cool, thanks for testing that, at least it's a small win. more to do i see... |
| 13:59 |
csharp |
senator: happy to help! |
| 14:02 |
|
ktomita_ joined #evergreen |
| 14:05 |
senator |
tsbere: so you mean the respond_complete near the end of sub create_hold aka open-ils.circ.holds.create.override ? |
| 14:07 |
senator |
i see tpac code calling open-ils.circ.holds.test_and_create.batch instead |
| 14:09 |
* Dyrcona |
sings: "Let's do the 'rebase' again...." |
| 14:11 |
Dyrcona |
senator: Note tsbere is talking about the targeter and not tpac. |
| 14:12 |
tsbere |
senator: test and create batch ends up calling open-ils.circ.holds.create (or the .override variant) if the "test" succeeds |
| 14:12 |
senator |
right, well i thought we're talking about whether the targeter must be waited on at hold placement time |
| 14:12 |
Dyrcona |
Also, pgtap is easy. :) |
| 14:13 |
senator |
ah there it is, i see |
| 14:13 |
tsbere |
senator: And that leads us into create_hold, which is where the targeter comes into play. I think the problem with hold *placement* is the "test" phase of that, the call to open-ils.circ.title_hold.is_possible call |
| 14:13 |
Dyrcona |
senator: OK. I know we have some internal tickets, related to checkin, that we like to blame on holds code being slow but we've not figured out what exactly is the problem. |
| 14:14 |
* graced |
thinks eeevil would be helpful about now... |
| 14:14 |
* Dyrcona |
shuts up, so as not to confuse things. |
| 14:36 |
rfrasur |
jeff++ |
| 14:38 |
Dyrcona |
So, another pgtap question. |
| 14:39 |
Dyrcona |
What should I do with views that were altered by the upgrade script? |
| 14:40 |
jeff |
as in, how should you test them? |
| 14:41 |
jeff |
sorry, in the context of pg_tap, i suppose that was a silly question on my part. |
| 14:42 |
* dbs |
doesn't understand the question |
| 14:42 |
tsbere |
Dyrcona: I think you can do tests as though you would for a table (columns and types for the columns) - Beyond that you may want to look at ensuring test data is there and pulling from the view to ensure you got the correct info, maybe? |
| 14:44 |
Dyrcona |
dbs: I'm working on tests for an upgrade script that alters existing view definitions. I'm also discovering that there is no easy way to check if a view definition (i.e. its underlying query) is correct. |
| 14:44 |
Dyrcona |
jeff: Basically, yes. |
| 14:44 |
Dyrcona |
Testing for the existence of the views is easy. |
| 14:45 |
Dyrcona |
Testing it the view has or hasn't certain columns is also easy. |
| 14:45 |
Dyrcona |
s/it/if/ |
| 14:45 |
csharp |
senator: https://bugs.launchpad.net/evergreen/+bug/1272316/comments/4 |
| 14:45 |
pinesol_green |
Launchpad bug 1272316 in Evergreen "much slower holds processing in 2.4+" (affected: 2, heat: 10) [Undecided,New] |
| 14:45 |
phasefx |
Dyrcona: test the views behavior with test data? |
| 14:45 |
senator |
csharp++ kmlussier++ |
| 14:46 |
dbs |
Dyrcona: given that the view probably surfaced incorrect data before, and that's why it has been upgraded, wouldn't you just insert data into the underlying tables that you know would trigger the bad "before" case and test that the view now shows "good" data? |
| 14:46 |
dbs |
what phasefx said, only more verbose :) |
| 14:46 |
Dyrcona |
dbs: The views are being changed not because they were incorrect before, but because columns are disappearing from a base table. |
| 14:47 |
tsbere |
Dyrcona: In that case I would mainly focus on "the view does not have the column that went away" |
| 14:47 |
phasefx |
Dyrcona: I would still pick some data that exercised the boundaries of whatever query powered the view.. find data that should fall in and outside of the view |
| 14:47 |
phasefx |
if it is indeed the query you're concerned with |
| 14:50 |
jeff |
beyond testing a view for columns being present / not-present and testing data types of those columns, i think you have to use data to test the underlying query of a view. |
| 14:50 |
Dyrcona |
phasefx: I'll give that some thought. Since we're talking about billing and 8 views (3 or which are only modified if they exist), that could be a lot of test data. |
| 14:51 |
jeff |
i'm not sure that "view's definition is [SQL HERE]" is a good test. |
| 14:51 |
Dyrcona |
jeff: I'm not sure "view's definition is ...." is easily done. |
| 14:52 |
jeff |
i don't see a way, short of extending. i'm just saying "maybe there isn't an easy way because it's not a great test" :-) |
| 14:52 |
phasefx |
when it comes to testing structure (which I think is not being advocated here), I think we could do that better / more centrally for everything |
| 14:52 |
|
stevenyvr joined #evergreen |
| 14:52 |
Dyrcona |
tsbere: Also, I don't recall at this point if the view columns changed. It may only be from statements in many cases. |
| 14:53 |
Dyrcona |
jeff: I agree. |
| 14:53 |
dbs |
SELECT definition FROM pg_catalog.pg_views WHERE schemaname = blah, viewname = blah; # but I agree with data view |
| 14:53 |
Dyrcona |
Are there transactions in the concerto data, and are bills included? |
| 14:54 |
Dyrcona |
If yes && yes, I may just test the views in live_t? |
| 14:54 |
dbs |
Dyrcona: Yes, but you would need to run the fine generator to actually generate bills |
| 14:54 |
dbs |
So no |
| 14:54 |
jeff |
i am interested in having or creating sample bills and payments, fwiw. |
| 14:55 |
phasefx |
care would have to be taken if we use the fine generator for test data, because of the day's date changing |
| 14:55 |
jeff |
i have some pretty fun test cases from our live data. ;-) |
| 14:56 |
Dyrcona |
dbs: (I am not knocking automated tests.) I think that speaks to my arguments about the complexity of testing things in a system as complicated and why automated tests will never catch everything. |
| 14:56 |
jeff |
i think we could have bills and payments in the sample data without need for using the fine generator to create the bills. |
| 14:56 |
Dyrcona |
jeff++ |
| 14:56 |
phasefx |
could and should |
| 14:56 |
dbs |
Dyrcona: Jesus, man, I never claimed automated tests will catch everything! |
| 14:56 |
Dyrcona |
dbs: OK. :) |
| 14:57 |
Dyrcona |
I don't mean to start a war. |
| 14:57 |
dbs |
jeff: I'm pretty sure berick and I kicked that around but didn't want to insert artificial data that might end up not matching what the fine_generator would actually produce |
| 14:57 |
dbs |
Dyrcona: You should have just let me win |
| 14:57 |
Dyrcona |
dbs++ |
| 14:57 |
dbs |
Dyrcona++ |
| 14:58 |
Dyrcona |
dbs: I'm writing tests. I think you did win. :) |
| 14:58 |
* dbs |
was looking for a wrecking ball |
| 14:58 |
Dyrcona |
Yeah, I can see issues with bogus data that doesn't match fine generator. |
| 14:58 |
phasefx |
hrmm. I think historical bills are fair game if they do indeed match what non-artificial data for that point in time looks like |
| 15:00 |
jeff |
dbs: yeah, was that conversation on list or on irc, or did it happen elsewhere? |
| 15:00 |
jeff |
i seem to recall it, and that this is at least somewhat of a rehash. |
| 15:00 |
dbs |
irc or launchpad, or perhaps a mix of both |
| 15:02 |
Dyrcona |
I'll leave out those tests for now. I can come back and add them later. |
| 15:03 |
|
BlerenMen joined #evergreen |
| 15:03 |
|
BlerenMen left #evergreen |
| 15:03 |
Dyrcona |
I think I'll move on to tests for branches that just add settings and permissions. Those are simple. |
| 15:03 |
* phasefx |
remembers having this sort of pain with sample data to power the receipt template editor, and we see how that turned out :( |
| 15:03 |
keynote2k |
Hi, all. Tony from Software Freedom Conservancy here. I'm looking for photos of Evergreen Conference 2012 in Indiana to use in Conservancy's FY2012 annual report. Anyone have any photos I could use (ideally, under a Creative Commons license)? |
| 15:03 |
jeff |
but i'd still like to figure out a way of having the test data, so maybe we can revisit. |
| 15:03 |
keynote2k |
if so: privmsg me. Thanks! |
| 15:04 |
jeff |
phasefx: yeah, good example. the data was buried and drifted significantly. i've been thinking about receipts and receipt macros lately. :-) |
| 15:04 |
phasefx |
jeff: yeah, I'm doing a talk on it at the EG conference, so I'll probably get the itch soon to make it better first :) |
| 15:04 |
jeff |
oh, receipts? |
| 15:04 |
phasefx |
two talks, one for QA, and one for customizing receipt templates |
| 15:05 |
jeff |
ah. i see why those intersected for you, then. :-) |
| 15:05 |
phasefx |
QA talk: need more tests :) |
| 15:06 |
jeff |
"Please join me after the receipt printer talk in the atrium, where we will be destroying a series of receipt printers using various weapons and solvents." |
| 15:06 |
phasefx |
haha |
| 15:06 |
jeff |
now THERE'S a fund raising opportunity. |
| 16:58 |
|
ElliotFriend joined #evergreen |
| 17:00 |
ElliotFriend |
@wunder 63033 |
| 17:00 |
pinesol_green |
ElliotFriend: The current temperature in Jost Farm, Florissant, Missouri is 35.4°F (4:00 PM CST on January 24, 2014). Conditions: Scattered Clouds. Humidity: 26%. Dew Point: 3.2°F. Windchill: 26.6°F. Pressure: 30.05 in 1018 hPa (Falling). |
| 17:14 |
ElliotFriend |
Dyrcona: Did we talk a couple (few) months back about using Git to push EG upgrades from a laptop to testing to production? |
| 17:14 |
ElliotFriend |
Or was that someone else? |
| 17:25 |
Dyrcona |
That was probably me. |
| 17:26 |
ElliotFriend |
Any chance you remember when that was? I'm trying to find the conversation for reference, but for the life of me can't |
| 18:45 |
jtaylorats |
Hard to say why those characters are in there. |
| 18:45 |
Dyrcona |
Oh, its easy to say why the characters are in there: Most software that works with MARC records just plain sucks. |
| 18:46 |
Dyrcona |
When I load records I usually run them through some Perl code to convert them to MarcXML, convert the charset if necessary, and to "scrub" the records of control characters and other junk, first. |
| 18:47 |
jtaylorats |
Partly curious why the admin tool has no problem with the insert. Can't say for sure but I don't think it scrambled anything. |
| 18:47 |
jtaylorats |
I'll have to do some more checking. This is a test load and not worrying about it at the moment but need to cover that base before the next load. |
| 18:48 |
jtaylorats |
Thanks all. |
| 18:48 |
jtaylorats |
Need to get out of here. |
| 18:48 |
jtaylorats |
Bye for now. |
| 18:48 |
jtaylorats |
Hopefully one day I can answer a question for someone :-) |
| 18:49 |
jtaylorats |
...and maybe it will even be the right answer ;-) |
| 18:50 |
jtaylorats |
WooHoo!!! Looks like the server went down....that means I can quit for the night. |
| 20:14 |
|
timhome joined #evergreen |
| 20:26 |
|
rsinger joined #evergreen |
| 23:05 |
|
zerick joined #evergreen |
| 11:42 |
Dyrcona |
It is theoretically OS agnostic now, the OS just has to support Firefox/Xulrunner. |
| 11:43 |
rfrasur |
Hmm, has anyone experimented with Chrome OS? |
| 11:43 |
Dyrcona |
In theory, yes, but it'll have all the usual browser compatibility issues. |
| 11:43 |
jeff |
rfrasur: the current web prototype should run fine on a chromebook. i haven't tested it yet. your issues come down to printing support and rfid support (if applicable) |
| 11:43 |
* rfrasur |
files this |
| 11:43 |
Dyrcona |
@hate printing |
| 11:43 |
pinesol_green |
Dyrcona: The operation succeeded. Dyrcona hates printing. |
| 11:43 |
jboyer-isl |
dbwells, jeff: Re: earlier today. Success! ccvm errors have been banished from our testing system. |
| 11:44 |
Dyrcona |
jboyer-isl++ |
| 11:44 |
rfrasur |
I'd like to go exclusively to email receipts that that's not likely to happen in the near future...not here at least. |
| 11:44 |
jeff |
i have put thought into printing from a web browser to a receipt printer on windows/linux/macos, but not yet put as much thought into printing to a receipt printer from a chromebook. |
| 14:34 |
Dyrcona |
Also, should my version of pg_prove be the same as pgtap on the server? |
| 14:35 |
* dbs |
has only run pg_prove on an all-in-one server install |
| 14:36 |
jeff |
pg_prove seems capable of running from another host. it uses psql to connect. |
| 14:36 |
Dyrcona |
All the tests are failing, and I don't think they should. |
| 14:37 |
Dyrcona |
I'll play with the options a bit. |
| 14:37 |
Dyrcona |
Parse errors: No plan found in TAP output |
| 14:37 |
Dyrcona |
Leads to believe I should get a more recent pgtap? |
| 14:38 |
dbs |
Which tests? What command are you running? |
| 14:39 |
Dyrcona |
pg_prove t # from Evergreen/Open-ILS/src/sql/Pg |
| 14:40 |
Dyrcona |
Ah, wait. Maybe pgtap is not actually loaded in the database I'm testing against. |
| 14:42 |
|
stevenyvr joined #evergreen |
| 14:42 |
Dyrcona |
I'll have to bug tsbere when he gets back to his desk. |
| 14:44 |
jeff |
Dyrcona: first error in the output is usually more helpful than the errors in the Test Summary Report |
| 14:44 |
|
bibliophylum joined #evergreen |
| 14:45 |
Dyrcona |
jeff: Yeah. select plan(2) says plan() doesn't exist. I'm trying to figure out the parameters that I need to use psql on the server itself to load pgtap in my database. |
| 14:45 |
jeff |
"Parse errors: No plan found in TAP output" can show for "i can't connect to postgresql", "there wasn't a database with that name", lack of plan() function, etc. |
| 14:52 |
eeevil |
"but what if someone gains access to a shell on the system?" oh .. wait... |
| 14:52 |
tsbere |
Dyrcona: No, it is allowed. If you talk TCPIP. You were talking socket. :P |
| 14:52 |
Dyrcona |
jeff: No, I was running pgtap from another machine remotely, but tried to install pgtap from the database server. |
| 14:52 |
jeff |
oh. got it. |
| 14:53 |
jeff |
"You can't test in here! This is the database server!" |
| 14:53 |
Dyrcona |
Ah, well. It's fixed with 'create extension pgtap schema public;' from the remote machine. |
| 14:53 |
Dyrcona |
:) |
| 14:53 |
Dyrcona |
Strangelove++ |
| 14:53 |
Dyrcona |
jeff++ |
| 14:53 |
|
ibogachenko joined #evergreen |
| 14:54 |
Dyrcona |
Oh, and now that pgtap is installed in the target database, all of the tests pass. |
| 14:55 |
jeff |
tests++ |
| 14:56 |
* Dyrcona |
finally gets around to writing pgtap tests. |
| 14:56 |
tsbere |
and due to adding a similar statement to our copy of the database create script used for restoring backups for various purposes we don't have to worry about that in the future :D |
| 15:00 |
Dyrcona |
Should I be concerned if the unbreak new encode test fails if I don't have the new version of Encode.pm? |
| 15:02 |
|
jbfink joined #evergreen |
| 15:06 |
dbs |
Dyrcona: maybe |
| 15:07 |
csharp |
I see that libmarc-xml-perl-1.0.2 has made it to Ubuntu, but the deb is for the upcoming trusty release |
| 15:11 |
gmcharlt |
if you don't mind mixing-and-matching package repositories, but are running Debian and would prefer a package over CPAN, 1.0.2 is available at http://debian.koha-community.org/ |
| 15:11 |
gmcharlt |
or you can grab it from sid |
| 15:12 |
* Dyrcona |
wishes he had more time to figure out packaging to make his own PPA or whatever for Ubuntu, or join MOTU even. |
| 15:24 |
dbs |
Dyrcona: hmm. well, I confirmed that the test still passes on new Encode, but sure is not good that it fails on old Encode.pm |
| 15:27 |
* dbs |
wondered for a moment if MARC::Charset might be involved but I have 1.35 here |
| 15:28 |
Dyrcona |
Ditto. |
| 15:31 |
Dyrcona |
I can't upgrade Encode.pm on this server because it might cause our training database problems. |
| 15:31 |
Dyrcona |
Our training Evergreen doesn't have the commit with the fix. |
| 15:33 |
dbwells |
Dyrcona: Is this a totally fresh DB? Anything which might cause a change to the MARC via triggers, such as varying config.internal_flag settings, or different org unit codes, could potentially cause this test to fail, as it requires the record be exactly as expected. |
| 15:34 |
Dyrcona |
dbwells: It has a copy of our production data in it, so that must be it. We're using id for tcn. |
| 15:42 |
|
rfrasur joined #evergreen |
| 15:45 |
dbwells |
I could be wrong, but I think that is the default. I wonder if just changing your top OU code to 'CONS' would let the test pass, since that gets inserted by maintaing_control_numbers? |
| 15:46 |
Dyrcona |
Heh. That's Ok. I'll just accept that the test fails on my setup. :) |
| 15:46 |
dbwells |
But you've got us all worried! ;) |
| 15:47 |
* dbs |
was going to ask if it was a stock db as well |
| 15:47 |
dbs |
dbwells++ |
| 15:48 |
Dyrcona |
Suppose I could alter the test before running it to do the update... |
| 15:48 |
|
jtaylorats joined #evergreen |
| 15:57 |
dbs |
I suppose we could make the test more resilient by regex_replace'ing the 003/035/901 fields out before running the md5sum |
| 15:57 |
dbs |
001 too I guess |
| 15:58 |
Dyrcona |
If it works on a brand new database, then I don't suppose it should change. |
| 15:59 |
Dyrcona |
Maybe some notes to the effect that it will fail in a database that already has been configured. |
| 16:13 |
|
jtaylorats joined #evergreen |
| 16:16 |
|
jwoodard joined #evergreen |
| 16:29 |
jeff |
hrm. incoming request that makes me think about merging reports and searching. |
| 16:31 |
Dyrcona |
On pgtap tests: I guess those that go with upgrade scripts should have XXXX. in the name where the version would go, looking at the examples in Open-ILS/src/sql/Pg/t. |
| 16:33 |
dbs |
Eh. My kneejerk reaction is that I don't think they should be tied to schema upgrades; just name what they're testing |
| 16:35 |
eeevil |
dbs: I though there was some logic that used the numbers? maybe not. If not, I'd say tests generated by a schema change /could/ (but needn't necessarily) carry the same number, and tests that just get added shouldn't have a number at all |
| 16:36 |
gmcharlt |
if we include any numbers at all in the filename, LP numbers would be the most useful in the long run |
| 16:36 |
gmcharlt |
at leat for regression test |
| 16:36 |
gmcharlt |
s |
| 16:37 |
dbs |
eeevil: no logic that I'm aware of |
| 16:37 |
dbs |
gmcharlt: yeah, lp numbers make sense for regress/ IMO |
| 16:37 |
phasefx |
I did one test for an upgrade script as an example/experiment, but I'm not sure I'd like to continue that practice |
| 16:37 |
Dyrcona |
Ok, sounds good to me. |
| 16:37 |
dbs |
"that I'm aware of" is a great big qualification :) |
| 16:37 |
Dyrcona |
After looking again, there is only that has an upgrade version number in the name. |
| 16:38 |
Dyrcona |
I suppose we don't want tests to check that a particular version exists in config.upgrade_log? |
| 16:38 |
phasefx |
part of me would like to see us use the make-pgtap script to encode the entire schema, and then start keeping that up-to-date |
| 16:39 |
|
jbfink joined #evergreen |
| 16:40 |
eeevil |
gmcharlt / dbs: that's fair. LP would be good, I agree |
| 16:40 |
eeevil |
Dyrcona: I was actually imagining that /that/ might be (partially) there, thus the numbering that senator did for his test on my recent change (sorry for the git churn on that, btw, all) |
| 16:40 |
gmcharlt |
eeevil: to be clear, I don't care much *how* the LP gets cited, but think that it should |
| 16:41 |
gmcharlt |
test names would be another way to do it, e.g. |
| 16:43 |
phasefx |
re: make-pgtap, the thought is that if you create an upgrade script that changes the schema, you'd also change a base set of tests that tracks the schema. How useful that would be for everyone, I'm not sure. Someone doing an upgrade could run the tests afterward to make sure it went well. For developers, eventually we might have automation that compares the results of upgrade scripts to |
| 16:43 |
phasefx |
pristine database installs (via those tests) |
| 16:43 |
|
akilsdonk joined #evergreen |
| 16:47 |
eeevil |
phasefx: not if I have /my/ way ;) |
| 16:48 |
eeevil |
(no more moving-baseline) |
| 09:31 |
jcamins |
Dyrcona: "The Do-it-yourself library management game for the entire family"? |
| 09:31 |
Dyrcona |
:) |
| 09:31 |
Dyrcona |
jcamins++ |
| 09:31 |
graced |
My new favorite fast game is We Didn't Play Test This At All. |
| 09:32 |
jcamins |
graced: lol |
| 09:32 |
graced |
I think one of our rounds was over in 10 seconds. |
| 09:32 |
dbs |
Monopoly: that's when you check out all of the latest Stephen King / George R.R. Martin copies to yourself |
| 09:34 |
phasefx |
Cards Against Humanity is probably my favorite party game :D |
| 09:34 |
graced |
And the phone app Cards Against Manatees |
| 09:34 |
paxed |
assuming "party game" means "needs to be played while intoxicated", then yes. |
| 09:35 |
phasefx |
We Didn't Play Test This, would be my 2nd. "Ah.. Zombies" |
| 09:35 |
graced |
Ah Zombies gets me every time for some reason |
| 09:36 |
phasefx |
graced++ I had forgotten the name of that game; I'm glad you mentioned it |
| 09:36 |
graced |
phasefx: we should play it at lunch on this Friday |
| 11:48 |
jtaylorats |
""$user",public" |
| 11:48 |
rfrasur |
Ben's cool. Reading his "what I think he meant was" version. |
| 11:50 |
rfrasur |
Hah! Love the last line "Long-range plans are a big deal for Open Source..." |
| 11:50 |
jtaylorats |
I've done a direct SQL insert as a test before I use the Copy option. |
| 11:51 |
jtaylorats |
Insert worked fine. It is when it goes to move it to the biblio.record_entry table that it triggers the failing functions. |
| 11:52 |
dbs |
huh, the actual notifications doc url is at http://en.flossmanuals.net/evergreen-in-action/sending-gentle-reminders-to-your-users/ |
| 11:53 |
jtaylorats |
Don't fix it...I may never find it again ;-) |
| 11:53 |
dbs |
jtaylorats: "\df oils_xpath" doesn't show anything? |
| 12:10 |
|
smyers_ joined #evergreen |
| 12:11 |
jtaylorats |
Set that from the evergreen-dev schema? |
| 12:11 |
jeff |
jboyer-isl: but that said, if the errors you've seen are those indexes failing to build (which can then be built after the pg_restore completes), it's likely unrelated to the issue jtaylorats has (though jtaylorats' issue might also be search path related :-) |
| 12:11 |
jboyer-isl |
jeff: That would explain why our upgrade testing db is not performing as fast as expected. |
| 12:11 |
csharp |
jtaylorats: (scrolling up) so "evergreen-dev" is a copy of your production database? |
| 12:12 |
jtaylorats |
Yes. |
| 12:12 |
csharp |
jtaylorats: then yes, try that |
| 12:14 |
jtaylorats |
PL/pgSQL function "fingerprint_trigger" line 10 at assignment |
| 12:14 |
jtaylorats |
SQL statement "INSERT INTO biblio.record_entry (marc, last_xact_id) VALUES (stage.marc, 'IMPORT')" |
| 12:14 |
jtaylorats |
PL/pgSQL function "staging_importer" line 5 at SQL statement |
| 12:14 |
jeff |
jboyer-isl: does your test db have an authority.by_heading index? |
| 12:14 |
jtaylorats |
The error has changed a bit. |
| 12:14 |
jtaylorats |
It showed more detail before. |
| 12:15 |
jeff |
jboyer-isl: the indexes authority.by_heading and authority.unique_by_heading_and_thesaurus are the common ones to fail at pg_restore time due to search_path issues. |
| 12:39 |
jtaylorats |
Well, sounds like I need to have them create a copy using a different method, assuming they did a dump/restore? |
| 12:39 |
jtaylorats |
That sound right? |
| 12:40 |
Dyrcona |
Oh. I was gonna say that we run a little script after pg_restore that fixes your problem. |
| 12:40 |
jtaylorats |
then check to see if that function exists in the test database? |
| 12:40 |
jtaylorats |
Works for me. |
| 12:40 |
jtaylorats |
What script? |
| 12:41 |
jtaylorats |
Will it insure that all the functions exist? I assume this problem could exist on many fronts??? |
| 12:41 |
dbs |
Dyrcona: we could add that script to http://en.flossmanuals.net/evergreen-in-action/care-and-feeding-of-evergreen/ for the logical backup section! |
| 12:42 |
pastebot |
"Dyrcona" at 64.57.241.14 pasted "fix_database.sql" (6 lines) at http://paste.evergreen-ils.org/56 |
| 12:43 |
jtaylorats |
Is this related to my problem or jeff's? |
| 15:06 |
jeff |
tsbere: which is why i'm using the exim logs -- they contain the email that we used. :-) |
| 15:07 |
jeff |
tsbere: but if we went with VERP and handled delayed bounces, etc, I'd probably encode the email address we have in the envelope. |
| 15:07 |
tsbere |
jeff: And how hard is it to say "fetch email from this patron by id, then pass it into said API, perhaps with the inclusion of the final bounce" ;) (also, the bounces I am concerned about are also the backscatter potential ones that are delevered from elsewhere) |
| 15:08 |
dbs |
jeff: yes, ours too (now that our uni is cutting over to google apps). went so far as to send myself a test message to play with it. But OTOH I'm still morally opposed to HTML email :) |
| 15:10 |
* tsbere |
also hates services that will strip anything that looks like one of their emails from the message, removes most of the headers from the message, and then wonder why you can't do anything to stop them from showing up again later, especially when multiple people got it and only one wants it to stop |
| 15:12 |
jeff |
yup. email. |
| 15:13 |
jeff |
dbs: and, contrats on groupwise -> google apps :-) |
