Evergreen ILS Website

IRC log for #evergreen, 2018-06-01

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat

All times shown according to the server's local time.

Time Nick Message
01:05 jonadab joined #evergreen
03:47 jlamos joined #evergreen
03:47 StomproJ joined #evergreen
03:47 bshum joined #evergreen
03:47 pastebot joined #evergreen
04:44 gsams joined #evergreen
06:30 pinesol_green News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
06:55 rlefaive joined #evergreen
07:13 rjackson_isl joined #evergreen
07:48 rjackson_isl joined #evergreen
07:49 rlefaive joined #evergreen
08:06 kmlussier joined #evergreen
08:09 kmlussier gmcharlt: Do you still want to be assigned to bug 1731272?
08:09 pinesol_green Launchpad bug 1731272 in Evergreen "web client: "Set default view" breaks record page loading" [High,Confirmed] https://launchpad.net/bugs/1731272 - Assigned to Galen Charlton (gmc)
08:17 collum joined #evergreen
08:48 idjit joined #evergreen
08:58 bos20k joined #evergreen
09:14 mmorgan joined #evergreen
09:15 lsach joined #evergreen
09:21 Dyrcona joined #evergreen
09:34 jvwoolf joined #evergreen
09:34 yboston joined #evergreen
09:46 Dyrcona All right. I think some of that strace information was off yesterday.
09:47 Dyrcona I believe I may have been using strace on the parent Apache process.
09:50 berick Dyrcona: that would explain a few things, for sure
09:50 Dyrcona Yeah. I think every actual thread that I tried to trace was blocked and/or strace reporting it being in x32 mode.
09:51 Dyrcona I'm looking at the options for ps more closely and doing strace on some threads that are working.
09:51 Dyrcona Mostly they block in select or accept.
09:51 mmorgan1 joined #evergreen
09:52 * csharp doesn't have any spinning procs at the moment, but will test along with for comparison
09:52 mmorgan2 joined #evergreen
09:52 Dyrcona I'm not looking at spinning processes. I'm actually looking at idle websocket workers.
09:53 mmorgan3 joined #evergreen
09:53 Dyrcona I think I found the one that sleeps and monitors. It jumps from nanosleep to restart_syscall.
09:54 Dyrcona Well, not quite. I think the strace connection interrupts the sleep.
09:55 berick one of the threads checks for activity and wakes by default every 5 seconds
09:55 Dyrcona If what I saw yesterday holds true, then I won't be able to strace the stuck processes.
09:55 Dyrcona nanosleep({5, 0}, 0x7f112a119e80)       = 0
09:55 Dyrcona Yeah.
09:57 mmorgan joined #evergreen
09:57 Dyrcona I think what works best if you know the pid of the stuck process is ps -L ${PID}
09:57 Dyrcona That seems to only list the pids of threads related to that process, one per line.
09:57 mmorgan1 joined #evergreen
09:58 * csharp saw a few of these in the websocket error logs from yesterday: [Thu May 31 13:47:29.346564 2018] [core:notice] [pid 22458] AH00051: child pid 29265 exit signal Segmentation fault (11), possible coredump in /etc/apache2-websockets
09:58 Dyrcona The -f option includes the parent PID as the second PID output, and that's the one that was looping in select which makes sense.
09:58 Dyrcona Yes, I have a lot of those.
09:59 Dyrcona Well, some on the bricks I checked.
09:59 Dyrcona No core dump though.
10:05 * Dyrcona decides to look for those messages in today's logs and see if any related pids are still around.
10:12 Dyrcona Looks like I signed in at the right moment. I have two running at 100% on brick 1.
10:13 Dyrcona Nagios is not reporting it. I don't know if that means nagios is not monitoring brick 1 for this or if it has not gone on long enough for nagios to notice, but I suspect the former.
10:16 Dyrcona Only 1 segmentation fault message on the bricks so far this morning (since 6:25 am), and that's not on a brick with currently spinning websockets apache2 procs.
10:18 Dyrcona Naturally, the PPID of the segfaulted proc is the main Apache2 listener and the PID that segfaulted is gone.
10:23 Dyrcona Oh! This is interesting!
10:27 Dyrcona csharp berick: https://pastebin.com/QkbYK45Y
10:27 Dyrcona I added some notes in "comments" at the top.
10:28 Dyrcona And, Nagios is reporting the high CPU apache processes now.
10:29 berick Dyrcona: huh, using 100% cpu, but strace shows them blocking on futex?
10:30 Dyrcona berick: Yes.
10:30 berick interesting, indeed
10:30 Dyrcona top is refreshing too quickly to copy and paste.
10:32 rjackson_isl Dyrcona: -d option can be used to adjust delay time (such as top -d 20 for 20 seconds)
10:33 berick and 's' command
10:33 Dyrcona Yeah. I'm looking up how to get it from ps.
10:33 Dyrcona The proc 27536 is running, though.
10:37 berick Dyrcona: can you find the last gateway log entry for that WS process?
10:38 Dyrcona RE: The x32 mode bit: "FWIW, the way that strace detects 32-bit mode is bogus, and I don't actually know how strace detects x32 mode. "
10:38 Dyrcona berick: I don't know how I would. Are the websockets pids logged in the gateway?
10:40 berick Dyrcona: yes, they should be
10:40 berick from my syslog setup:
10:40 berick 2018-06-01 10:39:22 eg-dev osrf_websocket_translator: [INFO:22818:./osrf_websocket_tran​slator.c:970:152777726422818110] WS disconnect from 127.0.0.1
10:40 Dyrcona I'll look.
10:40 berick direct logging to gateway.log should log them as well
10:40 Dyrcona The pid is after INFO.
10:40 berick yes
10:41 Dyrcona Looks like it is also buried in the "trace."
10:43 Dyrcona Jun  1 09:19:28 bh1 osrf_websocket_translator: [INFO:27464:./osrf_websocket_tran​slator.c:785:152785846327464112] [10.100.10.12] [] open-ils.pcrud open-ils.pcrud.search.actsc.atomic "d8a6790fd1d5b8bfc226cb17d2e02f35", {"owner":[107,106,2,1]}, {}
10:43 Dyrcona Oh, damn...
10:43 Dyrcona Anyway.... Hrm... There it is.
10:44 berick thanks Dyrcona
10:52 * csharp holds his hand over the appropriate part of the IRC screen until it scrolls out of view
10:55 csharp last message before the segfault from one of mine yesterday:
10:55 csharp open-ils.pcrud open-ils.pcrud.search.aou <STUFF>, {"parent_ou":null}, {"flesh":-1,"flesh_fields":{"​aou":["children","ou_type"]}}
10:55 Dyrcona Heh. berick++ for the magic to invalidate the session. :)
10:56 Dyrcona I had an "oh-no" second or two there.
10:56 berick thanks csharp
10:57 berick both innocuous requests
10:57 Dyrcona Seemingly.
10:57 csharp I'll try to get more data
10:57 csharp nothing's wrong right now but I'm sure it's just a matter of time :-)
10:57 berick i don't suppose either of you have log level set to debug for <gateway> ?
10:57 Dyrcona Well, I'll kill these processes.
10:58 Dyrcona Don't think so.
10:58 csharp berick: nope
10:58 NFPL joined #evergreen
10:59 Dyrcona At the risk of blowing things up, I'm going to kill -9 the one that is not responding rather than the ones that show up in top.
11:00 Dyrcona That killed the other two, and looks like another proc is spinning.
11:03 Dyrcona Ineterestingly, I am not finding this pid in the logs.
11:03 Dyrcona Oh... Interesting: gateway.06.log:2018-06-01 06:25:06 bh1 osrf_websocket_translator: [INFO:28537:./osrf_websocket_tran​slator.c:131:1527698458285373287] WS received SIGUSR1 - Graceful Shutdown
11:04 csharp there are a couple segfaulted procs I see in the websocket error.log on the brick that don't appear in the corresponding gateway logs
11:05 Dyrcona BTW, the above is the only message for this proc in today's logs.
11:05 Dyrcona And, nothing for yesterday, either.
11:07 Dyrcona So, this pid has apparently done nothing.
11:07 csharp I see lots of those for the past three days (so far) and they all happened at 6:25 a.m. - no idea why
11:07 Dyrcona Log rotation.
11:07 * jeff nods
11:08 csharp ah
11:08 berick csharp: lots of which?
11:08 csharp WS received SIGUSR1 - Graceful Shutdown
11:09 jeff likely logrotate called from cron renaming log files and then attempting to get the process to re-open log filehandles -- especially at that time of day.
11:10 csharp yeah /etc/crontab confirms log rotation
11:13 NFPL I've got what is hopefully a quick question. Why can I not create an alphanumeric password for new SIP users at the database level. It allows for numeric and conerts the Alphanumeric to MD5 but doesn't allow me to login or use the SIP user unless it is strictly numeric. I understand that the client wont allow me Alphanumeric passwords but didn't expect that issue in the database level
11:14 jeff NFPL: just to confirm, what version of Evergreen are you running?
11:15 Dyrcona So, the last log entry of this websockets process doing something useful is from Wednesday: ../../05/30/gateway.18.log:2018-05-30 18:15:43 bh1 osrf_websocket_translator: [INFO:28537:./osrf_websocket_tran​slator.c:785:1527698458285373287] [10.95.60.10] [] open-ils.actor open-ils.actor.ou_setting.ancestor_default.batch "103", ["format.date"], AUTHKEY
11:15 Dyrcona But, I didn't notice it spinning unitl today, after killing he other one that was spinning this morning.
11:15 * Dyrcona thinks bug in Apache or the websockets module.
11:17 jeff NFPL: I'd recommend setting the SIP user's password in the staff client, not in the database. It should deal with various gotchas and assumptions. Do the docs still suggest setting the password from the database somewhere?
11:27 csharp Dyrcona: berick: possibly related? https://github.com/disconne​ct/apache-websocket/pull/41
11:29 Dyrcona Could be. I'm looking at the other pull requests, too.
11:29 csharp this project looks pretty dang dead, tho
11:29 NFPL The staff client doesn't allow for alphanumeric and since these communicate with our system I was hoping to make them as secure as possible. I could use straight numeric, but was very curious as to why it wouldn't work as any password without alpha characters works DB side
11:30 Dyrcona NFPL: What version of Evergreen? How you set the password in the database recently changed.
11:30 NFPL 3.01
11:30 jeff https://github.com/jchampio/apache-websocket seems to be a much more maintained fork of the original
11:31 berick jeff: interesting...
11:31 csharp jeff++
11:32 jeff NFPL: The staff client not allowing for alphanumeric passwords sounds unusual to me. I'm not even sure that you can configure things that way.
11:32 Dyrcona jeff: Sure you can with a validation regex that only allows digits.
11:32 NFPL Equinox had changed the settings after making the initial users
11:33 Dyrcona NFPL: you have to use the new method to change/set a password in the database.
11:33 jeff Dyrcona: is there a regex that applies for passwords in the user editor?
11:33 csharp there is, as far as I remember
11:34 Dyrcona jeff: I thing/thought so, but I should really focus on websockets.
11:34 kmlussier I think it's a library setting.
11:34 NFPL The method I am using works for numeric but no alpha :(
11:34 csharp "Password format"
11:34 Dyrcona I have a function that I use to change passwords in the database.
11:34 NFPL But the DB doesn't chack that function does it?
11:35 Dyrcona NFPL: If Equinox changed things on your backend, then all bets are off. You'll have to find out what they did.
11:35 NFPL Just updating the db should be good enough because the trigger takes care of the MD5 conversion
11:35 Dyrcona NFPL: Not in 2.12+.
11:35 Dyrcona Maybe earlier....
11:36 csharp NFPL: there is a setting in Administration -> Local Administration -> Library Settings Editor named "Password format" - see if it has an entry for the library you're scoped for
11:36 NFPL Well it works as long as its numeric
11:36 jeff If you're unable to set a complex password in the staff client, I'd log in to the OPAC as the user and set a complex password there.
11:36 Dyrcona I don't think it should, and if it does, Equinox changed something.
11:36 jeff If you're unable to set a complex password in the OPAC, then I'd look into fixing the settings/customizations that are preventing you from setting a complex password in the OPAC.
11:36 csharp NFPL: since you're administered by Equinox, you might do better contacting their support directly
11:37 NFPL Its a function with a trigger. They may have set up that function?  Guess i will follow up with them then
11:37 csharp yeah
11:37 NFPL Thought it was just an EG thing
11:37 NFPL Well thanks anyway
11:37 csharp NFPL: anytime~
11:37 csharp s/~/!/
11:37 Dyrcona NFPL: To change the password in the database now, you have to get a salt, then update the password with a db function.
11:38 * Dyrcona will post his custom function to make it easier later.
11:38 Dyrcona I was looking at https://github.com/disconne​ct/apache-websocket/pull/39 which might also be relevant since we're using mpm_prefork.
11:39 Dyrcona The jchampio repository is definitely worth a look. I can test it on a vm soonish.
11:41 Dyrcona In fact, I'll do it now.
11:41 Dyrcona Well, get started now anyway.
11:43 * berick is trying it too
11:47 berick Dyrcona: beware it's more strict about checking the origin.  on my test VM I set WebSocketOriginCheck Off in the websocket apache config (because my Host doesn't match the apache host).  presumably not an issue on a real setup
11:47 berick it also supports whitelists, fyi
11:47 Dyrcona And, I see something about plugins.
11:47 berick otherwise, it seems to work as before, though
11:47 berick well, our osrf code is a "plugin"
11:48 Dyrcona Ok. I'm working out what I want for apx2 to install it, probably just apxs2 -i -c ....
11:48 Dyrcona Don't need to activate it again.
11:49 berick i followed the normal osrf instructions
11:50 Dyrcona Seems to have just worked for me...
11:51 Dyrcona I'll hit it with the staff client later.
11:54 NFPL_ joined #evergreen
11:56 Dyrcona But, first, lunch!
11:59 b_bonner left #evergreen
12:12 NFPL joined #evergreen
12:23 Dyrcona Well, so far, so good.
12:23 Dyrcona I've logged in and added a volume and copy to a bib record with the web staff client.
12:24 berick interestingly, i'm having issues with the new code.  but the issues look similar to what you have been reporting.  I don't get the CPU spike, but I can make the process lock up.  tracking that down now...
12:26 Dyrcona OK. I just checked the book out to myself. Are there automated tests that should be run?
12:28 berick there's no automated tests for websockets gateway
12:30 Dyrcona Worth asking, just in case. :)
12:42 jihpringle joined #evergreen
12:44 kmlussier joined #evergreen
12:58 jeff berick, Dyrcona: what distro are you each testing on?
12:58 Dyrcona Ubuntu 16.04, so is our production.
12:58 Dyrcona I've been pulled away to something else at the moment.
13:10 kmlussier joined #evergreen
13:12 berick 16.04 here too
13:16 Dyrcona So, I hit one pretty soon after it went to 100% CPU and not much interesting going on.
13:16 Dyrcona strace say the main proc is in x32 mode.
13:16 Dyrcona The second proc is waiting on a select
13:16 Dyrcona And the third is doing nanosleep over and over again.
13:18 Dyrcona I'm starting to think that strace says a proc is in x32 mode when it fails to connect/trace it and strace doesn't know why.
13:19 Dyrcona Kind of like the XUL client's generic network or server failure message.
13:22 jeffdavis It looks like switching from prefork to multiplex personality resolved our SIPServer problems.
13:22 jeffdavis miker++ # thanks for the encouragement on that
13:23 jeffdavis It makes me wonder if the prefork personality ought to be deprecated.
13:25 Dyrcona I think were still using prefork without issue.
13:25 Dyrcona s/were/we're/
13:28 jeffdavis Dyrcona: with EG 3.0+ ?
13:34 miker jeffdavis: oh, that's good news!
13:37 csharp we've been happily on multiplext for a couple of years
13:37 csharp multiplex, even
13:38 miker as have we (unsurprisingly)
13:42 Dyrcona jeffdavis: Yes.
13:42 Dyrcona But I should check the configuration. I could be wrong.
13:45 jvwoolf joined #evergreen
13:46 Dyrcona jeffdavis: Yes, we're still using prefork.
13:51 berick prefork here, still on 2.12 though
13:54 yboston joined #evergreen
13:58 Dyrcona Well, I have to go. Might be back later, might not.
14:15 * kmlussier notes that state laws requiring the ability to hear be an official meeting are outdated.
14:15 kmlussier Not to mention that minutes kept in an IRC meeting are far more informative than any minutes I've seen for an in-person meeting or conference call.
14:17 kmlussier Massachusetts has the same language in its laws.
14:38 David joined #evergreen
14:39 David Hi, I need a help with access new table. I have created an entry into fm_IDL.xml and restarted server. I am still getting an error that method is not found. Do I need to restart anything else to enable new table to be accessible?
14:40 StomproJ David, did you run autogen.sh?
14:40 StomproJ I think that updates some files from the field mapper definition.
14:41 csharp David: also, there are two installed fm_IDL.xml files - one in /openils/conf and one in /openils/var/web/reports
14:41 David Ah this is what I forgot to do. Thanks!
14:41 csharp you could symlink the reports one to the conf one
14:42 * csharp thinks since that's possible that should just be the way we do it
14:44 StomproJ I just found an odd situation with our database, 120 deleted bibs that have entries in metabib.browse_entry_def_map.  Anyone else remember seeing that?  Doing a set id=id update fixes them by forcing the triggers to run again.
14:44 David I am getting the following error when I run bash autogen.sh
14:44 David opensrf@ubuntu:/openils/bin$ bash ./autogen.sh  Updating Evergreen organization tree and IDL  Updating fieldmapper ./autogen.sh: line 82: /openils/var/web/opac/common/js//fmall.js: Permission denied
14:44 StomproJ David, did you chmod your /openils to opensrf?
14:44 StomproJ Sorry, chown not chmod
14:45 StomproJ It is one of the install steps.
14:45 David Yes, it has been running fine
14:46 StomproJ Who is the owner for fmall.js?
14:46 csharp StomproJ: this maybe? https://bugs.launchpad.net/evergreen/+bug/1773832
14:46 pinesol_green Launchpad bug 1773832 in Evergreen "Bib records that are both deleted and empty show in staff search" [Undecided,Confirmed]
14:47 csharp StomproJ: actually no, that's not it
14:47 David There is no such file: /openils/var/web/opac/common/js//fmall.js
14:47 csharp David: as root, do 'chown -R opensrf:opensrf /openils', then try again
14:48 David I can try that
14:48 David I know I already have that permissions
14:50 David That did not work.
14:50 David Somehow it is looking for /openils/var/web/opac/common/js//fmall.js file which does not exist
14:52 csharp David: try, as opensrf: 'touch /openils/var/web/opac/common/js/deleteme'
15:08 bos20k_ joined #evergreen
15:11 book` joined #evergreen
15:16 * jeff shakes fist at bug 28649
15:16 pinesol_green Launchpad bug 28649 in Launchpad itself "mail word wrapping breaks urls and other words" [Low,Triaged] https://launchpad.net/bugs/28649
15:24 * miker poke head in
15:24 * miker removes head
15:26 David I was able to run the autogen.sh but srfsh service cannot find the method for new tables
15:26 miker csharp: re symlinking fm_IDL.xml, we can't where the UI needs to be translated, unfortunately
15:39 csharp miker: ok :-/
15:43 miker csharp: because we use a dynamically identified DTD against the reports copy, to swap &-references with translated strings
16:02 kmlussier @quote [random]
16:02 pinesol_green kmlussier: Error: The command "random" is available in the LoveHate and Quote plugins.  Please specify the plugin whose command you wish to call by using its name as a command before "random".
16:02 kmlussier @quote random
16:03 pinesol_green kmlussier: Quote #174: "<rhamby> In reply to the previous query, I don't think Lebbeous is still working on this." (added by Dyrcona at 12:52 PM, September 13, 2017)
16:12 mmorgan @dessert
16:12 * pinesol_green grabs some Red Velvet Cake for mmorgan
16:13 frank_g Hi all, Is there any kind of bug in Report's cloning in web staff client? I am trying to clone a template but when I select the folder the next window is just showing a loading message.
16:13 * mmorgan doesn't like red velvet cake :-(
16:13 jihpringle frank_g: was the template created in the xul client or the web client?
16:14 frank_g xul client
16:14 jihpringle templates created in the xul client cannot currently be cloned in the web client
16:14 frank_g ahh ok, that responses my question, let my try creating a new template and then cloning it
16:16 frank_g jihpringle: yes, I tested it, thanks for your help
16:17 jihpringle you're welcome
16:17 David Hi, I am having an issue with accessing data that comes back from pcrud class inside the template. I am put the code and the output here. Can someone help as how I can access the data?
16:17 pastebot "david" at 64.57.241.14 pasted "Need to access the values inside the bless section" (9 lines) at http://paste.evergreen-ils.org/6687
16:25 berick David: you should be able to access field values with item.foo where 'foo' is one of the field names
16:25 berick that's assuming newtable.search_fields has <field> definitions in the IDL file
16:27 abneiman frank_g: Equinox has been contracted by MassLNC to address the cloning issue, the goal being 3.2
16:27 david_ joined #evergreen
16:28 david_ Can anyone help me accessing the data from pcrud call? Here is the pasted link: http://paste.evergreen-ils.org/6687
16:30 berick david_: :) I answered your other nick...
16:30 * berick re-pastes
16:30 berick david_: you should be able to access field values with item.foo where 'foo' is one of the field names.  that's assuming newtable.search_fields has <field> definitions in the IDL file
16:31 david_ berick: I tried to access it as item.fieldname and the page does not load
16:32 berick david_: then likely your IDL configuration is not quite right
16:33 david_ I think field idl is fine
16:33 david_ I am using the same name defined there
16:34 david_ Actually, I just found that I had a typo in my template for these fields. Thanks for the help
16:35 * berick nods
16:37 kmlussier frank_g / jihpringle: I thought xul report templates could be cloned in the web client, but that it was done through an inferior, less functional interface. The loading issue doesn't sound like it's expected.
16:37 * kmlussier hasn't looked at it in a few months and could be wrong.
16:39 frank_g abneiman: thant sounds good
16:41 jihpringle kmlussier: in 3.1 cloning a template from xul it doesn't work at all
16:42 jihpringle you can select the template, chose clone, click submit, chose the folder for it to go into, click submit and then you're taken to a screen that says "Loading"
16:42 jihpringle and nothing ever loads
16:44 kmlussier jihpringle: OK, good to know. I guess that makes it even more critical for the other fix to get in then.
17:03 mmorgan left #evergreen
17:17 jvwoolf left #evergreen
18:30 pinesol_green News from qatests: Testing Success <http://testing.evergreen-ils.org/~live>
20:59 dbs @later tell dbwells We seem to be missing the 3.1.1-3.1.2 version upgrade script in git?
20:59 pinesol_green dbs: The operation succeeded.

| Channels | #evergreen index | Today | | Search | Google Search | Plain-Text | summary | Join Webchat