| Time |
Nick |
Message |
| 01:49 |
|
troy__ joined #evergreen |
| 01:49 |
|
troy__ joined #evergreen |
| 06:00 |
pinesol |
News from qatests: Testing Success <http://testing.evergreen-ils.org/~live> |
| 07:15 |
|
rjackson_isl_hom joined #evergreen |
| 07:48 |
csharp |
Bmagic: re EBSCO - assuming it's EDS, we don't allow them real-time updates, but rather do a manual export that we upload to them monthly (but that may not be feasible for your situation) |
| 07:51 |
|
collum joined #evergreen |
| 07:58 |
JBoyer |
I don't think it's as simple as that, berick was saying he's seen a ton of them and they're not even using EBSCO for anything. (at least that's what I took away from the last time we talked about it) |
| 07:59 |
|
mantis1 joined #evergreen |
| 08:12 |
|
rfrasur joined #evergreen |
| 08:22 |
|
alynn26 joined #evergreen |
| 08:32 |
|
mmorgan joined #evergreen |
| 08:44 |
|
collum_ joined #evergreen |
| 08:46 |
|
troy___ joined #evergreen |
| 08:46 |
|
troy___ joined #evergreen |
| 08:50 |
|
Dyrcona joined #evergreen |
| 09:04 |
Bmagic |
csharp: we do the same |
| 09:06 |
|
jvwoolf joined #evergreen |
| 09:06 |
jvwoolf |
jeff++ |
| 09:06 |
jvwoolf |
JBoyer++ |
| 09:07 |
jvwoolf |
For the barcode printing information yesterday. Had to step away, but I appreciate it! |
| 09:09 |
berick |
csharp: JBoyer: we also do monthly batch exports to ebsco, which I hadn't put 2-2 together at the time, but in the end we also don't need/want the real time data. |
| 09:11 |
JBoyer |
Ah, ok. That is a lot closer to ok than "we're being scraped by a vendor we don't even use." Still not great but the eyebrow has been lowered. |
| 09:11 |
berick |
exactly |
| 09:17 |
Dyrcona |
We do a weekly ISBN dump for Novelist and a monthly MARC dump for EDS and send them to EBSCO. |
| 09:20 |
Dyrcona |
I suspect that the EBSCO "attacks" are them looking up data for the on-the-shelf feature of Novelist. |
| 09:33 |
Bmagic |
I can say with 95% certainty that unapi was the catalyst for the DOS yesterday. That's 6 bricks at 8 CPU's each, 48 CPU's maxed out. Today, we have a limit on that URL at nginx |
| 09:35 |
berick |
Bmagic: mind sharing your nginx rule? just curious |
| 09:36 |
Dyrcona |
We blocked the EBSCO IPs at the firewall because we don't use on the shelf. |
| 09:39 |
Bmagic |
berick: yep, I was thinking of committing it to a working branch just for sharing, and maybe inclusion for options for devops |
| 09:39 |
Bmagic |
I'll get something up today |
| 09:39 |
berick |
Bmagic++ wasn't trying to make you do a bunch of work, but thanks |
| 09:40 |
Bmagic |
You think it's worth an associated bug? |
| 09:40 |
berick |
Bmagic: i think that could be useful, yes, if nothing else for documentation purposes |
| 09:42 |
Dyrcona |
We should adjust the sample configuration for nginx. It should have TLSv1.3 added. |
| 10:26 |
Dyrcona |
Backporting sucks. |
| 10:27 |
Dyrcona |
I'm told that curbside isn't working on our training server with 3.5.2, but it works fine with 3.2.10 on production. I need to see if we're missing commits or I did a bad conflict resolution. It may be easier just to build a new branch. |
| 10:27 |
Dyrcona |
Not to mention the other stuff backported to training..... |
| 10:28 |
Dyrcona |
So much for working on Perl test today..... |
| 10:28 |
Dyrcona |
s/test/tests/ |
| 10:31 |
Dyrcona |
Doesn't help that I have a list of 18 commits for the curbside feature. I'll have to check that list is accurate. I suppose I could try the 3.5 backport branch, but I seem to recall not using it because it was missing something. |
| 10:33 |
Dyrcona |
Oh, that's right. There isn't a 3.5 curbside branch. |
| 10:38 |
Dyrcona |
Oorah! git log working/collab/gmcharlt/lp1879983_curbside_pickup_pr ^35fca97a4ec4f5009735544a26160060eaff7610 | grep -c ^commit |
| 10:38 |
pinesol |
Dyrcona: [evergreen|Bill Erickson] 3.5 Post-Beta Release Notes Additions - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=35fca97> |
| 10:49 |
Bmagic |
berick: bug 1913610 |
| 10:49 |
pinesol |
Launchpad bug 1913610 in Evergreen "Evergreen needs a way to mitigate DOS" [Undecided,New] https://launchpad.net/bugs/1913610 |
| 10:59 |
Dyrcona |
I wonder if bisect will help? I haven't found the missing code, yet. |
| 11:02 |
Dyrcona |
Bmagic: Your bug title sounds dire. |
| 11:02 |
Bmagic |
:) |
| 11:02 |
|
alynn26_away joined #evergreen |
| 11:19 |
Dyrcona |
OK. Where does curbsideDisabled() come from? |
| 11:20 |
Dyrcona |
grep -R curbsideDisabled Open-ILS/ turns up 1 use and 0 definitions for it. |
| 11:22 |
|
AFloyd__ joined #evergreen |
| 11:34 |
Dyrcona |
Well, after all that, it looks like a bad conflict resolution with the access key fix. |
| 11:35 |
csharp |
@who is under attack from EBSCO? |
| 11:35 |
pinesol |
yeats is under attack from EBSCO. |
| 11:36 |
berick |
heh |
| 11:49 |
Dyrcona |
BTW, https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/dyrcona/rel_3_5%2Bcurbside%2Bstripe_v3 |
| 11:49 |
Dyrcona |
That's curbside and Stripe V3 backported to rel_3_5 for those following along at home. |
| 11:55 |
Dyrcona |
Guess it's close enough to lunchtime. |
| 12:07 |
|
Cocopuff2018 joined #evergreen |
| 12:28 |
|
rfrasur joined #evergreen |
| 12:30 |
Bmagic |
csharp berick Dyrcona: with the new limit imposed, I'm trapping a lot of limits on URI's like this one: "GET /opac/extras/unapi?id=tag:acq.open-ils.org:biblio-record_entry/877750/-&format=marcxml-uris |
| 12:30 |
Bmagic |
The origin is from the bricks themselves... |
| 12:31 |
|
sandbergja_ joined #evergreen |
| 12:36 |
jeff |
(obligatory "the call is coming from inside the house!") |
| 12:37 |
berick |
Bmagic: that does look like the ones we were getting as well |
| 12:37 |
berick |
they were def. coming from ebsco, though |
| 12:38 |
Bmagic |
it seems like the Evergreen code.. perl? or somewhere is perhaps relaying an external request back as an internal request? |
| 12:38 |
Dyrcona |
I think the Evergreen bug should be marked a duplicate of the OpenSRF. |
| 12:39 |
Dyrcona |
Bmagic: What? |
| 12:39 |
Bmagic |
the Evergreen Z39.50 service comes to mind |
| 12:39 |
berick |
Bmagic: what makes you think it's coming from the brick? |
| 12:39 |
Bmagic |
Dyrcona: just spit balling, but since the requests are from itself |
| 12:39 |
Bmagic |
berick: the NGINX IP address recorded in the limiting error log |
| 12:40 |
berick |
Bmagic: hm, i wonder what the access log shows. ours was showing external IP |
| 12:40 |
Dyrcona |
Bmagic: Do you have the proper set up for apache to log the external IPs? That could actually use better documentation. I've had to figure it out twice. |
| 12:40 |
berick |
also what Dyrcona said |
| 12:40 |
Dyrcona |
Imagonna bug that. |
| 12:41 |
Bmagic |
these are nginx logs |
| 12:42 |
Dyrcona |
Z39.50 does talk to the local apache/nginx. We run ours on its own server without nginx. |
| 12:43 |
Bmagic |
I'm starting to think these are the aws dynamic load balancer internal IP's.... googling |
| 12:43 |
Dyrcona |
simple2zoom does that: converts Z39.50 to SRU and back again. |
| 12:50 |
Dyrcona |
launchpad-- bugs-- |
| 12:52 |
Bmagic |
yep, those requests are likely external, but my logs show the internal IP space because the load balancer isn't configured to reveal the source IP in the headers... Working on a solution. But yeah, these are external requests like you thought |
| 12:55 |
Bmagic |
the external IP needs to traverse the load balancer, then nginx, then apache2. :) pretty sure the last two are correct. The X-Forwarded and X-Real NGINX configs are in place like the ones shown in my opensrf working commit 08a7299b762ed366496a6a4fc14 |
| 12:57 |
Dyrcona |
Oh, I already did: bug 1908540 |
| 12:57 |
pinesol |
Launchpad bug 1908540 in Evergreen "Add Apache proxy logging example" [Wishlist,New] https://launchpad.net/bugs/1908540 |
| 12:58 |
|
collum joined #evergreen |
| 13:00 |
|
sandbergja_ joined #evergreen |
| 13:05 |
Bmagic |
Dyrcona++ |
| 13:08 |
Bmagic |
Dyrcona: the commit message is confusing me. CustomLog "|/usr/bin/logger -p local7.info" common -> CustomLog "|/usr/bin/logger -p local7.info" proxy niether of the LogFormat definitions are named "proxy", is that a mistake/typo. Should it be "proxy-common" or "proxy-combined" |
| 13:10 |
Bmagic |
asciidoc message rather |
| 13:13 |
Dyrcona |
Bmagic: Fixed, rebased, and force-pushed. |
| 13:13 |
Bmagic |
wowsa |
| 13:13 |
Bmagic |
Dyrcona++ |
| 13:14 |
Dyrcona |
I see it's gonna be one of those event-driven days where I don't get to what I planned to do. |
| 13:16 |
Dyrcona |
When I see bugs related to double clicking, I feel like saying, "Hello! Welcome to the Worldwide Web!" |
| 13:16 |
Dyrcona |
Double click to open an application was a dumb idea, because it is pretty much single click everywhere else. |
| 13:18 |
Dyrcona |
Now, to figure which thing to deal with next.... |
| 13:34 |
Bmagic |
format c: |
| 13:39 |
Dyrcona |
Command 'format' not found, did you mean.... |
| 13:41 |
|
mantis1 left #evergreen |
| 13:41 |
Dyrcona |
So, I'm leaning toward skipping this month's point releases again and going with the scheduled date in February. |
| 13:43 |
|
mantis1 joined #evergreen |
| 13:49 |
Bmagic |
Dyrcona: like that term: event-driven day. That has been my last 8 days. This new unapi nginx limitation has made a huge difference today already. It's stopped over 6k requests in the last 12 hours alone. So my event-driven days might be coming to an end. ::crosses fingers:: |
| 13:50 |
pinesol |
Showing latest 5 of 6 commits to Evergreen... |
| 13:50 |
pinesol |
[evergreen|Jane Sandberg] LP 1857351: set basic admin interface field order using route - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=7f23544> |
| 13:50 |
pinesol |
[evergreen|Jane Sandberg] LP 1857351: reorder a few other fm-editors - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=05abce1> |
| 13:50 |
pinesol |
[evergreen|Jane Sandberg] LP 1857351: admin-page grid column order respects specified fieldOrder - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=cedd847> |
| 13:50 |
pinesol |
[evergreen|Jane Sandberg] LP 1857351: fix field order for Recurring Fine Rules admin interface - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=85bfd65> |
| 13:50 |
pinesol |
[evergreen|Jane Sandberg] LP 1857351: fix field order for Course Terms - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=ced3a85> |
| 14:04 |
|
sandbergja_ joined #evergreen |
| 14:23 |
berick |
grabbing 1243 |
| 14:24 |
pinesol |
[evergreen|Bill Erickson] LP1894131 Sticky catalog holdings org select - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=1a9f22f> |
| 14:24 |
pinesol |
[evergreen|Bill Erickson] LP1894131 DB Stamp: Sticky Holdings Org - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=a8d4d1b> |
| 14:24 |
pinesol |
[evergreen|Bill Erickson] LP1901760 Shared Worker minor lint repair - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=45b0e0f> |
| 15:04 |
|
jihpringle joined #evergreen |
| 15:30 |
|
mantis1 left #evergreen |
| 16:04 |
|
khuckins joined #evergreen |
| 17:11 |
|
mmorgan left #evergreen |
| 17:33 |
|
Cocopuff2018 joined #evergreen |
| 18:01 |
pinesol |
News from qatests: Failed Installing Angular web client <http://testing.evergreen-ils.org/~live//archive/2021-01/2021-01-28_16:00:03/test.29.html> |
| 18:31 |
pinesol |
[evergreen|Terran McCanna] LP1855761 Typo when creating new carousel mapping - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=600eedd> |
| 18:31 |
pinesol |
[evergreen|Terran McCanna] LP1855761 Another typo fix - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=e7b5cdc> |
| 18:31 |
pinesol |
[evergreen|Michele Morgan] LP1855761 Fix additional occurrences of 'Succeessfully' - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=2066633> |
| 18:37 |
|
Cocopuff2018 joined #evergreen |
| 19:21 |
sandbergja |
Weird that the test was only failing on Chrome |
| 19:21 |
sandbergja |
But I'm glad that it failed somewhere, because it did rightly find a goofy typo |
| 19:23 |
pinesol |
[evergreen|Jane Sandberg] LP1857351 (follow-up): Fix typo that led to failing test - <http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=43a802a> |
| 19:34 |
|
sandbergja_ joined #evergreen |
| 20:12 |
|
troy__ joined #evergreen |
| 20:12 |
|
jonadab joined #evergreen |
| 20:17 |
|
kip joined #evergreen |
| 22:29 |
|
sandbergja__ joined #evergreen |
| 22:32 |
|
berick_ joined #evergreen |
| 22:33 |
|
yeats_ joined #evergreen |
| 22:34 |
|
dbs_ joined #evergreen |
