[feature] Rack attack #1419

Pralish · 2023-02-20T15:43:41Z

Addresses: #538

config/initializers/rack_attack.rb

donrestarone · 2023-02-27T13:40:00Z

@Pralish please also investigate why the server is unhealthy after terminating a request from rack-timeout.

The request times out:

A Rack::Timeout::RequestTimeoutException occurred in content#show:

  Request ran for longer than 15000ms 
  config/initializers/rack_timeout.rb:21:in `call'

and the puma Server hangs, and is unable to serve subsequent requests

EDIT:

Redis connection timeouts are causing the server to crash as well:

A Redis::TimeoutError occurred in background at 2023-02-28 00:12:35 UTC :

  Connection timed out
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb:58:in `block in _read_from_socket'

-------------------------------
Backtrace:
-------------------------------

  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb:58:in `block in _read_from_socket'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb:54:in `loop'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb:54:in `_read_from_socket'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb:47:in `gets'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/connection/ruby.rb:382:in `read'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:311:in `block in read'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:299:in `io'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:310:in `read'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:161:in `block in call'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:279:in `block (2 levels) in process'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:420:in `ensure_connected'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:269:in `block in process'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:356:in `logging'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:268:in `process'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:161:in `call'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:257:in `block in call_with_timeout'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:331:in `with_socket_timeout'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/client.rb:256:in `call_with_timeout'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis.rb:269:in `block in send_blocking_command'
  /home/ubuntu/.rbenv/versions/2.6.6/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis.rb:268:in `send_blocking_command'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/commands/lists.rb:270:in `_bpop'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-4.6.0/lib/redis/commands/lists.rb:167:in `brpop'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-namespace-1.8.2/lib/redis/namespace.rb:476:in `call_with_namespace'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/redis-namespace-1.8.2/lib/redis/namespace.rb:352:in `block (2 levels) in <class:Namespace>'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq/fetch.rb:47:in `block in retrieve_work'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq.rb:100:in `block in redis'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.5/lib/connection_pool.rb:63:in `block (2 levels) in with'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.5/lib/connection_pool.rb:62:in `handle_interrupt'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.5/lib/connection_pool.rb:62:in `block in with'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.5/lib/connection_pool.rb:59:in `handle_interrupt'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/connection_pool-2.2.5/lib/connection_pool.rb:59:in `with'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq.rb:97:in `redis'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq/fetch.rb:47:in `retrieve_work'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq/processor.rb:83:in `get_one'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq/processor.rb:95:in `fetch'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq/processor.rb:77:in `process_one'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq/processor.rb:68:in `run'


  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq/util.rb:56:in `watchdog'
  /var/www/violet/shared/bundle/ruby/2.6.0/gems/sidekiq-6.4.1/lib/sidekiq/util.rb:65:in `block in safe_thread'

-------------------------------
Data:
-------------------------------

  * data: {:sidekiq=>{}}

I discovered we are not aborting requests correctly, see here: #1434

Addresses: #1434 #1419 (comment)

Addresses: #1434 #1419 (comment) Co-authored-by: Pralish Kayastha <50227291+Pralish@users.noreply.github.com>

donrestarone · 2023-03-20T13:52:32Z

app/mailers/rack_attack_mailer.rb

@@ -0,0 +1,11 @@
+class RackAttackMailer < ApplicationMailer
+  def limit_exceeded(user, error_limit_exceeded = false)
+    @user = user


@Pralish you should CC the email to global admins as well

…rate_rack_attack

github-actions · 2023-03-20T16:56:30Z

Deployed review-app can be viewed at https://review-1419.violet-test.net

donrestarone · 2023-03-22T16:28:10Z

🧪 test plan

set ERROR_PER_MINUTE=1 and REQUEST_PER_MINUTE=2 ⌛ by running dokku config:set review-1419 ERROR_PER_MINUTE=1 REQUEST_PER_MINUTE=2
user should not be able to raise more than 1 error a minute ✅
user should not be able to make more than 2 requests per minute ✅

github-actions · 2023-03-22T17:38:44Z

Deployed review-app can be viewed at https://review-1419.violet-test.net

github-actions · 2023-03-24T15:50:35Z

Deployed review-app can be viewed at https://review-1419.violet-test.net

donrestarone · 2023-03-25T17:53:01Z

.env.test

@@ -9,4 +9,7 @@ APP_HOST=lvh.me:5250
 RECAPTCHA_SITE_KEY=6Lc6BAAAAAAAAChqRbQZcn_yyyyyyyyyyyyyyyyy
 RECAPTCHA_SECRET_KEY=6Lc6BAAAAAAAAKN3DRm6VA_xxxxxxxxxxxxxxxxx
 SECRET_KEY_BASE='38c72586473e364229897f24f1892f1dc5565776878aa4d8c6bf051258622bd2e923b926ab59b40f912b661216f764d993e8d6b8bbfbc33026e5c954b6c51f9b'
+REQUEST_PER_MINUTE=5
+ERROR_PER_MINUTE=3
+PERIOD_MULTIPLIER=2


@Pralish could you create a GitHub issue with documentation on how to tune this feature with these 3 variables?

github-actions · 2023-03-25T17:58:49Z

Deployed review-app can be viewed at https://review-1419.violet-test.net

rack attack

b78d9ff

Pralish added WIP early-alpha UNSAFE do not merge labels Feb 20, 2023

donrestarone reviewed Feb 20, 2023

View reviewed changes

config/initializers/rack_attack.rb Outdated Show resolved Hide resolved

Pralish added 4 commits February 21, 2023 20:45

update config

daa176c

separate var for error limit

4e8eba6

global admins should not be throttled

8d707b9

update config, write tests, send email

3159888

Pralish added 2 commits March 9, 2023 09:15

merged with master

8b9af17

error limit throttle

2ee8b9c

Pralish mentioned this pull request Mar 13, 2023

remove rack-timeout and integrate slowpoke #1454

Merged

donrestarone pushed a commit that referenced this pull request Mar 13, 2023

[feature] replace rack-timeout with slowpoke

df61ced

Addresses: #1434 #1419 (comment)

donrestarone mentioned this pull request Mar 13, 2023

[feature] replace rack-timeout with slowpoke #1457

Merged

donrestarone added a commit that referenced this pull request Mar 13, 2023

[feature] replace rack-timeout with slowpoke (#1457)

bd50172

Addresses: #1434 #1419 (comment) Co-authored-by: Pralish Kayastha <50227291+Pralish@users.noreply.github.com>

Pralish added 2 commits March 17, 2023 16:34

merged with master

91cce21

fix test

64f9f86

Pralish changed the title ~~[WIP] rack attack~~ Rack attack Mar 17, 2023

disable timeout for ember path

0c4fa55

donrestarone reviewed Mar 20, 2023

View reviewed changes

Pralish added 3 commits March 20, 2023 21:45

Merge branch 'master' of github.com:restarone/violet_rails into integ…

fb070d4

…rate_rack_attack

set env variable on test and bcc global admins

c270742

fix test

24e69ef

Pralish marked this pull request as ready for review March 20, 2023 16:46

Pralish added deploy-review-app and removed WIP early-alpha UNSAFE do not merge labels Mar 20, 2023

donrestarone changed the title ~~Rack attack~~ [feature] Rack attack Mar 22, 2023

Merge branch 'master' into integrate_rack_attack

3df2f90

donrestarone added the Passing UAT on Testnet ✅ label Mar 22, 2023

Merge branch 'master' into integrate_rack_attack

cd4fab9

donrestarone reviewed Mar 25, 2023

View reviewed changes

donrestarone changed the base branch from master to rc March 25, 2023 17:55

donrestarone merged commit f9d84ba into rc Mar 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature] Rack attack #1419

[feature] Rack attack #1419

Pralish commented Feb 20, 2023 •

edited by donrestarone

Loading

donrestarone commented Feb 27, 2023 •

edited

Loading

donrestarone Mar 20, 2023

github-actions bot commented Mar 20, 2023

donrestarone commented Mar 22, 2023 •

edited

Loading

github-actions bot commented Mar 22, 2023

github-actions bot commented Mar 24, 2023

donrestarone Mar 25, 2023

github-actions bot commented Mar 25, 2023

[feature] Rack attack #1419

[feature] Rack attack #1419

Conversation

Pralish commented Feb 20, 2023 • edited by donrestarone Loading

donrestarone commented Feb 27, 2023 • edited Loading

donrestarone Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot commented Mar 20, 2023

donrestarone commented Mar 22, 2023 • edited Loading

github-actions bot commented Mar 22, 2023

github-actions bot commented Mar 24, 2023

donrestarone Mar 25, 2023

Choose a reason for hiding this comment

github-actions bot commented Mar 25, 2023

Pralish commented Feb 20, 2023 •

edited by donrestarone

Loading

donrestarone commented Feb 27, 2023 •

edited

Loading

donrestarone commented Mar 22, 2023 •

edited

Loading