"deadlock detected" with parallel_tests and postgres #301

GCorbel · 2014-03-05T22:30:08Z

Hello,

In a rails application, I have this database.yml :

common: &common
  adapter: postgresql
  username: postgres
  password: postgres

test:
  <<: *common
  database: database_test<%= ENV['TEST_ENV_NUMBER'] %>

When I run my specs with time rake parallel:spec I have a lot of deadlock detected.

I added puts ENV['TEST_ENV_NUMBER'] In my spec_helper and it show the process number. In works when there is only one process.

What can I do?

The text was updated successfully, but these errors were encountered:

grosser · 2014-03-06T05:12:15Z

it sounds like it's trying to do things on the same database,
make sure it is really connected to different databases
``ActiveRecord::Base.connection.inspect` or something like that

maybe something else that causes the deadlocks or are you sure they are coming from postgres ?

GCorbel · 2014-03-08T16:02:24Z

I don't understand why but they seem to use all the same database.

I id this in my spec_helper :

puts ENV['TEST_ENV_NUMBER']
puts ActiveRecord::Base.connection.current_database

And it show :

3
database_test2 
....
4
database_test2
...

So they seeems to always use database_test2.

You can view my complete spec helper here : https://github.com/GCorbel/lescollectionneursassocies/blob/97321cba72327e5a31efae91f8e0cc61bb32d58c/spec/spec_helper.rb.

grosser · 2014-03-08T16:30:00Z

looks normal, maybe it's spring messing stuff up ?

On Sat, Mar 8, 2014 at 8:02 AM, Guirec Corbel notifications@github.comwrote:

I don't understand why but they seem to use all the same database.

I id this in my spec_helper :

puts ENV['TEST_ENV_NUMBER']
puts ActiveRecord::Base.connection.current_database

And it show :

3
database_test2
....
4
database_test2
...

So they seeems to always use database_test2.

You can view my complete spec helper here :
https://github.com/GCorbel/lescollectionneursassocies/blob/97321cba72327e5a31efae91f8e0cc61bb32d58c/spec/spec_helper.rb
.

Reply to this email directly or view it on GitHubhttps://github.com//issues/301#issuecomment-37101407
.

GCorbel · 2014-03-08T19:36:01Z

I don't thing than it's due to spring. I run the command rake and not spring rake.

GCorbel · 2014-03-08T19:50:26Z

It works when I do this my database.yml :

test:
  <<: *common
  database: lescollectionneurs_test
test1:
  <<: *common
  database: lescollectionneurs_test1
test2:
  <<: *common
  database: lescollectionneurs_test2
test3:
  <<: *common
  database: lescollectionneurs_test3
test4:
  <<: *common
  database: lescollectionneurs_test4

And this in my spec_helper :

ActiveRecord::Base.establish_connection("test#{ENV['TEST_ENV_NUMBER']}")

I have another problem, I do I18n.locale = :fr in my spec_helper but it does not seems to be considered because when my acceptances spec run, all translations are in english. I think than both bugs are related.

grosser · 2014-03-08T22:33:16Z

you could try printing debuggering in locale= to see where it is set from

does it also work with the old yml file + establish_connection "test" ?

ActiveRecord::Base.establish_connection("test")

is https://github.com/GCorbel/lescollectionneursassocies the project that's failing ?

GCorbel · 2014-03-08T22:41:16Z

I was wrong for translation. That was a mistake. I have another problem with the previous solution but this is not related to this issue. I will create another issue. Thanks for your help!

KurtPreston · 2014-09-04T15:32:10Z

I'm encountering the same issue. I've added the environment variable to config/database.yml, but it is always using my default test database unless I explicitly override it in spec/spec_helper as @GCorbel recommended.

Possibly relevant gems:

rspec 2.99.0
parallel_tests 1.0.6
activerecord 4.1.5

grosser · 2014-09-04T15:59:27Z

try <% puts "ENV IS #{ENV['TEST_ENV_NUMBER']}" %> in your database.yml ...
it should be different every time, if not please paste the commands used to
run the tests (should be printed by parallel_tests)

On Thu, Sep 4, 2014 at 8:32 AM, Kurt Preston notifications@github.com
wrote:

I'm encountering the same issue. I've added the environment variable to
config/database.yml, but it is always using my default test database
unless I explicitly override it in spec/spec_helper as @GCorbel
https://github.com/GCorbel recommended.

Possibly relevant gems:

rspec 2.99.0
parallel_tests 1.0.6
activerecord 4.1.5

—
Reply to this email directly or view it on GitHub
#301 (comment)
.

ipd · 2014-11-11T23:04:37Z

I have the same problem.

I added <% puts "ENV IS #{ENV['TEST_ENV_NUMBER']}" %> in my database.yml file as suggested by @grosser.

When i run rake parallel:speci get this:

ENV IS 
8 processes for 151 specs, ~ 18 specs per process
ENV IS

Which shows that the environment variable is not set, and that it is only being checked twice.

grosser · 2014-11-12T04:48:50Z

that sounds very strange ... can you see the commands parallel_tests
produces ?
what happens when you run them by hand ?

On Tue, Nov 11, 2014 at 3:04 PM, Ian Dickson notifications@github.com
wrote:

I have them same problem.

I added <% puts "ENV IS #{ENV['TEST_ENV_NUMBER']}" %> in my database.yml
file as suggested by @grosser https://github.com/grosser.

When i run rake parallel:speci get this:

ENV IS
8 processes for 151 specs, ~ 18 specs per process
ENV IS

Which shows that the environment variable is not set, and that it is only
being checked twice.

—
Reply to this email directly or view it on GitHub
#301 (comment)
.

jarkko · 2015-04-20T11:32:54Z

For the record, I'm having this same issue. I put in some debug information to rails_helper.rb, and here's what I found out:

For plain rake spec, ActiveRecord::Base.connection is not set before the line where environment is required. In fact, the whole ActiveRecord is an unitialized constant.
With parallel_rspec (or rake parallel:spec) this is not the case. ActiveRecord::Base.connection.current_database is always the *_test, without the number.

So it seems like with parallel test, the Rails environment is already loaded at that stage, and the db connection established. However, when that was done, the ENV var was not yet set. However, it is set correctly already from the first line of rails_helper.rb.

Investigating…

jarkko · 2015-04-20T11:56:15Z

OK, it was indeed spring springing up in action (no pun intended in the background). So if you run into this, make sure spring isn't running (ps auxww | grep spring) and if it is, stop it and remove spring from your gemfile.

grosser · 2015-04-20T14:43:16Z

removing spring from binstubs is good enough or using DISABLE_SPRING=1
rake xxx

I guess parallel_tests tasks should just blow up if spring is loaded ...

On Mon, Apr 20, 2015 at 4:56 AM, Jarkko Laine notifications@github.com
wrote:

OK, it was indeed spring springing up in action (no pun intended in the
background). So if you run into this, make sure spring isn't running (ps
auxww | grep spring) and if it is, stop it and remove spring from your
gemfile.

—
Reply to this email directly or view it on GitHub
#301 (comment)
.

grosser · 2015-11-30T19:37:39Z

can you make this a PR ... would be great to have this work out of the box :)

I don't know very much about rake tasks, so I don't know how to fix if it is running with rake, but this fixes bundle exec parallel_rspec.

yurygnutov · 2017-08-31T14:27:25Z

Hi all,

I experience kinda same problem with the gem when include that in Rails5 app. Even though, locally everything passed well with parallel_tests, on our GitLab CI the following problem:

$ bin/rake parallel:create
Database 'core_test' already exists
Database 'core_test' already exists
Database 'core_test' already exists
Database 'core_test' already exists

rake aborted!
ActiveRecord::StatementInvalid: PG::ObjectInUse: ERROR:  database "core_test" is being accessed by other users
DETAIL:  There is 1 other session using the database.
: DROP DATABASE IF EXISTS "core_test"

cat of database.yml during CI job execution clearly says that there everything well with the config.
Our gitlab-ci.yml looks like:

---
image: registry.core.com/core-ci:v1.0

services:
  - postgres:9.5
  - redis:3.2.1-alpine

variables:
  POSTGRES_DB: core_test
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: "test123"
  DATABASE_URL: "postgres://runner:test123@postgres/core_test"
  REDIS_URI: "redis://redis:6379"

cache:
  key: ${CI_BUILD_REF_NAME}
  paths:
    - node_modules/
    - bin/
    - vendor/bundle

before_script:
  - mkdir -p ~/.ssh && ssh-keyscan -t rsa gitlab.core.com >> ~/.ssh/known_hosts
  - eval $(ssh-agent -s)
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  - node -v
  - npm -v
  - ruby -v
  - bundler -v
  - npm install
  - npm run build
  - bin/bundle install --path vendor/bundle --binstubs --jobs $(nproc) "${FLAGS[@]}"

stages:
  - rspec

rspec:
  stage: rspec
  script:
    - cat config/database.yml
    - bin/rake db:migrate
    - bin/rake parallel:create
    - bin/rake parallel:prepare
    - bin/rake parallel:setup
    - bin/rake parallel:spec

tommyalvarez · 2018-02-28T16:58:55Z

@yurygnutov did you managed to solve this issue?

GCorbel closed this as completed Mar 8, 2014

soberstadt added a commit to soberstadt/parallel_tests that referenced this issue Nov 30, 2015

Disable Spring by default - grosser#301

e06f7eb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"deadlock detected" with parallel_tests and postgres #301

"deadlock detected" with parallel_tests and postgres #301

GCorbel commented Mar 5, 2014

grosser commented Mar 6, 2014

GCorbel commented Mar 8, 2014

grosser commented Mar 8, 2014

GCorbel commented Mar 8, 2014

GCorbel commented Mar 8, 2014

grosser commented Mar 8, 2014

GCorbel commented Mar 8, 2014

KurtPreston commented Sep 4, 2014

grosser commented Sep 4, 2014

ipd commented Nov 11, 2014

grosser commented Nov 12, 2014

jarkko commented Apr 20, 2015

jarkko commented Apr 20, 2015

grosser commented Apr 20, 2015

grosser commented Nov 30, 2015

yurygnutov commented Aug 31, 2017

tommyalvarez commented Feb 28, 2018

"deadlock detected" with parallel_tests and postgres #301

"deadlock detected" with parallel_tests and postgres #301

Comments

GCorbel commented Mar 5, 2014

grosser commented Mar 6, 2014

GCorbel commented Mar 8, 2014

grosser commented Mar 8, 2014

GCorbel commented Mar 8, 2014

GCorbel commented Mar 8, 2014

grosser commented Mar 8, 2014

GCorbel commented Mar 8, 2014

KurtPreston commented Sep 4, 2014

grosser commented Sep 4, 2014

ipd commented Nov 11, 2014

grosser commented Nov 12, 2014

jarkko commented Apr 20, 2015

jarkko commented Apr 20, 2015

grosser commented Apr 20, 2015

grosser commented Nov 30, 2015

yurygnutov commented Aug 31, 2017

tommyalvarez commented Feb 28, 2018