Integration tests are not working #354

Overseven · 2024-06-05T12:24:36Z

Currently, integration tests are not working. It falling with warden node never became running error.

Command:

cd tests && just test

Logs:

go test -run 'TestIntegration' -v ./integration_test.go
=== RUN   TestIntegration
=== RUN   TestIntegration/CreateSpace
setup test: CreateSpace
    warden_node.go:83: 
                Error Trace:    /Users/igor/work/cosmos/warden/wardenprotocol/tests/framework/exec/warden_node.go:83
                                                        /usr/local/go/src/runtime/asm_arm64.s:1222
                Error:          Received unexpected error:
                                exec: /var/folders/83/_nhy87k943n6968qf_t87lvw0000gn/T/TestIntegration3218378275/001/wardend --log_no_color start --home /var/folders/83/_nhy87k943n6968qf_t87lvw0000gn/T/TestIntegrationCreateSpace2494717894/001 --x-crisis-skip-assert-invariants
                                err: signal: killed
                                stdout: 
                                stderr: 
                Test:           TestIntegration/CreateSpace
    warden_node.go:91: 
                Error Trace:    /Users/igor/work/cosmos/warden/wardenprotocol/tests/framework/exec/warden_node.go:91
                                                        /Users/igor/work/cosmos/warden/wardenprotocol/tests/cases/create_space.go:26
                                                        /Users/igor/work/cosmos/warden/wardenprotocol/tests/integration_test.go:44
                Error:          Condition never satisfied
                Test:           TestIntegration/CreateSpace
                Messages:       warden node never became running
tests duration: 5.004322542s
--- FAIL: TestIntegration (15.37s)
    --- FAIL: TestIntegration/CreateSpace (5.00s)
FAIL
FAIL    command-line-arguments  15.782s
FAIL
error: Recipe `test` failed on line 3 with exit code 1

The text was updated successfully, but these errors were encountered:

Overseven · 2024-06-05T13:38:39Z

Also, it would be great to write more integration tests that will include all possible user actions.

Pitasi · 2024-06-05T14:45:43Z

it works for me (on current main branch) so it might be a timing issue. The expected output is this

❯ just test
go test -run 'TestIntegration' -v ./integration_test.go
=== RUN   TestIntegration
=== RUN   TestIntegration/CreateSpace
setup test: CreateSpace
run test: CreateSpace
tests duration: 1.402828709s
--- PASS: TestIntegration (9.93s)
    --- PASS: TestIntegration/CreateSpace (1.40s)
PASS
ok      command-line-arguments  10.242s

In order to understand if it's really just a matter of timing, could you increase the timeout here:

wardenprotocol/tests/framework/exec/warden_node.go

Line 93 in dbf46c8

}, 5*time.Second, 5*time.Millisecond, "warden node never became running")

from 5 seconds to something like 5 minutes? It should be plenty of time to start a wardend node :)

Overseven · 2024-06-05T14:56:16Z

Same result:

% just test

go test -run 'TestIntegration' -v ./integration_test.go
=== RUN   TestIntegration
=== RUN   TestIntegration/CreateSpace
setup test: CreateSpace
    warden_node.go:83: 
                Error Trace:    /Users/igor/work/cosmos/warden/wardenprotocol/tests/framework/exec/warden_node.go:83
                                                        /usr/local/go/src/runtime/asm_arm64.s:1222
                Error:          Received unexpected error:
                                exec: /var/folders/83/_nhy87k943n6968qf_t87lvw0000gn/T/TestIntegration1475181834/001/wardend --log_no_color start --home /var/folders/83/_nhy87k943n6968qf_t87lvw0000gn/T/TestIntegrationCreateSpace255166513/001 --x-crisis-skip-assert-invariants
                                err: signal: killed
                                stdout: 
                                stderr: 
                Test:           TestIntegration/CreateSpace
    warden_node.go:91: 
                Error Trace:    /Users/igor/work/cosmos/warden/wardenprotocol/tests/framework/exec/warden_node.go:91
                                                        /Users/igor/work/cosmos/warden/wardenprotocol/tests/cases/create_space.go:26
                                                        /Users/igor/work/cosmos/warden/wardenprotocol/tests/integration_test.go:44
                Error:          Condition never satisfied
                Test:           TestIntegration/CreateSpace
                Messages:       warden node never became running
tests duration: 5m0.012776042s
--- FAIL: TestIntegration (310.41s)
    --- FAIL: TestIntegration/CreateSpace (300.01s)
FAIL
FAIL    command-line-arguments  310.878s
FAIL
error: Recipe `test` failed on line 3 with exit code 1

Pitasi · 2024-06-05T15:03:49Z

can you change that same line to this:

	}, 5*time.Second, 5*time.Millisecond, "warden node never became running", "stdout", w.Stdout.String(), "stderr", w.Stderr.String())

so we can see the logs from the node

Overseven · 2024-06-05T15:11:02Z

Messages: warden node never became running%!(EXTRA string=stdout, string=, string=stderr, string=)
tests duration: 5.00473425s

Pitasi · 2024-06-05T15:25:09Z

ah gotcha, the right syntax for the log message would be something like this

	}, 5*time.Second, 5*time.Millisecond, "warden node never became running.\nstdout:\n%s\nstderr:\n%s", w.Stdout.String(), w.Stderr.String())

Overseven · 2024-06-05T15:27:41Z

I also tried this, but didn't get any logs:

require.Eventually(t, func() bool {
  stderr := w.Stderr.String()
  stdout := w.Stderr.String()
  if len(stderr) != 0 {
    println("!!! ERR", stderr)
  }
  if len(stdout) != 0 {
    println("!!! LOG", stdout)
  }
  ...
}

Pitasi · 2024-06-05T15:28:30Z

I also tried this, but didn't get any logs:

require.Eventually(t, func() bool {
  stderr := w.Stderr.String()
  stdout := w.Stderr.String()
  if len(stderr) != 0 {
    println("!!! ERR", stderr)
  }
  if len(stdout) != 0 {
    println("!!! LOG", stdout)
  }
  ...
}

stdout := w.Stderr.String()

should be

stdout := w.Stdout.String()

Overseven · 2024-06-05T15:29:24Z

}, 5time.Second, 5time.Millisecond, "warden node never became running.\nstdout:\n%s\nstderr:\n%s", w.Stdout.String(), w.Stderr.String())

Messages:       warden node never became running.
                                stdout:
                                
                                stderr:
tests duration: 5.006430208s

Overseven · 2024-06-05T15:30:28Z

stdout := w.Stderr.String()

should be

stdout := w.Stdout.String()

Fixed, but got the same result

Pitasi · 2024-06-05T15:32:24Z

so strange, it works for me

!!! LOG 5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000f2bae0, group}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000d67be0, icacontroller}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x1400052a780, acc}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000eb8eb0, crisis}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000b883d0, slashing}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000e7c780, gov}" version=0
5:30PM INF Upgrading IAVL storage for faster queries + execution on live state. This may take a while commit=436F6D6D697449447B5B5D3A307D module=server store_key="KVStoreKey{0x14000f59bc0, circuit}" version=0
...

Overseven · 2024-06-05T15:34:00Z

Do I need to make a snapshot or run a node before starting the test?

Pitasi · 2024-06-05T15:35:51Z

no, i just cloned this repo and only run just test and it worked for me, without doing anything else 🤔

Pitasi · 2024-06-05T15:40:50Z

I'm a bit out of ideas, maybe you can try running with a debugger attached? It's so weird that the stdout is not being captured

what OS are you using?

@optifat can you try to run these and see if they work for you?

Overseven · 2024-06-05T15:41:38Z

For more information - I'm running it on Mac M1. I noticed that some commands can't be executed without editing.
For example, this line from just localnet falling with error:

sed -i 's/cors_allowed_origins = \[\]/cors_allowed_origins = ["*"]/' ~/.warden/config/config.toml

Error:

sed: 1: "/Users/igor/.warden/con ...": command i expects \ followed by text

And it works only this way:

sed -i'' -e 's/cors_allowed_origins = \[\]/cors_allowed_origins = ["*"]/' ~/.warden/config/config.toml

Pitasi · 2024-06-05T15:51:34Z

I'm also on Mac and you're right, that sed command only works if you have GNU sed installed (brew install gnu-sed). Still I don't understand why the tests are not working :(

optifat · 2024-06-06T07:09:01Z

I'm a bit out of ideas, maybe you can try running with a debugger attached? It's so weird that the stdout is not being captured

what OS are you using?

@optifat can you try to run these and see if they work for you?

No, I got

run test: CreateSpace
    warden_cli.go:36: 
        	Error Trace:	/Users/pavel/warden/wardenprotocol/tests/framework/exec/warden_cli.go:36
        	            				/Users/pavel/warden/wardenprotocol/tests/cases/create_space.go:31
        	            				/Users/pavel/warden/wardenprotocol/tests/integration_test.go:46
        	Error:      	Received unexpected error:
        	            	exec: sh -c /var/folders/v2/cjl08_v9281cygqjl_vf60lm0000gn/T/TestIntegration1010986435/001/wardend tx warden new-space --from alice --yes --node tcp://127.0.0.1:64518 --home /var/folders/v2/cjl08_v9281cygqjl_vf60lm0000gn/T/TestIntegrationCreateSpace3004294498/001 --chain-id warden --keyring-backend test --keyring-dir /var/folders/v2/cjl08_v9281cygqjl_vf60lm0000gn/T/TestIntegrationCreateSpace3004294498/001
        	            	err: exit status 1
        	            	stdout: 
        	            	stderr: rpc error: code = Unknown desc = warden is not ready; please wait for first block: invalid height
        	Test:       	TestIntegration/CreateSpace
tests duration: 2.067708833s

Pitasi · 2024-06-14T09:03:45Z

I'm going to close this as we couldn't reproduce on several machines.

Overseven added bug Something isn't working test labels Jun 5, 2024

Overseven mentioned this issue Jun 5, 2024

x/intent: add unified MsgNewAction tx for creating new actions #321

Merged

Overseven added the protocol label Jun 5, 2024

Pitasi changed the title ~~Int tests are not working~~ Integration tests are not working Jun 14, 2024

Pitasi closed this as completed Jun 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integration tests are not working #354

Integration tests are not working #354

Overseven commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024 •

edited

Loading

Pitasi commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024 •

edited

Loading

Pitasi commented Jun 5, 2024

optifat commented Jun 6, 2024

Pitasi commented Jun 14, 2024

Integration tests are not working #354

Integration tests are not working #354

Comments

Overseven commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024

Overseven commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024 • edited Loading

Pitasi commented Jun 5, 2024

Pitasi commented Jun 5, 2024

Overseven commented Jun 5, 2024 • edited Loading

Pitasi commented Jun 5, 2024

optifat commented Jun 6, 2024

Pitasi commented Jun 14, 2024

Overseven commented Jun 5, 2024 •

edited

Loading

Overseven commented Jun 5, 2024 •

edited

Loading