Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to run pgbench in docker setup #596

Closed
Tracked by #2028
LizardWizzard opened this issue Sep 13, 2021 · 4 comments
Closed
Tracked by #2028

failed to run pgbench in docker setup #596

LizardWizzard opened this issue Sep 13, 2021 · 4 comments
Assignees
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug
Milestone

Comments

@LizardWizzard
Copy link
Contributor

I'm running pgbench -s5 -i in docker setup as it is in our console repo.

pgbench output:

NOTICE:  table "pgbench_accounts" does not exist, skipping
NOTICE:  table "pgbench_branches" does not exist, skipping
NOTICE:  table "pgbench_history" does not exist, skipping
NOTICE:  table "pgbench_tellers" does not exist, skipping
creating tables...
generating data (client-side)...
500000 of 500000 tuples (100%) done (elapsed 0.28 s, remaining 0.00 s)
vacuuming...
pgbench: fatal: query failed: ERROR:  could not read block 0 in rel 1663/16385/16399.2 from page server at lsn 0/08D8C7A8
CONTEXT:  while scanning block 0 of relation "public.pgbench_branches"
pgbench: query was: vacuum analyze pgbench_branches

In pageserver logs:

INFO Created database 1663/16385, copied 1014 blocks in 243 rels at 0/169F110, location: pageserver::restore_local_repo, pageserver/src/restore_local_repo.rs:646
INFO switched segment 0/1FFEB00 to 0/2002660, pageserver/src/walreceiver.rs:257
INFO switched segment 0/2FFFDF8 to 0/301D870, pageserver/src/walreceiver.rs:257
ERRO get_page_at_lsn: Timed out while waiting for WAL record at LSN 0/8D89210 to arrive, pageserver/src/page_service.rs:377
INFO switched segment 0/3FFF090 to 0/4004C40, pageserver/src/walreceiver.rs:257
INFO switched segment 0/4FFE860 to 0/5005EF8, pageserver/src/walreceiver.rs:257
ERRO get_page_at_lsn: Timed out while waiting for WAL record at LSN 0/8D89210 to arrive, pageserver/src/page_service.rs:377
INFO switched segment 0/5FFFAF8 to 0/6003658, pageserver/src/walreceiver.rs:257
INFO switched segment 0/6FFED28 to 0/7000D88, pageserver/src/walreceiver.rs:257
ERRO get_page_at_lsn: Timed out while waiting for WAL record at LSN 0/8D89210 to arrive, pageserver/src/page_service.rs:377
INFO switched segment 0/7FFFF80 to 0/8001FE0, pageserver/src/walreceiver.rs:257
WARN drop_segment called on non-existent relish 1663/16385/16389 at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16389_fsm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16389_vm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16386 at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16386_fsm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16386_vm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16395 at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16395_fsm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16395_vm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16392 at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16392_fsm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
WARN drop_segment called on non-existent relish 1663/16385/16392_vm at 0/8D8A6D0, pageserver/src/layered_repository.rs:820
ERRO get_page_at_lsn: Timed out while waiting for WAL record at LSN 0/8D8C7A8 to arrive, pageserver/src/page_service.rs:377

Timeouts are probably fine, because I had runs with timeouts but without this warnings

@lubennikovaav is it #502 or something new?

@LizardWizzard
Copy link
Contributor Author

Warning was fixed some time ago, wait lsn timeout error should be fixed in #630

@kelvich
Copy link
Contributor

kelvich commented Dec 22, 2021

@LizardWizzard let's check that again since backpressure was merged/fixed? #803

@kelvich kelvich added t/bug Issue Type: Bug c/storage/pageserver Component: storage: pageserver labels Dec 22, 2021
@LizardWizzard LizardWizzard self-assigned this Dec 22, 2021
@neondatabase-bot neondatabase-bot bot added this to the 2022/07 milestone Jul 5, 2022
@neondatabase-bot neondatabase-bot bot modified the milestones: 2022/07, 2022/08 Jul 25, 2022
@neondatabase-bot neondatabase-bot bot modified the milestones: 2022/08, 2023/03 Dec 20, 2022
@shanyp
Copy link
Contributor

shanyp commented Feb 23, 2023

@LizardWizzard could this be closed ? or do we need it for anything else ?

@LizardWizzard
Copy link
Contributor Author

LizardWizzard commented Feb 27, 2023

Lets close this in favor of #1587.

Current issue was discovered when I was running whole setup in docker on mac os, so it was quite slow. I no longer have this setup at hand, so I think it makes sense to add additional test for backpressure with CPU limits applied for pageserver process

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/storage/pageserver Component: storage: pageserver t/bug Issue Type: Bug
Projects
None yet
Development

No branches or pull requests

3 participants