-
-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] 230A Nondeterministic with --enable-tf 2 and/or --psy-rd >0 #124
Comments
I've landed a few psy-rd fixes in |
doesn't seem so. btw, this is not only on certain clips, it occurs on every clip i have tried. for reference though this was tested using the "sunflower_1080p25.y4m" test clip
Svt[info]: -------------------------------------------
Svt[info]: ------------------------------------------- |
Thanks @Ravice for trying it out. It's clear that there's still work left to do to bring to full determinism. It's a challenge for someone who uses an Apple Silicon dev environment, as ARM devices never had fully-deterministic encodes even with mainline SVT-AV1. I'll see what I (or other dev teams) can do. Are you still seeing crashes BTW? |
@juliobbv-p tentatively seems as though the LP>4 720p crashes are fixed 👍 forced LP6 1080p also appears to be going strong. |
update: SPY-RD is also nondeterministic at LP > 1
Svt[info]: -------------------------------------------
Svt[info]: -------------------------------------------
Svt[info]: ------------------------------------------- |
@Ravice interesting observations, as basically spy-rd boils down to a set of adjustments to distortion values based on block type and size, so I find it surprising that turning spy-rd on is causing nondeterministic behavior. Given that enable-tf also causes nondeterminism, there's probably some inherent brittleness where determinism in SVT-AV1 (on x86) is only guaranteed by certain hidden constraints that we're breaking with enable-tf 2, psy-rd, and/or spy-rd. Annoying. By any chance, are the no-spy LP5 files binary identical, or just happen to be the same size? |
LP1 files are binary identical, no-spy LP5 are also binary identical. |
@Ravice Maybe it'll have fixed |
built testing ec65071 sadly, seems not.
Svt[info]: -------------------------------------------
Svt[info]: Level of Parallelism: 1 -TP2 LP5 Svt[info]: Level of Parallelism: 1 |
Wow, that is an immense improvement in terms of consistency for psy-rd compared to what we experienced before. |
Overview
230A has determinism issues with both the --psy-rd parameter and --enable-tf 2 parameter, individually and in combination. impact on quality is somewhat mild, but at low bitrate (600-800) can be the difference between keeping or deleting some detail randomly.
both become deterministic when encoding with --lp 1
Branch
master f560b40 and testing 4486b1d
Reproduction
encode with 10-bit and --enable-tf 2 or --psy-rd >0 respectively with --lp >1
Platform (please complete the following information):
windows 10 msys2/Clang 18.1.8
cmake .. -DBUILD_SHARED_LIBS=0 -DCMAKE_BUILD_TYPE=Release -G "MSYS Makefiles" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_LINKER=lld -DCMAKE_CXX_FLAGS="-fuse-ld=lld -O3 -march=native -flto=thin" -DCMAKE_C_FLAGS="-fuse-ld=lld -O3 -march=native -flto=thin" -DCMAKE_C_FLAGS_INIT="-flto=16 -static -static-libgcc -static-libstdc++" -DCMAKE_EXE_LINKER_FLAGS="-flto -static -static-libgcc -static-libstdc++" -DBUILD_TESTING=0 -DBUILD_DEC=0 -DSVT_AV1_PGO=0 -DSVT_AV1_LTO=1 -DNATIVE=1 -DENABLE_AVX512=0
Additional context
by changing the thread counts in enc_handle.c the offending threads have perhaps been narrowed down from either end by individually increasing threads for LP1, and setting LP5 threads to 1.
--psy-rd
LP1 becomes nondeterministic with "enc_dec_process_init_count" > 1
LP1 becomes nondeterministic with "motion_estimation_process_init_count" > 1
LP5 psy-rd becomes deterministic if "enc_dec_process_init_count" == 1 and "motion_estimation_process_init_count" == 1 (this runs very slowly because of enc_dec 1)
--enable-tf 2
requires only that "motion_estimation_process_init_count" == 1 for LP5 to become deterministic
TF1 has no determinism issues tested all presets between 1 and 8
thread tests were performed at --preset 8, but the issue occurs with --preset 5 and 1-3 also.
additionally, 230A has introduced some crashes compared to 230 when encoding 720p at >LP4. this seems to always occur at the start of a minigop, but not always at the same place. crash is shown as a segfault/access violation. this also occurs at 1080p LP6 if you disable the condition that 1080p always encodes with LP5 settings, however 1080p LP5 has no problems. this may or may not be related to the determinism issues cause as it is also affected by LP.
The text was updated successfully, but these errors were encountered: