Decoding h264 10bit with avsw causes artifacts

[quamt]

When decoding h264 10bit video material there are artifacts in the output.
Normal h264 has no issue.
When piping via ffmpeg no issue too.
This seems to be only happening with h264 10bit video material.

`VCEEncC (x64) 6.06 by rigaya, build Nov  9 2020 20:22:42
AMF Version 1.4.16
reader: raw, avs, vpy, avhw [H.264/AVC, H.265/HEVC, MPEG2]
VCE available
device #0: Radeon RX 570`

`device #0: Radeon RX 570
H.264/AVC encode features
acceleration:   Hardware-accelerated
max profile:    High
max level:      5.2
max bitrate:    100000 kbps
ref frames:     1-16
Bframe support: no
HW instances:   1
max streams:    16

H.264/AVC encoder input:
Width:       64 - 4096
Height:      64 - 4096
alignment:   32
Interlace:   no
pix format:  NV12(native), YUV420P, YV12, BGRA, RGBA, ARGB
memory type: DX11(native), OPENCL, OPENGL, HOST

H.264/AVC encoder output:
Width:       64 - 4096
Height:      64 - 4096
alignment:   32
Interlace:   no
pix format:  NV12(native)
memory type: DX11(native), OPENCL, OPENGL, HOST

H.265/HEVC encode features
acceleration:   Hardware-accelerated
max profile:    main
max level:      6.2
max bitrate:    2147483 kbps
ref frames:     1-16
max streams:    16

H.265/HEVC encoder input:
Width:       192 - 4096
Height:      128 - 4096
alignment:   32
Interlace:   no
pix format:  NV12(native), P010(native), YUV420P, YV12, BGRA, RGBA, ARGB
memory type: DX11(native), OPENCL, OPENGL, HOST

H.265/HEVC encoder output:
Width:       192 - 4096
Height:      128 - 4096
alignment:   32
Interlace:   no
pix format:  NV12(native), P010(native)
memory type: DX11(native), OPENCL, OPENGL, HOST`

`ffmpeg     version: N-98575-g16c2ed4362
avutil     version:  56. 57. 100
avcodec    version:  58. 97. 102
avformat   version:  58. 49. 100
avfilter   version:   7. 87. 100
swresample version:   3.  8. 100`

Artifacts:

[rigaya]

Unfortunately I don't seem to have trouble with my H.264 10bit 420/422 files, encoded by x264 10bit.

I also tried samples on the web, but without any trouble.
(There is a sample on [3 Codecs, Framerates and Subtitles] > [15 Hi10 Anime Samples] > [[Underwater] Another - sample H264 Hi10P 1080p.avi])

Video Information from my test file (H.264 10bit 420).

[quamt]

@rigaya I tested the files from that page and they look OK.
When I use the files I have they get artifacts.
I tested it also with nvencc and the avsw and there are no artifacts in the encoded video.
This seems to only apply for vceenc and avsw.
When I pipe via ffmpeg using rawvideo it's OK.
I tried to let ffmpeg pipe it to vcodec libx264 and run the pipe to vceenc the artifacts appear too.

[quamt]

@rigaya I installed an earlier version of AMD the driver, as I thought the issue comes from there. But it is still creating artifacts.
Anything I can check?
Because the issue is happening on more than one 10bit X264 file.

[rigaya]

It's weird that some files comes with artifacts, and it's more weird that VCEEncC creates artifact but NVEncC does not, as it is using the same decode routine and the same libavcodec.dll when using --avsw.

Would you give me the ffmpeg/libx264 options to generates the 264 files which gives the artifacts? I cannot reproduce the issue with my options.

[quamt]

Ok I did some intense testing.
Somehow as soon as one of the following settings is used I get the artifacts.

--colorrange auto --videoformat auto --colormatrix auto --colorprim auto - --transfer auto

I tested with below settings and no artifacts but once the above is included (only one) an the artifacts are back.

“D:\rigaya\VCEEncC\VCEEncC64.exe” --avsw --interlace bff --vpp-afs preset=anime -i "D:\rigaya\test\test.mkv" --codec hevc --chromaloc auto --ref 3 --ltr 2 --b-pyramid --video-metadata clear --gop-len 120 --vpp-edgelevel strength=10.0,threshold=16.0,black=0,white=0 --audio-metadata clear --chapter-copy --sub-copy --sub-metadata clear --data-copy --attachment-copy --pe -u slow --audio-codec aac --vbr 2000 --max-bitrate 3000 --vbv-bufsize 3000 --qp-min 18 --qp-max 24 --motion-est q-pel -o "E:\VCEEnc_test.mkv"

Update
The same artifacts appear when I change - - ltr 2 to - - ltr 1

[rigaya]

Thank you for a great hint!

It seems to be problem when setting VUI information in VCEEncC. I've made a fix in VCEEnc 6.07 which should resolve this issue.

[quamt]

Didn't had this behaviour since the fix.
Will close this issue.
Thank you