--pa not working when using VBR mode

[GlitterKill]

I saw a note that --pa only works with VBR mode encodes but when I tested this with VBR I saw the following in the log output:

Pre-analysis is not supported on this device, disabled.
VCEEnc (x64) 7.00 (r1066) by rigaya, Apr 30 2022 18:34:01 (VC 1931/Win)
OS: Windows 11 x64 (22000) [UTF-8]
CPU: AMD Ryzen 7 3700X 8-Core Processor [4.24GHz] (8C/16T)
GPU: AMD Radeon RX 5700 XT, AMF Runtime 1.4.25 / SDK 1.4.24
Input Info: AviSynth+ 3.7.2 r3661(yv12)->p010 [AVX2], 1920x1080, 30000/1001 fps
Vpp Filters copyHtoD
cspconv(p010 -> yv12(16bit))
unsharp: radius 3, weight 0.5, threshold 10.0
cspconv(yv12(16bit) -> p010)
Output: H.265/HEVC main @ Level 4 (high tier)
1920x1080p 0:0 29.970fps (30000/1001fps)
Quality: slow
VBR: 4000 kbps
Max bitrate: 30000 kbps
QP: Min: 0, Max: 51
VBV Bufsize: 30000 kbps
Bframes: 0 frames
Pre Analysis: off
Motion Est: Q-pel
Slices: 1
GOP Len: 300 frames
Others: deblock vbaq

[rigaya]

I now have a test build which will allow using pre-analysis ( #54 ).

Please test with the test build uploaded at the links below.

• Win x86 link
• Win x64 link

[GlitterKill]

I downloaded the x64 build you linked and placed the files in staxrip/apps/encoders/vceenc (replacing existing files) but ran into an error when trying to do a VBR test encode. Here are the settings, error and log.

Log:

------------------------- System Environment -------------------------

StaxRip : v2.13.0
Windows : Windows 10 Pro 2009
Language : English (United States)
CPU : AMD Ryzen 7 3700X 8-Core Processor
GPU : Virtual Desktop Monitor, AMD Radeon RX 5700 XT
Resolution : 3440 x 1440
DPI : 96
Code Page : 1252

----------------------- Media Info Source File -----------------------

C:\Users\Gary\Downloads\1080test.mp4

General
Complete name : C:\Users\Gary\Downloads\1080test.mp4
Format : MPEG-4
Format profile : Base Media / Version 2
Codec ID : mp42 (mp42/mp41)
File size : 2.63 GiB
Duration : 59 min 3 s
Overall bit rate mode : Variable
Overall bit rate : 6 379 kb/s
Encoded date : UTC 2017-02-10 11:21:47
Tagged date : UTC 2017-02-10 11:22:08
TIM : 00:00:00:00
TSC : 30000
TSZ : 1001

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Main@L4.1
Format settings : CABAC / 3 Ref Frames
Format, CABAC : Yes
Format, Reference frames : 3 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 59 min 3 s
Bit rate : 6 057 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 29.970 (30000/1001) FPS
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.097
Stream size : 2.50 GiB (95%)
Language : English
Encoded date : UTC 2017-02-10 11:21:47
Tagged date : UTC 2017-02-10 11:21:47
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
Codec configuration box : avcC

Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : mp4a-40-2
Duration : 59 min 3 s
Bit rate mode : Variable
Bit rate : 317 kb/s
Maximum bit rate : 417 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 134 MiB (5%)
Language : English
Encoded date : UTC 2017-02-10 11:21:47
Tagged date : UTC 2017-02-10 11:21:47

----------------------------- Demux audio -----------------------------

MP4Box 2.1-DEV-rev79-gdf29bc8a0-x64-gcc11.2.0 Patman

C:\Temp\StaxRip\Apps\Support\MP4Box\MP4Box.exe -single 2 -out "C:\Users\Gary\Downloads\1080test_temp\ID1 English.m4a" C:\Users\Gary\Downloads\1080test.mp4

Start: 9:05:18 AM
End: 9:05:19 AM
Duration: 00:00:00

General
Complete name : C:\Users\Gary\Downloads\1080test_temp\ID1 English.m4a
Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom)
File size : 135 MiB
Duration : 59 min 3 s
Overall bit rate mode : Variable
Overall bit rate : 319 kb/s
Encoded date : UTC 2022-07-18 14:05:18
Tagged date : UTC 2022-07-18 14:05:18

Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Format profile : AAC@L2
Codec ID : mp4a-40-2
Duration : 59 min 3 s
Bit rate mode : Variable
Bit rate : 317 kb/s
Nominal bit rate : 9 b/s
Maximum bit rate : 325 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 134 MiB (99%)
Language : English
Encoded date : UTC 2022-07-18 14:05:18
Tagged date : UTC 2022-07-18 14:05:19

---------------------------- Configuration ----------------------------

Template : H265 VBR
Video Encoder Profile : AMD | H.265 VBR
Container/Muxer Profile : MP4 (mp4box)

--------------------------- AviSynth Script ---------------------------

AddAutoloadDir("C:\Temp\StaxRip\Apps\FrameServer\AviSynth\plugins")
LoadPlugin("C:\Temp\StaxRip\Apps\Plugins\Dual\L-SMASH-Works\LSMASHSource.dll")
Import("C:\Temp\StaxRip\Apps\Plugins\AVS\MultiSharpen\MultiSharpen.avsi")
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LSMASHVideoSource("C:\Users\Gary\Downloads\1080test.mp4")
MultiSharpen(1)
Prefetch(8)

------------------------- Source Script Info -------------------------

Width : 1920
Height : 1080
Frames : 106204
Time : 59:03.673
Framerate : 29.970029 (30000/1001)
Format : YUV420P8

------------------------- Target Script Info -------------------------

Width : 1920
Height : 1080
Frames : 106204
Time : 59:03.673
Framerate : 29.970029 (30000/1001)
Format : YUV420P8

---------------------- Media Info Audio Source 1 ----------------------

General
Complete name : C:\Users\Gary\Downloads\1080test_temp\ID1 English.m4a
Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom)
File size : 135 MiB
Duration : 59 min 3 s
Overall bit rate mode : Variable
Overall bit rate : 319 kb/s
Encoded date : UTC 2022-07-18 14:05:18
Tagged date : UTC 2022-07-18 14:05:18

Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Format profile : AAC@L2
Codec ID : mp4a-40-2
Duration : 59 min 3 s
Bit rate mode : Variable
Bit rate : 317 kb/s
Nominal bit rate : 9 b/s
Maximum bit rate : 325 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 134 MiB (99%)
Language : English
Encoded date : UTC 2022-07-18 14:05:18
Tagged date : UTC 2022-07-18 14:05:19

--------------------------- Video encoding ---------------------------

VCEEnc 7.10

C:\Temp\StaxRip\Apps\Encoders\VCEEnc\VCEEncC64.exe --avsdll C:\Temp\StaxRip\Apps\FrameServer\AviSynth\AviSynth.dll --codec hevc --output-depth 10 --quality slow --tier high --pe --pa --pa-sc High --pa-ss High --pa-caq-strength High --colormatrix bt709 --colorprim bt709 --transfer bt709 --vbaq --vbr 4000 -i C:\Users\Gary\Downloads\1080test_temp\1080test_new.avs -o C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc

---

C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc

VCEEnc (x64) 7.01 (r1134) by rigaya, Jul 14 2022 12:57:39 (VC 1932/Win)
OS: Windows 11 x64 (22000) [UTF-8]
CPU: AMD Ryzen 7 3700X 8-Core Processor [4.25GHz] (8C/16T)
GPU: AMD Radeon RX 5700 XT, AMF Runtime 1.4.25 / SDK 1.4.24
Input Info: AviSynth+ 3.7.2 r3661(yv12)->p010 [AVX2], 1920x1080, 30000/1001 fps
Vpp Filters copyHtoD
Output: H.265/HEVC main @ Level 4 (high tier)
1920x1080p 0:0 29.970fps (30000/1001fps)
Quality: slow
VBR: 4000 kbps
Max bitrate: 30000 kbps
QP: Min: 0, Max: 51
VBV Bufsize: 30000 kbps
Bframes: 0 frames
Pre Analysis: sc high, ss high, activity y, caq high
Motion Est: Q-pel
Slices: 1
GOP Len: 300 frames
VUI: matrix:bt709,colorprim:bt709,transfer:bt709
Others: deblock pe vbaq
Failed to send frame to encoder.

Start: 9:09:48 AM
End: 9:09:54 AM
Duration: 00:00:06

General
Complete name : C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc
File size : 0.00 Byte

------------------------- Error Muxing to MP4 -------------------------

Muxing to MP4 returned exit code: 1 (0x1)

---------------------------- Muxing to MP4 ----------------------------

MP4Box 2.1-DEV-rev79-gdf29bc8a0-x64-gcc11.2.0 Patman

C:\Temp\StaxRip\Apps\Support\MP4Box\MP4Box.exe -fps 29.970030 -add "C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc#video:name=" -add "C:\Users\Gary\Downloads\1080test_temp\ID1 English.m4a#audio:lang=eng:name=" -itags cover=C:\Users\Gary\Downloads\cover.png -new C:\Users\Gary\Downloads\1080test_new.mp4

[FileIn] Asked to read 5000 but got only 0
[FileIn] IO error EOF found after reading 0 bytes but file C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc size is 0
Cannot find track ID matching fragment #video
Error importing C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc#video:name=: At least one required element has not been found

---------------------------- Muxing to MP4 ----------------------------

MP4Box 2.1-DEV-rev79-gdf29bc8a0-x64-gcc11.2.0 Patman

C:\Temp\StaxRip\Apps\Support\MP4Box\MP4Box.exe -fps 29.970030 -add "C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc#video:name=" -add "C:\Users\Gary\Downloads\1080test_temp\ID1 English.m4a#audio:lang=eng:name=" -itags cover=C:\Users\Gary\Downloads\cover.png -new C:\Users\Gary\Downloads\1080test_new.mp4

[FileIn] Asked to read 5000 but got only 0
[FileIn] IO error EOF found after reading 0 bytes but file C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc size is 0
Cannot find track ID matching fragment #video
Error importing C:\Users\Gary\Downloads\1080test_temp\1080test_new_out.hevc#video:name=: At least one required element has not been found

Start: 9:09:54 AM
End: 9:09:54 AM
Duration: 00:00:00

[rigaya]

I've tested further for pre-analysis, it seems like pre-analysis seems to work only for H.264 encoding with RX5xxx series. (I've tested with RX5500XT). This might be hardware or driver limitation.

I'm not sure for RX6xxx series as I don't have one.

[arctus256]

I can confirm pre-analysis in HEVC works on RX6xxx series with (in particular RX6800).

Tested settings:

E:\video\test\test_temp\test_new_out.hevc

VCEEnc (x64) 7.01 (r1134) by rigaya, Jul 14 2022 12:57:39 (VC 1932/Win)
OS: Windows 10 x64 (19044) [UTF-8]
CPU: AMD Ryzen 7 5800X 8-Core Processor [4.85GHz] (8C/16T)
GPU: AMD Radeon RX 6800, AMF Runtime 1.4.25 / SDK 1.4.24
Input Info: AviSynth+ 3.7.2 r3661(yv12)->nv12 [AVX2], 1912x1080, 24000/1001 fps
Vpp Filters copyHtoD
Output: H.265/HEVC main @ Level 4 (main tier)
1912x1080p 0:0 23.976fps (24000/1001fps)
Quality: slow
VBR: 8250 kbps
Max bitrate: 12000 kbps
QP: Min: 0, Max: 51
VBV Bufsize: 12000 kbps
Bframes: 0 frames
Pre Analysis: sc high, ss high, activity y, caq high
Motion Est: Q-pel
Slices: 1
GOP Len: 240 frames
VUI: matrix:bt709,colorprim:bt709,transfer:bt709
Others: deblock pe vbaq

[rigaya]

@arctus256 Thank you for the information about RX6xxx series. Nice to hear it is supported for HEVC also in RX6xxx series.

So I think pre-analysis supports looks like below at this time. (I've checked RX460 and pre-analysis did not work for H.264)

Pre-analysis Support

GPU	H.264	HEVC
RX4xx, RX5xx series or before	no	no
RX5xxx series	yes	no
RX6xxx series	yes	yes

[rigaya]

VCEEnc 7.01 has the fix included for the pre-analysis issue, reflecting the test build change, I think I'll have this issue closed.

Thank you for reporting and having interest on this issue.