[BugFix] Fix buffer index with FCD #2372

Job	Run time
build-wheel-linux (3.8, cp38-cp38, --extra-index-url https://download.pytorch.org/whl/cpu, "['cpu...	1m 4s
build-wheel-linux (3.9, cp39-cp39, --extra-index-url https://download.pytorch.org/whl/cpu, "['cpu...	1m 6s
build-wheel-linux (3.10, cp310-cp310, --extra-index-url https://download.pytorch.org/whl/cpu, "['...	1m 3s
build-wheel-linux (3.11, cp311-cp311, --extra-index-url https://download.pytorch.org/whl/cpu, "['...	1m 10s
build-wheel-mac (3.8, 3.8)	59s
build-wheel-mac (3.9, 3.9)	49s
build-wheel-mac (3.10, 3.10.3)	1m 2s
build-wheel-mac (3.11, 3.11)	52s
build-wheel-windows (3.8, 3.8)	3m 0s
build-wheel-windows (3.9, 3.9)	3m 9s
build-wheel-windows (3.10, 3.10.3)	3m 11s
build-wheel-windows (3.11, 3.11)	2m 58s
test-wheel (linux, ubuntu-20.04, 3.8)	59s
test-wheel (linux, ubuntu-20.04, 3.9)	59s
test-wheel (linux, ubuntu-20.04, 3.10)	1m 5s
test-wheel (linux, ubuntu-20.04, 3.11)	1m 22s
test-wheel (mac, macos-latest, 3.8)	1m 47s
test-wheel (mac, macos-latest, 3.9)	2m 3s
test-wheel (mac, macos-latest, 3.10)	2m 5s
test-wheel (mac, macos-latest, 3.11)	1m 10s
test-wheel-windows (3.8)	2m 19s
test-wheel-windows (3.9)	2m 16s
test-wheel-windows (3.10)	2m 13s
test-wheel-windows (3.11)	2m 20s
	41m 1s

Provide feedback