Fix video batching to videollava

[merveenoyan]

cc @zucchini-nlp works with no problems

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

Thanks ❤️

Can we activate tests, it is being skipped now for PIL inputs?

[merveenoyan]

@zucchini-nlp one thing confused me, you are passing numpy tests but don't pass numpify=True when creating dummy inputs (also they pass the assertion of being numpy arrays) I couldn't make any change to the numpy test to make it for PIL

[zucchini-nlp]

Yes, I made bad decision when constructing tests tailoring only for torch and numpy. This one has better dummy video preparation, can copy from it maybe

transformers/tests/models/llava_next_video/test_image_processing_llava_next_video.py

Lines 99 to 124 in 3aefb4e

	def prepare_video_inputs(self, equal_resolution=False, numpify=False, torchify=False):
	images = prepare_image_inputs(
	batch_size=self.batch_size,
	num_channels=self.num_channels,
	min_resolution=self.min_resolution,
	max_resolution=self.max_resolution,
	equal_resolution=equal_resolution,
	numpify=numpify,
	torchify=torchify,
	)
	# let's simply copy the frames to fake a long video-clip
	if numpify or torchify:
	videos = []
	for image in images:
	if numpify:
	video = image[None, ...].repeat(8, 0)
	else:
	video = image[None, ...].repeat(8, 1, 1, 1)
	videos.append(video)
	else:
	videos = []
	for pil_image in images:
	videos.append([pil_image] * 8)
	return videos

[zucchini-nlp]

Great, thanks!

[amyeroberts]

Thanks for fixing!

FWIW - I think the current "numpify" and "torchify" logic is pretty bad and repeated everywhere. We should look into making this clearer and consolidate in the future!