Add BPE pre-tokenization for Qwen2. #11840

Job	Run time
Push Docker image to Docker Hub (light, .devops/main.Dockerfile, linux/amd64,linux/arm64)	28m 22s
Push Docker image to Docker Hub (full, .devops/full.Dockerfile, linux/amd64,linux/arm64)	33m 33s
Push Docker image to Docker Hub (server, .devops/server.Dockerfile, linux/amd64,linux/arm64)	27m 26s
Push Docker image to Docker Hub (light-cuda, .devops/main-cuda.Dockerfile, linux/amd64)	19m 46s
Push Docker image to Docker Hub (full-cuda, .devops/full-cuda.Dockerfile, linux/amd64)	20m 44s
Push Docker image to Docker Hub (server-cuda, .devops/server-cuda.Dockerfile, linux/amd64)	20m 43s
Push Docker image to Docker Hub (light-rocm, .devops/main-rocm.Dockerfile, linux/amd64,linux/arm64)	20m 0s
Push Docker image to Docker Hub (full-rocm, .devops/full-rocm.Dockerfile, linux/amd64,linux/arm64)	12m 24s
Push Docker image to Docker Hub (server-rocm, .devops/server-rocm.Dockerfile, linux/amd64,linux/a...	12m 7s
Push Docker image to Docker Hub (light-intel, .devops/main-intel.Dockerfile, linux/amd64)	8m 9s
Push Docker image to Docker Hub (server-intel, .devops/server-intel.Dockerfile, linux/amd64)	8m 21s
	3h 31m 35s

Provide feedback