ggml : move AMX to the CPU backend

[slaren]

• Move AMX code to CPU backend
• Enable disabled types in AMX backend (Q4_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_XS)
• Change C++ standard to C++17
• Enable ccache for HIP windows CI

[Djip007]

Can this function be call now this is a extra cpu buffer, only weight can be store in this buffer type?

[slaren]

It's not called at the moment, but it doesn't hurt to have it.

[Djip007]

OK, I was just asking so as not to break everything in my PR.

[Djip007]

simply remove the ggml_barrier and add it if needed after parallel_for_ggml ?

[slaren]

It's not really needed in any cases, I will just remove it in #10606.

[Djip007]

Is this fp16 matmul faster than LLAMAFILE fp16 sgemm?
if not, now this backend is in the CPU backend it may not be needed.

[slaren]

Yes, I believe it is significantly faster than llamafile sgemm. In my tests it's about 40% faster at pp512.

[Djip007]

Strange, it doesn't use AMX and yet it looks like the same way of doing things as with tinyblas. I'll have to look at this more closely 😎

[slaren]

It does use AVX512, although the implementation looks a lot simpler than in sgemm.

llama.cpp/ggml/src/ggml-cpu/amx/mmq.cpp

Lines 1332 to 1372 in 3420909

	template <int BLOCK_M, int BLOCK_N, int BLOCK_K>
	struct tinygemm_kernel_avx<float, ggml_fp16_t, float, BLOCK_M, BLOCK_N, BLOCK_K> {
	static void apply(int K, const float * RESTRICT A, const ggml_fp16_t * RESTRICT B, float * RESTRICT C, int ldc) {
	constexpr int ROWS = BLOCK_M;
	constexpr int COLS = BLOCK_N;
	assert(BLOCK_K == 16);
	__m512 va;
	__m512 vb[COLS];
	__m512 vc[ROWS * COLS];
	auto loadc = [&](auto idx) {
	vc[idx] = _mm512_setzero_ps();
	};
	Unroll<ROWS * COLS>{}(loadc);
	auto compute = [&](auto idx, auto k) {
	constexpr int row = idx / COLS;
	constexpr int col = idx % COLS;
	if constexpr (col == 0) {
	va = _mm512_loadu_ps(A + row * K + k);
	}
	if constexpr (row == 0) {
	vb[col] = _mm512_cvtph_ps(_mm256_loadu_si256((const __m256i *)(B + col * K + k)));
	}
	vc[idx] = _mm512_fmadd_ps(va, vb[col], vc[idx]);
	};
	for (int k = 0; k < K; k += 16) {
	Unroll<ROWS * COLS>{}(compute, k);
	}
	auto storec = [&](auto idx) {
	constexpr int row = idx / COLS;
	constexpr int col = idx % COLS;
	C[row * ldc + col] = _mm512_reduce_add_ps(vc[idx]);
	};
	Unroll<ROWS * COLS>{}(storec);
	}
	};

[Djip007]

llama.cpp/ggml/src/ggml-cpu/llamafile/sgemm.cpp

Lines 421 to 445 in 3420909

	template <int RM, int RN>
	NOINLINE void gemm(int64_t m0, int64_t m, int64_t n0, int64_t n) {
	int64_t ytiles = (m - m0) / RM;
	int64_t xtiles = (n - n0) / RN;
	int64_t tiles = xtiles * ytiles;
	int64_t duty = (tiles + nth - 1) / nth;
	int64_t start = duty * ith;
	int64_t end = start + duty;
	if (end > tiles)
	end = tiles;
	for (int64_t job = start; job < end; ++job) {
	int64_t ii = m0 + job / xtiles * RM;
	int64_t jj = n0 + job % xtiles * RN;
	D Cv[RN][RM] = {};
	for (int64_t l = 0; l < k; l += KN)
	for (int64_t j = 0; j < RN; ++j)
	for (int64_t i = 0; i < RM; ++i)
	Cv[j][i] = madd(load<V>(A + lda * (ii + i) + l),
	load<V>(B + ldb * (jj + j) + l),
	Cv[j][i]);
	for (int64_t j = 0; j < RN; ++j)
	for (int64_t i = 0; i < RM; ++i)
	C[ldc * (jj + j) + (ii + i)] = hsum(Cv[j][i]);
	}
	}

[mingfeima]

I did not put AMX in the old patch because the 6th gen Xeon is not released at that moment. Since it is on sale now, i think it is good to add amx-f16 to the gemm.

[ericcurtin]

👍 for "Change C++ standard to C++17"... Only ancient platforms don't support that these days, and probably ones you don't want to run AI workloads on anyway...

[mingfeima]

big thumbs up for "Switch to C++17". Actually the forced Unroll from my amx patch does not work properly on C++11, I checked the assembly and confirmed, on C++17 the unrolling is totally workable.

[slaren]

I also changed the parameters of the functions called by Unroll to constexpr in #10606, following the comments that you had left in the code, but I didn't check the generated code. Good to know that it works as expected.