integer overflow in malloc()

[setharnold]

openjpeg/src/bin/jp3d/opj_jp3d_compress.c

Line 401 in 06f7d41

parameters->cp_matrice = (int *) malloc(numlayers * matrix_width * sizeof(int));

            parameters->tcp_numlayers = numlayers;
            matrix_width = parameters->numresolution[0] + parameters->numresolution[1] + parameters->numresolution[2];
            parameters->cp_matrice = (int *) malloc(numlayers * matrix_width * sizeof(int));
            s = s + 2;

Hello, note the malloc() argument may suffer from an integer multiplication overflow if these parameters are not bounded elsewhere in the code. I'd like to suggest that all such malloc(a*b) calls check their parameters for overflow, or switch to calloc(), as appropriate. (With three inputs, this one may take more work than usual.)

Thanks

[stweil]

parameters->numresolution is [3, 3, 1], so matrix_width is 7. Therefore limiting numlayers would be sufficient here. What is a reasonable limit for that value?

The missing check for the result of malloc being NULL is more important.

[setharnold]

If matrix_width is always 7 then this is an easy fix:

• clean up and return an error if matrix_width != 7
• use calloc(numlayers, matrix_width * sizeof(int))

Thanks