Caffe returns different values for same input repeated

[eriophora]

I'm not sure if this is Caffe, or something I'm doing, but it's concerning. I'm using pycaffe. While debugging, I happened to run a block of 28 images, 224x224x3 each, all identical (i.e., the same values). The outputs of my net (based on GoogLeNet) oscillates:

i.e.,

# b is a 28x3x224x224 matrix of float 32s
out = net.forward_all(**{net1.inputs[0]:b})

Out[1]:
{'loss3/estimator': array([[-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502],
        [-0.33872247],
        [-0.38100502]], dtype=float32)}

As you can see, the values oscillate between -0.38100502 and -0.33872247. Anyone have any idea why this is occuring / has observed it in the past? The net is instantiated as:

class _GPUMgr(caffe.Net):
    def __init__(self, model_file, pretrained_file):
        '''
        model_file : the caffe model to use
        pretrained_file : array of model weights
        '''
        caffe.Net.__init__(self, model_file, pretrained_file, caffe.TEST)
        in_ = self.inputs[0]
        self.image_dims = np.array(self.blobs[in_].data.shape[1:])

    def __call__(self, data_array):
        '''
        Actually executes the prediction, provided with an
        N x 3 x H x W array of N images that have already
        been preprocessed and resized.
        '''
        logging.debug('Running chunk of %i images on GPU' % (
            data_array.shape[0]))
        if type(data_array).__module__ != np.__name__:
            raise TypeError("data_array type is %s, must be %s" % (
                str(type(data_array)), str(np.__name__)))
        if data_array.dtype != np.dtype('float32'):
            raise ValueError("data_array must be float32")
        if np.any(data_array.shape[1:] != self.image_dims):
            raise ValueError(
                "data_array must have shape N x %i x %i x %i" % (
                self.image_dims[0], self.image_dims[1], self.image_dims[2]))
        out = self.forward_all(**{self.inputs[0]: data_array})
        predictions = np.exp(out[self.outputs[0]])
        return list(predictions[:,0])

(the values above were not obtained by the __call__() function, and hence were not exponentiated)

[seanbell]

Are you using cuDNN? I believe the output you get from it is non-deterministic.

[eriophora]

I am, and that is very interesting thank you!

[seanbell]

I think it's worth looking into this more, since many people have reported similar issues with cuDNN.

[longjon]

Yes, this is worth noting and understanding. I'm closing in favor of #3168; discuss there as needed.