A model saved with Python3.5 cannot be loaded in Python3.6

[Dapid]

... at least if it includes a Lambda layer.

Minimal example:

Run save_model.py on Python3.5:

from keras.layers import Input, Lambda
from keras.models import Model

def noop(x):
    return x

inp = Input(shape=(None, 1))
output = Lambda(noop)(inp)

model = Model(inputs=inp, outputs=output)
model.save('result.h5')

Loading the model from Python3.5 also works:

from keras.models import load_model
load_model('result.h5')

But Python3.6 raises a SystemError:

XXX lineno: 7, opcode: 0
Traceback (most recent call last):
  File "load_model.py", line 3, in <module>
    load_model('result.h5')
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/models.py", line 243, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/models.py", line 317, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 144, in deserialize_keras_object
    list(custom_objects.items())))
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/topology.py", line 2524, in from_config
    process_node(layer, node_data)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/topology.py", line 2481, in process_node
    layer(input_tensors[0], **kwargs)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/engine/topology.py", line 619, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/david/.virtualenvs/py36/lib/python3.6/site-packages/keras/layers/core.py", line 663, in call
    return self.function(inputs, **arguments)
  File "save_model.py", line 7, in self_outer
    return x
SystemError: unknown opcode

Contrary to what was stated in bug #7297, the problem is indeed in Keras, since the error is triggered before any call to the backend. The problem appears in both Tensorflow and Theano. Also, both in Keras 2.1.5 and master.

[Dapid]

Digging a bit more, I found that the marshal serial format has changed a bit between Python 3.5 and 3.6. The change is not documented in the release notes, but the documentation doesn't promise to keep it.

I am following the logic used by Keras to serialise Lambda layers, mostly found in https://github.com/keras-team/keras/blob/master/keras/utils/generic_utils.py#L171-L241

import marshal
import codecs
import types


def noop(x):
    return x


raw_code = marshal.dumps(noop.__code__)
code36 = codecs.encode(raw_code, 'base64').decode('ascii')

# Running the same on Python35
code35 = '4wEAAAAAAAAAAQAAAAEAAABDAAAAcwQAAAB8AABTKQFOqQApAdoBeHIBAAAAcgEAAAD6BzxzdGRp\nbj7aBG5vb3ABAAAAcwAAAAA=\n'

# Now we decode:
raw_code_35 = codecs.decode(code35.encode('ascii'), 'base64')
raw_code_36 = codecs.decode(code36.encode('ascii'), 'base64')

f_35 = types.FunctionType(marshal.loads(raw_code_35), globals())
f_36 = types.FunctionType(marshal.loads(raw_code_36), globals())

# Test the function
assert f_36(0) == 0

# This raises:
f_35(0)

[tadbeer]

Similar issue occurs when loading a python 3.5 built model on a python 3.6 notebook in Google Colab. Is there any workaround?
Also, does doing the procedure other way round cause errors too, i.e. loading a python 3.6 built Keras model on python 3.5.
I need to operate in a python 3.5 anaconda environment built on my local machine only, because tensorflow-gpu is currently supported for python 3.5 only.

[Dapid]

@tadbeer the best way is to save the architecture as code, and the weights in an h5. This will be compatible across versions.

Saving on yaml or json will only work across versions if you don't have Lambda layers.

[tadbeer]

thanks @Dapid ! Writing the model architecture in the code itself and then loading weights on top of it worked. I was thus able to load a model built on my machine in python 3.5 to Google Collab in python 3.6.

[fchollet]

A Lambda layer contains arbitrary code. That is not safely serializable, at all.

We serialize this arbitrary code on a best-effort basis. There are no guarantees whatsoever that it will be loadable on a different system or a different Python version.

In order to safely serialize a custom expression, you need to store its code somewhere. Not as bytecode -- you need the original source code.

So you can do:

1. Keep the code around and just save the weights. As you said:

save the architecture as code, and the weights in an h5. This will be compatible across versions

2. Create a custom layer with a get_config method that serializes it (optionally a custom from_config). When you load the model's config, use a CustomObjectScope to specify your custom layer class.

[burhr2]

Hi!
In my case, I had a Lambda layer as I built a siamese network. I have tried to save model architecture as a JSON and model weights separately but still, I was receiving errors (SystemError: unknown opcode) when loading both models from JSON and its weights (model.load_weights('siamese_model.h5')) even though I trained and test the model on TensorFlow 2.3.0 with python 3.8.

The workaround was to rebuild my architecture and then use model.load_weights('trained_model.h5'). More detailed in here

Detailed example

Training the model

do preprocessing of your inputs

.....

Then compile the model

do the training

save your trained model weights

to load the model for future use first rebuild ur model again

and then call model.load_weights('')

The important part is model.save_weights('trained_model.h5'), rebuilding the architecture from codes and model.load_weights('trained_model.h5')

Addition:
To extract embedding from the trained model please see my post #8748 (comment)

[yeohhoo]

Hi! In my case, I had a Lambda layer as I built a siamese network. I have tried to save model architecture as a JSON and model weights separately but still, I was receiving errors (SystemError: unknown opcode) when loading both models from JSON and its weights (model.load_weights('siamese_model.h5')) even though I trained and test the model on TensorFlow 2.3.0 with python 3.8.

The workaround was to rebuild my architecture and then use model.load_weights('trained_model.h5'). More detailed in here

Detailed example

Training the model

do preprocessing of your inputs

.....

Then compile the model

do the training

save your trained model weights

to load the model for future use first rebuild ur model again

and then call model.load_weights('')

The important part is model.save_weights('trained_model.h5'), rebuilding the architecture from codes and model.load_weights('trained_model.h5')

Addition: To extract embedding from the trained model please see my post #8748 (comment)

hello, i read your code. But i can't understand why you use the ModelCheckpoint callback function. it seems you don't use the ".ckpt". Could you tell me the reason? Thanks