[Feature discussion] Improving model save format

[Miffyli]

Continuation on the discussion in #312 related to improving save file format.

Issue with current way of storing/loading models

• Cloudpickle can have issues with different version of Python (mentioned here)
• Cloudpickle itself mentions that it shouldn't be used for long-term storage (something RL people might want).
• Since everything is stored into one serialized file, a single corrupted part prevents easily loading rest of the things. Example: My saved model had Tensorflow code that did not work on different Python version, thus resulting error upon loading. I only needed the model parameters, but I couldn't load them because one of the class items broke deserialization.
• A minor point, but cloudpickle is Python-specific, so accessing anything inside these files requires going through Python+ cloudpickle.

Suggestion
Using Python-native Zipfile to manually construct a zipped archive. Each entry in the zip file would be a separately serialized object. E.g. A2C would have one entry for "gamma", another for "n_steps" and so on. Finally the zip file would include one more entry for the model parameters, serialized with numpy's savez or just with pickle as is done now.

Since many learning algorithms store policies, obs/action spaces, functions and other custom Python objects, it might make sense to store these using pickle. However now if these become non-deserializable (e.g. new Python version), we can still load other items. This works especially well with load_parameters, which could directly access the parameters in zip file rather than trying to deserialize the whole file.

An alternative would be to use JSON to serialize class parameters, but this may not play well with custom functions and the likes. JSON would offer very human-readable and long-lasting files, though. I also took a look at jsonpickle, but I am not sure if relying on another library is a better idea than sticking with pickle for serializing things.

[araffin]

Hello,

I will be on holidays soon, so I'll let @AdamGleave @erniejunior @hill-a discuss this issue (if they have time), otherwise, I'll try to come by to you at the end of august.

[ernestum]

I am a big fan of storing the class parameters in json files. Its easy to parse from other context and will last long enough for all practical purposes.
Not so big fan of the one file per parameter within a zip approach. It seems a bit convoluted to me.

[araffin]

I'm back ;)
after encountering some issues due to different package version, i'm definitely for that feature.

I would favor yaml over json (easier to write and it can handle some python objects (cf rl zoo)) and maybe use zip format to have only one file ar the end? (containing npy arrays and serialized params)

The main issue i see is ensuring that we can serialize all python objects.

Last thing: i would keep the cloudpickle format as an option for backward compat at first.

[Miffyli]

@araffin
Welcome back, hopefully all refreshed! ^^

JSON vs YAML: YAML indeed is more human-readable, but I would go with JSON as it seems more commonly supported (and still seems to require 3rd party libraries for Python). With JSON files, somebody could parse them in other languages/scripts easier. We can use the indent argument of json.dump to make it prettier for human consumption.

Zipping: I agree with @erniejunior "one file per parameter" approach is quite convoluted. How about .zip archive with one JSON file and one file with the parameters (e.g. numpy file)?

Unserializable objects: I figure still will rely on cloudpickle/pickle. How about serializing them into bytes/base64 and storing in the JSON file much like other parameters? This would be tested with json.dump to see which objects are serializable, and if not then use cloudpickle. This will result in messier JSON file, but still readable by other languages.

Backward compatibility: Yes, this would definitely be implemented for loading and as optional setting for saving (just in case).

[araffin]

still seems to require 3rd party libraries for Python

Ok, good point.

We can use the indent argument of json.dump to make it prettier for human consumption.

yes, good idea.

How about .zip archive with one JSON file and one file with the parameters (e.g. numpy file)?

Yes, that was my idea + maybe pickle file for what cannot be properly put into one or the other

How about serializing them into bytes/base64 and storing in the JSON file much like other parameters?

This would prevent from using/loading it from another language no?
Btw I think the first step would be to reference/identify what cannot be put into a JSON/numpy file and figure out how to change that, no?

[Miffyli]

If we encode the bytes of non-JSON-able objects into a string and save it in the JSON, then other languages can still read the JSON file all nice but can't really do much with these byte-strings (they do not have any meaning outside Python/Cloudpickle/Pickle).

Did a quick run of what can be JSONiable and what needs pickling. As expected, spaces and policy are the main issue as they contain classes and functions. However, in my opinion, these can stay byte-serialized as they only have proper meaning in Python. These are also the parts that will likely break something when changing Python or library versions.

One thing we can do with this much like Keras does: When loading model, we can replace some of the items from a file with something else. E.g. A2C.load(..., custom_objects={'policy': other_policy}) would skip reading policy from the file and instead uses the provided one other_policy. This is already in load function but I would move it bit deeper so that it could prevent loading invalid objects (and thus crashing).

<class 'stable_baselines.a2c.a2c.A2C'>
        gamma                     <class 'float'>                          JSON      
        n_steps                   <class 'int'>                            JSON      
        vf_coef                   <class 'float'>                          JSON      
        ent_coef                  <class 'float'>                          JSON      
        max_grad_norm             <class 'float'>                          JSON      
        learning_rate             <class 'float'>                          JSON      
        alpha                     <class 'float'>                          JSON      
        epsilon                   <class 'float'>                          JSON      
        lr_schedule               <class 'str'>                            JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.acer.acer_simple.ACER'>
        gamma                     <class 'float'>                          JSON      
        n_steps                   <class 'int'>                            JSON      
        q_coef                    <class 'float'>                          JSON      
        ent_coef                  <class 'float'>                          JSON      
        max_grad_norm             <class 'int'>                            JSON      
        learning_rate             <class 'float'>                          JSON      
        lr_schedule               <class 'str'>                            JSON      
        rprop_alpha               <class 'float'>                          JSON      
        rprop_epsilon             <class 'float'>                          JSON      
        replay_ratio              <class 'int'>                            JSON      
        replay_start              <class 'int'>                            JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.acktr.acktr_disc.ACKTR'>
        gamma                     <class 'float'>                          JSON      
        nprocs                    <class 'int'>                            JSON      
        n_steps                   <class 'int'>                            JSON      
        vf_coef                   <class 'float'>                          JSON      
        ent_coef                  <class 'float'>                          JSON      
        vf_fisher_coef            <class 'float'>                          JSON      
        max_grad_norm             <class 'float'>                          JSON      
        learning_rate             <class 'float'>                          JSON      
        kfac_clip                 <class 'float'>                          JSON      
        lr_schedule               <class 'str'>                            JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.deepq.dqn.DQN'>
        checkpoint_path           <class 'NoneType'>                       JSON      
        param_noise               <class 'bool'>                           JSON      
        learning_starts           <class 'int'>                            JSON      
        train_freq                <class 'int'>                            JSON      
        prioritized_replay        <class 'bool'>                           JSON      
        prioritized_replay_eps    <class 'float'>                          JSON      
        batch_size                <class 'int'>                            JSON      
        target_network_update_freq <class 'int'>                            JSON      
        checkpoint_freq           <class 'int'>                            JSON      
        prioritized_replay_alpha  <class 'float'>                          JSON      
        prioritized_replay_beta0  <class 'float'>                          JSON      
        prioritized_replay_beta_iters <class 'NoneType'>                       JSON      
        exploration_final_eps     <class 'float'>                          JSON      
        exploration_fraction      <class 'float'>                          JSON      
        learning_rate             <class 'float'>                          JSON      
        gamma                     <class 'float'>                          JSON      
        verbose                   <class 'int'>                            JSON      
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.ppo2.ppo2.PPO2'>
        gamma                     <class 'float'>                          JSON      
        n_steps                   <class 'int'>                            JSON      
        vf_coef                   <class 'float'>                          JSON      
        ent_coef                  <class 'float'>                          JSON      
        max_grad_norm             <class 'float'>                          JSON      
        learning_rate             <class 'float'>                          JSON      
        lam                       <class 'float'>                          JSON      
        nminibatches              <class 'int'>                            JSON      
        noptepochs                <class 'int'>                            JSON      
        cliprange                 <class 'float'>                          JSON      
        cliprange_vf              <class 'float'>                          JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON      
<class 'stable_baselines.trpo_mpi.trpo_mpi.TRPO'>
        gamma                     <class 'float'>                          JSON      
        timesteps_per_batch       <class 'int'>                            JSON      
        max_kl                    <class 'float'>                          JSON      
        cg_iters                  <class 'int'>                            JSON      
        lam                       <class 'float'>                          JSON      
        entcoeff                  <class 'float'>                          JSON      
        cg_damping                <class 'float'>                          JSON      
        vf_stepsize               <class 'float'>                          JSON      
        vf_iters                  <class 'int'>                            JSON      
        hidden_size_adversary     <class 'int'>                            JSON      
        adversary_entcoeff        <class 'float'>                          JSON      
        expert_dataset            <class 'NoneType'>                       JSON      
        g_step                    <class 'int'>                            JSON      
        d_step                    <class 'int'>                            JSON      
        d_stepsize                <class 'float'>                          JSON      
        using_gail                <class 'bool'>                           JSON      
        verbose                   <class 'int'>                            JSON      
        policy                    <class 'abc.ABCMeta'>                    Cloudpickle
        observation_space         <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        action_space              <class 'gym.spaces.discrete.Discrete'>   Cloudpickle
        n_envs                    <class 'int'>                            JSON      
        _vectorize_action         <class 'bool'>                           JSON      
        policy_kwargs             <class 'dict'>                           JSON

[araffin]

Great, I agree on both points ;).
@hill-a @erniejunior @AdamGleave could you comment on that so we can start the implementation?

[AdamGleave]

Things being human-readable where possible seems useful for debugging. I don't think there's any need to support languages other than Python, though: even if we're careful with the serialization format, it'd be very hard to make use of this outside of Stable Baselines, since the TensorFlow graph would have to be reconstructed exactly. (If this was a desiderata, we should probably save the graph and weights together.)

This is unrelated, but one thing that continually bugs me when saving models: if you're using VecNormalize, you have to separately save the normalization statistics. It might be nice to have functionality to save both a policy and the set of transformations it used?

[Miffyli]

@AdamGleave

I agree on both points. Hence I would just cloudpickle/pickle the non-JSON-able objects and leave it at that.

One open point is the storing of model parameters: Should this be Numpy savez object or something more universal? I figure savez (and save) format is long-lasting and easy enough for people to use the values somewhere else-

I did quick experiments with storing objects with this new format, and here is the JSON (class parameters) part of the saved model. Objects that are serialized with cloudpickle (:serialized:) also include first-level members of the object so human reader can get some idea what the serialization contains. These are not used in any way when reading the file (only :serialized: is read and deserialized).

Example JSON file of class parameters

{
    "gamma": 0.99,
    "timesteps_per_batch": 1024,
    "max_kl": 0.01,
    "cg_iters": 10,
    "lam": 0.98,
    "entcoeff": 0.0,
    "cg_damping": 0.01,
    "vf_stepsize": 0.0003,
    "vf_iters": 3,
    "hidden_size_adversary": 100,
    "adversary_entcoeff": 0.001,
    "expert_dataset": null,
    "g_step": 1,
    "d_step": 1,
    "d_stepsize": 0.0003,
    "using_gail": false,
    "verbose": 0,
    "policy": {
        ":type:": "<class 'abc.ABCMeta'>",
        ":serialized:": "gASVMgAAAAAAAACMIHN0YWJsZV9iYXNlbGluZXMuY29tbW9uLnBvbGljaWVzlIwJTWxwUG9saWN5lJOULg==",
        "__module__": "stable_baselines.common.policies",
        "__doc__": "\n    Policy object that implements actor critic, using a MLP (2 layers of 64)\n\n    :param sess: (TensorFlow session) The current TensorFlow session\n    :param ob_space: (Gym Space) The observation space of the environment\n    :param ac_space: (Gym Space) The action space of the environment\n    :param n_env: (int) The number of environments to run\n    :param n_steps: (int) The number of steps to run for each environment\n    :param n_batch: (int) The number of batch to run (n_envs * n_steps)\n    :param reuse: (bool) If the policy is reusable or not\n    :param _kwargs: (dict) Extra keyword arguments for the nature CNN feature extraction\n    ",
        "__init__": "<function MlpPolicy.__init__ at 0x7feb24b73268>",
        "__abstractmethods__": "frozenset()",
        "_abc_registry": "<_weakrefset.WeakSet object at 0x7feb24c340f0>",
        "_abc_cache": "<_weakrefset.WeakSet object at 0x7feb24c34128>",
        "_abc_negative_cache": "<_weakrefset.WeakSet object at 0x7feb24c34198>",
        "_abc_negative_cache_version": 52
    },
    "observation_space": {
        ":type:": "<class 'gym.spaces.discrete.Discrete'>",
        ":serialized:": "gASVTgsAAAAAAACME2d5bS5zcGFjZXMuZGlzY3JldGWUjAhEaXNjcmV0ZZSTlCmBlH2UKIwBbpRLCowFc2hhcGWUKYwFZHR5cGWUjAVudW1weZSMBWR0eXBllJOUjAJpOJRLAEsBh5RSlChLA4wBPJROTk5K/////0r/////SwB0lGKMCW5wX3JhbmRvbZSMFG51bXB5LnJhbmRvbS5fcGlja2xllIwSX19yYW5kb21zdGF0ZV9jdG9ylJOUjAdNVDE5OTM3lIWUUpR9lCiMDWJpdF9nZW5lcmF0b3KUaBSMBXN0YXRllH2UKIwDa2V5lIwVbnVtcHkuY29yZS5tdWx0aWFycmF5lIwMX3JlY29uc3RydWN0lJOUaAiMB25kYXJyYXmUk5RLAIWUQwFilIeUUpQoSwFNcAKFlGgKjAJ1NJRLAEsBh5RSlChLA2gOTk5OSv////9K/////0sAdJRiiULACQAAW6DUsAzWsNOFcUrEouywwBF3V4lPOLX1Xz38oPU+Ft9cmSdH+UdKVn9ZtIUnq6RP6Mn1mqpv9eamqbx1OvrRab59kHcR+kFSd8mlTnwjWpsL797ktjG68OFSTa52PM4iqJpJOdzQFK7Ya4YlvS238Xav/nWTBJgNoxEQ82f0pka7olRWNoR54gIVszlaFWckVM0PArDWId+aJ+TWDqHpOkHom217lPxCqiHJvG/jZ3CyOLA5ID7nwjQRQ2x/6FLJnW2hAPwje7eDUmmSxhm3J2OAuzqZPI30/o9P7OiFAN6aM4wb8Mg3fVua1xIxsgI/88AkcSMhpYXgHnxhf6UkXb/c3pwQ6HQhCWrPEaMgonDwGDMazfV2oSaUmhzWxVP27GpNKRX2OryVeJoKAcCvsXRXkrigucUQiESyksdsTE1HQYmxZeAM2PhNL3hG8jwaUZqsIADZk1vUGMQXX5C9ZMUkHg/ESKviQwacFSKZtt/7RF8iSFB/0zFF/JwQ+xtwueDgwWRB2MnS8q/JvkC7uriGXG1qDEfjaNAM751mZy/X4PQv0AH+YE4o5pYplmO/+hwix/xec3whdl4+KAinMUpLGJMGKERO/i174h8YAmhMepSEN6yQPuTszOCwDJOZGdfZ+ap3SYzC6PIG3ELgXf69Ek++j0zvpRkLP1rCLprf76pd9hgSLrX2kaT1jAbUgnJhdJdEhyJy+4JbXSxc9P6hPLiHZeicFHzXSWTE0hJ0H8PuP8RNEEJe+oL9E4xgSJLLNXFhQd63zVkg6v9bxR0X1pzag8kC1UtF6oH9xhBUoAavFA2J6J+PsHzTsZE5YOGw1YhKv8camBtB+InSXLeL56wwXYc/m+D+fWJeL1P8/hHDB5VuqIajBaEHppbTtH3osXyd9ewomecdRmLYaI6GP+Vy2wlu8/AlCp7fcZChr4JOwcuIqfhU+Ij063ALAQWwglAuHzgO/BAL6Wqw36LCkiARi4WbSROuj3CRYLqhk4AizvA4zDEz7cUCjWm4tS73WkDVeN0U4sa3PDY2uNpwF7Nj9jl6+IwSCv2HDqHvPkOcjwORfQMaJx7i990yst8p+naj13pZCQB9semWWGxbi3LY21UVG8lg5lvmizJBHFpCYTBV4AlaHnk9BBXnmnsZftuFtHvDDXAV3nvvuouP2qQhy3VE3dbmH9xazRQRy4dENoB4ydClla5yAzv6G3fxMs1uxOf5Vo+MZrlmWxCUsitH6A0p2qq1ulMixe73W51u/bcb6JquDCgtFF8xIm8ZGzP+jyvQSi0oPCWH+dIKipQlc7thg8fJ3zxQ0GvNAzmXqoOXr5+OkClJnGA+WWGYEl4Jy5y5UKOKtTmv91BZbciXByuGnGEDk4z/aWrdHBRzLrIqRpo8rvu/XRDpSt1Crh/woK4l1CtohhHfgybR0+dZzHoew5m9Dd0slOdkUFB0YJDFAouVDhbaKfmmxZO14ZS3Lt43yBEiv67U5cHwNdudZP7bnQV8bP2JF0a9atXC+h/iHBZgMJSYvByflIEFMkZUAUyeq3mDeudXTj6f6EIOduOuqGLNgZui9Sy303WUpyD+IHZQt+PwbFyae+gaGQ4wFN7zp0B2fnNYwb7K0RQImf4/4gkABgQp/J5h4JVQXxt+f06dJLHroswT2MhUR2b/7go0O/D7GewQaB4SWLmRpzmCFsOHp7akEiRPFLQrOXvg0jgiUUptf0cW+LuyeLo3bYbs9k8bkvOX/VgbdSujE6SvGA6oSe6ztUT11dvw8+aft0IWMa9EnJzILXa1NShJVM6nTu2r2cZRG7vaqQE4OJTr8XQZw8qMgBbStI/NRLfgbSA+XJIVV/L6ixpRYQCC5lHlbHcnSMoJi0pSYZ5fr71eWjDijhULcH6csQnIO/YNZKKJF3Q0xbxaqP6aq7C3RsIRoQsls2sSv9/uZ4IQ4D/ioe8m397X/Zs6QJZ/hYK0nXZn1c6dDSOaDJibVtkKBUTdWhbo6e93y1bwMWC+PYkUkxAmEx2Y1SnntwoUmiJYH179BCQLignL1omY+HagH5ZP5odULYNdU/A4oJHoSEgPxYDtnta4VRsu21fN5a9Tx/+0rA08cghouw32yH4A7qjboJdpX3b7YhKinhhzjFtNLmjy79k6o5KDLM9vb6oswJFAp82SEFkLCiycVvUTTE7pwOB3PpRzcHUdHTBfnPF0FuPp//hjNv7srG5XjzuiH9wvmK1oYxsnvm2vqcomlBE0gjiW/nsqXybwJtqpGxYu8ipbqsYpqDN+QzRtu+S+jRfnwdkcAazG9cpasrAoUDhWTYzVbgNJBdraafYa6WxJNXdlB26/CkgXQEAJ28qZDLccx7h20V+0NH/Mj9fMd2+jOq9Ksd50k5MQqzWosk4dynbUhbGM/atehHVf10fxTWftyXBL1aI+FvaGKJj/NxcxnhWhNwFn5zd0tBgI6BpXZup/vdsMrc5S+ZrRafbQi5Rhhjr88gZHmOcBMljLe7cOLtmXhqk/HrI+47ftamQ8Oy6MGY+F6RcxSOWlQpU4w3MjhfnKqw9NEc18+Bjg9O/+60oj9SNNteMW8xL/d2D6+PS0k8R18oFQl/zOk9yBbgeqYkYTZ39pFygk1KdbbgybFkj/VMwkWEKGy+qIFWP2NWbVHf0nqd/U8rVfvo9PHcDF92HphtkYGqqMgbwfZhQWJfXO8Ijw7f+vvWZzgpt2khJSwYaBiJH9czE2rg5ps4CDlQ64nKnGWE/USQCAfyicP7MesC2+wE1gmLMR4WCDLHq7YhaRHTpUPmYqWx9xuB9AVIIL147OZPeYTfqTHDI53kbVnw4nqds4pAa64MvQtsZT/r46alHXeWJDvIwF3S2E1hHoGsXSt62RPtZpbcZ1hl4wZqaM3tkuX6ZIWsFJ7+Q8pGrPrvWdaFRMg/g/QDE0SmWu706c3q0yPYoEdqxQldbeEveY81V0sR6rRftLnWWGJ637ujgDNYujxwCZJpJ52hy8gdhTgli78HKd9nhoqx2wo3BS403A9CCAcSOCTFR8zCde6foFoPCp0pUp6DAtNLwvpmdNBTRlfF4cLh38FUvfM8r6gwNs3WFQ63pyy9fIHFPI6ApUFcgZ1VbmnA9jUOiJ6KKhQygLFZ/q91ZCqCYTgDfUUj8UqWXfpa4Ucmljc6KX2JDgDhKFa5wvYUiIHO6LdMClSU6sc5FxVpMJOx+jFgD2O7HII9kueQM9QjKjFo7H4ZxX1uX88a+oV04JkJDIDSd3iTFqqfOn//1KrCutmxdLoyHJ0PeizgDmcQbFsrGi0qya1LsW2PdIuwXscMRJ7ZRJlHSUYowDcG9zlEsDdYwJaGFzX2dhdXNzlEsAjAVnYXVzc5RHAAAAAAAAAAB1YnViLg==",
        "n": 10,
        "shape": [],
        "dtype": "int64",
        "np_random": "RandomState(MT19937)"
    },
    "action_space": {
        ":type:": "<class 'gym.spaces.discrete.Discrete'>",
        ":serialized:": "gASVTgsAAAAAAACME2d5bS5zcGFjZXMuZGlzY3JldGWUjAhEaXNjcmV0ZZSTlCmBlH2UKIwBbpRLCowFc2hhcGWUKYwFZHR5cGWUjAVudW1weZSMBWR0eXBllJOUjAJpOJRLAEsBh5RSlChLA4wBPJROTk5K/////0r/////SwB0lGKMCW5wX3JhbmRvbZSMFG51bXB5LnJhbmRvbS5fcGlja2xllIwSX19yYW5kb21zdGF0ZV9jdG9ylJOUjAdNVDE5OTM3lIWUUpR9lCiMDWJpdF9nZW5lcmF0b3KUaBSMBXN0YXRllH2UKIwDa2V5lIwVbnVtcHkuY29yZS5tdWx0aWFycmF5lIwMX3JlY29uc3RydWN0lJOUaAiMB25kYXJyYXmUk5RLAIWUQwFilIeUUpQoSwFNcAKFlGgKjAJ1NJRLAEsBh5RSlChLA2gOTk5OSv////9K/////0sAdJRiiULACQAAW6DUsAzWsNOFcUrEouywwBF3V4lPOLX1Xz38oPU+Ft9cmSdH+UdKVn9ZtIUnq6RP6Mn1mqpv9eamqbx1OvrRab59kHcR+kFSd8mlTnwjWpsL797ktjG68OFSTa52PM4iqJpJOdzQFK7Ya4YlvS238Xav/nWTBJgNoxEQ82f0pka7olRWNoR54gIVszlaFWckVM0PArDWId+aJ+TWDqHpOkHom217lPxCqiHJvG/jZ3CyOLA5ID7nwjQRQ2x/6FLJnW2hAPwje7eDUmmSxhm3J2OAuzqZPI30/o9P7OiFAN6aM4wb8Mg3fVua1xIxsgI/88AkcSMhpYXgHnxhf6UkXb/c3pwQ6HQhCWrPEaMgonDwGDMazfV2oSaUmhzWxVP27GpNKRX2OryVeJoKAcCvsXRXkrigucUQiESyksdsTE1HQYmxZeAM2PhNL3hG8jwaUZqsIADZk1vUGMQXX5C9ZMUkHg/ESKviQwacFSKZtt/7RF8iSFB/0zFF/JwQ+xtwueDgwWRB2MnS8q/JvkC7uriGXG1qDEfjaNAM751mZy/X4PQv0AH+YE4o5pYplmO/+hwix/xec3whdl4+KAinMUpLGJMGKERO/i174h8YAmhMepSEN6yQPuTszOCwDJOZGdfZ+ap3SYzC6PIG3ELgXf69Ek++j0zvpRkLP1rCLprf76pd9hgSLrX2kaT1jAbUgnJhdJdEhyJy+4JbXSxc9P6hPLiHZeicFHzXSWTE0hJ0H8PuP8RNEEJe+oL9E4xgSJLLNXFhQd63zVkg6v9bxR0X1pzag8kC1UtF6oH9xhBUoAavFA2J6J+PsHzTsZE5YOGw1YhKv8camBtB+InSXLeL56wwXYc/m+D+fWJeL1P8/hHDB5VuqIajBaEHppbTtH3osXyd9ewomecdRmLYaI6GP+Vy2wlu8/AlCp7fcZChr4JOwcuIqfhU+Ij063ALAQWwglAuHzgO/BAL6Wqw36LCkiARi4WbSROuj3CRYLqhk4AizvA4zDEz7cUCjWm4tS73WkDVeN0U4sa3PDY2uNpwF7Nj9jl6+IwSCv2HDqHvPkOcjwORfQMaJx7i990yst8p+naj13pZCQB9semWWGxbi3LY21UVG8lg5lvmizJBHFpCYTBV4AlaHnk9BBXnmnsZftuFtHvDDXAV3nvvuouP2qQhy3VE3dbmH9xazRQRy4dENoB4ydClla5yAzv6G3fxMs1uxOf5Vo+MZrlmWxCUsitH6A0p2qq1ulMixe73W51u/bcb6JquDCgtFF8xIm8ZGzP+jyvQSi0oPCWH+dIKipQlc7thg8fJ3zxQ0GvNAzmXqoOXr5+OkClJnGA+WWGYEl4Jy5y5UKOKtTmv91BZbciXByuGnGEDk4z/aWrdHBRzLrIqRpo8rvu/XRDpSt1Crh/woK4l1CtohhHfgybR0+dZzHoew5m9Dd0slOdkUFB0YJDFAouVDhbaKfmmxZO14ZS3Lt43yBEiv67U5cHwNdudZP7bnQV8bP2JF0a9atXC+h/iHBZgMJSYvByflIEFMkZUAUyeq3mDeudXTj6f6EIOduOuqGLNgZui9Sy303WUpyD+IHZQt+PwbFyae+gaGQ4wFN7zp0B2fnNYwb7K0RQImf4/4gkABgQp/J5h4JVQXxt+f06dJLHroswT2MhUR2b/7go0O/D7GewQaB4SWLmRpzmCFsOHp7akEiRPFLQrOXvg0jgiUUptf0cW+LuyeLo3bYbs9k8bkvOX/VgbdSujE6SvGA6oSe6ztUT11dvw8+aft0IWMa9EnJzILXa1NShJVM6nTu2r2cZRG7vaqQE4OJTr8XQZw8qMgBbStI/NRLfgbSA+XJIVV/L6ixpRYQCC5lHlbHcnSMoJi0pSYZ5fr71eWjDijhULcH6csQnIO/YNZKKJF3Q0xbxaqP6aq7C3RsIRoQsls2sSv9/uZ4IQ4D/ioe8m397X/Zs6QJZ/hYK0nXZn1c6dDSOaDJibVtkKBUTdWhbo6e93y1bwMWC+PYkUkxAmEx2Y1SnntwoUmiJYH179BCQLignL1omY+HagH5ZP5odULYNdU/A4oJHoSEgPxYDtnta4VRsu21fN5a9Tx/+0rA08cghouw32yH4A7qjboJdpX3b7YhKinhhzjFtNLmjy79k6o5KDLM9vb6oswJFAp82SEFkLCiycVvUTTE7pwOB3PpRzcHUdHTBfnPF0FuPp//hjNv7srG5XjzuiH9wvmK1oYxsnvm2vqcomlBE0gjiW/nsqXybwJtqpGxYu8ipbqsYpqDN+QzRtu+S+jRfnwdkcAazG9cpasrAoUDhWTYzVbgNJBdraafYa6WxJNXdlB26/CkgXQEAJ28qZDLccx7h20V+0NH/Mj9fMd2+jOq9Ksd50k5MQqzWosk4dynbUhbGM/atehHVf10fxTWftyXBL1aI+FvaGKJj/NxcxnhWhNwFn5zd0tBgI6BpXZup/vdsMrc5S+ZrRafbQi5Rhhjr88gZHmOcBMljLe7cOLtmXhqk/HrI+47ftamQ8Oy6MGY+F6RcxSOWlQpU4w3MjhfnKqw9NEc18+Bjg9O/+60oj9SNNteMW8xL/d2D6+PS0k8R18oFQl/zOk9yBbgeqYkYTZ39pFygk1KdbbgybFkj/VMwkWEKGy+qIFWP2NWbVHf0nqd/U8rVfvo9PHcDF92HphtkYGqqMgbwfZhQWJfXO8Ijw7f+vvWZzgpt2khJSwYaBiJH9czE2rg5ps4CDlQ64nKnGWE/USQCAfyicP7MesC2+wE1gmLMR4WCDLHq7YhaRHTpUPmYqWx9xuB9AVIIL147OZPeYTfqTHDI53kbVnw4nqds4pAa64MvQtsZT/r46alHXeWJDvIwF3S2E1hHoGsXSt62RPtZpbcZ1hl4wZqaM3tkuX6ZIWsFJ7+Q8pGrPrvWdaFRMg/g/QDE0SmWu706c3q0yPYoEdqxQldbeEveY81V0sR6rRftLnWWGJ637ujgDNYujxwCZJpJ52hy8gdhTgli78HKd9nhoqx2wo3BS403A9CCAcSOCTFR8zCde6foFoPCp0pUp6DAtNLwvpmdNBTRlfF4cLh38FUvfM8r6gwNs3WFQ63pyy9fIHFPI6ApUFcgZ1VbmnA9jUOiJ6KKhQygLFZ/q91ZCqCYTgDfUUj8UqWXfpa4Ucmljc6KX2JDgDhKFa5wvYUiIHO6LdMClSU6sc5FxVpMJOx+jFgD2O7HII9kueQM9QjKjFo7H4ZxX1uX88a+oV04JkJDIDSd3iTFqqfOn//1KrCutmxdLoyHJ0PeizgDmcQbFsrGi0qya1LsW2PdIuwXscMRJ7ZRJlHSUYowDcG9zlEsDdYwJaGFzX2dhdXNzlEsAjAVnYXVzc5RHAAAAAAAAAAB1YnViLg==",
        "n": 10,
        "shape": [],
        "dtype": "int64",
        "np_random": "RandomState(MT19937)"
    },
    "n_envs": 1,
    "_vectorize_action": true,
    "policy_kwargs": {}
}

[araffin]

Should this be Numpy savez object or something more universal?

Using numpy format sounds reasonable.

I did quick experiments with storing objects with this new format, and here is the JSON

Nice ;)