Errors when reading file from multiple processes

[dchandan]

I am encountering HDF error in a program where the file id's of input files are shared across multiple python processes for concurrent reading. If I substitute the netCDF4 library for the PyNIO library, then no errors are produced. Would you know why the netCDF4 library is not working for concurrent reads? I installed netCDF4 through the enthought python distribution.

[shoyer]

Are you attempting to use the same netCDF4.Dataset object across multiple processes? My understanding is that netCDF4.Dataset objects cannot be pickled, and hence, cannot be transferred between processes. However, in my experience it works fine if the Dataset objects are created in the separate processes.

[jswhit]

do you mean that you get an error with PyNIO, but not netCDF4, or vice-versa?

[dchandan]

I am afraid I reached my conclusions far too quickly. Although I thought that PyNIO was passing my tests, it turned out that was pure luck, and that in fact on subsequent runs I encountered the same problem as with netcdf4. I followed up with Stephan's comment and he is right - the Dataset object cannot be shared among multiple processes. The problem with PyNIO was the same (although they don't call it Dataset). I ended up with opening the files from multiple processes and that worked out with both netCDF4 and PyNIO. Sorry for prematurely reporting this as an issue!