Always cache spec?

[rly]

Is there a reason why we would not want to cache the spec when writing a file?

Advantages of always caching spec: fewer complex use cases and interactions to handle and tests to make.

Parallel issue on hdmf: hdmf-dev/hdmf#76

[oruebel]

For NWB:N files I don't see a reason why you would not always want to cache the spec. The amount of data is minimal and it helps ensure that the file remains accessible.

The only use-case I could think of where you may not want to do this is if you want to write partial files, e.g., a single TimeSeries container, rather than a full NWB:N file. The question of partial files has come up in the context of integration with data management. However, even in this case, I think you are probably fine with caching the spec.

[bendichter]

The tests are considerable slower when caching the spec

…

On Wed, May 29, 2019, 5:36 PM Oliver Ruebel ***@***.***> wrote: For NWB:N files I don't see a reason why you would not always want to cache the spec. The amount of data is minimal and it helps ensure that the file remains accessible. The only use-case I could think of where you may not want to do this is if you want to write partial files, e.g., a single TimeSeries container, rather than a full NWB:N file. The question of partial files has come up in the context of integration with data management. However, even in this case, I think you are probably fine with caching the spec. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#967?email_source=notifications&email_token=AAGOEETYINEULAEGJGNZQHTPX4OSLA5CNFSM4HQY3TDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWRAKOY#issuecomment-497157435>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGOEEWU2KY4VTIAQS7VSPDPX4OSLANCNFSM4HQY3TDA> .

[t-b]

The tests are considerable slower when caching the spec

Which just means that writing the spec to the file takes a lot of time or?

[bendichter]

I'm not sure if it's writing or reading, just something I noticed. I think it would be worthwhile to profile this to see what the time penalty is.

[t-b]

Related: #497

[rly]

Would there be any issues in loading namespaces by default on read? If specs are cached by default, they should be read by default, right?

i.e. make load_namespaces=true the default when calling NWBHDF5IO(path, 'r', load_namespaces=true)

pynwb/src/pynwb/__init__.py

Lines 188 to 230 in 0eea97e

	class NWBHDF5IO(_HDF5IO):
	@docval({'name': 'path', 'type': str, 'doc': 'the path to the HDF5 file'},
	{'name': 'mode', 'type': str,
	'doc': 'the mode to open the HDF5 file with, one of ("w", "r", "r+", "a", "w-")'},
	{'name': 'load_namespaces', 'type': bool,
	'doc': 'whether or not to load cached namespaces from given path', 'default': False},
	{'name': 'manager', 'type': BuildManager, 'doc': 'the BuildManager to use for I/O', 'default': None},
	{'name': 'extensions', 'type': (str, TypeMap, list),
	'doc': 'a path to a namespace, a TypeMap, or a list consisting paths \
	to namespaces and TypeMaps', 'default': None},
	{'name': 'file', 'type': h5py.File, 'doc': 'a pre-existing h5py.File object', 'default': None},
	{'name': 'comm', 'type': "Intracomm", 'doc': 'the MPI communicator to use for parallel I/O',
	'default': None})
	def __init__(self, **kwargs):
	path, mode, manager, extensions, load_namespaces, file_obj, comm =\
	popargs('path', 'mode', 'manager', 'extensions', 'load_namespaces', 'file', 'comm', kwargs)
	if load_namespaces:
	if manager is not None:
	warn("loading namespaces from file - ignoring 'manager'")
	if extensions is not None:
	warn("loading namespaces from file - ignoring 'extensions' argument")
	if 'w' in mode:
	raise ValueError("cannot load namespaces from file when writing to it")
	# XXX: Leaving this here in case we want to revert to this strategy for
	# loading cached namespaces
	# ns_catalog = NamespaceCatalog(NWBGroupSpec, NWBDatasetSpec, NWBNamespace)
	# super(NWBHDF5IO, self).load_namespaces(ns_catalog, path)
	# tm = TypeMap(ns_catalog)
	# tm.copy_mappers(get_type_map())
	tm = get_type_map()
	super(NWBHDF5IO, self).load_namespaces(tm, path)
	manager = BuildManager(tm)
	else:
	if manager is not None and extensions is not None:
	raise ValueError("'manager' and 'extensions' cannot be specified together")
	elif extensions is not None:
	manager = get_manager(extensions=extensions)
	elif manager is None:
	manager = get_manager()
	super(NWBHDF5IO, self).__init__(path, manager=manager, mode=mode, file=file_obj, comm=comm)

[bendichter]

I think that would replace any custom classes that have already been imported and/or registered