Releases: seung-lab/BigArrays.jl
used in zebrafish dataset processing and analysis
v1.3.0
add Info data structure to better manipulate the configuration
to run benchmark, we need to manipulate the parameters, so we add a new Infos module to represent the configuration and easy manipulation.
this release was done before a major change.
I'll change the Key-value Store to pointing to the layer path rather than the specific mip level. the design was inspired by cloud-volume. This design will allow more advanced features in the future. The future change will break some code, so this is the last backward compatible release.
first release for Julia 1.0/0.7
upgraded to Julia 1.0
smarter compression
automatically decompress data using gzip magic number.
release for Julia 0.6
- parsing info here rather than in backends.
remove some Base functions to avoid `type piracy`
fixed the issue:
#21
asynchronized IO based on Julia 0.5.2
- use channel to control number of concurrent requests
- still have a memory leak issue in getindex function.
support key-value backends
version based on Julia 0.5
features
once an Associate type was defined and values were retrievable using Dict indexing, it could be used as a backend of BigArray. So it is highly extensible.
current backends include:
- google cloud storage
- aws s3
test
It was used for pinky40 dataset processing as a big test.
problems
speed
the speed is pretty slow due to single processing reading, should be extensible using multiple threads in the future. Current Julia do not support threading iteration of tuples!
compatibility of neuroglancer
current binary chunks were compressed using blosclz, and config file is config.json while neuroglancer uses a json file called info. The basic idea is the same, but implementation is different, so should be strait forward to modify for compatibility of neuroglancer.
Julia 0.5 version used in production for zebra fish
Merge pull request #7 from seung-lab/dev Julia 0.5 version used in production for zebra fish