Large single-tile image and memory: need low level streaming implementation

[gcode-importer]

Originally reported on Google Code with ID 94

The site 'http://marsoweb.nas.nasa.gov/HiRISE/hirise_images/'
allows the download of JP2 images.

I have downloaded

ESP_011277_1825_RED: Crater in Meridiani Planum with Layering

The downloaded file 'ESP_011277_1825_RED.JP2' has a size of
90'573'764 Byte.

On my LINUX 32-bit machine all viewers fail to show this image.

-----------------------------------
j2k_dump -i ESP_011277_1825_RED.JP2 > index

[ERROR] read error: passed the end of the codestream
   (start = -1307983864, current = -1217410100, end = -1217410100
[ERROR] tcd_decode: incomplete bistream
Segmentation fault
-----------------------------------
My simple JP2 reader shows:

[224]marker(0xff90)
sot  tile_nr(0) Psot(1'415'700'102) TPsot(0) TNsot(1)

-----------------------------------
free -m (i.e. megabytes)
             total    used    free  shared  buffers  cached
Mem:          2025     807    1217       0       58     685
-/+ buffers/cache:      64    1961
Swap:           23       0      23
-----------------------------------

Does this mean that my memory and swap space is too small for
such an image?

winfried

Reported by szukw000 on 2011-09-26 06:24:14

[gcode-importer]

Perhaps it was the swap space. After extending the swap space to
8 GB, I got:

j2k_dump -v -i ESP_011277_1825_RED.JP2 -o index

Setting decoding area to 0,0,0,0
[INFO] Stream reached its end !
[ERROR] Stream too short
ERROR -> j2k_dump: failed read the tile header

winfried

Reported by szukw000 on 2011-09-28 07:21:10

[gcode-importer]

I have now tested another HiRISE image: ESP_014256_1300_RED.JP2.
It has a size of 259'194'623 bytes.

Currently

 http://openjpeg.googlecode.com/svn/branches/openjpeg-1.5

returns revision 950.

In openjpeg-1.5-revision-950/application/codec/

both 'j2k_dump' and 'j2k_to_image' fail with 'Segmentation fault'.

'j2k_to_image' fails in 'libopenjpeg/t1.c'.

line 1556: y == 0, x == 0, tile_w == 14617, tile_dp == NULL

line 1557: clbk_h == 64

line 1558: cblk_w == 58

line 1559: j == 0, i == 0, tmp == -1025

winfried

Reported by szukw000 on 2011-09-28 08:20:53

[gcode-importer]

Adding direct link:
http://hirise-pds.lpl.arizona.edu/download/PDS/RDR/ESP/ORB_011200_011299/ESP_011277_1825/ESP_011277_1825_RED.JP2

Reported by malaterre on 2012-01-11 09:15:46

[gcode-importer]

Reported by malaterre on 2012-05-29 13:18:23

[gcode-importer]

This image is not tiled. This will be hard to deal with even using v2.

Reported by malaterre on 2012-10-15 15:28:12

[gcode-importer]

Reported by malaterre on 2014-02-25 15:27:22

• Labels added: Priority-Low
• Labels removed: Priority-Medium

[gcode-importer]

Issue 122 has been merged into this issue.

Reported by malaterre on 2014-02-27 08:14:23

[gcode-importer]

Reported by malaterre on 2014-02-27 08:14:38

[gcode-importer]

Reported by malaterre on 2014-02-27 08:15:11

[gcode-importer]

Issue 151 has been merged into this issue.

Reported by malaterre on 2014-02-27 13:32:29

[gcode-importer]

Reported by malaterre on 2014-03-06 11:50:49

• Blocking: Improve support for region of interest #39

[gcode-importer]

Reported by malaterre on 2014-03-06 11:51:35

[gcode-importer]

Reported by malaterre on 2014-03-07 09:11:23

• Blocking: OpenJPEG allocates more memory than stricly needed #275

[rouault]

The only missing block to finally solve the issue is support in opj_decompress itself for large images : #1012
Support in the lib for partial decoding with low memory requirements is now completed per #1010