zstast extract. size does not match expected size error

[GeorginaGibson]

Question criteria

• I added a descriptive title here.
• I searched the zstash GitHub Discussions to find a similar question and didn't find it.
• I searched the zstash documentation.

What is the deadline?

Ideally within a week. I need the files for analysis.

Describe your question

I am running into trouble trying to extract files from my archive with zstash but I hit an error about the unexpected size of the first file. I am 95% sure I have successfully run this command on the same archive previously. Any advice would be appreciated. Is there a way to by-pass this first file if it is corrupt?

---

I created my archive several moths ago with:
zstash create --hpss=/home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC --maxsize 256 . 2>&1 | tee zstash_create_20241004.log

And then I updated it with:
zstash update --hpss=/home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC 2>&1 | tee zstash_update_20241004.log

I can see that the files I am interested in are listed when I do:
zstash ls --hpss=/home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC

I was trying to extract with:
zstash extract --hpss=/home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC "*timeSeriesStatsMonthly.*nc"

Are there are any possible answers you came across?

I have not found a solution

What machine were you running on?

Perlmutter

Environment

/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/bin/zstash

Minimal Complete Verifiable Example (MCVE)

zstash extract --hpss=/home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC          "*timeSeriesStatsMonthly.*nc"

INFO: Transferring file from HPSS: zstash/index.db
INFO: Transferring file from HPSS: zstash/000001.tar
INFO: zstash/000001.tar exists. Checking expected size matches actual size.
INFO: 000001.tar: expected size=274093875200 != 1843243=actual_size
INFO: Retrying HPSS get: 1 tries remaining.
INFO: zstash/000001.tar exists. Checking expected size matches actual size.
INFO: 000001.tar: expected size=274093875200 != 1843243=actual_size
INFO: Transferring file from HPSS: zstash/000001.tar
INFO: zstash/000001.tar exists. Checking expected size matches actual size.
INFO: 000001.tar: expected size=274093875200 != 1843243=actual_size
Traceback (most recent call last):
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/bin/zstash", line 10, in <module>
    sys.exit(main())
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/lib/python3.10/site-packages/zstash/main.py", line 69, in main
    extract()
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/lib/python3.10/site-packages/zstash/extract.py", line 45, in extract
    failures: List[FilesRow] = extract_database(args, cache, keep_files)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/lib/python3.10/site-packages/zstash/extract.py", line 283, in extract_database
    failures = extractFiles(matches, keep_files, keep, cache, cur, args)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/lib/python3.10/site-packages/zstash/extract.py", line 504, in extractFiles
    raise e
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.10.0_login/lib/python3.10/site-packages/zstash/extract.py", line 493, in extractFiles
    raise RuntimeError(
RuntimeError: zstash/000001.tar size does not match expected size.

Relevant log output

No response

Anything else we need to know?

No response

[forsyth2]

@golaz do you have any suggestions here? This may or may not have a related solution to #362, which has a similar, though distinct, problem of being unable to verify file integrity (bad checksum as opposed to does not match expected size)

[forsyth2]

@GeorginaGibson Do you have the file path to the directory you ran zstash create --hpss=/home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC --maxsize 256 . 2>&1 | tee zstash_create_20241004.log in? I don't haver permission to access the HPSS directory /home/g/georgina itself.

[GeorginaGibson]

It was here on Perlmutter. /pscratch/sd/g/georgina/e3sm_scratch/pm-cpu/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC.
I think I have the permissions set there so that you can take a look. There are a few tar files in there I manually downloaded to make sure content was still there. Thanks!

[forsyth2]

I don't haver permission to access the HPSS directory /home/g/georgina itself.

HPSS is a little tricky to work on since some commands don't work. I'm not sure if you can chmod your permissions there, but if not, maybe do the following:

hsi
cd /home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC/zstash
du 000001.tar

I want to see if the file size matches the "expected size" of 274093875200 or the "actual size" of 1843243. That is, is the file full-size on HPSS and just not extracted fully or is not even full-size on HPSS?

I see in /pscratch/sd/g/georgina/e3sm_scratch/pm-cpu/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC/zstash_update_20241004.log that tar size=1843243

zstash create --hpss=/home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC --maxsize 256 . 2>&1 | tee zstash_create_20241004.log

It's possible this command causes problems, per the discussion at #335 (it's best to avoid sending output to a log file in the directory being archived). But even then, we'd expect a bad checksum error, not the mismatched sizes error.

[GeorginaGibson]

du on the file showed its size matches the ‘actual size’ at 1843243. I think I have manage to change the permissions on /home/g/georgina Thanks for looking into this. Georgina Georgina Gibson Research Associate Professor International Arctic Research Center E-mail: ***@***.***

…

On Tue, Mar 4, 2025 at 12:09 PM forsyth2 ***@***.***> wrote: I don't haver permission to access the HPSS directory /home/g/georgina itself. HPSS is a little tricky to work on since some commands don't work. I'm not sure if you can chmod your permissions there, but if not, maybe do the following: hsicd /home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC/zstash du 000001.tar I want to see if the file size matches the "expected size" of 274093875200 or the "actual size" of 1843243. That is, is the file full-size on HPSS and just not extracted fully or is not even full-size on HPSS? I see in /pscratch/sd/g/georgina/e3sm_scratch/pm-cpu/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC/zstash_update_20241004.log that tar size=1843243 zstash create --hpss=/home/g/georgina/E3SM/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC --maxsize 256 . 2>&1 | tee zstash_create_20241004.log It's possible this command causes problems, per the discussion at #335 <#335> (it's best to avoid sending output to a log file in the directory being archived). But even then, we'd expect a bad checksum error, not the mismatched sizes error. — Reply to this email directly, view it on GitHub <#364 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADBZOZ4SDV4CRCIBMQQMXBD2SX25XAVCNFSM6AAAAABYA3EQC6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMZZGI3DGOI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[maltrud]

@GeorginaGibson @forsyth2 I just had a quick look and I seems that 00000[0-2].tar are useless. you've lost all of the data (overwritten?) that zstash thinks should be there according to the index.db file. all of the other tar files seem ok. the restarts begin in 000019.tar so you should be able to regenerate files if necessary.

[GeorginaGibson]

Thanks Mat. Not sure how they would have been overwritten.. maby during a zstash update (I added 5 more years to the archive at one point). Did you manually look (and find) where the restarts were ? Or did you get a grep type command to work ? That was what I was trying when it balked and complained about the first file.

[maltrud]

you can do the following to get a readable version of the index.db file:

sqlite3 index.db "select * from files;" > & index.txt

I did this on your index.db file. here's few lines from the index.txt file:
.....
2056|archive/ocn/hist/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC.mpaso.hist.am.timeSeriesStatsMonthlyMin.1960-12-01.nc|342714960|2024-09-30 05:23:14|09a78b1d24cd312d42860c097614dc0c|000019.tar|123340490752
2057|archive/rest/1950-01-02-00000/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC.cpl.r.1950-01-02-00000.nc|1326455952|2024-09-17 17:30:17|45f6314b07d2840215defa70d79ac5c9|000019.tar|123683207680
2058|archive/rest/1950-01-02-00000/TL319_r05_ARRM10to60E2r1.JRA-MOSART-BGC.elm.h1.1950-01-01-00000.nc|4546416556|2024-09-20 17:40:21|e3fd2888126dc88ebb5c562245f08509|000019.tar|125009665536
.....
this shows the first instance of the word "rest" and that the file is in 000019.tar. this is actually how I always retrieve specific files instead of doing it with zstash directly.