Support for updating tmpfs size according to Image size #7484

lkunjumon · 2021-04-29T16:36:14Z

Why I did it

while sonic upgrade, Image will be extracted to tmpfs for installation so tmpfs size should be larger than image size. Image installation will fail if image size is larger than tmpfs size.

we are facing below error while installing debug image with size greater than tmpfs which is 1.5g in marvell armhf platform.

sonic-installer install
New image will be installed, continue? [y/N]: y
Downloading image...
...99%, 1744 MB, 708 KB/s, 0 seconds left...
Installing image SONiC-OS-202012.0-dirty-20210311.224845 and setting it as default...
Command: bash /tmp/sonic_image
tar: installer/fs.zip: Wrote only 7680 of 10240 bytes
tar: installer/onie-image-arm64.conf: Cannot write: No space left on device
tar: Exiting with failure status due to previous errors
Verifying image checksum ... OK.
Preparing image archive ...

How I did it

compare downloaded image size with tmpfs size, if size less than image size update the tmpfs size according to image size.

How to verify it

Install an Image with size larger than tmpfs. we verified by installing debug image with size 1.9gb which is larger than tmpfs size 1.5gb.

Which release branch to backport (provide reason below if selected)

201811
201911
202006
202012

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

qiluo-msft · 2021-04-29T19:55:25Z

installer/sharch_body.sh

 # Untar and launch install script in a tmpfs
 cur_wd=$(pwd)
 export cur_wd
 archive_path=$(realpath "$0")
 tmp_dir=$(mktemp -d)
 if [ "$(id -u)" = "0" ] ; then
    mount -t tmpfs tmpfs-installer $tmp_dir || exit 1
+    mount_size=$(df $tmp_dir | tail -1 | tr -s ' ' | cut -d' ' -f4)
+    if [ "$mount_size" -lt "$((image_size*3))" ]; then


image_size*3

This seems an estimation. How do you know *3 is enough? Could you get the accurate required size? #Closed

that must be generated from the build process.

use this to get the original size?

lgh@p330:~$ gzip -l docker-sonic-vs.gz compressed uncompressed ratio uncompressed_name 242618472 684168704 64.5% docker-sonic-vs

This script untars sonic-platform.bin, which is a ONIE image. We cannot use gzip command.
Will update the logic by getting image size from the "tar" command, excluding the ONIE installer script

Blueve · 2021-05-08T04:02:07Z

I hit same issue on Celestica platform.
The tmpfs is 991M by default but some of image size are larger than 1G

lkunjumon · 2021-05-17T13:03:54Z

Updated logic to get image size from "tar" command.

qiluo-msft · 2021-05-17T23:11:31Z

installer/sharch_body.sh

@@ -25,13 +25,19 @@ fi

 echo " OK."

+image_size=$(( $(sed -e '1,/^exit_marker$/d' "$0"  | tar --to-stdout -xf - | wc -c) / 1024))


/

Do you want to get the ceiling of division?

okay.will update with ceiling.

Sorry, how do you get the ceiling? Also there are 2 more division below.

Instead of x / 1024,
(x + 1023) / 1024 will be much safer.

we can avoid ceiling here as we already adding ceiling to image_size by 1G in this command " mount_size=$(((image_size/1024/1024)+1)) " .Here if we add it will be ceiled by 1KB only ,so it is not necessary.

Okay, you are using padding instead of ceiling. Do you think 1G padding is too large?

"mount_size=$(((image_size/1024/1024)+1))" Here we are adding ceiling for image_size and extra free space for the mount , so extra free space will be less than 1GB.

My only concern is the extreme padding size 1G seem too large. Others look good to me.

@qiluo-msft,
is it fine to update 32MB extra space? Find below changes. If this is fine, we will push the changes.

mount_size=$((((image_size+1023)/1024) + 32))
mount -o remount,size="${mount_size}M" -t tmpfs tmpfs-installer $tmp_dir || exit 1

32M padding is good. However you divided 3 times, so I think you formula will not work for corner cases.
This is my math

bytes = $(sed -e '1,/^exit_marker$/d' "$0" | tar --to-stdout -xf - | wc -c) image_size=$((bytes + 1023) / 1024) mount_size=$((image_size + 31) / 32) * 32 mount -o remount,size="${mount_size}M"

32M padding is good. However you divided 3 times, so I think you formula will not work for corner cases.
This is my math

bytes = $(sed -e '1,/^exit_marker$/d' "$0" | tar --to-stdout -xf - | wc -c) image_size=$((bytes + 1023) / 1024) mount_size=$((image_size + 31) / 32) * 32 mount -o remount,size="${mount_size}M"

Updated PR with your comments. Please review.

qiluo-msft · 2021-05-17T23:12:18Z

installer/sharch_body.sh

 # Untar and launch install script in a tmpfs
 cur_wd=$(pwd)
 export cur_wd
 archive_path=$(realpath "$0")
 tmp_dir=$(mktemp -d)
 if [ "$(id -u)" = "0" ] ; then
    mount -t tmpfs tmpfs-installer $tmp_dir || exit 1
+    mount_size=$(df $tmp_dir | tail -1 | tr -s ' ' | cut -d' ' -f4)
+    if [ "$mount_size" -lt "$((image_size*3))" ]; then


image_size

Is image_sizes*1 is good enough? #Closed

I have addressed the comment. Please review

lkunjumon · 2021-05-28T09:47:40Z

retest please

radha-danda · 2021-06-09T15:20:05Z

@lguohan, @qiluo-msft, we have answered the queries on this PR. Kindly let us know if you have any further queries. Can this PR be merged?

Signed-off-by: lkunjumon <lkunjumon@marvell.com>

radha-danda · 2021-06-21T06:32:23Z

@lguohan, @qiluo-msft, we have answered the queries on this PR. Kindly let us know if you have any further queries. Can this PR be merged?

radha-danda · 2021-06-21T06:34:39Z

@lguohan, can this PR be merged as it is approved by Qi Luo

#### Why I did it while sonic upgrade, Image will be extracted to tmpfs for installation so tmpfs size should be larger than image size. Image installation will fail if image size is larger than tmpfs size. we are facing below error while installing debug image with size greater than tmpfs which is 1.5g in marvell armhf platform. sonic-installer install <url> New image will be installed, continue? [y/N]: y Downloading image... ...99%, 1744 MB, 708 KB/s, 0 seconds left... Installing image SONiC-OS-202012.0-dirty-20210311.224845 and setting it as default... Command: bash /tmp/sonic_image tar: installer/fs.zip: Wrote only 7680 of 10240 bytes tar: installer/onie-image-arm64.conf: Cannot write: No space left on device tar: Exiting with failure status due to previous errors Verifying image checksum ... OK. Preparing image archive ... #### How I did it compare downloaded image size with tmpfs size, if size less than image size update the tmpfs size according to image size. #### How to verify it Install an Image with size larger than tmpfs. we verified by installing debug image with size 1.9gb which is larger than tmpfs size 1.5gb.

dflynn-Nokia · 2021-09-03T13:41:46Z

Please cherry-pick this PR to the 202012 and 202106 releases.

#### Why I did it while sonic upgrade, Image will be extracted to tmpfs for installation so tmpfs size should be larger than image size. Image installation will fail if image size is larger than tmpfs size. we are facing below error while installing debug image with size greater than tmpfs which is 1.5g in marvell armhf platform. sonic-installer install <url> New image will be installed, continue? [y/N]: y Downloading image... ...99%, 1744 MB, 708 KB/s, 0 seconds left... Installing image SONiC-OS-202012.0-dirty-20210311.224845 and setting it as default... Command: bash /tmp/sonic_image tar: installer/fs.zip: Wrote only 7680 of 10240 bytes tar: installer/onie-image-arm64.conf: Cannot write: No space left on device tar: Exiting with failure status due to previous errors Verifying image checksum ... OK. Preparing image archive ... #### How I did it compare downloaded image size with tmpfs size, if size less than image size update the tmpfs size according to image size. #### How to verify it Install an Image with size larger than tmpfs. we verified by installing debug image with size 1.9gb which is larger than tmpfs size 1.5gb.

Support for updating tmpfs size according to Image size

33bc329

lkunjumon requested a review from qiluo-msft as a code owner April 29, 2021 16:36

qiluo-msft reviewed Apr 29, 2021

View reviewed changes

Add support to get actual size of compressed image.

25d32a1

qiluo-msft reviewed May 17, 2021

View reviewed changes

Updating tmpfs size according to Image size

3c6c399

Updated review comments and added extra 32MB free space for mount size.

53e8c97

Signed-off-by: lkunjumon <lkunjumon@marvell.com>

qiluo-msft approved these changes Jun 16, 2021

View reviewed changes

qiluo-msft merged commit 90801dc into sonic-net:master Jun 21, 2021

Blueve added Request for 202012 Branch Request for 202106 Branch labels Sep 9, 2021

qiluo-msft added the Included in 202012 Branch label Sep 13, 2021

judyjoseph added the Included in 202106 Branch label Sep 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for updating tmpfs size according to Image size #7484

Support for updating tmpfs size according to Image size #7484

lkunjumon commented Apr 29, 2021 •

edited

Loading

qiluo-msft Apr 29, 2021 •

edited

Loading

lguohan May 1, 2021

lguohan May 8, 2021 •

edited

Loading

antony-rheneus May 17, 2021 •

edited

Loading

Blueve commented May 8, 2021

lkunjumon commented May 17, 2021

qiluo-msft May 17, 2021

lkunjumon May 18, 2021

qiluo-msft May 21, 2021

lkunjumon May 27, 2021

qiluo-msft Jun 1, 2021

lkunjumon Jun 1, 2021 •

edited

Loading

qiluo-msft Jun 9, 2021

radha-danda Jun 14, 2021

qiluo-msft Jun 14, 2021

lkunjumon Jun 16, 2021

qiluo-msft May 17, 2021 •

edited

Loading

lkunjumon May 21, 2021

lkunjumon commented May 28, 2021

radha-danda commented Jun 9, 2021

radha-danda commented Jun 21, 2021

radha-danda commented Jun 21, 2021

dflynn-Nokia commented Sep 3, 2021

		@@ -25,13 +25,19 @@ fi

		echo " OK."

		image_size=$(( $(sed -e '1,/^exit_marker$/d' "$0" \| tar --to-stdout -xf - \| wc -c) / 1024))

Support for updating tmpfs size according to Image size #7484

Support for updating tmpfs size according to Image size #7484

Conversation

lkunjumon commented Apr 29, 2021 • edited Loading

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

qiluo-msft Apr 29, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lguohan May 8, 2021 • edited Loading

Choose a reason for hiding this comment

antony-rheneus May 17, 2021 • edited Loading

Choose a reason for hiding this comment

Blueve commented May 8, 2021

lkunjumon commented May 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lkunjumon Jun 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiluo-msft May 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lkunjumon commented May 28, 2021

radha-danda commented Jun 9, 2021

radha-danda commented Jun 21, 2021

radha-danda commented Jun 21, 2021

dflynn-Nokia commented Sep 3, 2021

lkunjumon commented Apr 29, 2021 •

edited

Loading

qiluo-msft Apr 29, 2021 •

edited

Loading

lguohan May 8, 2021 •

edited

Loading

antony-rheneus May 17, 2021 •

edited

Loading

lkunjumon Jun 1, 2021 •

edited

Loading

qiluo-msft May 17, 2021 •

edited

Loading