public archive sdk #2426

arturrez · 2024-12-11T00:26:04Z

Why this should be merged

resolves #2425
it provides sdk pkg to access public archive data for fuji to speedup avalanchego bootstrap if connected to fuji
current timing

1 node 27.877
2 nodes 31.086
7 nodes 1:25.34

How this works

only fuji is supported for now.

sdk provides a functionality to Download public archive data into tmp file. This data can be un-tar into destination folder.
this sdk is used inside of the CLI. In case of the error all downloaded data is cleaned and bootstrap continues on like nothing happened.

How this was tested

./bin/avalanche node local start  --num-nodes 1 --fuji --avalanchego-path ~/src/github.com/ava-labs/avalanchego/build/avalanchego fuji1
./bin/avalanche node local start  --num-nodes 2 --fuji --avalanchego-path ~/src/github.com/ava-labs/avalanchego/build/avalanchego fuji2

How is this documented

n/a

sdk/publicarchive/downloader.go

… achilles-sdk

# Conflicts: # pkg/node/local.go

sukantoraymond · 2025-01-07T06:31:23Z

pkg/node/local.go

@@ -490,6 +498,9 @@ func UpsizeLocalNode(
 		nodeConfig = map[string]interface{}{}
 	}
 	nodeConfig[config.NetworkAllowPrivateIPsKey] = true
+	if network.Kind == models.Fuji {
+		nodeConfig[config.IndexEnabledKey] = false // disable index for Fuji


what does it mean if this flag is disabled?

more info https://docs.avax.network/nodes/configure/configs-flags#--index-enabled-boolean

dataset for avalanchego created with index and without are not interchangeable

sukantoraymond · 2025-01-07T06:48:10Z

sdk/publicarchive/downloader.go

+}
+
+// NewDownloader returns a new Downloader
+// network: the network to download from ( fuji or mainnet). todo: add mainnet support


lets remove this mainnet support part

sukantoraymond · 2025-01-07T06:49:06Z

sdk/publicarchive/downloader.go

+			}, nil
+		}
+	default:
+		return Downloader{}, fmt.Errorf("unsupported network ID: %d", network.ID)


I think we can just print currently public archive is only supported on fuji

sukantoraymond · 2025-01-07T09:23:25Z

sdk/publicarchive/downloader.go

+	return nil
+}
+
+func (d Downloader) UnpackTo(targetDir string) error {


Is there no library for this?

we also have code for this

sukantoraymond · 2025-01-07T09:37:26Z

sdk/publicarchive/downloader.go

+		switch header.Typeflag {
+		case tar.TypeDir:
+			d.logger.Debug("Creating directory", zap.String("path", targetPath))
+			if err := os.MkdirAll(targetPath, os.FileMode(header.Mode)); err != nil {
+				d.logger.Error("Failed to create directory", zap.Error(err))
+				return fmt.Errorf("failed to create directory: %w", err)
+			}
+		case tar.TypeReg:
+			d.logger.Debug("Ensure parent directory exists for ", zap.String("path", targetPath))
+			if err := os.MkdirAll(filepath.Dir(targetPath), os.FileMode(0o755)); err != nil {
+				d.logger.Error("Failed to create parent directory for file", zap.Error(err))
+				return fmt.Errorf("failed to create parent directory for file: %w", err)
+			}
+			d.logger.Debug("Creating file", zap.String("path", targetPath))
+			outFile, err := os.Create(targetPath)
+			if err != nil {
+				d.logger.Error("Failed to create file", zap.Error(err))
+				return fmt.Errorf("failed to create file: %w", err)
+			}
+			defer outFile.Close()
+			copied, err := io.CopyN(outFile, tarReader, header.Size)
+			if err != nil {
+				d.logger.Error("Failed to write file", zap.Error(err))
+				return fmt.Errorf("failed to write file: %w", err)
+			}
+			if copied < header.Size {
+				d.logger.Error("Incomplete file write", zap.String("path", targetPath))
+				return fmt.Errorf("incomplete file write for %s", targetPath)
+			}
+			extractedSize += header.Size
+			d.logger.Debug("Written bytes", zap.Int64("bytes", extractedSize))
+		default:
+			d.logger.Debug("Skipping file", zap.String("path", targetPath))
+		}


might be better for us to use library

i didn't find one that I liked

I am open for the suggestions

sukantoraymond · 2025-01-07T09:52:14Z

pkg/node/helper.go

+	if err := publicArcDownloader.Download(); err != nil {
+		return fmt.Errorf("failed to download public archive: %w", err)
+	}
+	// defer publicArcDownloader.CleanUp()


does this need to be deleted

good catch. it should be enabled and was commented out for the debug

sukantoraymond · 2025-01-07T09:59:19Z

pkg/node/helper.go

+
+	wg := sync.WaitGroup{}
+	mu := sync.Mutex{}
+	var firstErr error


maybe we should rename this err var

err is used in

if firstErr == nil { firstErr = fmt.Errorf("failed to unpack public archive: %w", err)

so it should be some other name

sukantoraymond · 2025-01-07T10:05:47Z

pkg/node/helper.go

+		ux.Logger.Info("unpacking public archive to %s", target)
+
+		// Skip if target already exists
+		if _, err := os.Stat(target); err == nil {


should we override data instead of skip?

no we should skip as data is already there

sdk/publicarchive/helpers.go

felipemadero · 2025-01-07T19:05:52Z

sdk/publicarchive/downloader.go

+	"github.com/cavaliergopher/grab/v3"
+	"go.uber.org/zap"
+
+	sdkConstants "github.com/ava-labs/avalanche-cli/sdk/constants"


inside sdk I shall call this one constants, and the other one cliConstants

felipemadero · 2025-01-07T19:06:57Z

sdk/publicarchive/downloader.go

+// logLevel: the log level
+func NewDownloader(
+	network network.Network,
+	logLevel logging.Level,


can it receive a logger instead of log level

that is to have full control on the logs

felipemadero · 2025-01-07T19:09:13Z

sdk/publicarchive/downloader.go

+	return nil
+}
+
+func (d Downloader) UnpackTo(targetDir string) error {


we also have code for this

felipemadero · 2025-01-07T19:10:53Z

sdk/constants/constants.go

@@ -10,5 +10,6 @@ const (
 	APIRequestLargeTimeout = 2 * time.Minute

 	// node
-	WriteReadUserOnlyPerms = 0o600
+	WriteReadUserOnlyPerms    = 0o600
+	WriteReadUserOnlyDirPerms = 0o700


UserOnlyReadWriteExecPerms ?

felipemadero · 2025-01-07T19:12:44Z

pkg/node/helper.go

+	return nil
+}
+
+func CleanUpClusterNodeData(rootDir string, nodesNames []string) error {


maybe this is not needed to be public

agree. fixed

felipemadero · 2025-01-07T19:13:21Z

pkg/node/helper.go

@@ -555,3 +560,75 @@ func GetNodeData(endpoint string) (
 		"0x" + hex.EncodeToString(proofOfPossession.ProofOfPossession[:]),
 		nil
 }
+
+func SeedClusterData(


Can it have another name. Something related to downloading blockchain db for the cluster

changed to DownloadPublicArchive

felipemadero

LGTM

arturrez added 9 commits December 9, 2024 18:21

sdk public archive

1ba4d47

lint additional checks

d2618b3

add unittests

eb1359d

fixes

fd5ec0c

add readme add e2e unittest. fixes

167a623

r4 testing

20907d5

r4r

9ae7c69

nits

710408c

put destroy node on error back

6f9cee2

arturrez requested a review from a team as a code owner December 11, 2024 00:26

arturrez requested a review from sukantoraymond December 11, 2024 00:26

arturrez changed the title ~~Achilles sdk~~ public archive sdk Dec 11, 2024

github-advanced-security bot found potential problems Dec 11, 2024

View reviewed changes

sdk/publicarchive/downloader.go Fixed Show fixed Hide fixed

arturrez and others added 8 commits December 11, 2024 12:06

add another security check

a531f62

lint

d4e03c9

Merge branch 'main' into achilles-sdk

c48bc75

Merge branch 'main' into achilles-sdk

ef97841

add missing license

dcb2780

Merge branch 'achilles-sdk' of github.com:ava-labs/avalanche-cli into…

8592aba

… achilles-sdk

Merge branch 'main' into achilles-sdk

a92819f

# Conflicts: # pkg/node/local.go

Merge branch 'main' into achilles-sdk

10b4d47

sukantoraymond reviewed Jan 7, 2025

View reviewed changes

felipemadero reviewed Jan 7, 2025

View reviewed changes

address feedback

6066e3a

felipemadero approved these changes Jan 8, 2025

View reviewed changes

Merge branch 'main' into achilles-sdk

24102cb

arturrez merged commit 3e3ccce into main Jan 8, 2025
37 checks passed

arturrez deleted the achilles-sdk branch January 8, 2025 18:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

public archive sdk #2426

public archive sdk #2426

arturrez commented Dec 11, 2024 •

edited

Loading

sukantoraymond Jan 7, 2025

arturrez Jan 7, 2025

arturrez Jan 7, 2025

sukantoraymond Jan 7, 2025

arturrez Jan 7, 2025

sukantoraymond Jan 7, 2025

arturrez Jan 7, 2025

sukantoraymond Jan 7, 2025 •

edited

Loading

felipemadero Jan 7, 2025

sukantoraymond Jan 7, 2025

arturrez Jan 7, 2025

arturrez Jan 7, 2025

sukantoraymond Jan 7, 2025

arturrez Jan 7, 2025

sukantoraymond Jan 7, 2025

arturrez Jan 7, 2025

sukantoraymond Jan 7, 2025

arturrez Jan 7, 2025

felipemadero Jan 7, 2025

arturrez Jan 7, 2025

felipemadero Jan 7, 2025

felipemadero Jan 7, 2025

arturrez Jan 7, 2025

felipemadero Jan 7, 2025

felipemadero Jan 7, 2025

arturrez Jan 7, 2025

felipemadero Jan 7, 2025

arturrez Jan 7, 2025

felipemadero Jan 7, 2025

arturrez Jan 7, 2025

felipemadero left a comment

public archive sdk #2426

public archive sdk #2426

Conversation

arturrez commented Dec 11, 2024 • edited Loading

Why this should be merged

How this works

How this was tested

How is this documented

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sukantoraymond Jan 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

felipemadero left a comment

Choose a reason for hiding this comment

arturrez commented Dec 11, 2024 •

edited

Loading

sukantoraymond Jan 7, 2025 •

edited

Loading