cmd, core, eth, les, node: chain freezer on top of db rework #19244

karalabe · 2019-03-08T14:08:17Z

This PR implements a long desired feature of moving ancient chain segments out of the active database into immutable append-only flat files.

There are a few goals of this new mechanism:

By storing less data in the active key/value database, compactions should in theory be a lot faster since there's less to shuffle around. In practice we already prefixed chain items with their block numbers, so compactions might not benefit too much in themselves.
By storing less data in the active key/value database, the same amount of cache and file descriptor allowance should provide better performance, since more state trie nodes can be kept in active memory.
By moving the ancient chain items into their own flat files, we can split the database into an active set requiring a fast SSD disk and an immutable set for which a slow HDD disk suffices. This should permit more people to run Ethereum nodes without huge SSDs (--datadir.ancient=/path/on/hdd).
By keeping ancient chain items in flat files, we can piggyback on the operating system's built in memory manager and file cacher, allowing Geth to potentially max out available memory without actually having to hog it. This should reduce the strain on Go's GC, reduce the amount of memory used by Geth, while at the same time increase overall system stability.

ligi · 2019-03-08T15:41:25Z

core/rawdb/accessors_chain.go

@@ -51,6 +54,24 @@ func DeleteCanonicalHash(db ethdb.Deleter, number uint64) {
 	}
 }

+// readAllHashes retrieves all the hashes assigned to blocks at a certain heights,


Suggested change

// readAllHashes retrieves all the hashes assigned to blocks at a certain heights,

// readAllHashes retrieves all the hashes assigned to blocks at certain heights,

ligi · 2019-03-08T15:43:02Z

core/rawdb/database.go

@@ -22,10 +22,44 @@ import (
 	"github.com/ethereum/go-ethereum/ethdb/memorydb"
 )

+// freezerdb is a databse wrapper that enabled freezer data retrievals.


Suggested change

// freezerdb is a databse wrapper that enabled freezer data retrievals.

// freezerdb is a databse wrapper that enables freezer data retrievals.

Matthalp-zz · 2019-03-08T15:34:54Z

core/rawdb/freezer.go

+	// freezerBlockGraduation is the number of confirmations a block must achieve
+	// before it becomes elligible for chain freezing. This must exceed any chain
+	// reorg depth, since the freezer also deletes all block siblings.
+	freezerBlockGraduation = 60000


Consider making this a variable that is passed into the freezer so that its value can correspond to eth/downloader/MaxForkAncestry in some way.

Matthalp-zz · 2019-03-08T15:41:44Z

core/rawdb/freezer.go

+	freezer := &freezer{
+		tables: make(map[string]*freezerTable),
+	}
+	for _, name := range []string{"hashes", "headers", "bodies", "receipts", "diffs"} {


Nit: IMHO "diffs" is an overloaded terminology. I would have used tds.

Matthalp-zz · 2019-03-08T15:43:48Z

core/rawdb/freezer.go

+//   reserving it for go-ethereum. This would also reduce the memory requirements
+//   of Geth, and thus also GC overhead.
+type freezer struct {
+	tables map[string]*freezerTable // Data tables for storing everything


I understand that using a map for the freezerTable adds some niceties like being able to iterate across the tables. However, I would consider making each freezer table an explicit member of the struct. It makes it immediately clear what the freezer is storing without having to look at newFreezer.

Matthalp-zz · 2019-03-08T15:45:27Z

core/rawdb/freezer.go

+		}
+		freezer.tables[name] = table
+	}
+	// Truncate all data tables to the same length


Just giving my two wei: a method called truncateAllTableToEqualLength would remove this comment and decrease the cognitive load to know what the remaining code is doing.

loving your usage of my two wei 👍

Matthalp-zz · 2019-03-08T15:49:24Z

core/rawdb/freezer.go

+	freezerBatchLimit = 30000
+)
+
+// freezer is an memory mapped append-only database to store immutable chain data


Can you point me to where the memory mapping is happening? My understanding is that os.File is not guaranteed to be memory-mapped as its left up to the OS.

Matthalp-zz · 2019-03-08T15:51:19Z

core/rawdb/freezer.go

+	return nil
+}
+
+// Ancient retrieves an ancient binary blob from the append-only immutable files.


It seems like the kind is around because each table is not coupled to the data it stores. This could be changed if it was placed within the ChainDB proposed in #19200 (as discussed previously).

Matthalp-zz · 2019-03-08T15:54:26Z

core/rawdb/freezer_table.go

+	"github.com/ethereum/go-ethereum/common"
+	"github.com/ethereum/go-ethereum/log"
+	"github.com/ethereum/go-ethereum/metrics"
+	"github.com/golang/snappy"


Has there been any benchmarking to determine if snappy compression is helpful (e.g. low overhead, high data savings)? Perhaps it is useful for bodies and receipts (and headers?), but may not do much for the other pieces of data.

Have you considered making this an optional parameter for each freezerTable instance?

From an earlier test

16 blocks.

as one binary file, uncompressed 332K ,compressed 254K.

As 16 indivudually compressed files, 257K

Afaik, we haven't tested that on headers

Though we cut off a lot of redundant data from the receipts, so I expect snappy to be less effective since the charts were plotted.

Storage-wise it seems worthwhile. Is the decompression overhead for reads something to consider?

Matthalp-zz · 2019-03-08T16:01:42Z

core/rawdb/freezer.go

+	for {
+		// Retrieve the freezing threshold. In theory we're interested only in full
+		// blocks post-sync, but that would keep the live database enormous during
+		// dast sync. By picking the fast block, we still get to deep freeze all the


s/dast/fast

Matthalp-zz · 2019-03-08T16:29:58Z

core/rawdb/freezer.go

+			}
+			// Inject all the components into the relevant data tables
+			if err := f.tables["hashes"].Append(f.frozen, hash[:]); err != nil {
+				log.Error("Failed to deep freeze hash", "number", f.frozen, "hash", hash, "err", err)


It seems like that if one of the subsequent freezerTable.Appends fails and then the import is retried (without reopening the freezer) then this will result in a panic since the entry will already have been written? Is that the failure model you want?

Matthalp-zz · 2019-03-08T16:31:43Z

core/rawdb/freezer.go

+		if n := len(ancients); n > 0 {
+			context = append(context, []interface{}{"hash", ancients[n-1]}...)
+		}
+		log.Info("Deep froze chain segment", context...)


This is another example of something I would have extracted out to not distract from the core functionality in this method.

core/rawdb/freezer_table.go

Matthalp-zz

Left a few comments. They really center around two points:
(1) Using atomic within code regions that are already guarded by locks
(2) File handle management. It seems like the freezer will have quite a few file handles open across the tables being used, which may compete with LevelDB's handle budget without being careful (which I know you are well aware of since you raised the point to me a while back)

Matthalp-zz · 2019-03-26T19:55:48Z

core/rawdb/freezer_table.go

+	errOutOfBounds = errors.New("out of bounds")
+)
+
+// indexEntry contains the number/id of the file that the data resides in, aswell as the


s/aswell/as well/

Matthalp-zz · 2019-03-26T19:57:56Z

core/rawdb/freezer_table.go

+	offset  uint32 // stored as uint32 ( 4 bytes)
+}
+
+const indexEntrySize = 6


Nit: move const to the top of the file

Whilst I generally agree with that, this number depends on the above type declaration. Moving it elsewhere would give the false sense that it can be modified freely.

Matthalp-zz · 2019-03-26T20:00:21Z

core/rawdb/freezer_table.go

+// offset within the file to the end of the data
+// In serialized form, the filenum is stored as uint16.
+type indexEntry struct {
+	filenum uint32 // stored as uint16 ( 2 bytes)


Is the index's storage footprint such a concern that you have to use uint[16|32] instead of uint64? Using uint64 seems simpler, consistent with the BlockChain API, and is the common word size in modern processors. It just feels like an additional piece of context to remember without having any substantial impact on a running system.

In-memory wise it doesn't matter tbh, it could even be a plain int. On-disk storage wise we know it won't ever exceed uint16, so no point in wasting more disk space on zeroes.

I hear you. Just figured I would point it out.

Matthalp-zz · 2019-03-26T20:06:44Z

core/rawdb/freezer_table.go

+
+// newTable opens a freezer table with default settings - 2G files and snappy compression
+func newTable(path string, name string, readMeter metrics.Meter, writeMeter metrics.Meter) (*freezerTable, error) {
+	return newCustomTable(path, name, readMeter, writeMeter, 2*1000*1000*1000, false)


Nit: Consider making this a constant. Also should this be 2 * 1024 * 1024 * 1024?

Yeah, I did wonder about this. I think I'd go with your suggestion.

We just need to take extra care not to ever overflow it, since many file systems + 32bit architectures are hard limited to that number.

Ah, scratch 32 bit arch, that should handle 4GB (or 3 in case of windows, lol).

Meh. ISO standards FTW

Matthalp-zz · 2019-03-26T20:09:15Z

core/rawdb/freezer_table.go

+	t.releaseFilesAfter(0, false)
+	// Open all except head in RDONLY
+	for i := uint32(0); i < t.headId; i++ {
+		if _, err = t.openFile(i, os.O_RDONLY); err != nil {


Are there any issues with file handler management here? The scenario I'm imagining is when multiple tables each have a bunch of files open (and LevelDB is also active).

The current historical chain is about 80GB in size, so that's +- 40 fds. These will probably be opened either way if someone syncs from us. I think 40 is safe, but if we grow we probably want to eventually add fd management. That seems an overcomplication at this point however.

Agree. I was just asking to serve as a sanity check.

Matthalp-zz · 2019-03-26T20:10:23Z

core/rawdb/freezer_table.go

+	t.lock.Lock()
+	defer t.lock.Unlock()
+	// If out item count is corrent, don't do anything
+	if atomic.LoadUint64(&t.items) <= items {


Is the atomic.LoadUint64 needed here if the write lock has already been acquired?

The locking mechanism was designed to allow reading and appending to be concurrently runable (since append is just pushes some data that readers are unaware of, and at the very end bumps the item count, making it available).

If append does not write lock, just uses atomic, then this write lock here doesn't protect the items counter. Append will only obtain a write lock if it needs to switch to a new file. That said, Append and Truncate may not really be safe to use concurrently, as they might race on what the final value of items should be.

If it were me I would be more conservative and lock more frequently -- especially for writes. I feel like any slow down would be negligible and it ensures the code is more safe. Just my two wei.

Matthalp-zz · 2019-03-26T20:12:43Z

core/rawdb/freezer_table.go

+}
+
+// releaseFilesAfter closes all open files with a higher number, and optionally also deletes the files
+func (t *freezerTable) releaseFilesAfter(num uint32, remove bool) {


Nit: I find the remove option to be misleading for a method about releasing. I would consider adding a second method that makes this more explicit. They can still use a common helper function.

Matthalp-zz · 2019-03-26T20:15:33Z

core/rawdb/freezer_table.go

+		return errClosed
+	}
+	// Ensure only the next item can be written, nothing else
+	if atomic.LoadUint64(&t.items) != item {


Is the atomic.LoadUint64 necessary if you've already acquired a read lock?

Yes, Append can run concurrently (since it doesn't change internals, just adds something to the end of the file and them atomically makrs it available).

karalabe · 2019-03-27T23:23:12Z

Nitpick to do: make data file numbers zero padded to 3 digits.

TL;DR

-rw-r--r-- 1 root root 1999998703 Mar 27 16:37 receipts.0.cdat
-rw-r--r-- 1 root root 1999997444 Mar 27 16:54 receipts.1.cdat
-rw-r--r-- 1 root root 1999999967 Mar 27 19:48 receipts.10.cdat
-rw-r--r-- 1 root root 1999999260 Mar 27 20:08 receipts.11.cdat
-rw-r--r-- 1 root root 1999994597 Mar 27 20:30 receipts.12.cdat
-rw-r--r-- 1 root root 1999993520 Mar 27 20:51 receipts.13.cdat
-rw-r--r-- 1 root root 1999992762 Mar 27 21:12 receipts.14.cdat
-rw-r--r-- 1 root root 1999990541 Mar 27 21:33 receipts.15.cdat
-rw-r--r-- 1 root root 1999988154 Mar 27 21:53 receipts.16.cdat
-rw-r--r-- 1 root root 1999995262 Mar 27 22:14 receipts.17.cdat
-rw-r--r-- 1 root root 1273323457 Mar 27 22:27 receipts.18.cdat
-rw-r--r-- 1 root root 1999989717 Mar 27 17:07 receipts.2.cdat
-rw-r--r-- 1 root root 1999995186 Mar 27 17:21 receipts.3.cdat
-rw-r--r-- 1 root root 1999999174 Mar 27 17:40 receipts.4.cdat
-rw-r--r-- 1 root root 1999990962 Mar 27 18:03 receipts.5.cdat
-rw-r--r-- 1 root root 1999995600 Mar 27 18:26 receipts.6.cdat
-rw-r--r-- 1 root root 1999997331 Mar 27 18:47 receipts.7.cdat
-rw-r--r-- 1 root root 1999993270 Mar 27 19:08 receipts.8.cdat
-rw-r--r-- 1 root root 1999998447 Mar 27 19:28 receipts.9.cdat

holiman · 2019-03-28T07:50:10Z

Here are some charts from the test with @rjl493456442 's changes that was run in fast-sync.

This is the master. It has a very intense download of blocks/headers/state for about 4 hours, then a long drawn-out state download which takes roughly another 8 hours.

For the experimental, it looks a bit different. It has an 'intense' download lasting six hours, followed by one hours of less intense download, and finishes the fast sync in a total of 7 hours.

So 7 hours instead of 12. I'm not quite sure why.

The malloc/gc overhead was more stable on experimental:

holiman · 2019-03-28T08:11:11Z

More stats on the data

root@mon07:/datadrive/geth/geth/chaindata# du -h .
88G    ./ancient
153G    .
root@mon06:/datadrive/geth/geth/chaindata# du -h .
154G    .

So the master levedb stores 154G of data, whereas the experimental has 65G in levedb and 88G in the ancient

Matthalp-zz · 2019-03-28T13:29:27Z

@holiman The results make sense to me. There is less pressure on LevelDB since it's now responsible for handling half the amount of data as before, which reduces its operation overhead (e.g. compaction). The LevelDB storage hierarchy is also organized by powers of 10, so going from ~150 GB to half that means that one less level is needed.

Matthalp-zz · 2019-03-28T13:31:56Z

Nonetheless it's probably worth running it a few times. I'm not sure the exact benchmark setup but if a different peer is being used for each sync then that could introduce some variation as well.

holiman · 2019-03-28T20:25:30Z

Yeah - we did just that. We swapped the machines, wiped the node-ids and ran again. It's still running, but this time it looks like the master will finish ahead of this PR. So that was probably a one-off thing - the big difference for statesync is the quality of your peers.

holiman · 2019-03-29T07:25:25Z

So on the second run, they were roughly equal -- this PR making a fast sync in ~10h, the master half an hour faster. Also, on the second run, the difference in mallocs and gc was not as pronounced:

OBS Since we switched machines, the colours are swapped in that ^ chart, this PR is the blue line.

holiman · 2019-04-24T09:50:10Z

cmd/geth/chaincmd.go

@@ -367,9 +367,12 @@ func exportPreimages(ctx *cli.Context) error {

 func copyDb(ctx *cli.Context) error {
 	// Ensure we have a source chain directory to copy
-	if len(ctx.Args()) != 1 {
+	if len(ctx.Args()) < 1 {
 		utils.Fatalf("Source chaindata directory path argument missing")


Suggested change

utils.Fatalf("Source chaindata directory path argument missing")

utils.Fatalf("Source directory paths missing (chaindata and ancient)")

holiman · 2019-04-24T09:51:05Z

cmd/geth/chaincmd.go

 		utils.Fatalf("Source chaindata directory path argument missing")
 	}
+	if len(ctx.Args()) < 2 {


Don't we by default put ancient inside the chain directory? Shouldn't we default to that here too?

By default yes, but if we want to copydb from a database where the ancient is outside?

holiman · 2019-04-24T09:57:10Z

cmd/utils/flags.go

@@ -1569,7 +1576,7 @@ func MakeChainDatabase(ctx *cli.Context, stack *node.Node) ethdb.Database {
 	if ctx.GlobalString(SyncModeFlag.Name) == "light" {
 		name = "lightchaindata"
 	}
-	chainDb, err := stack.OpenDatabase(name, cache, handles, "")
+	chainDb, err := stack.OpenDatabaseWithFreezer(name, cache, handles, "", "")


Shouldn't you pass the cfg.DatabaseFreezer in here somehow?

rjl493456442 · 2019-04-26T03:34:46Z

core/blockchain.go

+				return i, fmt.Errorf("containing header #%d [%x…] unknown", block.Number(), block.Hash().Bytes()[:4])
+			}
+			// Compute all the non-consensus fields of the receipts
+			if err := receiptChain[i].DeriveFields(bc.chainConfig, block.Hash(), block.NumberU64(), block.Transactions()); err != nil {


We don't need to derive the receipt fields here since they will be discarded during the RLP encoding anyway.

rjl493456442 · 2019-04-26T03:35:25Z

core/blockchain.go

+				continue
+			}
+			// Compute all the non-consensus fields of the receipts
+			if err := receiptChain[i].DeriveFields(bc.chainConfig, block.Hash(), block.NumberU64(), block.Transactions()); err != nil {


We don't need to derive the receipt fields here since they will be discarded during the RLP encoding anyway.

rjl493456442 · 2019-04-26T04:21:25Z

core/rawdb/freezer_table.go

+		t.lock.RLock()
+	}
+
+	defer t.lock.RUnlock()


The lock is used to protect map[uint32]*os.File as well as head and index file descriptor. Since os.File.Write is not thread safe, should we use Lock here instead of RLock?

I think maybe we should add a freezer.md where we explain things like the concurrency assumptions and the data layout. To answer the question, I think No; we do not cater for multiple callers of Append, tha aim is to prevent concurrent calling of Append/Truncate.

rjl493456442 · 2019-04-26T04:22:13Z

eth/downloader/downloader.go

+		// feed us valid blocks until head. All of these blocks might be written into
+		// the ancient store, the safe region for freezer is not enough.
+		if d.checkpoint != 0 && d.checkpoint > MaxForkAncestry+1 {
+			d.ancientLimit = height - MaxForkAncestry - 1


d.ancientLimit = height - MaxForkAncestry - 1 -> d.ancientLimit = d.checkpoint - MaxForkAncestry - 1

holiman · 2019-04-26T10:04:27Z

Data taken from mon07, comparing the effectiveness of snappy versus encoding plan.

Sorry for the crappyness of the labeling, but aside from that, in order of size:

bodies: 76 e9 versus 55 e9 (snappy wins)
receipts 61 e9 versus 22 e9 (snappy wins)
hashes 260 e6 versus 240 e6 (plain would be smaller)
difficulty 91 e6versus 75 e6 (plain would be smaller).
headers 4.1 e9 versus 3.2 e9 (snappy wins)

Also, the index for each type is takes rougly 44MB on disk each.

Matthalp-zz · 2019-04-30T13:52:55Z

core/blockchain.go

@@ -218,6 +219,39 @@ func NewBlockChain(db ethdb.Database, cacheConfig *CacheConfig, chainConfig *par
 	if err := bc.loadLastState(); err != nil {
 		return nil, err
 	}
+	if frozen, err := bc.db.Ancients(); err == nil && frozen >= 1 {


Nit: frozen > 0 is slightly clearer.

Matthalp-zz · 2019-04-30T14:03:53Z

core/blockchain.go

+			rawdb.DeleteBody(db, hash, num)
+			rawdb.DeleteReceipts(db, hash, num)
+		}
+		// Todo(rjl493456442) txlookup, bloombits, etc


Nit: s/Todo/TODO (and possibly end in a ':'):

// TODO(rjl493456442): txlookup, bloombits, etc

Matthalp-zz · 2019-04-30T14:08:11Z

core/blockchain.go

+	//
+	// Notably, it can happen that system crashes without truncating the ancient data
+	// but the head indicator has been updated in the active store. Regarding this issue,
+	// system will self recovery by truncating the extra data during the setup phase.


s/recovery/recover/

Matthalp-zz · 2019-04-30T15:13:07Z

core/rawdb/freezer.go

+
+		case *number-freezerBlockGraduation <= f.frozen:
+			log.Debug("Ancient blocks frozen already", "number", *number, "hash", hash, "frozen", f.frozen)
+			time.Sleep(freezerRecheckInterval)


Should this be an error?

* freezer: implement split files for data * freezer: add tests * freezer: close old head-file when opening next * freezer: fix truncation * freezer: more testing around close/open * rawdb/freezer: address review concerns * freezer: fix minor review concerns * freezer: fix remaining concerns + testcases around truncation * freezer: docs * freezer: implement multithreading * core/rawdb: fix freezer nitpicks + change offsets to uint32 * freezer: preopen files, simplify lock constructs * freezer: delete files during truncation

* all: freezer style syncing core, eth, les, light: clean up freezer relative APIs core, eth, les, trie, ethdb, light: clean a bit core, eth, les, light: add unit tests core, light: rewrite setHead function core, eth: fix downloader unit tests core: add receipt chain insertion test core: use constant instead of hardcoding table name core: fix rollback core: fix setHead core/rawdb: remove canonical block first and then iterate side chain core/rawdb, ethdb: add hasAncient interface eth/downloader: calculate ancient limit via cht first core, eth, ethdb: lots of fixes * eth/downloader: print ancient disable log only for fast sync

* freezer: disable compression on hashes and difficulties * core/rawdb: address review concerns * core/rawdb: address review concerns

* core, eth: some fixes for freezer * vendor, core/rawdb, cmd/geth: add db inspector * core, cmd/utils: check ancient store path forceily * cmd/geth, common, core/rawdb: a few fixes * cmd/geth: support windows file rename and fix rename error * core: support ancient plugin * core, cmd: streaming file copy * cmd, consensus, core, tests: keep genesis in leveldb * core: write txlookup during ancient init * core: bump database version

karalabe · 2019-05-16T13:45:22Z

@matthalp This PR did get a bit more complex than I anticipated, though that's because we decided to split the data files into 2GB chunks. Without that the code was really really simple.

That said, the operations we are doing are relatively straightforward. The complexity comes mostly from optimizations. With our current code we have a very nice control over how much resources we waste (0 memory, 10-20 file descriptors vs. caches + files for leveldb), and we control how much computational overhead we have (0 vs. compaction for leveldb).

Yes, leveldb might be a tad nicer, but by rolling our own files we can take advantage of all the properties of the data we are storing and optimize for our specific use case.

holiman · 2019-05-22T09:37:34Z

core/blockchain.go

+
+				// Always keep genesis block in active database.
+				if b.NumberU64() != 0 {
+					deleted = append(deleted, b)


This can cause a huge spike in memory, our monitoring node spiked at nearly 40G memory shoving four million blocks from leveldb to ancients

317787106 · 2023-04-24T07:01:32Z

If I want to run an ethereum node from genesis block with --syncmode "snap", but don't want to freeze levledb, is there a switch to disable this feature ? Or can I edit some codes to disable it ?

holiman · 2023-04-24T07:56:08Z

No, there is no such switch. It should be fairly trivial to disable the live conversion from leveldb to freezer, but there's also the initial download-to-freezer which is maybe a bit less trivial.
Why would you want to disable it though?

317787106 · 2023-04-24T08:08:23Z

@holiman, I want to konw the space-saving rate about the conversion from leveldb to freezer. If i disable the freezer firstly and then enable that feature, we can calculate the rate of space. Is there any experimental data?

karalabe requested a review from holiman as a code owner March 8, 2019 14:08

This was referenced Mar 8, 2019

implement chain freezer, clean up database accesses #17814

Closed

ethdb: introduce vectordb for persistent sequential data storage #19229

Closed

ligi requested changes Mar 8, 2019

View reviewed changes

Matthalp-zz reviewed Mar 8, 2019

View reviewed changes

holiman reviewed Mar 13, 2019

View reviewed changes

core/rawdb/freezer_table.go Outdated Show resolved Hide resolved

karalabe force-pushed the freezer-2 branch from bc733f7 to 1aa8834 Compare March 26, 2019 11:35

Matthalp-zz reviewed Mar 26, 2019

View reviewed changes

karalabe force-pushed the freezer-2 branch 2 times, most recently from bd9760b to 5810ffe Compare March 27, 2019 12:02

karalabe force-pushed the freezer-2 branch from 5810ffe to ffbf258 Compare April 24, 2019 09:29

holiman reviewed Apr 24, 2019

View reviewed changes

karalabe requested a review from zsfelfoldi as a code owner April 25, 2019 14:59

karalabe force-pushed the freezer-2 branch from 6828915 to ebca031 Compare April 25, 2019 15:04

karalabe added this to the 1.9.0 milestone Apr 25, 2019

rjl493456442 reviewed Apr 26, 2019

View reviewed changes

holiman mentioned this pull request Apr 26, 2019

freezer: disable compression on hashes and difficulties karalabe/go-ethereum#14

Merged

Matthalp-zz reviewed Apr 30, 2019

View reviewed changes

adamschmideg added the status:triage label May 7, 2019

karalabe removed the status:triage label May 14, 2019

karalabe force-pushed the freezer-2 branch 2 times, most recently from a971ccd to c3bb1b3 Compare May 15, 2019 11:40

karalabe and others added 8 commits May 16, 2019 10:39

cmd, core, eth, les, node: chain freezer on top of db rework

006c21e

core/rawdb: add file lock for freezer

b6cac42

core/rawdb: support starting offset for future deletion

331de17

freezer: disable compression on hashes and difficulties (#14)

42c746d

* freezer: disable compression on hashes and difficulties * core/rawdb: address review concerns * core/rawdb: address review concerns

cosensus, core, eth, params, trie: fixes + clique history cap

536b3b4

karalabe force-pushed the freezer-2 branch from c3bb1b3 to 536b3b4 Compare May 16, 2019 07:41

adamschmideg added the status:triage label May 16, 2019

karalabe removed the status:triage label May 16, 2019

karalabe force-pushed the freezer-2 branch from 79c0a43 to 996e2de Compare May 16, 2019 12:34

cmd/geth, core/rawdb: seamless freezer consistency, friendly removedb

9eba3a9

karalabe force-pushed the freezer-2 branch from b50a100 to 9eba3a9 Compare May 16, 2019 14:02

karalabe merged commit f5d89cd into ethereum:master May 16, 2019

holiman reviewed May 22, 2019

View reviewed changes

ehnuje mentioned this pull request Jan 7, 2021

database: added Delete and Replay to Batch interface klaytn/klaytn#841

Merged

9 tasks

gzliudan mentioned this pull request Dec 19, 2024

upgrade package common XinFinOrg/XDPoSChain#768

Merged

19 tasks

gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Dec 19, 2024

common: improve functions of StorageSize (ethereum#19244)

6bf4046

gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Dec 21, 2024

common: improve functions of StorageSize (ethereum#19244)

4c3b77b

gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Dec 26, 2024

common: improve functions of StorageSize (ethereum#19244)

a3c189e

gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Dec 28, 2024

common: improve functions of StorageSize (ethereum#19244)

a747a98

gzliudan mentioned this pull request Jan 23, 2025

reduce chain database size XinFinOrg/XDPoSChain#570

Open

	// readAllHashes retrieves all the hashes assigned to blocks at a certain heights,
	// readAllHashes retrieves all the hashes assigned to blocks at certain heights,

	// freezerdb is a databse wrapper that enabled freezer data retrievals.
	// freezerdb is a databse wrapper that enables freezer data retrievals.

	utils.Fatalf("Source chaindata directory path argument missing")
	utils.Fatalf("Source directory paths missing (chaindata and ancient)")

cmd, core, eth, les, node: chain freezer on top of db rework #19244

cmd, core, eth, les, node: chain freezer on top of db rework #19244

Conversation

karalabe commented Mar 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Matthalp-zz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karalabe commented Mar 27, 2019

holiman commented Mar 28, 2019

holiman commented Mar 28, 2019

Matthalp-zz commented Mar 28, 2019 • edited Loading

Matthalp-zz commented Mar 28, 2019

holiman commented Mar 28, 2019

holiman commented Mar 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holiman commented Apr 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karalabe commented May 16, 2019 • edited Loading

Choose a reason for hiding this comment

317787106 commented Apr 24, 2023 • edited Loading

holiman commented Apr 24, 2023

317787106 commented Apr 24, 2023

Matthalp-zz commented Mar 28, 2019 •

edited

Loading

karalabe commented May 16, 2019 •

edited

Loading

317787106 commented Apr 24, 2023 •

edited

Loading