Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implements Bulk processing dictionary API #114

Merged
merged 2 commits into from
Apr 5, 2022

Conversation

Barbayar
Copy link
Contributor

@Barbayar Barbayar commented Apr 2, 2022

Description

Implements Bulk processing dictionary API.

Bulk processing dictionary API

It allows you to compress files with a pre-trained dictionary. Because, the compress file doesn't contain a dictionary, you can save some bytes. The save is significant if your files are smaller and similar.

P.S. With Streaming API, we can compress/decompress files with a pre-trained dictionary. But, in order to compress/decompress multiple files, we need to re-create writer/reader every time we compress/decompress. It's very inefficient, because it needs to load the dictionary every time.

Usage

p, err := NewBulkProcessor(dict, BestSpeed)
if err != nil {
	fmt.Fatalf("failed to create a BulkProcessor")
}

for payload := range in {
	compressed, err := p.Compress(nil, payload)
	if err != nil {
		fmt.Fatalf("failed to compress")
	}

	uncompressed, err := p.Decompress(nil, compressed)
	if err != nil {
		fmt.Fatalf("failed to decompress")
	}
}

@Barbayar Barbayar requested a review from Viq111 April 2, 2022 23:00
Copy link
Collaborator

@Viq111 Viq111 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! The structure looks straight forward, I've added a couple of comments to make it more clear and robust for an external audience

zstd_bulk.go Outdated Show resolved Hide resolved
zstd_bulk.go Outdated Show resolved Hide resolved
zstd_bulk.go Show resolved Hide resolved
zstd_bulk.go Outdated Show resolved Hide resolved
zstd_bulk.go Outdated Show resolved Hide resolved
zstd_bullk_test.go Outdated Show resolved Hide resolved
zstd_bullk_test.go Outdated Show resolved Hide resolved
zstd_bullk_test.go Outdated Show resolved Hide resolved
zstd_bullk_test.go Outdated Show resolved Hide resolved
zstd_bullk_test.go Show resolved Hide resolved
@Barbayar Barbayar requested a review from Viq111 April 4, 2022 23:41
Copy link
Collaborator

@Viq111 Viq111 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! this looks great!

@Barbayar Barbayar merged commit 393c3c1 into 1.x Apr 5, 2022
@Barbayar Barbayar deleted the barbayar.dashzeveg/bulk-processing-api branch April 5, 2022 20:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants