Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add reconstruction from systematic chunks #36

Merged
merged 4 commits into from
Dec 20, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions reed-solomon-novelpoly/src/errors.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ pub enum Error {

#[error("Shards do have inconsistent lengths: first = {first}, other = {other})")]
InconsistentShardLengths { first: usize, other: usize },

#[error("Shard is empty")]
EmptyShard,
}

/// Result alias to simplify API.
Expand Down
54 changes: 53 additions & 1 deletion reed-solomon-novelpoly/src/novel_poly_basis/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,9 @@ impl ReedSolomon {
Ok(shards)
}

/// each shard contains one symbol of one run of erasure coding
/// Reconstruct from chunks.
///
/// The result may be padded with zeros. Truncate the output to the expected byte length.
pub fn reconstruct<S: Shard>(&self, received_shards: Vec<Option<S>>) -> Result<Vec<u8>> {
let gap = self.n.saturating_sub(received_shards.len());

Expand Down Expand Up @@ -190,6 +192,10 @@ impl ReedSolomon {
})
.expect("Existential shard count is at least k shards. qed");

if first_shard_len == 0 {
return Err(Error::EmptyShard);
}

// make sure all shards have the same length as the first one
if let Some(other_shard_len) = received_shards[(first_shard_idx + 1)..].iter().find_map(|shard| {
shard.as_ref().and_then(|shard| {
Expand Down Expand Up @@ -231,6 +237,52 @@ impl ReedSolomon {

Ok(acc)
}

/// Reconstruct from the set of systematic chunks.
/// Systematic chunks are the first `k` chunks, which contain the initial data.
///
/// Provide a vector containing chunk data. If too few chunks are provided, recovery is not
/// possible.
/// The result may be padded with zeros. Truncate the output to the expected byte length.
pub fn reconstruct_from_systematic<S: Shard>(&self, chunks: Vec<S>) -> Result<Vec<u8>> {
let Some(first_shard) = chunks.first() else {
return Err(Error::NeedMoreShards { have: 0, min: self.k, all: self.n });
};
if chunks.len() < self.k {
return Err(Error::NeedMoreShards { have: chunks.len(), min: self.k, all: self.n });
}

let shard_len = AsRef::<[[u8; 2]]>::as_ref(first_shard).len();

if shard_len == 0 {
return Err(Error::EmptyShard);
}

if let Some(length) = chunks.iter().find_map(|c| {
let length = AsRef::<[[u8; 2]]>::as_ref(c).len();
if length != shard_len {
Some(length)
} else {
None
}
}) {
return Err(Error::InconsistentShardLengths { first: shard_len, other: length });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized this is being checked 3 times: here, in erasure-coding crate and in availability-recovery..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right, I'll move all checks here. I only see it being checked here and in erasure-coding crate. Where is it being checked in availability-recovery?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll remove checks from polkadot once this is merged and released. The only check that needs to remain in polkadot is for the even length of shards. This is needed because WrappedShard will panic if the length is not even and this function expects a WrappedShard.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is it being checked in availability-recovery?

I meant in your systematic recovery PR, but it will probably use this function instead, so nevermind.

The only check that needs to remain in polkadot is for the even length of shards. This is needed because WrappedShard will panic if the length is not even and this function expects a WrappedShard.

I see, thanks for the clarification.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant in your systematic recovery PR, but it will probably use this function instead, so nevermind.

Ah, yes, I'll switch my PR to use this function

}

let mut systematic_bytes = Vec::with_capacity(shard_len * 2 * self.k);

for i in 0..shard_len {
for chunk in chunks.iter().take(self.k) {
// No need to check for index out of bounds because i goes up to shard_len and
// we return an error for non uniform chunks.
let chunk = AsRef::<[[u8; 2]]>::as_ref(chunk)[i];
systematic_bytes.push(chunk[0]);
systematic_bytes.push(chunk[1]);
}
}

Ok(systematic_bytes)
}
}

#[cfg(test)]
Expand Down
17 changes: 17 additions & 0 deletions reed-solomon-novelpoly/src/novel_poly_basis/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,23 @@ impl Arbitrary for ArbitraryData {
}
}

#[test]
fn round_trip_systematic_quickcheck() {
fn property(available_data: ArbitraryData, n_validators: u16) {
let n_validators = n_validators.max(2);
let rs = CodeParams::derive_parameters(n_validators as usize, (n_validators as usize - 1) / 3 + 1)
.unwrap()
.make_encoder();
let kpow2 = rs.k;
let chunks = rs.encode::<WrappedShard>(&available_data.0).unwrap();
let mut res = rs.reconstruct_from_systematic(chunks.into_iter().take(kpow2).collect()).unwrap();
res.truncate(available_data.0.len());
assert_eq!(res, available_data.0);
}

QuickCheck::new().quickcheck(property as fn(ArbitraryData, u16))
}

#[test]
fn round_trip_quickcheck() {
fn property(available_data: ArbitraryData, n_validators: u16) {
Expand Down
Loading