DA Compression Witness Generation and Assignments (#1232)

* Remove deprecate tests

* Remove zstd encoding of literals

* Remove raw/rle blocks

* Remove block raw/rle scenarios

* Remove lstream tag

* Remove raw/rle bytes processing

* Literals witness row assignment

* Remove huffman code component from witness

* Add witness rows to sequence section

* Add debug flags

* Add sequence instruction table witness

* Finish recovering original input. sequence decoding done

* Add witness rows for sequence FSE tables

* Add witness rows for sequence header

* Remove debug flags

* fmt

* Add debug flags

* Add debug flags

* Add debug flags

* Add unit tester

* Update Fse construction

* Correct fse construction

* Correct tag length

* Temporarily recover const

* Assign literals header rows

* Assign columns

* Resolve merge compile errors

* Modify tag config assignment

* Correct block config from witgen

* Assign additional seq bitstream fields

* Correct tags

* configure gadgtes

* Add more data fields onto fse

* Correct fse assignment

* fmt

* Remove debug flags

* fmt

* Fix witness assignment error

* fmt

* Increase fse fixed capacity

* Correct bitstream column assignmnet

* Correct assignment for bitstring table

* fmt

* Correct comment

* Remove FseSymbol

* Rename FSE section

* Recover tests

* Correct block header assignment

* Correct fse decoder assignment

* fmt

* Remove constants

* Add debug flags

* Correct offset increment

* Assign padding

* Isolate gates

* Remove gates

* Remove gates

* Remove gates

* Remove gates

* Adjust gates

* Recover gates

* fix: q_enable fixed column to avoid active gates on unusable rows

* wip dbg: non-padded rows except first row

* fix: tag transition has been fixed

* fix(witgen): tag is_change=true

* wip(dbg:gate): continue same tag

* fix: compilation (u8 table load) and fhd gate works

* fix: fcs OK || unusable_rows=14

* test: fcs is OK

* fix(gate): tag=BlockHeader OK

* test(lookup): tag=BlockHeader Block_Size OK

* test(gate): processing block content OK

* fix(gate+lookup): literals header and literals header table

* fix(gate): tag=RawBytes OK

* fix(gate): tag=SequencesHeader (header decoder fixed order of bits)

* fix(fse): sorted state table (partially)

* wip(fse sorted states): gates active on unusable rows?

* fix(fse): fse table and sorted table OK

* witgen: bitstring table OK

* Correct skipped bitstream rows

* Recover gadgets

* Recover gadgets

* Recover gadgets

* Recover gates

* Recover gates

* Recover component

* Recover constraint

* Recover constraints

* Recover gates

* Recover gates

* Recover constraints

* Recover constraints

* Recover constraints

* Recover constraints

* fix(lookup): var-bit-packing and other rows of tag=FseCode

* restore gate

* fix(lookup): bitstream table lookup (start + end)

* Correct sequence data init

* Correct fse in sequence data decoding

* Recover gates

* Recover constraints

* fix(lookup): interleaved order lookup OK

* fix(gate): tag=Null OK

* Correct states, symbols, values in bitstream decoding witness

* Adjust gates

* Adjust fixed table

* Remove debug flag

* fix(fse_decoder): is_mlt and is_mot expr computation

* fse table lookup OK

* Correct seq_idx

* Recover constraints

* Recover gate

* fix: constraints updated

* chore: degree-overflow fix

* Remove is_next_null condition

* Add next_nb

* Correct tail end bit read idx

* fix: update various cases of nil

* Correct nil row index

* Correct bitstream decoding idx

* `SequenceInstructionTable` and `SequenceExecutionConfig` (#1259)

* rebase to upstream

Signed-off-by: noelwei <fan@scroll.io>

* wip of addrtable

Signed-off-by: noelwei <fan@scroll.io>

* all gates

Signed-off-by: noelwei <fan@scroll.io>

* change the lookup purpose

Signed-off-by: noelwei <fan@scroll.io>

* fix according to reviews

* purge unused seqvaluetable

Signed-off-by: noelwei <fan@scroll.io>

* change zero testing to corresponding gadget
complete assignment

Signed-off-by: noelwei <fan@scroll.io>

* trivial fixing

Signed-off-by: noelwei <fan@scroll.io>

* purge the duplicated works

Signed-off-by: noelwei <fan@scroll.io>

* unit test (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* seq exec circuit (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* seq exec circuit (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* seq exec circuit (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* output region: gates and lookups (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* pass unittest for seqinst table

Signed-off-by: noelwei <fan@scroll.io>

* seq exec: complete the seq num lookup

Signed-off-by: noelwei <fan@scroll.io>

* add seq exec info in witgen

Signed-off-by: noelwei <fan@scroll.io>

* assign and unit tests (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* refactor for better assignment

Signed-off-by: noelwei <fan@scroll.io>

* assignments and unit tests (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* induce debug utilities in AddressRow

Signed-off-by: noelwei <fan@scroll.io>

* pass first unit test (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* chore: integrate seq-inst-table and seq-exec-config into decoder-config

* refactor to low degree

Signed-off-by: noelwei <fan@scroll.io>

* more unittest for seq exec (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* pass unit tests

Signed-off-by: noelwei <fan@scroll.io>

* add assign entry

Signed-off-by: noelwei <fan@scroll.io>

* update some witgens, pass decoder's unit test

Signed-off-by: noelwei <fan@scroll.io>

* integrate seq exec into decoder (WIP)

Signed-off-by: noelwei <fan@scroll.io>

* temporary disable 3 lookups and unit test pass for the rest

Signed-off-by: noelwei <fan@scroll.io>

* all of the unit test passed

Signed-off-by: noelwei <fan@scroll.io>

* trivial updates: head condition in seq exec and exported cells

Signed-off-by: noelwei <fan@scroll.io>

* clear the warnings

Signed-off-by: noelwei <fan@scroll.io>

* chore: fmt

---------

Signed-off-by: noelwei <fan@scroll.io>
Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me>

* chore: most clippy issues/warnings resolved

* wip(dbg): decode encoded batch data

* fix: literals header's size format bits

* DA-Compression (Missing Pieces for single encoded batch) (#1263)

* Fix repeated match byte slice:

* Add batch witgen test

* Adjust test

* fix: handle predefined table appropriately in table reconstruction

* Skip predefined fse tables:

* Add copy constraints and assigned exports

* wip dbg: expected vs got

* fix: rlc_acc and rlc, handle is_reverse approprately

* fix decoder rlc (#1266)

Signed-off-by: noelwei <fan@scroll.io>
Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me>

* fix: constraints/lookups OK for batch0000

* fix: literal header assignment fix, batch127 failure

* Add tag transition fix for fse

* fix: fse table handling of prob=-1 and state_idx/sym_count_acc

* fix: fse -> seq data order

* fix: handle first real symbol in prob=-1 table cases

---------

Signed-off-by: noelwei <fan@scroll.io>
Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me>
Co-authored-by: Ho <noel.wei@gmail.com>

* commit from @noel2004: f181535

* Dbg (large block) (#1270)

* data and test

* wip dbg: offset > currently decoded (overflow subtraction)

* chore: bitwise op table generic over op and range of lhs/rhs

* wip dbg: more blobs (witgen fail)

* fix: repeat offset usage (ll ==0) and dont include skipped states

* Account for nil row in fse

* Fix fse nil row byte skip

* Correct is_update_state for nil row in sequence data

* fix (fse state increasing, but may not be in u8) and fmt

* chore: remove println

---------

Co-authored-by: Ray Gao <qg2153@columbia.edu>

* Refactor/Cleanup Witgen code (#1271)

* Remove debug flags

* Remove aux data

* fmt and add comments

* Recover test

* Remove import

* Recover test

* Recover test

* Assign padding

* Remove unused variables

* Remove initial assignment

* Add comments

* fmt

* fmt

* fmt

* fmt

* Organize comments:

* tag_value unused | refactor max_len

* more refactor and clippy

* fix: handle multi-block witgen

---------

Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me>

* chore: refactor zstd config into init fn

* fix(tag=sequences_data): columns can change iff conditions are met

* Support multi-block (#1273)

* add test

* detour zstd dep

Signed-off-by: noelwei <fan@scroll.io>

* tbt: test for multi-blob

* update zstd dep

Signed-off-by: noelwei <fan@scroll.io>

* chore: remove unused patch

* update zstd (scroll repo)

Signed-off-by: noelwei <fan@scroll.io>

* Correct multi-block back reference target bytes (#1275)

Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me>

* minor fix in assignment (block_info and seq_info)

* dbg: fix some issues

* dbg: fix bitstring table lookup for byte_idx delta

* fix: repeated offsets are carried forward into next block

* wip test: large-multi-block

* update zstd, set windowlog

Signed-off-by: noelwei <fan@scroll.io>

* fix: is_init continue if nil=1

* refactor bitstring table to separate out bitstrings of different len

---------

Signed-off-by: noelwei <fan@scroll.io>
Co-authored-by: noelwei <fan@scroll.io>
Co-authored-by: Ray Gao <qg2153@columbia.edu>

* fix: handle padding in blob-data and batch-data configs (integration tests OK)

---------

Signed-off-by: noelwei <fan@scroll.io>
Co-authored-by: Rohit Narurkar <rohit.narurkar@proton.me>
Co-authored-by: Ho <noel.wei@gmail.com>
Co-authored-by: noelwei <fan@scroll.io>

aggregator/Cargo.toml

@@ -36,7 +36,7 @@ revm-primitives = "3.1.0"
 
 # da-compression
 bitstream-io = "2.2.0"
-zstd = { version = "0.13", features = ["experimental"] }
+zstd = { git = "https://github.com/scroll-tech/zstd-rs", branch = "hack/mul-block", features = ["experimental"]}
 
 [dev-dependencies]
 

aggregator/src/aggregation.rs

@@ -18,7 +18,7 @@ pub(crate) use barycentric::{
 };
 pub(crate) use batch_data::BatchDataConfig;
 pub(crate) use blob_data::BlobDataConfig;
-pub(crate) use decoder::{DecoderConfig, DecoderConfigArgs};
+pub(crate) use decoder::{witgen, DecoderConfig, DecoderConfigArgs};
 pub(crate) use rlc::RlcConfig;
 
 pub use circuit::AggregationCircuit;

aggregator/src/aggregation/batch_data.rs

@@ -264,7 +264,12 @@ impl BatchDataConfig {
                     * is_padding_curr.expr()
                     * (boundary_count_curr - boundary_count_prev),
                 // bytes rlc is accumulated appropriately
-                is_data.expr() * (bytes_rlc_prev * r + byte - bytes_rlc_curr),
+                is_data.expr()
+                    * is_padding_curr.expr()
+                    * (bytes_rlc_curr.expr() - bytes_rlc_prev.expr()),
+                is_data.expr()
+                    * (1.expr() - is_padding_curr.expr())
+                    * (bytes_rlc_prev * r + byte - bytes_rlc_curr),
             ]
         });
 
@@ -425,8 +430,10 @@ impl BatchDataConfig {
         let mut count = 0u64;
         let mut bytes_rlc_acc = Value::known(Fr::zero());
         for (i, row) in rows.iter().enumerate() {
-            bytes_rlc_acc = bytes_rlc_acc * challenge_value.keccak_input()
-                + Value::known(Fr::from(row.byte as u64));
+            if !row.is_padding {
+                bytes_rlc_acc = bytes_rlc_acc * challenge_value.keccak_input()
+                    + Value::known(Fr::from(row.byte as u64));
+            }
             let byte = region.assign_advice(
                 || "byte",
                 self.byte,

aggregator/src/aggregation/blob_data.rs

@@ -1,18 +1,19 @@
 use std::io::Write;
 
+use gadgets::util::Expr;
 use halo2_ecc::bigint::CRTInteger;
 use halo2_proofs::{
     circuit::{AssignedCell, Layouter, Region, Value},
     halo2curves::bn256::Fr,
-    plonk::{Advice, Column, ConstraintSystem, Error},
+    plonk::{Advice, Column, ConstraintSystem, Error, Expression, SecondPhase, Selector},
     poly::Rotation,
 };
 use itertools::Itertools;
 use zkevm_circuits::{table::U8Table, util::Challenges};
 
 use crate::{
-    aggregation::rlc::POWS_OF_256,
-    blob::{init_zstd_encoder, BatchData, BLOB_WIDTH, N_BLOB_BYTES, N_DATA_BYTES_PER_COEFFICIENT},
+    aggregation::{decoder::witgen::init_zstd_encoder, rlc::POWS_OF_256},
+    blob::{BatchData, BLOB_WIDTH, N_BLOB_BYTES, N_DATA_BYTES_PER_COEFFICIENT},
     RlcConfig,
 };
 
@@ -28,27 +29,92 @@ use crate::{
 /// fact the zstd encoded form of the raw batch data represented in BatchDataConfig.
 #[derive(Clone, Debug)]
 pub struct BlobDataConfig {
+    /// Selector to mark the first row in the layout, enabled at offset=0.
+    q_first: Selector,
+    /// Whether the row is enabled or not. We need exactly N_BLOB_BYTES rows, enabled from offset=1
+    /// to offset=N_BLOB_BYTES.
+    q_enabled: Selector,
     /// The byte value at this row.
     byte: Column<Advice>,
+    /// Whether or not this is a padded row. This can be the case if not all bytes in the blob
+    /// (4096 * 31) could be filled. Padded bytes must be 0 and bytes_rlc must continue while in
+    /// the padded region.
+    is_padding: Column<Advice>,
+    /// running RLC of bytes seen so far. It remains unchanged once padded territory starts.
+    bytes_rlc: Column<Advice>,
 }
 
 pub struct AssignedBlobDataExport {
     pub bytes_rlc: AssignedCell<Fr, Fr>,
 }
 
 impl BlobDataConfig {
-    pub fn configure(meta: &mut ConstraintSystem<Fr>, u8_table: U8Table) -> Self {
+    pub fn configure(
+        meta: &mut ConstraintSystem<Fr>,
+        challenges: &Challenges<Expression<Fr>>,
+        u8_table: U8Table,
+    ) -> Self {
         let config = Self {
+            q_enabled: meta.selector(),
+            q_first: meta.complex_selector(),
             byte: meta.advice_column(),
+            is_padding: meta.advice_column(),
+            bytes_rlc: meta.advice_column_in(SecondPhase),
         };
 
         meta.enable_equality(config.byte);
+        meta.enable_equality(config.bytes_rlc);
 
         meta.lookup("BlobDataConfig (0 < byte < 256)", |meta| {
             let byte_value = meta.query_advice(config.byte, Rotation::cur());
             vec![(byte_value, u8_table.into())]
         });
 
+        meta.create_gate("BlobDataConfig: first row", |meta| {
+            let is_first = meta.query_selector(config.q_first);
+
+            let byte = meta.query_advice(config.byte, Rotation::cur());
+            let bytes_rlc = meta.query_advice(config.bytes_rlc, Rotation::cur());
+            let is_padding_next = meta.query_advice(config.is_padding, Rotation::next());
+
+            vec![
+                is_first.expr() * byte,
+                is_first.expr() * bytes_rlc,
+                is_first.expr() * is_padding_next,
+            ]
+        });
+
+        meta.create_gate("BlobDataConfig: main gate", |meta| {
+            let is_enabled = meta.query_selector(config.q_enabled);
+
+            let is_padding_curr = meta.query_advice(config.is_padding, Rotation::cur());
+            let is_padding_prev = meta.query_advice(config.is_padding, Rotation::prev());
+            let delta = is_padding_curr.expr() - is_padding_prev.expr();
+
+            let byte = meta.query_advice(config.byte, Rotation::cur());
+
+            let bytes_rlc_curr = meta.query_advice(config.bytes_rlc, Rotation::cur());
+            let bytes_rlc_prev = meta.query_advice(config.bytes_rlc, Rotation::prev());
+
+            vec![
+                // if is_padding: byte == 0
+                is_enabled.expr() * is_padding_curr.expr() * byte.expr(),
+                // is_padding is boolean
+                is_enabled.expr() * is_padding_curr.expr() * (1.expr() - is_padding_curr.expr()),
+                // is_padding transitions from 0 -> 1 only once
+                is_enabled.expr() * delta.expr() * (1.expr() - delta.expr()),
+                // bytes_rlc updates in the non-padded territory
+                is_enabled.expr()
+                    * (1.expr() - is_padding_curr.expr())
+                    * (bytes_rlc_prev.expr() * challenges.keccak_input() + byte.expr()
+                        - bytes_rlc_curr.expr()),
+                // bytes_rlc remains unchanged in padded territory
+                is_enabled.expr()
+                    * is_padding_curr.expr()
+                    * (bytes_rlc_curr.expr() - bytes_rlc_prev.expr()),
+            ]
+        });
+
         assert!(meta.degree() <= 4);
 
         config
@@ -62,33 +128,36 @@ impl BlobDataConfig {
         batch_data: &BatchData,
         barycentric_assignments: &[CRTInteger<Fr>],
     ) -> Result<AssignedBlobDataExport, Error> {
-        let assigned_bytes = layouter.assign_region(
+        let (assigned_bytes, bytes_rlc) = layouter.assign_region(
             || "BlobData bytes",
-            |mut region| self.assign_rows(&mut region, batch_data),
+            |mut region| self.assign_rows(&mut region, batch_data, &challenge_value),
         )?;
 
         layouter.assign_region(
             || "BlobData internal checks",
             |mut region| {
                 self.assign_internal_checks(
                     &mut region,
-                    challenge_value,
                     rlc_config,
                     barycentric_assignments,
                     &assigned_bytes,
                 )
             },
-        )
+        )?;
+
+        Ok(AssignedBlobDataExport { bytes_rlc })
     }
 
+    #[allow(clippy::type_complexity)]
     pub fn assign_rows(
         &self,
         region: &mut Region<Fr>,
         batch_data: &BatchData,
-    ) -> Result<Vec<AssignedCell<Fr, Fr>>, Error> {
+        challenges: &Challenges<Value<Fr>>,
+    ) -> Result<(Vec<AssignedCell<Fr, Fr>>, AssignedCell<Fr, Fr>), Error> {
         let batch_bytes = batch_data.get_batch_data_bytes();
         let blob_bytes = {
-            let mut encoder = init_zstd_encoder();
+            let mut encoder = init_zstd_encoder(None);
             encoder
                 .set_pledged_src_size(Some(batch_bytes.len() as u64))
                 .map_err(|_| Error::Synthesis)?;
@@ -99,32 +168,74 @@ impl BlobDataConfig {
         };
         assert!(blob_bytes.len() <= N_BLOB_BYTES, "too many blob bytes");
 
+        self.q_first.enable(region, 0)?;
+        for i in 1..=N_BLOB_BYTES {
+            self.q_enabled.enable(region, i)?;
+        }
+
+        for col in [self.byte, self.bytes_rlc, self.is_padding] {
+            region.assign_advice(
+                || "advice at q_first=1",
+                col,
+                0,
+                || Value::known(Fr::zero()),
+            )?;
+        }
+
         let mut assigned_bytes = Vec::with_capacity(N_BLOB_BYTES);
-        for (i, &byte) in blob_bytes
-            .iter()
-            .chain(std::iter::repeat(&0))
-            .take(N_BLOB_BYTES)
-            .enumerate()
-        {
+        let mut bytes_rlc = Value::known(Fr::zero());
+        let mut last_bytes_rlc = None;
+        for (i, &byte) in blob_bytes.iter().enumerate() {
+            let byte_value = Value::known(Fr::from(byte as u64));
+            bytes_rlc = bytes_rlc * challenges.keccak_input() + byte_value;
             assigned_bytes.push(region.assign_advice(
                 || "byte",
                 self.byte,
-                i,
-                || Value::known(Fr::from(byte as u64)),
+                i + 1,
+                || byte_value,
             )?);
+            region.assign_advice(
+                || "is_padding",
+                self.is_padding,
+                i + 1,
+                || Value::known(Fr::zero()),
+            )?;
+            last_bytes_rlc =
+                Some(region.assign_advice(|| "bytes_rlc", self.bytes_rlc, i + 1, || bytes_rlc)?);
         }
 
-        Ok(assigned_bytes)
+        let last_bytes_rlc = last_bytes_rlc.expect("at least 1 byte guaranteed");
+        for i in blob_bytes.len()..N_BLOB_BYTES {
+            assigned_bytes.push(region.assign_advice(
+                || "byte",
+                self.byte,
+                i + 1,
+                || Value::known(Fr::zero()),
+            )?);
+            region.assign_advice(
+                || "is_padding",
+                self.is_padding,
+                i + 1,
+                || Value::known(Fr::one()),
+            )?;
+            region.assign_advice(
+                || "bytes_rlc",
+                self.bytes_rlc,
+                i + 1,
+                || last_bytes_rlc.value().cloned(),
+            )?;
+        }
+
+        Ok((assigned_bytes, last_bytes_rlc))
     }
 
     pub fn assign_internal_checks(
         &self,
         region: &mut Region<Fr>,
-        challenge_value: Challenges<Value<Fr>>,
         rlc_config: &RlcConfig,
         barycentric_assignments: &[CRTInteger<Fr>],
         assigned_bytes: &[AssignedCell<Fr, Fr>],
-    ) -> Result<AssignedBlobDataExport, Error> {
+    ) -> Result<(), Error> {
         rlc_config.init(region)?;
         let mut rlc_config_offset = 0;
 
@@ -151,10 +262,6 @@ impl BlobDataConfig {
             pows_of_256
         };
 
-        // read randomness challenges for RLC computations.
-        let r_keccak =
-            rlc_config.read_challenge1(region, challenge_value, &mut rlc_config_offset)?;
-
         ////////////////////////////////////////////////////////////////////////////////
         //////////////////////////////////// LINKING ///////////////////////////////////
         ////////////////////////////////////////////////////////////////////////////////
@@ -194,12 +301,6 @@ impl BlobDataConfig {
             region.constrain_equal(limb3.cell(), blob_crt.truncation.limbs[2].cell())?;
         }
 
-        ////////////////////////////////////////////////////////////////////////////////
-        //////////////////////////////////// EXPORT ////////////////////////////////////
-        ////////////////////////////////////////////////////////////////////////////////
-
-        let bytes_rlc =
-            rlc_config.rlc(region, assigned_bytes, &r_keccak, &mut rlc_config_offset)?;
-        Ok(AssignedBlobDataExport { bytes_rlc })
+        Ok(())
     }
 }