epic: Reviews of Ichigo paper

[tikikun]

review it!

[tikikun]

It's important to cite SpiritLM from META to make our paper more robust, but we really have similiar results.

[tikikun]

We need similar analysis but with our own way of alignment using transcription prompting, probably need to refer to qwen audio technique also

[tikikun]

@hahuyhoang411 Also remember to add the visualization for non speech pattern

[tikikun]

intent classification also need to pay attention

[tikikun]

@hahuyhoang411 should review

[hahuyhoang411]

Was discussing with @tuanlda78202 he pointed out a few missing key numbers . Will add in

[tuanlda78202]

Review of Ichigo Paper

1. Figure Inference Pipeline: The main figure should mention its inference pipeline, current figure may lead to confusion with the training pipeline. It is recommended to have separate figures for each stage, clearly illustrating the Whisper Encoder (continuous embedding) and VQ (codebooks) separately. This could be labeled as Quantizer Whisper Encoder.

2. Pre-training Dataset: The paper lacks clarity regarding the type of data used for pre-training (only semantic tokens). It is unclear why only ASR data is utilized (theoretically, any sound data could be used). Additionally, the processing and saving of the pre-training data (stored on disk) and the number of tokens used for this stage are not specified. It would be beneficial to provide a clearer explanation, specifically using raw sound data, generating semantic tokens using Quantizer Whisper Encoder, and saving them on disk for efficient training of all stages.

3. Post-training: The post-training stage should be clearly defined after the header, with specific statements like T2T, S2T (speech-to-text) and ST2T (transcription). This will prevent confusion among readers.

4. Reader Questions: Readers may question why our paper does not include fine-tuning on the S2S task. They may assume that since we have a semantic token encoder, we only need a decoder to generate sound.

5. Transcribe Task: The transcribe task should be explicitly defined to enhance understanding. It should be noted as S+T -> T, with input consisting of Instruction (text tokens) and Audio (semantic tokens) and output as Text tokens.

6. Noise Part: Readers may be confused about the pipeline and the number of semantic tokens produced. It is unclear why 512 semantic tokens result in 513^n. Additionally, the generation of refuse answers from LLMs is not explicitly mentioned. It would be beneficial to clarify this aspect as a figure pipeline: random noise -> Ichigo -> Text (generated refuse answers from LLMs).

cc @tikikun @hahuyhoang411

[tikikun]

#132

@hahuyhoang411

[hahuyhoang411]

On going review