Improve audio-to-text pipeline by enabling flash-attention. [$750]

[JJassonn69]

Bounty Overview

We have identified an opportunity to improve the current audio-to-text pipeline in Livepeer AI Network by enabling flash-attention that will speed up the pipeline significantly allowing for faster and almost realtime operation. We are seeking the community and bounty hunters support to quickly implement this optimisation so it can be available to developers working with Livepeer.

Required Skillset.

• Proven experience working with deep learning frameworks such as PyTorch, particularly in implementing attention mechanisms and optimising model performance.
• Strong experience with Python.

Bounty Requirements

To successfully resolve this bounty, you must:

1. Enable the optimisation on the existing pipeline by enabling memory efficient flash attention.
2. Ensure that devices that don't yet support the optimisation should safely fallback to working Scaled Dot-Product Attention SDPA implementation .
3. Create a separate docker container image similar to PR #185 to avoid dependencies issues with other pipelines.

Scope Exclusions

• None. All areas related to the issue are within scope.

Implementation Tips

1. Consult the documentation of the flash-attention from pytorch to better understand how to enable it in audio-to-text pipeline.
2. Validate performance improvements in the Flash Attention-enabled pipeline and ensure proper fallback functionality in unsupported devices.

How to Apply

This bounty has been assigned to Prakarsh, who has expressed interest in addressing this issue. If you have been officially assigned this issue:

• Communicate with the team as needed for context or clarification.
• Provide regular updates on your progress in the issue thread.

Warning

Please ensure the issue is assigned to you before starting work. To avoid duplication of efforts, unassigned issue submissions will not be accepted.

[JJassonn69]

Posted to notion. Closing this.