feat: Implement Chatting with PDF using Video Subtitles

[umeshSinghVerma]

Feature: Chatting with PDF 🗨️📄

Overview:
The goal of this feature is to allow users to interactively chat with the content of a YouTube video by using its subtitles. Users can ask questions about the video, and the AI will respond based on the information provided in the subtitles. 🤖

Implementation Steps:

1. Extract Subtitles:

   • Utilize the YouTube Data API to fetch the subtitles (captions) of the video. You can use the captions endpoint to get the available captions and then fetch the relevant language. 📺
   • Parse the subtitle text to clean it up and create a structured format that can be used for querying. 📝

2. Store Subtitles:

   • Store the extracted subtitles in Chrome's unlimited storage so that they can be accessed when the user wants to chat about the video. You can structure it as key-value pairs, with the video ID as the key and the subtitle text as the value. 💾

3. Set Up AI Integration:

   • Choose an AI model (e.g., OpenAI's GPT or Gemini) for processing user queries. 🧠
   • Create a function that sends the subtitle content to the AI model along with the user’s questions.

4. Build Chat Interface:

   • Create a user-friendly chat interface within the FrameTagger extension where users can type their questions about the video. 💬
   • Display the AI responses in a conversational format. 🗨️

5. Handle User Queries:

   • Implement logic to process the user's questions:
     • If the question pertains to specific content in the video, filter the relevant parts of the subtitles.
     • Send the filtered text to the AI model for a more contextual response. 🔍

6. Response Display:

   • Show the AI's responses in real-time within the chat interface. ⏱️
   • Optionally, allow users to navigate to specific timestamps in the video based on the AI's response (if applicable). 🎥

7. Testing and Feedback:

   • Conduct user testing to gather feedback on the functionality and improve the interaction based on user experience. 📊

Potential Challenges:

• Contextual Understanding: Ensuring the AI model understands the context of the video based on the subtitles. 🤔
• Response Relevance: Fine-tuning the AI responses to ensure they are relevant to the user’s queries. 🎯
• Performance: Making sure the extraction and querying process is efficient and does not slow down the user experience. ⚡

Benefits:

• Enhances user engagement by providing an interactive way to understand video content. 🌟
• Helps students clarify doubts and deepen their understanding of the material covered in the videos. 🎓

[atharva9980]

Hi @umeshSinghVerma , I'd like to work on this issue. Could you please assign it to me? Thanks!

[umeshSinghVerma]

Hi @atharva9980!

Thanks for your interest in this feature! I've assigned the issue to you. Before you dive in, please try out FrameTagger by downloading it from the Chrome Web Store. Let me know if you encounter any bugs or have suggestions for improvements. Also, don't forget to star ⭐ the repository if you haven't already—it helps us grow!

Feel free to reach out if you have any questions. Excited to see your contribution!

Hi @umeshSinghVerma , I'd like to work on this issue. Could you please assign it to me? Thanks!