ValueError: AES key must be either 16, 24, or 32 bytes long

[github-user-en]

Checklist

• I'm asking a question
• I've looked through the README and FAQ for similar questions
• I've searched the bugtracker for similar questions including closed ones

Question

Please help me diagnose this error log and enlighten me with what I need to do to download the complete video. Any help would be much appreciated!

C:\Users\AAI-Shagun\AppData\Roaming\youtube-dlg>youtube-dl.exe --hls-prefer-native --hls-use-mpegts -v "https://5ad66d227b0c1.streamlock.net/edusecure/_definist/64/1212/5042/121513/64_1212_5042_121513_0.mp4/chunklist_w1422491268.m3u8?isPrimary=true&ref=http%3A%2F%2Ftheiashub.edugyaan.com&sessionId=1l0gahmb2ueads99d190bfcnn4&userId=231644&username=singhal.shagun%40gmail.com&videoId=121513"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--hls-prefer-native', '--hls-use-mpegts', '-v', 'https://5ad66d227b0c1.streamlock.net/edusecure/_definist/64/1212/5042/121513/64_1212_5042_121513_0.mp4/chunklist_w1422491268.m3u8?isPrimary=true&ref=http%3A%2F%2Ftheiashub.edugyaan.com&sessionId=1l0gahmb2ueads99d190bfcnn4&userId=231644&username=singhal.shagun%40gmail.com&videoId=121513']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2019.07.16
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.14393
[debug] exe versions: ffmpeg 3.2.4, ffprobe 3.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[generic] chunklist_w1422491268.m3u8?isPrimary=true&ref=http://theiashub.edugyaan.com&sessionId=1l0gahmb2ueads99d190bfcnn4&userId=231644&username=singhal.shagun@gmail: Requesting header
[generic] chunklist_w1422491268.m3u8?isPrimary=true&ref=http://theiashub.edugyaan.com&sessionId=1l0gahmb2ueads99d190bfcnn4&userId=231644&username=singhal.shagun@gmail: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://5ad66d227b0c1.streamlock.net/edusecure/_definist/64/1212/5042/121513/64_1212_5042_121513_0.mp4/chunklist_w1422491268.m3u8?isPrimary=true&ref=http%3A%2F%2Ftheiashub.edugyaan.com&sessionId=1l0gahmb2ueads99d190bfcnn4&userId=231644&username=singhal.shagun%40gmail.com&videoId=121513'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 1016
[download] Destination: chunklist_w1422491268-chunklist_w1422491268.m3u8isPrimary=true&ref=http -theiashub.edugyaan.com&sessionId=1l0gahmb2ueads99d190bfcnn4&userId=231644&username=singhal.shagun@gmail.mp4
[download] 2.7% of ~687.24MiB at Unknown speed ETA Unknown ETATraceback (most recent call last):
File "main.py", line 19, in
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl_init.py", line 474, in main
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl_init.py", line 464, in _real_main
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 2008, in download
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 807, in extract_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 861, in process_ie_result
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 1642, in process_video_result
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 1915, in process_info
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\YoutubeDL.py", line 1854, in dl
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\downloader\common.py", line 366, in download
File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpaxmfe_l6\build\youtube_dl\downloader\hls.py", line 174, in real_download
File "C:\Python\Python34\lib\site-packages\Crypto\Cipher\AES.py", line 95, in new
File "C:\Python\Python34\lib\site-packages\Crypto\Cipher\AES.py", line 59, in init
File "C:\Python\Python34\lib\site-packages\Crypto\Cipher\blockalgo.py", line 141, in init
ValueError: AES key must be either 16, 24, or 32 bytes long

C:\Users\AAI-Shagun\AppData\Roaming\youtube-dlg>

[dstftw]

Most likely broken key URL in manifest.

[github-user-en]

I am able to watch the videos just fine in the browser. However, when I try to download them, I get this error. Also, this error occurs irrespective of which video I try to download and irrespective of whether I try to download using the playlist.m3u8 file or the chunklist_w*********.m3u8 file.

For further diagnosis' sake, please find the URLs in the attached excel file.
URLs.xlsx

I need to download the videos because there's a limit that I can watch videos only up to 4 times on the website.

[remitamine]

I think the key request, requires specific cookies/headers, try to pass them with --cookies/--add-header option.

[github-user-en]

I think the key request, requires specific cookies/headers, try to pass them with --cookies/--add-header option.

The cookies.txt extension for Chrome says that:
# HTTP Cookie File by Genuinous @genuinous.
# No cookies for edugyaan.com.
# Download all cookies.

Regarding the possible requirement of a specific header, youtube-dl is using:
"http_headers": { "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.96 Safari/537.36", "Accept-Encoding": "gzip, deflate", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept-Language": "en-us,en;q=0.5" }

Chrome, however, showed me these Headers:

Which of the Chrome headers do I need to pass along? And is there a more efficient way to pass them than typing each of them in the command using --add-header?

[remitamine]

Which of the Chrome headers do I need to pass along?

the headers you're looking at are not the ones that I'm referring to, I said the headers of the key request, not the manifest request(in the screenshot it contains authenticate?wowzasessionid...), and you have to try I can't tell which one to use, it depends on the website.

And is there a more efficient way to pass them than typing each of them in the command using --add-header?

by adding support for the original website.

[github-user-en]

@remitamine
I see that you added the 'account needed' label. The credentials may be found in the excel below.

[remitamine]

don't post credentials publically, it's better to change the password.

[github-user-en]

@remitamine
After logging into the account, follow these steps:

1. Click on "Quality Enrichment Programme (QEP) - Basic".
2. Click on "1. Full Course : QEP"
3. From the list on the right hand side, click on "QEP Basic Session 4 - June 26".

Just to remind you, please don't play a video because I can play a video 4 times only. You can get the manifest URL etc, without actually playing any video... it starts loading them automatically.

[github-user-en]

I get ERROR: batch file could not be read when I run the following command:

youtube-dl.exe --add-header "Request Method":"GET" --add-header "Remote Address":"3.0.51.212:443" --add-header "Referrer Policy":"no-referrer-when-downgrade" --add-header "Accept":"*/*" --add-header "Accept-Encoding":"gzip, deflate" -add-header "Accept-Language":"en-US,en;q=0.9" --add-header "Connection":"keep-alive" --add-header "Cookie":"PHPSESSID=jnottoh8pfgqvq7gekf4bq1qf5" --add-header "DNT":"1" --add-header "Host":"theiashub.edugyaan.com" --add-header "Referer":"http://theiashub.edugyaan.com/video/embededView/120418" "https://5ad66d227b0c1.streamlock.net/edusecure/_definist/64/1212/5042/120418/64_1212_5042_120418_0.mp4/chunklist_w1289291518.m3u8?isPrimary=true&ref=http%3A%2F%2Ftheiashub.edugyaan.com&sessionId=jnottoh8pfgqvq7gekf4bq1qf5&userId=231644&username=singhal.shagun%40gmail.com&videoId=120418" --hls-prefer-native --hls-use-mpegts --add-header "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"

Any ideas?

[remitamine]

the manifests served from this website are (partially) encode parts of the Key URL.
you can try to use this patch to correct the key URL and see if it will produce a valid file.

diff --git a/youtube_dl/downloader/hls.py b/youtube_dl/downloader/hls.py
index b59aad73f..a7b060fd8 100644
--- a/youtube_dl/downloader/hls.py
+++ b/youtube_dl/downloader/hls.py
@@ -13,6 +13,7 @@ from .external import FFmpegFD
 
 from ..compat import (
     compat_urllib_error,
+    compat_urllib_parse_unquote,
     compat_urlparse,
     compat_struct_pack,
 )
@@ -184,6 +185,7 @@ class HlsFD(FragmentFD):
                     if decrypt_info['METHOD'] == 'AES-128':
                         if 'IV' in decrypt_info:
                             decrypt_info['IV'] = binascii.unhexlify(decrypt_info['IV'][2:].zfill(32))
+                        decrypt_info['URI'] = compat_urllib_parse_unquote(decrypt_info['URI'])
                         if not re.match(r'^https?://', decrypt_info['URI']):
                             decrypt_info['URI'] = compat_urlparse.urljoin(
                                 man_url, decrypt_info['URI'])

make sure that you use the Native HLS Downloader:

youtube-dl --hls-prefer-native [M3U8_URL]

[github-user-en]

Which file should I put this code in? Would I need to recompile things? I'm somewhat lost...

[remitamine]

you can look up online for how to obtain source code from a git repository, apply a git patch and running the source code with python.

[github-user-en]

@remitamine
Although it took me an entire night to understand your approach, but I'm still up against a roadblock..

First, kindly confirm whether the steps I performed are correct:

1. Install Git for Windows, Visual Studio Code, GNU Make, Python, Pandoc & Nostests.
2. Fork Youtube-dl's repository to my github account and then clone it on my local machine.
3. In the clone downloaded on my machine, in the hls.py file, insert those two statements.
4. Then run the command python -m youtube_dl --hls-prefer-native [m3u8 URL] to see if the issue is solved.

If the above understanding is correct, then please check the associated logs. I'm still getting an error...though, it's different from the earlier one.

Also, the command I mentioned in step 4 above doesn't seem to be using the HLS downloader. Youtube-DL is using FFMPEG, which might be the cause of the issue now. Possibly, I'm framing the command in an incorrect manner?
Logs.log

[remitamine]

the command I mentioned in step 4 above doesn't seem to be using the HLS downloader

install PyCrypto or PyCryptodome.

[github-user-en]

Hi @remitamine,

I installed PyCrypto and downloaded two of those videos. However, I face the following issues:

1. The video downloads were interrupted multiple times with the error, ERROR: Did not get any data blocks.
2. When I got this error, I would re-run the command and the video download would resume. Although I could download the complete videos by resuming the operation 3-4 times, with the 1st video, I got the message Malformed aac bitstream detected. Install ffmpeg or avconv to fix this automatically. Also, the video couldn't be played either by VLC or by PotPlayer.
3. So when I tried downloading the 2nd video, I placed ffpmeg.exe file in youtube-dl folder (where setup.py is present in the cloned code). Although I didn't get the malfomed aac bitstream message this time, but the downloaded video still couldn't be viewed using VLC or PotPlayer.

In the file properties, Windows has mostly blank fields.

VLC doesn't even show the duration of the video:

Any ideas what's going wrong here?

For the malformed aac bitstream, I tried the suggestions on this reddit thread but they didn't work. If it doesn't cause us to deviate from the thread title, your ideas on that too would be invaluable.

[github-user-en]

@remitamine
Some thing strange happended after I created a local EXE of youtube-dl as per your advice here. I put the EXE in a folder with the other FFMPEG files (ffmpeg.exe, ffplay.exe, ffprobe.exe). And then, I ran this batch file:
ExecuteTillDownloaded(Successful).zip

The result was:
Log.txt

The following caught my eye:

1. It threw the warning: "WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg". But even till yesterday, hls downloader supported the download....including our modified source code.
2. The ffmpeg continued downloading later fragments even though it failed to download many fragments in between.
3. Although it says "[download] 100% of 120.16MiB in 02:44:15", the fact is that the downloaded file is only 35 minutes 18 seconds long; not 2 hours 44 minutes 15 seconds long as it should be.
4. Every second of this 35 minutes 18 seconds works, in contrast to the earlier HLS downloads which didn't work, despite being complete downloads.

Any advice? Would we be able to make HLS downloader work as earlier, and have the ffmpeg files in the same folder to allow youtube-dl to fix the malformed bitstream?

[remitamine]

It threw the warning: "WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg". But even till yesterday, hls downloader supported the download....including our modified source code.

PyCrypto missing from executable

The ffmpeg continued downloading later fragments even though it failed to download many fragments in between.

possibly similar to #15914.

anyway, you have to look for a solution for the problem you have, I'm not here to fix every problem and answer every question(especially when the problem is caused by the website serving partially broken files)

[github-user-en]

@remitamine
The file that I'm downloading using HLS downloader is remaining encrypted. That's why I'm unable to view the video, despite being able to download the videos completely.

Any suggestions regarding the code to write the output file in decrypted form?

[remitamine]

Any suggestions regarding the code to write the output file in decrypted form?

you have to fix the way to obtain the AES key.

and again, I'm not highly interested in this issue.