Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Very rare 403 error when streaming from YouTube #417

Closed
sasjafor opened this issue Feb 25, 2019 · 19 comments · Fixed by #428
Closed

Very rare 403 error when streaming from YouTube #417

sasjafor opened this issue Feb 25, 2019 · 19 comments · Fixed by #428
Labels

Comments

@sasjafor
Copy link

Very rarely I get a 403 error when trying to stream a YouTube video with my discord bot.
However, I can then immediately retry to play the exact same video and then it works.

I haven't found a consistent way of reproducing this error and it is quite rare, maybe 1 in 1000 stream attempts would be my guess. Also continuously trying to stream the same video doesn't seem to be able to reproduce the error. So any attempt would have to probably use a collection of a lot of different videos (1000+) and have a lot of time, but hopefully there is another way of figuring this out.

Here is the stack trace (which seems quite useless, as it's just the status code that miniget receives):

Error: Status code: 403
  at ClientRequest.httpLib.get (/REDACTED/node_modules/miniget/lib/index.js:123:19)
  at Object.onceWrapper (events.js:315:30)
  at emitOne (events.js:116:13)
  at ClientRequest.emit (events.js:211:7)
  at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:543:21)
  at HTTPParser.parserOnHeadersComplete (_http_common.js:112:17)
  at TLSSocket.socketOnData (_http_client.js:440:20)
  at emitOne (events.js:116:13)
  at TLSSocket.emit (events.js:211:7)
  at addChunk (_stream_readable.js:263:12) +0ms

The error always seems to appear twice (I only posted one to keep the text shorter), completely identically. I thought your miniget module would only retry on a 500 error, so I'm not sure why it shows up twice.

I am using the newest ytdl-core like this stream = ytdl(url, opts);.
And I'm using discord.js on the master branch, where I pass the stream like this this.conn.play(this.stream, opts);

I think if ytdl-core could simply retry once after receiving a 403, this might solve the problem. Of course a 403 with This video is not available. would need no retry. If I'm not mistaken ytdl-core doesn't retry on a receiving a 403 as of now, of course rightly so as normally a 403 means that the video is not available.

In this case, I think something might be going wrong on YouTube's side, but we can't just wait for them to fix it. And I know that this is quite a difficult problem, as it is not easily reproducible.

If you do not want to change anything in this module, because this error is very vague, I would gladly try out a changed version myself, I might just need a hint how I could add a retry when a 403 is received.

@fent
Copy link
Owner

fent commented Feb 26, 2019

I've seen this a few times in tests. If you'd like to test if simply retrying the request fixes it, you can change this line

https://github.com/fent/node-miniget/blob/master/lib/index.js#L124

to use >= 400, so that it considers 4xx error codes recoverable, and will retry the request.

If this doesn't work, we'd have to retry on ytdl-core level, by making a fresh call to ytdl.getInfo(), which would be a bit more complicated.

@Ensjam
Copy link

Ensjam commented Feb 27, 2019

Same issue

@Ensjam
Copy link

Ensjam commented Feb 27, 2019

Error: Status code: 403 at ClientRequest.httpLib.get C:\Users\Administrator\Desktop\pc\node_modules\miniget\lib\index.js:123:19) at Object.onceWrapper (events.js:316:30) at emitOne (events.js:115:13) at ClientRequest.emit (events.js:210:7) at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_clien t.js:565:21) at HTTPParser.parserOnHeadersComplete (_http_common.js:116:23) at TLSSocket.socketOnData (_http_client.js:454:20) at emitOne (events.js:115:13) at TLSSocket.emit (events.js:210:7) at addChunk (_stream_readable.js:266:12)

@omarroth
Copy link

omarroth commented Feb 27, 2019

YouTube sometimes will provide a different signature type for URLs that appears to be base64'd, rather than the usual 012345A.BCFDE. It uses the same decrypt function, but it needs to be appended as &sig=, rather than &signature=. YouTube provides the parameter name as a &sp=, so for example:

s=TguFguaOdq1UFgre5zoeOCrM0DHS0Hf91-bHsYT6qrwrWEICwSI72VcO0ooo0lxeJ6TZJweetJPa4Oj8HxwwwFBd5yfgIARww2IxgLAgLA
sp=sig

becomes:

&sig=ALgxI2wwRAIgfy5dBFwwwxH8jO4aPJteewJZT6Jexl0ooo0OcV27ISwCIEWrwrq6TYsHb-19fH0SHD0MrCOeFz5ergTU1qdOaugo

So the /signature/ on this line would need to take that into account.

It appears to be more common on music videos, appearing roughly 1 out of 40. Here's the relevant line in Invidious and a relevant PR in youtube-dl.

@Ensjam
Copy link

Ensjam commented Feb 27, 2019

I tried to change this line like mentioned above
https://github.com/fent/node-miniget/blob/master/lib/index.js#L124
and i got another error :

Error: Too many redirects
    at ClientRequest.httpLib.get (\node_modules\miniget\lib\index.js:115:32)
    at Object.onceWrapper (events.js:277:13)
    at ClientRequest.emit (events.js:189:13)
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:556:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:109:17)
    at TLSSocket.socketOnData (_http_client.js:442:20)
    at TLSSocket.emit (events.js:189:13)
    at addChunk (_stream_readable.js:284:12)
    at readableAddChunk (_stream_readable.js:265:11)
    at TLSSocket.Readable.push (_stream_readable.js:220:10)

@Ensjam
Copy link

Ensjam commented Mar 4, 2019

Any update on this issue ?

@kimyearho
Copy link

kimyearho commented Apr 5, 2019

I have the same problem.
I tend to have problems relatively often. Only the ytdl quality option is used and is the same as the default usage.

@ImaCrea
Copy link

ImaCrea commented Apr 15, 2019

same issue here :/

@Ensjam
Copy link

Ensjam commented Apr 15, 2019

Any updates on this issue ?

@Yosifz8
Copy link

Yosifz8 commented May 13, 2019

base64'd

Do you have more info about it? Because of the new "sig" is in a different format:
4L_xI2wwRQIhAPLQeE2AGXNs-Gw1E7TH4KLxpt7eehdtNbdDtg%3D0PD_cAiA45Z3kDz8gRkA5s3VFg1pk6dXYgQUMZwSckB_LR1M4YA%3D%3DA%3D9

And the old format is:
1D2CD6A596CCE255E770BE98F7251422F451A069.5740D4BA74381729F978779A78800CAA8CC64645640

Do i have to use the same decrypt method? or need to do something else?

@omarroth
Copy link

Unfortunately I haven't had much time to look into it, but you use the same decrypt function. Only thing to keep in mind is that "sig" should be URI-unescaped, so:

4L_xI2wwRQIhAPLQeE2AGXNs-Gw1E7TH4KLxpt7eehdtNbdDtg=0PD_cAiA45Z3kDz8gRkA5s3VFg1pk6dXYgQUMZwSckB_LR1M4YA==A=9

You should be able to tell if the function has been decrypted succesfully because all the padding (=) will be on the end.

@fent fent closed this as completed in #428 May 14, 2019
@fent
Copy link
Owner

fent commented Feb 18, 2020

it's not decrypted. what's the video id?

@matheustavaresdev
Copy link

matheustavaresdev commented Feb 18, 2020 via email

@fent
Copy link
Owner

fent commented Feb 18, 2020 via email

@matheustavaresdev
Copy link

matheustavaresdev commented Feb 18, 2020 via email

@drepram
Copy link

drepram commented Jun 4, 2020

I had the same error today and I fixed it by generating another API key Google Developer Console

The error was:

{ Error: Request failed with status code 403
    at createError (/Users/acp/Works/Megana/ytscrape/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/Users/acp/Works/Megana/ytscrape/node_modules/axios/lib/core/settle.js:18:12)
    at IncomingMessage.handleStreamEnd (/Users/acp/Works/Megana/ytscrape/node_modules/axios/lib/adapters/http.js:202:11)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  config:
   { adapter: [Function: httpAdapter],
     transformRequest: { '0': [Function: transformRequest] },
     transformResponse: { '0': [Function: transformResponse] },
     timeout: 0,
     xsrfCookieName: 'XSRF-TOKEN',
     xsrfHeaderName: 'X-XSRF-TOKEN',
     maxContentLength: -1,
     validateStatus: [Function: validateStatus],
     headers:
      { Accept: 'application/json, text/plain, */*',
        'User-Agent': 'axios/0.18.1' },
     method: 'get',
     url:
      'https://www.googleapis.com/youtube/v3/search?q=APBN%20dan%20APBD%20dalam%20Pembangunan%20Ekonomi%20Kelas%2011&part=snippet&maxResults=12&key=AIzaSyBtKhNioZvkXRU-k-gXTOfn1oFSR4nCG48',
     data: undefined },
  request:
   ClientRequest {
     _events:
      [Object: null prototype] {
        socket: [Function],
        abort: [Function],
        aborted: [Function],
        error: [Function],
        timeout: [Function],
        prefinish: [Function: requestOnPrefinish] },
     _eventsCount: 6,
     _maxListeners: undefined,
     output: [],
     outputEncodings: [],
     outputCallbacks: [],
     outputSize: 0,
     writable: true,
     _last: true,
     chunkedEncoding: false,
     shouldKeepAlive: false,
     useChunkedEncodingByDefault: false,
     sendDate: false,
     _removedConnection: false,
     _removedContLen: false,
     _removedTE: false,
     _contentLength: 0,
     _hasBody: true,
     _trailer: '',
     finished: true,
     _headerSent: true,
     socket:
      TLSSocket {
        _tlsOptions: [Object],
        _secureEstablished: true,
        _securePending: false,
        _newSessionPending: false,
        _controlReleased: true,
        _SNICallback: null,
        servername: 'www.googleapis.com',
        alpnProtocol: false,
        authorized: true,
        authorizationError: null,
        encrypted: true,
        _events: [Object],
        _eventsCount: 8,
        connecting: false,
        _hadError: false,
        _handle: [TLSWrap],
        _parent: null,
        _host: 'www.googleapis.com',
        _readableState: [ReadableState],
        readable: true,
        _maxListeners: undefined,
        _writableState: [WritableState],
        writable: false,
        allowHalfOpen: false,
        _sockname: null,
        _pendingData: null,
        _pendingEncoding: '',
        server: undefined,
        _server: null,
        ssl: [TLSWrap],
        _requestCert: true,
        _rejectUnauthorized: true,
        parser: null,
        _httpMessage: [Circular],
        [Symbol(res)]: [TLSWrap],
        [Symbol(asyncId)]: 6,
        [Symbol(lastWriteQueueSize)]: 0,
        [Symbol(timeout)]: null,
        [Symbol(kBytesRead)]: 0,
        [Symbol(kBytesWritten)]: 0,
        [Symbol(connect-options)]: [Object] },
     connection:
      TLSSocket {
        _tlsOptions: [Object],
        _secureEstablished: true,
        _securePending: false,
        _newSessionPending: false,
        _controlReleased: true,
        _SNICallback: null,
        servername: 'www.googleapis.com',
        alpnProtocol: false,
        authorized: true,
        authorizationError: null,
        encrypted: true,
        _events: [Object],
        _eventsCount: 8,
        connecting: false,
        _hadError: false,
        _handle: [TLSWrap],
        _parent: null,
        _host: 'www.googleapis.com',
        _readableState: [ReadableState],
        readable: true,
        _maxListeners: undefined,
        _writableState: [WritableState],
        writable: false,
        allowHalfOpen: false,
        _sockname: null,
        _pendingData: null,
        _pendingEncoding: '',
        server: undefined,
        _server: null,
        ssl: [TLSWrap],
        _requestCert: true,
        _rejectUnauthorized: true,
        parser: null,
        _httpMessage: [Circular],
        [Symbol(res)]: [TLSWrap],
        [Symbol(asyncId)]: 6,
        [Symbol(lastWriteQueueSize)]: 0,
        [Symbol(timeout)]: null,
        [Symbol(kBytesRead)]: 0,
        [Symbol(kBytesWritten)]: 0,
        [Symbol(connect-options)]: [Object] },
     _header:
      'GET /youtube/v3/search?q=APBN%20dan%20APBD%20dalam%20Pembangunan%20Ekonomi%20Kelas%2011&part=snippet&maxResults=12&key=AIzaSyBtKhNioZvkXRU-k-gXTOfn1oFSR4nCG48 HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.18.1\r\nHost: www.googleapis.com\r\nConnection: close\r\n\r\n',
     _onPendingData: [Function: noopPendingOutput],
     agent:
      Agent {
        _events: [Object],
        _eventsCount: 1,
        _maxListeners: undefined,
        defaultPort: 443,
        protocol: 'https:',
        options: [Object],
        requests: {},
        sockets: [Object],
        freeSockets: {},
        keepAliveMsecs: 1000,
        keepAlive: false,
        maxSockets: Infinity,
        maxFreeSockets: 256,
        maxCachedSessions: 100,
        _sessionCache: [Object] },
     socketPath: undefined,
     timeout: undefined,
     method: 'GET',
     path:
      '/youtube/v3/search?q=APBN%20dan%20APBD%20dalam%20Pembangunan%20Ekonomi%20Kelas%2011&part=snippet&maxResults=12&key=AIzaSyBtKhNioZvkXRU-k-gXTOfn1oFSR4nCG48',
     _ended: true,
     res:
      IncomingMessage {
        _readableState: [ReadableState],
        readable: false,
        _events: [Object],
        _eventsCount: 3,
        _maxListeners: undefined,
        socket: [TLSSocket],
        connection: [TLSSocket],
        httpVersionMajor: 1,
        httpVersionMinor: 1,
        httpVersion: '1.1',
        complete: true,
        headers: [Object],
        rawHeaders: [Array],
        trailers: {},
        rawTrailers: [],
        aborted: false,
        upgrade: false,
        url: '',
        method: null,
        statusCode: 403,
        statusMessage: 'Forbidden',
        client: [TLSSocket],
        _consuming: true,
        _dumped: false,
        req: [Circular],
        responseUrl:
         'https://www.googleapis.com/youtube/v3/search?q=APBN%20dan%20APBD%20dalam%20Pembangunan%20Ekonomi%20Kelas%2011&part=snippet&maxResults=12&key=AIzaSyBtKhNioZvkXRU-k-gXTOfn1oFSR4nCG48',
        redirects: [] },
     aborted: undefined,
     timeoutCb: null,
     upgradeOrConnect: false,
     parser: null,
     maxHeadersCount: null,
     _redirectable:
      Writable {
        _writableState: [WritableState],
        writable: true,
        _events: [Object],
        _eventsCount: 2,
        _maxListeners: undefined,
        _options: [Object],
        _redirectCount: 0,
        _redirects: [],
        _requestBodyLength: 0,
        _requestBodyBuffers: [],
        _onNativeResponse: [Function],
        _currentRequest: [Circular],
        _currentUrl:
         'https://www.googleapis.com/youtube/v3/search?q=APBN%20dan%20APBD%20dalam%20Pembangunan%20Ekonomi%20Kelas%2011&part=snippet&maxResults=12&key=AIzaSyBtKhNioZvkXRU-k-gXTOfn1oFSR4nCG48' },
     [Symbol(isCorked)]: false,
     [Symbol(outHeadersKey)]:
      [Object: null prototype] { accept: [Array], 'user-agent': [Array], host: [Array] } },
  response:
   { status: 403,
     statusText: 'Forbidden',
     headers:
      { vary: 'X-Origin, Origin,Accept-Encoding',
        'content-type': 'application/json; charset=UTF-8',
        date: 'Thu, 04 Jun 2020 10:14:51 GMT',
        expires: 'Thu, 04 Jun 2020 10:14:51 GMT',
        'cache-control': 'private, max-age=0',
        'x-content-type-options': 'nosniff',
        'x-frame-options': 'SAMEORIGIN',
        'content-security-policy': 'frame-ancestors \'self\'',
        'x-xss-protection': '1; mode=block',
        server: 'GSE',
        'accept-ranges': 'none',
        'transfer-encoding': 'chunked',
        'alt-svc':
         'h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"',
        connection: 'close' },
     config:
      { adapter: [Function: httpAdapter],
        transformRequest: [Object],
        transformResponse: [Object],
        timeout: 0,
        xsrfCookieName: 'XSRF-TOKEN',
        xsrfHeaderName: 'X-XSRF-TOKEN',
        maxContentLength: -1,
        validateStatus: [Function: validateStatus],
        headers: [Object],
        method: 'get',
        url:
         'https://www.googleapis.com/youtube/v3/search?q=APBN%20dan%20APBD%20dalam%20Pembangunan%20Ekonomi%20Kelas%2011&part=snippet&maxResults=12&key=AIzaSyBtKhNioZvkXRU-k-gXTOfn1oFSR4nCG48',
        data: undefined },
     request:
      ClientRequest {
        _events: [Object],
        _eventsCount: 6,
        _maxListeners: undefined,
        output: [],
        outputEncodings: [],
        outputCallbacks: [],
        outputSize: 0,
        writable: true,
        _last: true,
        chunkedEncoding: false,
        shouldKeepAlive: false,
        useChunkedEncodingByDefault: false,
        sendDate: false,
        _removedConnection: false,
        _removedContLen: false,
        _removedTE: false,
        _contentLength: 0,
        _hasBody: true,
        _trailer: '',
        finished: true,
        _headerSent: true,
        socket: [TLSSocket],
        connection: [TLSSocket],
        _header:
         'GET /youtube/v3/search?q=APBN%20dan%20APBD%20dalam%20Pembangunan%20Ekonomi%20Kelas%2011&part=snippet&maxResults=12&key=AIzaSyBtKhNioZvkXRU-k-gXTOfn1oFSR4nCG48 HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.18.1\r\nHost: www.googleapis.com\r\nConnection: close\r\n\r\n',
        _onPendingData: [Function: noopPendingOutput],
        agent: [Agent],
        socketPath: undefined,
        timeout: undefined,
        method: 'GET',
        path:
         '/youtube/v3/search?q=APBN%20dan%20APBD%20dalam%20Pembangunan%20Ekonomi%20Kelas%2011&part=snippet&maxResults=12&key=AIzaSyBtKhNioZvkXRU-k-gXTOfn1oFSR4nCG48',
        _ended: true,
        res: [IncomingMessage],
        aborted: undefined,
        timeoutCb: null,
        upgradeOrConnect: false,
        parser: null,
        maxHeadersCount: null,
        _redirectable: [Writable],
        [Symbol(isCorked)]: false,
        [Symbol(outHeadersKey)]: [Object] },
     data: { error: [Object] } } }

@asmirbe
Copy link

asmirbe commented Jul 30, 2021

same issue how can we keeps continue even if a 403 error appear right now stream is stopped and download is canceled how to fix this please ?

@TimeForANinja
Copy link
Collaborator

@asmirbelkic please keep the discussion in the #932 thread

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.