Deviantart 404 problem #488

fatefarer · 2019-11-26T11:13:38Z

Hello. When trying to grab any gallery I get the same type of message

[downloader.http][warning] '404 Not Found' for 'https://www.deviantart.com/download/739386335/dc87m5b-eddbd940-e7c0-481b-a65c-9b599d403e8b.jpg?token=bd5f065c2f02e4e05baa47e026b2331054f71b2d&ts=1574766971'
[download][error] Failed to download deviantart_739386335_Christian Maverick [Commission].jpg

Funnily enough, It worked perfectly just yesterday. I even tested it on the same gallery.
Sorry in advance for being a total and utter noob, but I will try to provide any information necessary. I am using the Win version, Python installed, gallery-dl version 1.11.2-dev. The command used is

py -3 -m gallery_dl -u [USERNAME] -p [PASSWORD] "URL"

Twi-Hard · 2019-11-26T14:28:00Z

They just changed a bunch of stuff. It might not be relevant though.
https://www.deviantart.com/team/journal/Eclipse-Change-Log-November-20-2019-820940679
https://www.deviantart.com/team/journal/Pagination-for-Browse-and-Search-820827828

sledgehammer93 · 2019-11-26T16:23:20Z

A (temporary) solution that I just found is to set the originals setting in gallery-dl.conf to false, like so:
"deviantart": { "extra": true, "flat": true, "folders": false, "journals": "html", "mature": true, "metadata": true, "original": false, "quality": 100, "wait-min": 0, } },
This will re-download some of the original files as .jpg's, but it appears to work on my end.

a-washing-machine · 2019-11-26T16:25:15Z

Can confirm. Everything worked fine for the last couple days until some point late at night yesterday.

The token it gets for the URL seems to be invalid, maybe?

I'm using this configuration for deviantart:

{
    "extractor":
    {
          "deviantart":
        {
			"refresh-token": "..................................",
			"client-id": "..................................",
            "client-secret": "..................................",			
			"extra": true,
			"metadata": true,
            "postprocessors": [{
                "name": "metadata",
                "mode": "custom",
                "format": "{description}\n"
            }]
        }
    }

}

Tried with and without refresh token, then re-ran gallery_dl oauth:deviantart just to see if a new refresh-token would help, but no.

Noticed it mainly failing for stash items, but on further investigation, quite a lot of regular deviations are failing too.

Examples:

https://www.deviantart.com/shnider/art/Twilight-Sharkle-464517039
https://sta.sh/010hpi1rjmac

...Worked when I downloaded them yesterday, now they result in a 404:

gallery-dl.exe -v https://www.deviantart.com/shnider/art/Twilight-Sharkle-464517039
[gallery-dl][debug] Version 1.11.1
[gallery-dl][debug] Python 3.7.5 - Windows-7-6.1.7601-SP1
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.6
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/shnider/art/Twilight-Sharkle-464517039'
[gallery-dl][debug] Updating urllib3 ciphers
[deviantart][debug] Using custom API credentials (client-id <<REDACTED>>)
[deviantart][debug] Using DeviantartDeviationExtractor for 'https://www.deviantart.com/shnider/art/Twilight-Sharkle-464517039'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /_napi/da-browse/shared_api/deviation/extended_fetch?deviationid=464517039&username=shnider&type=art&include_session=false HTTP/1.1" 200 None
[deviantart][debug] Active postprocessor modules: [MetadataPP]
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /download/464517039/d7ok7n3-7d4ba5d3-add0-4211-9de0-20a76f0d620d.png?token=fa0d2f5674cd1fd18ee4bf59561af7caa40b0487&ts=1574785613 HTTP/1.1" 404 None
[downloader.http][warning] '404 Not Found' for 'https://www.deviantart.com/download/464517039/d7ok7n3-7d4ba5d3-add0-4211-9de0-20a76f0d620d.png?token=fa0d2f5674cd1fd18ee4bf59561af7caa40b0487&ts=1574785613'
[download][error] Failed to download deviantart_464517039_Twilight Sharkle.png


gallery-dl.exe -v https://sta.sh/010hpi1rjmac
[gallery-dl][debug] Version 1.11.1
[gallery-dl][debug] Python 3.7.5 - Windows-7-6.1.7601-SP1
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.6
[gallery-dl][debug] Starting DownloadJob for 'https://sta.sh/010hpi1rjmac'
[gallery-dl][debug] Updating urllib3 ciphers
[deviantart][debug] Using custom API credentials (client-id <<REDACTED>>)
[deviantart][debug] Using DeviantartStashExtractor for 'https://sta.sh/010hpi1rjmac'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): sta.sh:443
[urllib3.connectionpool][debug] https://sta.sh:443 "GET /010hpi1rjmac HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/256A4866-90B4-350E-BC91-CBA6C7E949D9 HTTP/1.1" 200 1214
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids%5B0%5D=256A4866-90B4-350E-BC91-CBA6C7E949D9&mature_content=true HTTP/1.1" 200 307
[deviantart][debug] Active postprocessor modules: [MetadataPP]
[urllib3.connectionpool][debug] https://sta.sh:443 "GET /download/3706115701405332/daqg37x-89d31e0b-ae4a-42d0-9199-d773e5e8b109?token=d0ff59a1dccb967d619af5ed8eb80ca88c9f3a9b&ts=1574785093 HTTP/1.1" 404 None
[downloader.http][warning] '404 Not Found' for 'https://sta.sh/download/3706115701405332/daqg37x-89d31e0b-ae4a-42d0-9199-d773e5e8b109?token=d0ff59a1dccb967d619af5ed8eb80ca88c9f3a9b&ts=1574785093'
[download][error] Failed to download deviantart_649082445_Flatcomm.part

Hrxn · 2019-11-26T17:00:37Z

/addlabel: site-change

mikf · 2019-11-26T17:01:09Z

Download links like

https://www.deviantart.com/download/464517039/d7ok7n3-7d4ba5d3-add0-4211-9de0-20a76f0d620d.png?token=fa0d2f5674cd1fd18ee4bf59561af7caa40b0487&ts=1574785613
https://sta.sh/download/3706115701405332/daqg37x-89d31e0b-ae4a-42d0-9199-d773e5e8b109?token=d0ff59a1dccb967d619af5ed8eb80ca88c9f3a9b&ts=1574785093

now only redirect to the real download URL if you are logged in.

Clicking the download button on a deviation page without an active DeviantArt session (in a private browser window, for example) will redirect you to the login page, and opening any download links, like the ones above, will give a 404 page. But this all works when logged in.

On a more positive note: Download links from their OAuth API got fixed and work again, so I will most likely revert a bunch of changes made during #436 and everything except scraps and direct deviation links should work again.

Possible solutions for right now:

Revert back to v1.10.5.
This version was the last that used OAuth API download links, which now work again.
Export your DeviantArt cookies, enable cookies-update, and use the cookies.txt file with --cookies (Login with --username and -password hasn't been implemented for DeviantArt yet)

Twi-Hard · 2019-11-26T20:24:24Z

Is it possible at all to get scraps and direct links to work again? A lot of the art I've been downloading has been scraps (I'm not sure how much of it was direct links).

This commit (partially) reverts 27b5b24, 94eb7c6, and a437e78. Download URLs from the 'extended_fetch' endpoint are now only usable for logged in users, while those from the respective OAuth API endpoint are working again. Everything except scraps and direct deviation links should be fixed, and those two categories will work with exported cookies. (#488) TODO: - "native" login with --username and --password - better handling of internally stored cookies

mikf · 2019-11-26T22:52:32Z

359c3bc should fix original image downloads for everything except the two categories that don't use the OAuth API (scraps and direct links), but those will work with exported DeviantArt cookies.

#445 had some discussion about cookie usage on DeviantArt, if anyone is interested.

Next on the To-Do list: login support with --username and --password, so you don't need to rely exported browser cookies.

fatefarer · 2019-11-27T09:19:39Z

Could you, please, explain how to revert to a previous version? Upgrade commands seem to work only towards later versions. It still shows 1.11.2 after switching the executable in PATH for the older one and running the upgrade command. Again, sorry for dumb questions, I guess it's common knowledge around here.
Thanks in advance.

mikf · 2019-11-27T11:18:17Z

You can tell pip to install a specific version by using gallery-dl==1.10.5 as install target,
but getting the latest dev version (1.11.2-dev) also works, since it has now been fixed on master.

# 1.10.5
$ python3 -m pip install gallery-dl==1.10.5

# 1.11.2-dev
$ python3 -m pip install --upgrade https://github.com/mikf/gallery-dl/archive/master.tar.gz

fatefarer · 2019-11-27T12:44:17Z

Both ways work like a charm. Thank you very much.

espressoelf · 2019-12-03T20:45:35Z

I'm having some problems with the dA plug-in, too, which seem to be related to the changes. At first I was using dA with a public access token and default settings, which resulted in temporary filenames for images with special characters (like a colon) in the title. Now I'm logged in via OAuth and try to use a custom filename format: "filename": "{published_time}.{target[filename]}.{target[extension]}" Since the site changes, target[extension] and extension are empty.

Does the API no longer provide the original file extension? Not using a custom filename format works, but that means being back at the initial problem again.

mikf · 2019-12-03T21:21:30Z

Hmm, you are right, extension (and target[extension]) are now empty, because the download URLs don't have a filename extension anymore. But that's OK, because gallery-dl will use a files's MIME type from the Content-Type header to determine its extension if that field is empty.

Also: extension and target[extension], as well as filename and target[filename] have the same values, and its always better to use the top-level field if possible, especially for extension. The MIME type guess, for example, only works for extension and not target[extension].

temporary filenames

What do you mean by that? Could you provide an example DeviantArt link where this happens?

espressoelf · 2019-12-03T21:47:06Z

Example: https://www.deviantart.com/evelar/art/Subject-BA-XX81-822258460 ("Subject: BA-XX81"), which 404s btw., single images seem to be broken or OAuth is acting up, I don't know. (Edit: I'm using the current dev. version.)
But https://www.deviantart.com/evelar/gallery/ worked, should be deviantart_822258460_Subject: BA-XX81.png, according to the output, but actually is D3DUT9~N.PNG. It's a XFS filesystem, OS is OpenSUSE (Tumbleweed), v5.3.12-1 Kernel. That's why I prefer dA's filename, which would've been subject__ba_xx81_by_evelar-ddljuos. Prefixed with deviationId (822258460) it would be better inside an archive, but that's just my opinion.

mikf · 2019-12-03T22:12:21Z

Single images as well as Scraps aren't possible to fetch with the OAuth API. For single images you'd need their old UUID which in the new Eclipse interface is no longer available, and the OAuth API has no (documented) endpoint for scraps. Both therefore have to use the new public endpoints, which require you to be logged in to download the original image versions.

dA's filename with its numeric deviation ID in front would be "{index} {filename}.{extension}" as filename, and, if your filesystem doesn't like certain characters, you can remove them with path-restrict and path-remove.

espressoelf · 2019-12-03T22:46:57Z

index and deviationId have the same value, but I'll go with your suggestions, since you know way better than I do what to prefer. I'll also use the options you provided, thank you.

One thing, though: XFS is perfectly fine with colons in filenames, so it must be another issue. But I'll just go with path-restrict for now.

cloudywings2 · 2019-12-05T15:09:23Z

Export your DeviantArt cookies, enable cookies-update, and use the cookies.txt file with --cookies (Login with --username and -password hasn't been implemented for DeviantArt yet)

How do you do this? I've tried adding the following line to the Deviantart settings to give it access to the browser cookies

"cookies_update": "C:/Users/UserName/AppData/Local/Google/Chrome/User Data/Default/Cookies",

but it doesn't work.

mikf · 2019-12-05T21:00:19Z

It should be

"cookies": "C:/path/to/cookies.txt",
"cookies-update": true,

where cookies is the path to a Netscape cookies.txt file, and cookies-update allows gallery-dl to modify said file.

Specifying the location where Chrome stores its cookies won't work, because gallery-dl doesn't know how to read cookies from such a file. You'll have to use a browser addon that can export your cookies as a cookies.txt file. I'm personally using export-cookies-txt in Firefox, but I can't give you any recommendations for Chrome.

Edit: Downloading single images and scraps currently wont work, regardless of your cookie setup: #505

a-washing-machine mentioned this issue Nov 26, 2019

DeviantArt Stash download can lead to infinite loop (rare) #489

Closed

mikf added the site:change label Nov 26, 2019

fatefarer closed this as completed Nov 27, 2019

mikf reopened this Nov 30, 2019

mikf pinned this issue Nov 30, 2019

This was referenced Nov 30, 2019

[DeviantArt] Scraps 404 #498

Closed

deviantart download 404 error #495

Closed

lllusion3469 mentioned this issue Dec 1, 2019

Deviantart broke download urls when not logged in danbooru/danbooru#4219

Closed

kattjevfel mentioned this issue Dec 5, 2019

[deviantart] KeyError - 'files' #505

Closed

mikf unpinned this issue Dec 20, 2019

mikf added the fixed label Dec 20, 2019

mikf closed this as completed Dec 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deviantart 404 problem #488

Deviantart 404 problem #488

fatefarer commented Nov 26, 2019

Twi-Hard commented Nov 26, 2019 •

edited

Loading

sledgehammer93 commented Nov 26, 2019 •

edited

Loading

a-washing-machine commented Nov 26, 2019

Hrxn commented Nov 26, 2019

mikf commented Nov 26, 2019

Twi-Hard commented Nov 26, 2019

mikf commented Nov 26, 2019

fatefarer commented Nov 27, 2019 •

edited

Loading

mikf commented Nov 27, 2019

fatefarer commented Nov 27, 2019

espressoelf commented Dec 3, 2019

mikf commented Dec 3, 2019 •

edited

Loading

espressoelf commented Dec 3, 2019 •

edited

Loading

mikf commented Dec 3, 2019

espressoelf commented Dec 3, 2019

cloudywings2 commented Dec 5, 2019

mikf commented Dec 5, 2019 •

edited

Loading

Deviantart 404 problem #488

Deviantart 404 problem #488

Comments

fatefarer commented Nov 26, 2019

Twi-Hard commented Nov 26, 2019 • edited Loading

sledgehammer93 commented Nov 26, 2019 • edited Loading

a-washing-machine commented Nov 26, 2019

Hrxn commented Nov 26, 2019

mikf commented Nov 26, 2019

Twi-Hard commented Nov 26, 2019

mikf commented Nov 26, 2019

fatefarer commented Nov 27, 2019 • edited Loading

mikf commented Nov 27, 2019

fatefarer commented Nov 27, 2019

espressoelf commented Dec 3, 2019

mikf commented Dec 3, 2019 • edited Loading

espressoelf commented Dec 3, 2019 • edited Loading

mikf commented Dec 3, 2019

espressoelf commented Dec 3, 2019

cloudywings2 commented Dec 5, 2019

mikf commented Dec 5, 2019 • edited Loading

Twi-Hard commented Nov 26, 2019 •

edited

Loading

sledgehammer93 commented Nov 26, 2019 •

edited

Loading

fatefarer commented Nov 27, 2019 •

edited

Loading

mikf commented Dec 3, 2019 •

edited

Loading

espressoelf commented Dec 3, 2019 •

edited

Loading

mikf commented Dec 5, 2019 •

edited

Loading