Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Deviantart] An unexpected error occurred: KeyError - 'deviation'. #6254

Closed
sophi228 opened this issue Sep 30, 2024 · 10 comments
Closed

[Deviantart] An unexpected error occurred: KeyError - 'deviation'. #6254

sophi228 opened this issue Sep 30, 2024 · 10 comments
Labels
regression site:bug

Comments

@sophi228
Copy link

Hii! I'm using gallery-dl for over a year now, but since the 1.27.5 update, I'm having trouble downloading some literature labeled texts/fanfics from Deviantart. I hope I did nothing wrong. Here are the debugging information:

[gallery-dl][debug] Version 1.27.5
[gallery-dl][debug] Python 3.10.4 - Windows-10-10.0.22631-SP0
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.7
[gallery-dl][debug] Configuration Files ['%APPDATA%\gallery-dl\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/starvinglunatic/art/Against-the-world-chapter-1-50968347'
[deviantart][debug] Using DeviantartDeviationExtractor for 'https://www.deviantart.com/starvinglunatic/art/Against-the-world-chapter-1-50968347'
[deviantart][debug] Using custom API credentials (client-id 29899)
[deviantart][info] Requesting public access token
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "POST /oauth2/token HTTP/1.1" 200 126
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/starvinglunatic HTTP/1.1" 200 390
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /StarvingLunatic/art/50968347 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/016057D5-33D2-EC3D-D242-0E9DCF39368A HTTP/1.1" 200 818
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids%5B0%5D=016057D5-33D2-EC3D-D242-0E9DCF39368A&mature_content=true HTTP/1.1" 200 936
[deviantart][info] Collecting folder information for 'StarvingLunatic'
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/folders?username=StarvingLunatic&offset=0&limit=50&mature_content=true HTTP/1.1" 200 5920
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/50619809-A51F-09B2-C01C-EAD5CCA7D6BE?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/50619809-A51F-09B2-C01C-EAD5CCA7D6BE?username=StarvingLunatic&offset=24&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/50619809-A51F-09B2-C01C-EAD5CCA7D6BE?username=StarvingLunatic&offset=48&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/50619809-A51F-09B2-C01C-EAD5CCA7D6BE?username=StarvingLunatic&offset=72&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/50619809-A51F-09B2-C01C-EAD5CCA7D6BE?username=StarvingLunatic&offset=96&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/F3EF72FC-51E3-8CEB-083B-258D6C414FC6?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/F3EF72FC-51E3-8CEB-083B-258D6C414FC6?username=StarvingLunatic&offset=24&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 3035
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/F097CA37-DD8F-C43D-C2E0-72A3818309D6?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 2929
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/425DD360-D28F-F5A1-76CA-C12D689B5334?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 4664
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/7D26849B-B1F3-11F6-037B-94C1E401517B?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 5006
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/8E72B18C-CE61-9EA9-121F-9DB5ED29895B?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 1708
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/E9F0DC3B-6A44-AA53-9627-5AC5679D082E?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 4028
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/60246A9F-71FD-8EDC-F48B-B203A2164613?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/61AE6FFD-72C5-BD25-2E14-2F3E09A15DBC?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 4591
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/F2BCC72F-2341-3114-F2C2-E13EC2669938?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 3598
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/A4BD78D5-3C92-B73C-E5DA-94086806DFAC?username=StarvingLunatic&offset=0&limit=24&mature_content=true&mode=newest HTTP/1.1" 200 3529
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/comments/deviation/016057D5-33D2-EC3D-D242-0E9DCF39368A?maxdepth=5&offset=0&limit=50&mature_content=true HTTP/1.1" 200 1340
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET / HTTP/1.1" 200 None
[deviantart][debug] Sleeping 2.00 seconds (request)
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /_puppy/dadeviation/init?deviationid=50968347&username=StarvingLunatic&type=journal&include_session=false&expand=deviation.related&da_minor_version=20230710&csrf_token=NS3h3x8dLZW5ODct.sklnfk.JUdnAfMtfmv5xVaWEcoMG9wxPvNmDb0aD4NZVInzMBA HTTP/1.1" 400 118
[deviantart][error] An unexpected error occurred: KeyError - 'deviation'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[deviantart][debug]
Traceback (most recent call last):
File "C:\Users\klaiu\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\job.py", line 151, in run
for msg in extractor:
File "C:\Users\klaiu\AppData\Local\Programs\Python\Python310\lib\site-packages\gallery_dl\extractor\deviantart.py", line 185, in items
content = self.eclipse_api.deviation_extended_fetch(
KeyError: 'deviation'
@BlueSpiders99
Copy link

I am having a similar error with DeviantArt; my KeyError is 'category'.

[gallery-dl][debug] Version 1.25.8 - Executable
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.19045
[gallery-dl][debug] requests 2.31.0 - urllib3 1.26.16
[gallery-dl][debug] Configuration Files ['.\\config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/REDACTED/favourites/96088431/new'
[deviantart][debug] Using DeviantartCollectionExtractor for 'https://www.deviantart.com/REDACTED/favourites/96088431/new'
[deviantart][debug] Using custom API credentials (client-id 26320)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/REDACTED HTTP/1.1" 200 330
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/collections/folders?username=REDACTED&offset=0&limit=50&mature_content=true HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/collections/A5E0B283-9E2A-A78F-6A13-4CC68A5E19A0?username=REDACTED&offset=0&limit=24&mature_content=true HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids%5B0%5D=654092F0-958C-5AD6-84A3-149BBF2D947F&deviationids%5B1%5D=89E8CD81-D7BC-36C2-111E-A2005C93
4BC0&deviationids%5B2%5D=E52BFD20-C24D-538C-9995-5ABB5B52EF94&deviationids%5B3%5D=48D5AD22-5AEB-7BAF-6797-A98DB07BD83B&deviationids%5B4%5D=D421C09E-AF16-30F6-6E78-0460691682AC&deviationids%5B5%5D=34BF31C7-5019-A
4FB-43AC-EBBDE596B076&deviationids%5B6%5D=FF814C5A-9514-C713-EEC9-7D2C5FD53371&deviationids%5B7%5D=6DF734EB-763F-7F79-19C8-C26401373F7B&deviationids%5B8%5D=D7569B5C-4379-50EF-1AD3-4143FBD0B194&deviationids%5B9%5
D=74B3391C-CE0A-2997-CDF2-783E2E02B7B0&deviationids%5B10%5D=FF799C92-5D1F-21DC-C529-1D87046DBCDF&deviationids%5B11%5D=8201446E-C3A0-4544-863E-79B2DE8E7ED2&deviationids%5B12%5D=67DEF4C4-5713-A715-FB48-3413E3867EE
4&deviationids%5B13%5D=258EA935-1260-988C-6111-D53D4A200D83&deviationids%5B14%5D=C9B1895E-51E1-754D-BF7D-048A090D2046&deviationids%5B15%5D=0FE6EEB7-ED69-C1B6-3A24-47BEF2070B70&deviationids%5B16%5D=11A2FA65-5B22-
56CC-C955-86EFD6FB0D02&deviationids%5B17%5D=69CDC46C-B41B-CCB9-D494-8682B9CB230B&deviationids%5B18%5D=AED86DB2-4F87-98E1-76A9-D7C103EA5F05&deviationids%5B19%5D=5847262A-F4E5-5A78-ACC1-EBC3744011B1&deviationids%5
B20%5D=AE920495-917D-129E-B2DA-8650B865B09E&deviationids%5B21%5D=8922BB5E-3031-16ED-15F9-1E21FE015C19&deviationids%5B22%5D=F4D1FEC2-4D88-4096-B43A-ED3F782C0625&deviationids%5B23%5D=3963E09C-4865-AF0E-3328-B706F35A42E0&mature_content=true HTTP/1.1" 200 None
[deviantart][error] An unexpected error occurred: KeyError - 'category'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues . 
[deviantart][debug]
Traceback (most recent call last):
  File "gallery_dl\job.pyc", line 96, in run
  File "gallery_dl\extractor\deviantart.pyc", line 112, in items
  File "gallery_dl\extractor\deviantart.pyc", line 751, in prepare
  File "gallery_dl\extractor\deviantart.pyc", line 197, in prepare
KeyError: 'category'

@mikf
Copy link
Owner

mikf commented Sep 30, 2024

I hope I did nothing wrong.

You didn't, don't worry. If anything it was the DA devs who broke their API and caused the empty journal/status/literature issue to begin with, and me who implemented a bad and buggy workaround for it (#6196, #6207).

There is currently no fix. You could revert to v1.27.4 to get rid of the exception, but the downloaded literature items would all be empty.

@BlueSpiders99

[gallery-dl][debug] Version 1.25.8 - Executable

You are using an outdated version. This was fixed in v1.27.3 (269b280)

@mikf mikf added this to v1.27.6 Sep 30, 2024
@mikf mikf moved this to DeviantArt Journals/Literature in v1.27.6 Sep 30, 2024
@mikf mikf closed this as completed by moving to DeviantArt Journals/Literature in v1.27.6 Sep 30, 2024
@mikf mikf reopened this Sep 30, 2024
@mikf
Copy link
Owner

mikf commented Sep 30, 2024

Never mind, found the problem:

gallery-dl/gallery_dl/extractor/deviantart.py

Line 188 in f3f2749

"journal",

This needs to be "art" or DA complains:

{
  "error": "invalid_request",
  "errorDescription": "Unknown user/deviationid combination.",
  "errorCode": 404,
  "status": "error"
}

@sophi228
Copy link
Author

Wow, congrats! I'm using the version directly from the GitHub using the Pip installation. Is there a way to update this myself or I will have to wait until the next update? Thanks in advance and I hope this isn't a dumb question

mikf added a commit that referenced this issue Oct 1, 2024
@mikf
[deviantart] fix & improve journal/literature extraction (#6254, #6207)
ed859f0 
fetch text from HTML __INITIAL_STATE__,
since the API doesn't reliably work and is unusable for sta.sh journals
@sophi228
Copy link
Author

sophi228 commented Oct 1, 2024

Thanks for the fix! I tested with 2 different accounts and they both worked like a charm! <3

mikf added a commit that referenced this issue Oct 2, 2024
@mikf
[deviantart] extract journal HTML from webpage (#6254, #6207, #6196)
7dbd53e
@mikf
Copy link
Owner

mikf commented Oct 2, 2024

The solution from 7dbd53e (get journal/literature HTML directly from webpage) should be better and more accurate than the previous __INITIAL_STATE__ markup.

@mikf mikf closed this as completed Oct 2, 2024
@rewrib
Copy link

rewrib commented Nov 9, 2024 •
edited
Loading

@mikf I think paywalled literature might still be affected by this. Do you have an idea what might be the issue here? Otherwise I will try to debug this myself, as this will hard to reproduce, if you don't have access to paid literature yourself.

Traceback (most recent call last):
  File "/home/user/gallery_dl/job.py", line 151, in run
    for msg in extractor:
               ^^^^^^^^^
  File "/home/user/gallery_dl/extractor/deviantart.py", line 180, in items
    journal = self._extract_journal(deviation)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/gallery_dl/extractor/deviantart.py", line 391, in _extract_journal
    html = self._textcontent_to_html(deviation, content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/gallery_dl/extractor/deviantart.py", line 402, in _textcontent_to_html
    markup = html["markup"]
             ~~~~^^^^^^^^^^
KeyError: 'markup'

@mikf
Copy link
Owner

mikf commented Nov 9, 2024

You'd theoretically need to pass logged in cookies to be able to access paywalled literature, but you currently can't as cookies are generally disabled for OAuth sites including DA:

gallery-dl/gallery_dl/extractor/deviantart.py

Line 34 in 1ddbcda

cookies_domain = None

(Removing this line allows cookies)

@rewrib
Copy link

rewrib commented Nov 10, 2024

You'd theoretically need to pass logged in cookies to be able to access paywalled literature, but you currently can't as cookies are generally disabled for OAuth sites including DA:

gallery-dl/gallery_dl/extractor/deviantart.py

Line 34 in 1ddbcda

cookies_domain = None

(Removing this line allows cookies)

Thanks that fixed it for me

@mikf
Copy link
Owner

mikf commented Nov 10, 2024 •
edited
Loading

@rewrib Passing cookies is now possible without code modifications and the exception should be fixed as well: 7916c8b

@mikf mikf mentioned this issue Nov 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
regression site:bug
Projects
No open projects
v1.27.6
Status: DeviantArt Journals/Literature
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mikf @rewrib @BlueSpiders99 @sophi228