Twitter: detect when account is blocked

[trip54654]

Twitter will occasionally block your account for scraping. (Until now I could always unblock it by solving their puzzle-captcha.) When this happens gallery-dl thinks that it couldn't find the twitter user it's trying to access.

It would be great if it could detect getting blocked and return a special error code for it.

[mikf]

I would need Twitter's API response when this happens to know what to check for.
Could you (or someone else) post the last file generated with --write-pages for this situation?

[trip54654]

I did that, and it wrote a file with these contents:

{"errors":[{"message":"Authorization: Denied by access control: To protect our users from spam and other malicious activity, this account is temporarily locked. Please log in to https://twitter.com to unlock your account.; Missing LdapGroup(weaverbird-admins)","locations":[{"line":3,"column":5}],"path":["user","result"],"extensions":{"name":"AuthorizationError","source":"Client","bounce":{"bounce_location":"https://twitter.com/account/access","sub_error_code":0},"code":326,"kind":"Permissions","tracing":{"trace_id":"12af2b68b62ae10d"}},"bounce":{"bounce_location":"https://twitter.com/account/access","sub_error_code":0},"code":326,"kind":"Permissions","name":"AuthorizationError","source":"Client","tracing":{"trace_id":"12af2b68b62ae10d"}}],"data":{"user":{}}}

[trip54654]

By the way, if anyone is curious: I could unblock it and continue scraping by logging in with the browser and solving the captcha thing.

[mikf]

This error now gets a proper error message and you can tell it to wait until you unblocked your account (although I don't know if Twitter will actually let you continue). (d53db6e)

[trip54654]

Thank you!

[trip54654]

@mikf I noticed that this returns the same error code as some other failures. For example, if a user's tweets are protected, it also returns 16. Could this get a separate exit code? So far I'm not trying to parse gallery-dl output, because that would mess with download process display etc., so I only have the exit code.

[Technotron21]

By the way, if anyone is curious: I could unblock it and continue scraping by logging in with the browser and solving the captcha thing.

Were there more steps to it? Because after I initially unblocked it on my browser, it still acts like it couldn't authenticate me, even when I tried logging in from different browsers. Is there something in the config files that I might be missing?

[trip54654]

For example, if a user's tweets are protected, it also returns 16.

I'm solving this for now by assuming the account is blocked if it doesn't work for other random users 3 times in a row. The probability that 3 random users have protected their tweets is low enough.

Were there more steps to it? Because after I initially unblocked it on my browser, it still acts like it couldn't authenticate me, even when I tried logging in from different browsers.

When I can access an account from my browser, gallery-dl also works. Maybe you have to delete the gallery-dl cache file to force gallery-dl to login again.

[mikf]

@trip54654 You can set a custom exit code with something like

"actions": {"error:Account temporarily locked": "status = 127"}