Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JAVLibrary scrape not working. #338

Closed
trestlesky opened this issue Oct 8, 2020 · 3 comments
Closed

JAVLibrary scrape not working. #338

trestlesky opened this issue Oct 8, 2020 · 3 comments
Labels

Comments

@trestlesky
Copy link

Describe the bug
As of yesterday, it looks like JavLibrary scrapes are not working. As a preface, I have a python script that runs before downloading any torrent files. This script takes the JAV ID in the torrent and attempts to fetch a user rating from JavLibrary. It then downloads the torrent if it's above a certain rating. Monday evening, I found that script stopped working because JavLibrary either enabled or increased Cloudfare protection (my request.get in my script started returning 503s). I haven't been able to get that script to work since. Now, I had some JAV video files already downloaded that sill needed to be scraped. I found that I wasn't getting any response scraping with JavLibrary. I wonder if it's connected somehow? I'm no Java expert by any means, but I'm assuming it's possible for Cloudfare to thwart programs like this?

Give an example
Any file scraped with just JavLibrary comes up blank. I even tested older files that had already been successfully scraped (by renaming them back to the original ID) with no success.

@trestlesky trestlesky added the bug label Oct 8, 2020
@yamadasgamer
Copy link

yamadasgamer commented Oct 10, 2020

same (cloudflare protection web)
this is the error in the log

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403, URL=http://www.r18.com/common/search/floor=movies/searchword=SDAB00004/
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:178)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:167)
at moviescraper.doctord.controller.siteparsingprofile.specific.R18ParsingProfile.searchResultOnR18(R18ParsingProfile.java:402)
at moviescraper.doctord.controller.siteparsingprofile.specific.R18ParsingProfile.getSearchResults(R18ParsingProfile.java:436)
at moviescraper.doctord.model.Movie.scrapeMovie(Movie.java:773)
at moviescraper.doctord.controller.amalgamation.ScrapeAmalgamatedMovieWorker$1.doInBackground(ScrapeAmalgamatedMovieWorker.java:180)
at moviescraper.doctord.controller.amalgamation.ScrapeAmalgamatedMovieWorker$1.doInBackground(ScrapeAmalgamatedMovieWorker.java:169)
at javax.swing.SwingWorker$1.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at javax.swing.SwingWorker.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

@capt1234
Copy link

capt1234 commented Oct 19, 2020

Right now Javlibrary is working again! Time: 4.30 PM (German Time), Date: 19th October 2020.

I am using the fork from User zuko7177 from the post in Issue #335.

@trestlesky
Copy link
Author

Right now Javlibrary is working again! Time: 4.30 PM (German Time), Date: 19th October 2020.

I am using the fork from User zuko7177 from the post in Issue #335.

Can confirm, seems to be working again. I don't see anything different in the source though, so it's possible JavLibrary relaxed their cloudfare policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants