Net Scan: 72TB BAB&anOptions, 210 TB app_vars

[jspenguin2017]

I'm not sure why but I can't manage to keep 10 worker children alive, so I'm downscaling my cluster to 4 worker servers (8 children), I finished processing 72 TB out of 210 TB of data.

In order to prevent maintainers from burning out, I will now add a daily cap on how many links I post:

BlockAdBlock  500
app_vars      10
Other         100

In addition, I will not post more than 500 links combined per day.

---

Note that some domains may be NSFW.

[jspenguin2017]

@gwarser
I thought we are sorting alphabetically?

The Internet is much much bigger than 1 million websites, it's close to 2 billion. I use Common Crawl, which (I think) crawls top 40 million websites or something.

Maybe it's time for generic scriptlet?

[okiehsch]

http://ad1.ink/GUHa3Za is a parked domain.
http://www.binnenland.nl the domain is for sale.

[jspenguin2017]

Whitelisted, and whitelisted.

[okiehsch]

Can you test http://3050.pw/, I see literally nothing.

[jspenguin2017]

Yea, the website is a bit broken, but BAB still works...

[jspenguin2017]

It's up to you, if you think it's not worth fixing, I'll just whitelist it.

[okiehsch]

Yes, I meant nothing until BAB kicks in 😆.
I will pass, and add the rest, I tested approximately every 5th domain.

[jspenguin2017]

Alright, whitelisted.

I'm still working on my Chromium as a Service, it refuses to load extensions in headless mode and I can't find a working tutorial to make EC2 headful...

[jspenguin2017]

Main type (FAB, anOptions, etc.) is from the scanner output, which may not be accurate.
Sub type (hard / soft) is tested.

[okiehsch]

We should probably add a warning that some of the listed domains may be NSFW.

Main type (FAB, anOptions, etc.) is from the scanner output, which may not be accurate.

Yes, I fixed https://www.1x1trainer.net and the site was not using FAB.

[jspenguin2017]

OK, updated opening.

[okiehsch]

https://www.anton-hilft.de is not using anOptions, it uses antiblock.org.
No other false detections in the first "hard batch".

[okiehsch]

Second "hard batch" domains are all valid.

[jspenguin2017]

Well yea, I tested those ones.

I'll be testing everything from now on, either manually or semi-automated with Puppeteer.

[dumbusernameidk]

didn't want to open an issue since it's related to this.

nextdoordolls.com (nsfw) uses anOptions as well. Never got it to trigger but probably should add.