Restricting access for automated bots

[rgueldenpfennig]

Hello dear Squidlr users!

Since I implemented the support of additional social media platforms, I noticed that certain bots are using the Squidlr website to crawl the content of various platforms (especially Instagram, Facebook and TikTok).
It seems that the access is automated.

So the Squidlr crawler is being crawled 😏 I already expected something like that and in general I don't care much about it. As long as the access is not excessive. But currently those bots produce roughly 40% of the traffic, that Squidlr handles daily.

I already adjusted the rate limiting per IP. So you won't be able do send a dozen of requests to Squidlr in a short period of time. The usage of an "average" internet user, who wants to download some videos, is still possible, of course.

But I need to tackle the bots. Why? Because hosting Squidlr is not for free. The needed infrastructure costs money, especially when it's usage increases.

tl;dr
I will begin to restrict the access to bots in the upcoming weeks by implementing various measurements.

If you are an owner of the affected bots or automatic services and you want to find a solution, you can get in touch with me.

So long, I wish everyone a good time using Squidlr! 📺

Robin