[Feature] Add configuration option to use IP Access Rules instead of IP lists to avoid the 10k entries limitation

[aleksandarmomic]

IP lists get filled very often because of the limitation. Is it possible to implement the bouncer to use the IP Access Rules feature in cloudflare instead of the lists? It still has 50k limitation which won't make full soon because now, when connected to the central api, rules hover around 10.

[HawksRepos]

Would be appreciated if this could be implemented. Having a 10k limit on Cloudflare that’s constantly full means that new IP’s are not able to be added from the bouncer

[PhasecoreX]

The nice thing about the list is that everything in the list is managed by Crowdsec, and it's just one entry in the IP Access rules to reference the list. I already have preexisting rules in the IP Access Rules that I wouldn't want to be deleted. Also, if I need to modify my own rules, I don't want to sift through 10k+ rules to find it.

My idea would be to just have it so that the "top 10k" are added to Cloudflare, however that's determined. For example it could be whatever IPs most recently attacked your server, and the rest being the most recently globally banned IPs. Not sure if that can be done or not.

Either way, if this feature is added, I'd want a configuration option to choose where these IPs are being sent to, the existing list or the IP Access Rules.

[ibracorp-sycotix]

Definitely would like to see this happen. Please do implement this.

Cheers from IBRACORP

[sbs2001]

One more advantage of IP access rules would be support for multiple remediations for free accounts. We could somewhat differentiate what's created by crowdsec by attaching note to the rules created by bouncer.

The only blocking issue with using IP access rules, is that there's no way to create them in bulk in one API call.
So for each banned IP, we'll need one API call. For typical crowdsec list we'll need 10k+ calls. And Cloudflare has rate limit of 1200 calls per 5 minutes. We could respect that limit, but it's going to be really really slow.

Other than that cloudflare calls Ip Access Rules as "legacy" at https://blog.cloudflare.com/introducing-ip-lists/#ip-prefix-boundaries

[HawksRepos]

Is it possible to combine the options? so, general blocked IP's from the CrowdSec cluster go on the IP lists, but then additional IP's from one's own server are added via IP Access Rules with a note/description? That way you get around the 10k limit and the API call limitation with IP Access Rules. I doubt IPs will be added all that often, so a few API calls to adjust that list should be easy enough.

[HoudayerPierre]

Hello, I have a new install and the list on CF is empty beacause the limit is already reach. For old install, not having latest IP is not too bad, but for new installs it's not good at all 😟 .

/var/log/crowdsec-cloudflare-bouncer.log:

msg="HTTP status 400: This account has reached the maximum number of items (10020)"

Maybe just add a config (by default ?) to limit at 9500, 9800 ... of latest IPs so we keep a little of range for actuals accessing IPs on our server.

Beacause aving no way to fill the IP list acctually is not great :(

[aleksandarmomic]

The only blocking issue with using IP access rules, is that there's no way to create them in bulk in one API call. So for each banned IP, we'll need one API call. For typical crowdsec list we'll need 10k+ calls. And Cloudflare has rate limit of 1200 calls per 5 minutes. We could respect that limit, but it's going to be really really slow.

But it's going to be slow on first run until all the ips are there. Currently crowdsec doesn't update 1200 ips every minute, so updating will be performant. It's just the initial populating that will take 10 minutes or so. From my viewpoint, that's a non issue. No one starts up crowdsec in a rush to have an immediate protection.

Other than that cloudflare calls Ip Access Rules as "legacy" at https://blog.cloudflare.com/introducing-ip-lists/#ip-prefix-boundaries

As far as I can see they call it legacy because IP lists are newer and have more flexible ip ranges whereas IP access rules have a certain number of them. It should not be a blocker for crowdsec because it bans real ips instead of ip ranges.

One more advantage of IP access rules would be support for multiple remediations for free accounts. We could somewhat differentiate what's created by crowdsec by attaching note to the rules created by bouncer.

That would be an awesome addition!

[LtSich]

@aleksandarmomic : don't forget some people have multiple account to publish banlist.
On my side I have access to 30 cloudflare account, and I update the banlist on each account.
Then now imagine to ban just 20 IP, this mean 600 calls on the API !
And now you not only ban 20 IP, but you unban 10 more... This make 900 call on the API !

Where with the list system this make only 30 calls...

I have 1 cloudflare bouncer on my central server, and I manage every list from here.
With the legacy system I will need to deploy 1 bouncer for each client on each server to update only for the client account...
This mean that I need to ask the client to configure 1 API access, but most of them just don't know how to do that.

Actually I manage everything from my account, each client just give me access with the "members" function.

The IP list is clearly the system to use, maybe Cloudflare need to increase that list size.
Or maybe CS need to adapt and make possible to use multiple list.
Cloudflare allow you 10 list if you take some subscription to at least 1 domain.

Then you should be able to go up to 100k IP on the banlist.
That's probably the best option.

crowdsec_blocklist_1
crowdsec_blocklist_2
....

[aleksandarmomic]

@aleksandarmomic : don't forget some people have multiple account to publish banlist. On my side I have access to 30 cloudflare account, and I update the banlist on each account. Then now imagine to ban just 20 IP, this mean 600 calls on the API ! And now you not only ban 20 IP, but you unban 10 more... This make 900 call on the API !

Where with the list system this make only 30 calls...

I have 1 cloudflare bouncer on my central server, and I manage every list from here. With the legacy system I will need to deploy 1 bouncer for each client on each server to update only for the client account... This mean that I need to ask the client to configure 1 API access, but most of them just don't know how to do that.

Actually I manage everything from my account, each client just give me access with the "members" function.

The IP list is clearly the system to use, maybe Cloudflare need to increase that list size. Or maybe CS need to adapt and make possible to use multiple list. Cloudflare allow you 10 list if you take some subscription to at least 1 domain.

Then you should be able to go up to 100k IP on the banlist. That's probably the best option.

crowdsec_blocklist_1 crowdsec_blocklist_2 ....

That is a good point. I am not proposing to remove the use of lists, but maybe have it set up in the config to use rules or list(s). So we, small self-hosters, have the option to use that. :)

[sbs2001]

Cloudflare allow you 10 list if you take some subscription to at least 1 domain.
Then you should be able to go up to 100k IP on the banlist.

That seems logical, but unfortunately doesn't work that way. From https://developers.cloudflare.com/firewall/cf-firewall-rules/rules-lists#entitlements

Regardless of plan, you can store up to a total of 10,000 items, spread across all of your lists.

So even if you have 10 lists you can only store 10k items atmost. The subscription buys only more lists, but not more overall "capacity". On Fastly bouncer we do something like this, since they allow unlimited lists, so we create 100+ lists and loadbalance/distribute the bad IPs among them. Maybe CF would be inclined to increase the capacity, to match with fastly :)

[ahmaddxb]

Hello, I have a new install and the list on CF is empty beacause the limit is already reach. For old install, not having latest IP is not too bad, but for new installs it's not good at all 😟 .

/var/log/crowdsec-cloudflare-bouncer.log:

msg="HTTP status 400: This account has reached the maximum number of items (10020)"

Maybe just add a config (by default ?) to limit at 9500, 9800 ... of latest IPs so we keep a little of range for actuals accessing IPs on our server.

Beacause aving no way to fill the IP list acctually is not great :(

i have this issue on a new install, how did you fix this?

[HoudayerPierre]

Hello, I have a new install and the list on CF is empty beacause the limit is already reach. For old install, not having latest IP is not too bad, but for new installs it's not good at all 😟 .
/var/log/crowdsec-cloudflare-bouncer.log:
msg="HTTP status 400: This account has reached the maximum number of items (10020)"

Maybe just add a config (by default ?) to limit at 9500, 9800 ... of latest IPs so we keep a little of range for actuals accessing IPs on our server.
Beacause aving no way to fill the IP list acctually is not great :(

i have this issue on a new install, how did you fix this?

No fix, I manually add IPs in the CF list by copy paste from the dashboard / console from time to time, not good but better than nothing...

Hoping that one day we can have a ubiquiti (UDM) bouncer to get CF + UDM firewall bouncers ;)

[ahmaddxb]

Hello, I have a new install and the list on CF is empty beacause the limit is already reach. For old install, not having latest IP is not too bad, but for new installs it's not good at all 😟 .
/var/log/crowdsec-cloudflare-bouncer.log:
msg="HTTP status 400: This account has reached the maximum number of items (10020)"

Maybe just add a config (by default ?) to limit at 9500, 9800 ... of latest IPs so we keep a little of range for actuals accessing IPs on our server.
Beacause aving no way to fill the IP list acctually is not great :(

i have this issue on a new install, how did you fix this?

No fix, I manually add IPs in the CF list by copy paste from the dashboard / console from time to time, not good but better than nothing...

Hoping that one day we can have a ubiquiti (UDM) bouncer to get CF + UDM firewall bouncers ;)

Oh so what is this limit referring too. I just started using this bouncer today and from the very start I have had this error.

P.S (UDM) bouncer would be great.

[sbs2001]

Those facing #77 (comment) , any chance you have multiple accounts in /etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml ?

[ahmaddxb]

Those facing #77 (comment) , any chance you have multiple accounts in /etc/crowdsec/bouncers/crowdsec-cloudflare-bouncer.yaml ?

only have one account id and one zone id

[sbs2001]

Strange, could you try:

1. stop the bouncer,if it's running using sudo systemctl stop crowdsec-cloudflare-bouncer
2. Clear cache sudo /usr/local/bin/crowdsec-cloudflare-bouncer -d
3. Start the bouncer sudo systemctl start crowdsec-cloudflare-bouncer

[ahmaddxb]

Im running cloudflare-bouncer in docker, i tried removing the container and pulling it again but same issue happens

[HoudayerPierre]

Only one account and zone id too.

Stopped the bouncer, cleared the cache, started it again. Same error, HTTP status 400: This account has reached the maximum number of items (10020)

[yashodhank]

I tried reinstalling bouncer with no luck.
initially it worked, when I excluded one of the zone/domain from the api, it started throwing error as follows:

sudo crowdsec-cloudflare-bouncer -d
INFO[18-02-2022 09:30:22] Getting all IP lists                          account_id=b4*
INFO[18-02-2022 09:30:23] ip list crowdsec_challenge already exists     account_id=b4*
INFO[18-02-2022 09:30:23] removing ip list dependencies                 account_id=b4*
INFO[18-02-2022 09:30:23] listing zones                                 account_id=b4*
INFO[18-02-2022 09:30:23] found 8 zones on this account                 account_id=b4*
INFO[18-02-2022 09:30:23] deleting rules containing $crowdsec_challenge  account_id=b4*
INFO[18-02-2022 09:30:25] deleting filters containing $crowdsec_challenge  account_id=b4*
INFO[18-02-2022 09:30:27] deleting ip list crowdsec_challenge           account_id=b4*
FATA[18-02-2022 09:30:27] HTTP status 400: filters.api.ip_list_in_use (10031)
FATA[18-02-2022 09:30:27] HTTP status 400: filters.api.ip_list_in_use (10031)

[sbs2001]

@yashodhank could you try deleting rules containing $crowdsec_challenge from the excluded zone ? The bouncer should take care of this tho. Any chance your excluded zone has lot of firewall rules ?

[sbs2001]

I can confirm #77 (comment) . This is caused because the bulk API call tries to create >10k items in IP list and fails thus nothing ends up in the IP list

[yashodhank]

Thank you

Any chance your excluded zone has lot of firewall rules ?

There is just one firewall rule which allows letsencrypt to validate file in .well-known directory when domain is behind Cloudflare proxy.
Initially when I installed bouncer and ran command till where we start bouncer service, it went well. When I noticed bouncer is not taking into consideration sites/zone/domains that are not even hosted on particular server but all those domains part of cloudflare account have received rules update, I decided to excluded it and added one domain name into API Tokens exclusion list for which I do not wish to have any rules maintained by the bouncer.

could you try deleting rules containing $crowdsec_challenge from the excluded zone ?

I did but somehow script stops there by throwing exact error.

[aleksandarmomic]

What's the current info on implementation of this feature? Just a simple config for those who want to use it, nothing forced. It should still default to lists unless explicitly configured to use rules.

[LtSich]

The IP list from CAPI is so big now that we clearly need to select what to send to CF.
I'm currently making change on my custom CF bouncer to first send the IP blocked on my network, and then only add the most "recent" IP (order by duration DESC) from CAPI and limit the total list to 10k entry...
The good thing could be that CF allow us to maintain bigger IP list....

[aleksandarmomic]

#80 to me looks like a workaround instead of a solution to the issue. Right now it is a plausible solution while the community ip list is just above 10k. But this solution is not future proof. Crowdsec is growing and with it grows the community ip list. I see that because I implemented crowdsec a 3 weeks ago and it started with 9600 ips. Now after 3 weeks it's around 10800. With this growth this solution will very soon become obsolete as we will miss more and more potential attackers. IP Access Rules still look like a more future proof solution, but not for an indefinite amount of time.

[modem7]

Maybe using a scenario like this might be more beneficial? https://crowdsec.net/blog/how-to-beat-application-ddos/
(crowdsecurity/http-ddos-by-ASN)

Potentially leveraging ASN's rather than the full IP address list would lower the amount of entries used.

Alternatively, do some logic to see how many of the IP addresses can be grouped into CIDR ranges and apply that instead (albeit, heavy handed if not careful).

As to the how, or if this even a remotely suitable way to go.......that's another question entirely.

[buixor]

Hello !

The new release of the bouncer v0.0.9 limits the number of IPs to 10k to avoid hitting the issue (and favors the latest decisions). It is mostly a band-aid for now, but at least it hopefully limits the risk of stuff breaking.

The next crowdsec release will add the option to select the relevant scenarios for a bouncer, so that it doesn't waste its list space with for example IPs known for ssh-bruteforce (or other non-relevant scenarios).

I let this issue open as of now to keep track of progress.

[lordraiden]

Hello !

The new release of the bouncer v0.0.9 limits the number of IPs to 10k to avoid hitting the issue (and favors the latest decisions). It is mostly a band-aid for now, but at least it hopefully limits the risk of stuff breaking.

The next crowdsec release will add the option to select the relevant scenarios for a bouncer, so that it doesn't waste its list space with for example IPs known for ssh-bruteforce (or other non-relevant scenarios).

I let this issue open as of now to keep track of progress.

What about options to prioritize the most aggressive or most recent 10k IP's and the ones attacking our server?

Someone mentioned the use of IP Access rules, doesn't this solve the problem for now? it allows 50k rules
https://support.cloudflare.com/hc/en-us/articles/217074967-Configuring-IP-Access-Rules

[buixor]

Hello, IP access rules are considered deprecated after our discussions with cloudflare. However, with 1.3.3 we introduced the ability to filter IPs (on LAPI) based on scenarios, so that we can avoid polluting the lists with ie. ssh-bf IPs.

I do believe it should be enough 👍

I keep open for reference, but will tag as won't fix for now

[lordraiden]

Hello, IP access rules are considered deprecated after our discussions with cloudflare. However, with 1.3.3 we introduced the ability to filter IPs (on LAPI) based on scenarios, so that we can avoid polluting the lists with ie. ssh-bf IPs.

I do believe it should be enough 👍

I keep open for reference, but will tag as won't fix for now

Is documented somewhere how to do this? Is this configuration attached or linked to a bouncer or it applies to all? Thanks

[LaurenceJJones]

I have updated the config with the relevant keys and short docs on how to apply the filter. Closing issue for now.