IP pools: disallow deleting fleet-wide default pool?

[david-crespo]

We create a default pool at the fleet level at rack init. In #3985 (comment) and the comment below I mentioned the possibility that you could delete the default fleet-wide pool and therefore have nothing to fall back to if there is no silo default.

omicron/nexus/db-queries/src/db/datastore/ip_pool.rs

Lines 75 to 80 in 01e730a

	/// Look up the default IP pool for the current silo. If there is no default
	/// at silo scope, fall back to the next level up, namely the fleet default.
	/// There should always be a default pool at the fleet level, though this
	/// query can theoretically fail if someone is able to delete that pool or
	/// make another one the default and delete that.
	pub async fn ip_pools_fetch_default(

A simple way to prevent this is to require there to always be a fleet-wide default pool by:

• disallowing deletion of a fleet-wide default
• only allowing make default on a fleet-wide pool, but not remove default (see IP pools: change/remove default #4002)

However, it may be that we actually want to allow operators to remove the default fleet-level pool, thereby forcing end users to either use a silo default (if one is configured) or always explicitly specify a pool at instance create. We will have to listen to customers on this based on their experience with IP pools as currently implemented.