Skip to content

Commit

Permalink
Merge branch 'develop' into issues/488
Browse files Browse the repository at this point in the history
  • Loading branch information
jantman authored Dec 7, 2020
2 parents bc217b6 + 21b73e8 commit c51d822
Show file tree
Hide file tree
Showing 11 changed files with 997 additions and 141 deletions.
23 changes: 21 additions & 2 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,16 @@ Changelog
10.0.0 (2020-12-07)
-------------------

**Important:** This release makes significant changes to how Trusted Advisor is used.
IMPORTANT - Breaking Changes
++++++++++++++++++++++++++++

* This release makes significant changes to how Trusted Advisor is used; see below.
* This release requires the following new IAM permissions: ``eks:ListClusters``, ``eks:DescribeCluster``, ``eks:ListNodegroups``, ``eks:ListFargateProfiles``, ``eks:DescribeFargateProfile``, ``kinesis:DescribeLimits``.
* This release introduces a number of new limits, as well as new services. Please see below for details.
* This release **removes** the ``EC2/Security groups per VPC`` limit, which no longer exists, and adds the new ``EC2/VPC security groups per Region`` limit.

All Changes
+++++++++++

* `Issue #466 <https://github.com/jantman/awslimitchecker/issues/466>`__ - **Significant** changes to Trusted Advisor support.

Expand All @@ -18,7 +27,17 @@ Changelog
* My current intent is to leave Trusted Advisor support in this state until Service Quotas is available in China and GovCloud, at which point I plan on completely removing all Trusted Advisor support.
* Migrate CI builds from travis-ci.org to travis-ci.com.
* `Issue #503 <https://github.com/jantman/awslimitchecker/issues/503>`__ - Fix ``Units set to "None"`` error when retrieving load balancer data from Service Quotas. We now allow the (A|E)LB per Region quota with a unit of either "Count" (prior to November 2020) or "None" (November 2020 on).
* `PR #490 <https://github.com/jantman/awslimitchecker/pull/490>`__ - Add missing RDS limits: ``Manual Cluster Snapshots``, ``Custom Endpoints Per DB Cluster``, ``DB Instance Roles``, and ``DB Cluster Roles``. Thanks to `sebasrp <https://github.com/sebasrp>`__ for this contribution!
* `Issue #489 <https://github.com/jantman/awslimitchecker/issues/489>`__ / `PR #490 <https://github.com/jantman/awslimitchecker/pull/490>`__ - Add missing RDS limits: ``Manual Cluster Snapshots``, ``Custom Endpoints Per DB Cluster``, ``DB Instance Roles``, and ``DB Cluster Roles``. Thanks to `sebasrp <https://github.com/sebasrp>`__ for this contribution!
* `Issue #472 <https://github.com/jantman/awslimitchecker/issues/472>`__ / `PR #494 <https://github.com/jantman/awslimitchecker/pull/494>`__ - Add support for the ``EKS`` service, and 8 new limits for it. Thanks to `sebasrp <https://github.com/sebasrp>`__ for this contribution!
* `Issue #495 <https://github.com/jantman/awslimitchecker/issues/495>`__ / `PR #496 <https://github.com/jantman/awslimitchecker/pull/496>`__ - Add support for the ``Kinesis`` service, and one new limit for it. Thanks to `sebasrp <https://github.com/sebasrp>`__ for this contribution!
* `PR #499 <https://github.com/jantman/awslimitchecker/pull/499>`__ - Set quota_name for VPC "Entries per route table" limit, so that the current limit will be automatically retrieved from Service Quotas. Thanks to `patuck <https://github.com/patuck>`__ for this contribution!
* `Issue #498 <https://github.com/jantman/awslimitchecker/issues/498>`__ - Fix multiple issues relating to VPC limits:

* Update the EC2 / ``Rules per VPC security group`` limit to support retrieving the current limit value from Service Quotas.
* Remove the ``EC2/Security groups per VPC`` limit, which no longer exists.
* Add the new ``EC2/VPC security groups per Region`` limit.

* `Issue #501 <https://github.com/jantman/awslimitchecker/issues/501>`__ - Update ``VPC/Network interfaces per Region`` limit for new calculation method.

.. _changelog.9_0_0:

Expand Down
2 changes: 2 additions & 0 deletions awslimitchecker/services/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,10 @@
from awslimitchecker.services.elasticache import _ElastiCacheService
from awslimitchecker.services.elasticbeanstalk import _ElasticBeanstalkService
from awslimitchecker.services.elb import _ElbService
from awslimitchecker.services.eks import _EksService
from awslimitchecker.services.firehose import _FirehoseService
from awslimitchecker.services.iam import _IamService
from awslimitchecker.services.kinesis import _KinesisService
from awslimitchecker.services.lambdafunc import _LambdaService
from awslimitchecker.services.rds import _RDSService
from awslimitchecker.services.redshift import _RedshiftService
Expand Down
24 changes: 13 additions & 11 deletions awslimitchecker/services/ec2.py
Original file line number Diff line number Diff line change
Expand Up @@ -640,12 +640,12 @@ def _get_limits_spot(self):
def _find_usage_networking_sgs(self):
"""calculate usage for VPC-related things"""
logger.debug("Getting usage for EC2 VPC resources")
sgs_per_vpc = defaultdict(int)
sg_count = 0
rules_per_sg = defaultdict(int)
for sg in self.resource_conn.security_groups.all():
if sg.vpc_id is None:
continue
sgs_per_vpc[sg.vpc_id] += 1
sg_count += 1
"""
see: https://github.com/jantman/awslimitchecker/issues/431
Expand Down Expand Up @@ -676,12 +676,10 @@ def _find_usage_networking_sgs(self):
)
rules_per_sg[sg.id] = max(counts)
# set usage
for vpc_id, count in sgs_per_vpc.items():
self.limits['Security groups per VPC']._add_current_usage(
count,
aws_type='AWS::EC2::VPC',
resource_id=vpc_id,
)
self.limits['VPC security groups per Region']._add_current_usage(
sg_count,
aws_type='AWS::EC2::SecurityGroup',
)
for sg_id, count in rules_per_sg.items():
self.limits['Rules per VPC security group']._add_current_usage(
count,
Expand Down Expand Up @@ -727,14 +725,16 @@ def _get_limits_networking(self):
:rtype: dict
"""
limits = {}
limits['Security groups per VPC'] = AwsLimit(
'Security groups per VPC',
limits['VPC security groups per Region'] = AwsLimit(
'VPC security groups per Region',
self,
500,
2500,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EC2::SecurityGroup',
limit_subtype='AWS::EC2::VPC',
quotas_name='VPC security groups per Region',
quotas_service_code='vpc'
)
limits['Rules per VPC security group'] = AwsLimit(
'Rules per VPC security group',
Expand All @@ -744,6 +744,8 @@ def _get_limits_networking(self):
self.critical_threshold,
limit_type='AWS::EC2::SecurityGroup',
limit_subtype='AWS::EC2::VPC',
quotas_name='Inbound or outbound rules per security group',
quotas_service_code='vpc'
)
limits['VPC Elastic IP addresses (EIPs)'] = AwsLimit(
'VPC Elastic IP addresses (EIPs)',
Expand Down
254 changes: 254 additions & 0 deletions awslimitchecker/services/eks.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
"""
awslimitchecker/services/eks.py
The latest version of this package is available at:
<https://github.com/jantman/awslimitchecker>
################################################################################
Copyright 2015-2018 Jason Antman <jason@jasonantman.com>
This file is part of awslimitchecker, also known as awslimitchecker.
awslimitchecker is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
awslimitchecker is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with awslimitchecker. If not, see <http://www.gnu.org/licenses/>.
The Copyright and Authors attributions contained herein may not be removed or
otherwise altered, except to add the Author attribution of a contributor to
this work. (Additional Terms pursuant to Section 7b of the AGPL v3)
################################################################################
While not legally required, I sincerely request that anyone who finds
bugs please submit them at <https://github.com/jantman/awslimitchecker> or
to me via email, and that you send any contributions or improvements
either as a pull request on GitHub, or to me via email.
################################################################################
AUTHORS:
Jason Antman <jason@jasonantman.com> <http://www.jasonantman.com>
################################################################################
"""

import abc # noqa
import logging

from .base import _AwsService
from ..limit import AwsLimit
from ..utils import paginate_dict

logger = logging.getLogger(__name__)


class _EksService(_AwsService):

service_name = 'EKS'
api_name = 'eks' # AWS API name to connect to (boto3.client)
quotas_service_code = 'eks'

def find_usage(self):
"""
Determine the current usage for each limit of this service,
and update corresponding Limit via
:py:meth:`~.AwsLimit._add_current_usage`.
"""
logger.debug("Checking usage for service %s", self.service_name)
self.connect()
for lim in self.limits.values():
lim._reset_usage()
self._find_clusters_usage()
self._have_usage = True
logger.debug("Done checking usage.")

def _find_clusters_usage(self):
clusters_info = paginate_dict(
self.conn.list_clusters,
alc_marker_path=['nextToken'],
alc_data_path=['clusters'],
alc_marker_param='nextToken'
)

cluster_list = clusters_info['clusters']

for cluster in cluster_list:
describe_cluster_response = self.conn.describe_cluster(
name=cluster
)
security_group_id_list = describe_cluster_response['cluster'][
'resourcesVpcConfig']['securityGroupIds']
self.limits[
'Control plane security groups per cluster']._add_current_usage(
len(security_group_id_list),
resource_id=cluster,
aws_type='AWS::EKS::Cluster'
)
public_access_cidrs_list = describe_cluster_response['cluster'][
'resourcesVpcConfig']['publicAccessCidrs']
self.limits[
'Public endpoint access CIDR ranges per cluster'
]._add_current_usage(
len(public_access_cidrs_list),
resource_id=cluster,
aws_type='AWS::EKS::Cluster'
)

list_nodegroup_response = paginate_dict(
self.conn.list_nodegroups,
clusterName=cluster,
alc_marker_path=['nextToken'],
alc_data_path=['nodegroups'],
alc_marker_param='nextToken'
)
nodegroup_list = list_nodegroup_response['nodegroups']
self.limits['Managed node groups per cluster']._add_current_usage(
len(nodegroup_list),
resource_id=cluster,
aws_type='AWS::EKS::Cluster')

list_fargate_profiles_response = paginate_dict(
self.conn.list_fargate_profiles,
clusterName=cluster,
alc_marker_path=['nextToken'],
alc_data_path=['fargateProfileNames'],
alc_marker_param='nextToken'
)
fargate_profiles_list = list_fargate_profiles_response[
'fargateProfileNames'
]
self.limits['Fargate profiles per cluster']._add_current_usage(
len(fargate_profiles_list),
resource_id=cluster,
aws_type='AWS::EKS::FargateProfile')

for fargate_profile_name in fargate_profiles_list:
fargate_info = self.conn.describe_fargate_profile(
clusterName=cluster,
fargateProfileName=fargate_profile_name
)
profile_selectors = fargate_info['fargateProfile']['selectors']
self.limits['Selectors per Fargate profile']._add_current_usage(
len(profile_selectors),
resource_id="{}.{}".format(cluster, fargate_profile_name),
aws_type='AWS::EKS::FargateProfile')

for selector in profile_selectors:
label_pairs = selector['labels']
self.limits[
'Label pairs per Fargate profile selector'
]._add_current_usage(
len(label_pairs),
resource_id=(
"{}.{}.{}".format(
cluster,
fargate_profile_name,
selector
)
),
aws_type='AWS::EKS::FargateProfile')

self.limits['Clusters']._add_current_usage(
len(cluster_list),
resource_id=self._boto3_connection_kwargs['region_name'],
aws_type='AWS::EKS::Cluster')

def get_limits(self):
"""
Return all known limits for this service, as a dict of their names
to :py:class:`~.AwsLimit` objects.
:returns: dict of limit names to :py:class:`~.AwsLimit` objects
:rtype: dict
"""
if self.limits != {}:
return self.limits
limits = {}
limits['Clusters'] = AwsLimit(
'Clusters',
self,
100,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EKS::Cluster',
)
limits['Control plane security groups per cluster'] = AwsLimit(
'Control plane security groups per cluster',
self,
4,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EKS::Cluster',
)
limits['Managed node groups per cluster'] = AwsLimit(
'Managed node groups per cluster',
self,
30,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EKS::Nodegroup',
)
limits['Nodes per managed node group'] = AwsLimit(
'Nodes per managed node group',
self,
100,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EKS::Cluster',
)
limits['Public endpoint access CIDR ranges per cluster'] = AwsLimit(
'Public endpoint access CIDR ranges per cluster',
self,
40,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EKS::Cluster',
)
limits['Fargate profiles per cluster'] = AwsLimit(
'Fargate profiles per cluster',
self,
10,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EKS::FargateProfile',
)
limits['Selectors per Fargate profile'] = AwsLimit(
'Selectors per Fargate profile',
self,
5,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EKS::FargateProfile',
)
limits['Label pairs per Fargate profile selector'] = AwsLimit(
'Label pairs per Fargate profile selector',
self,
5,
self.warning_threshold,
self.critical_threshold,
limit_type='AWS::EKS::FargateProfile',
)
self.limits = limits
return limits

def required_iam_permissions(self):
"""
Return a list of IAM Actions required for this Service to function
properly. All Actions will be shown with an Effect of "Allow"
and a Resource of "*".
:returns: list of IAM Action strings
:rtype: list
"""
return [
"eks:ListClusters",
"eks:DescribeCluster",
"eks:ListNodegroups",
"eks:ListFargateProfiles",
"eks:DescribeFargateProfile",
]
Loading

0 comments on commit c51d822

Please sign in to comment.