feat(helm): Support HPA for supersetNode and supersetWorker

[tenkian4]

SUMMARY

Support auto-scaling the supersetNode and supersetWorker using HPA with CPU/MEM metrics.
When enabled in helm values, will create HPA objects and omit the static number of replicas from the deployments.
HPA templates have been derived from the standard templates generated using helm create.
The default is disabled.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

HPA Objects are created after the values are configured and feature enabled:

HPA Is reacting to CPU load and increasing replicas:

Pods created

TESTING INSTRUCTIONS

Apply with custom values.yaml and set the following options :

supersetNode:
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 80
    targetMemoryUtilizationPercentage: 80

The same options can be set for supersetWorker
The helm install/upgrade will then create the HPA objects and starts monitoring the resource usage and scale replicas.

To simulate CPU load we added a sidecar which generates load as the HPA averages across all containers in the pods.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[craig-rueda]

Thanks for your contribution! Looks like you're missing License info from the new HPA files, but otherwise LGTM.

[tenkian4]

Thanks for your contribution! Looks like you're missing License info from the new HPA files, but otherwise LGTM.

We have added the licence info copied from one of the other templates to the 2 new files.

Thanks.

[craig-rueda]

I think you're missing newlines at the end of those files as well

[tenkian4]

I think you're missing newlines at the end of those files as well

Added. Thanks.

[fzhan]

Seems flask isn't reading the cache correctly when in HPA mode:

HTTPException
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1788, in dispatch_request
self.raise_routing_exception(req)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1770, in raise_routing_exception
raise request.routing_exception # type: ignore
File "/usr/local/lib/python3.10/site-packages/flask/ctx.py", line 351, in match_request
result = self.url_adapter.match(return_rule=True) # type: ignore
File "/usr/local/lib/python3.10/site-packages/werkzeug/routing/map.py", line 624, in match
raise NotFound() from None
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.
2024-05-07 04:57:06,714:WARNING:superset.views.base:HTTPException
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1788, in dispatch_request
self.raise_routing_exception(req)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1770, in raise_routing_exception
raise request.routing_exception # type: ignore
File "/usr/local/lib/python3.10/site-packages/flask/ctx.py", line 351, in match_request
result = self.url_adapter.match(return_rule=True) # type: ignore
File "/usr/local/lib/python3.10/site-packages/werkzeug/routing/map.py", line 624, in match
raise NotFound() from None
werkzeug.exceptions.NotFound: 404 Not Found: The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.