Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add configs for Presto #204

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions deployment/sandbox/flyte_generated.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -641,6 +641,33 @@ metadata:
namespace: flyte
---
apiVersion: v1
data:
config.yaml: |
plugins:
presto:
environment: ""
defaultRoutingGroup: "adhoc"
defaultUser: "flyte-default-user"
routingGroupConfigs:
- name: "adhoc"
limit: 250
projectScopeQuotaProportionCap: 0.7
namespaceScopeQuotaProportionCap: 0.7
- name: "etl"
limit: 100
projectScopeQuotaProportionCap: 0.7
namespaceScopeQuotaProportionCap: 0.7
refreshCacheConfig:
name: "presto"
syncPeriod: 5s
workers: 15
lruCacheSize: 2000
kind: ConfigMap
metadata:
name: flyte-presto-config-h6cfm99ftc
namespace: flyte
---
apiVersion: v1
data:
config.yaml: |
propeller:
Expand Down Expand Up @@ -1147,6 +1174,8 @@ spec:
ephemeral-storage: 100Mi
memory: 100Mi
volumeMounts:
- mountPath: /etc/flyte/config-presto
name: presto-config-volume
- mountPath: /etc/flyte/config-qubole
name: qubole-config-volume
- mountPath: /etc/flyte/config-container
Expand All @@ -1159,6 +1188,9 @@ spec:
name: plugin-config-volume
serviceAccountName: flytepropeller
volumes:
- configMap:
name: flyte-presto-config-h6cfm99ftc
name: presto-config-volume
- configMap:
name: flyte-qubole-config-9tcd8mk2c2
name: qubole-config-volume
Expand Down
4 changes: 4 additions & 0 deletions kustomize/overlays/sandbox/propeller/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,14 @@ configMapGenerator:
- name: flyte-qubole-config
files:
- plugins/qubole/config.yaml
- name: flyte-presto-config
files:
- plugins/presto/config.yaml

patches:
- propeller.yaml
# add the volumemount for each plugin configmap
- plugins/spark/propeller-patch.yaml
- plugins/container/propeller-patch.yaml
- plugins/qubole/propeller-patch.yaml
- plugins/presto/propeller-patch.yaml
19 changes: 19 additions & 0 deletions kustomize/overlays/sandbox/propeller/plugins/presto/config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
plugins:
presto:
environment: ""
defaultRoutingGroup: "adhoc"
defaultUser: "flyte-default-user"
routingGroupConfigs:
- name: "adhoc"
limit: 250
projectScopeQuotaProportionCap: 0.7
namespaceScopeQuotaProportionCap: 0.7
- name: "etl"
limit: 100
projectScopeQuotaProportionCap: 0.7
namespaceScopeQuotaProportionCap: 0.7
refreshCacheConfig:
name: "presto"
syncPeriod: 5s
workers: 15
lruCacheSize: 2000
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# This file is only for volume mounts. The configmap itself that's being mounted is sufficiently different that
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote this comment at one point I think. At the time, it made sense. At this point, I have no idea what it means.

# there's no benefit to having it in this folder, since the entire thing gets overridden anyways.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: flytepropeller
namespace: flyte
spec:
template:
spec:
volumes:
- name: presto-config-volume
configMap:
name: flyte-presto-config
containers:
- name: flytepropeller
volumeMounts:
- name: presto-config-volume
mountPath: /etc/flyte/config-presto