Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dd-apm): Added initial config for dd apm #1617

Merged
merged 7 commits into from
Jun 9, 2021
Merged

feat(dd-apm): Added initial config for dd apm #1617

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1a6c864
feat(dd-apm): Added initial config for dd apm
May 27, 2021
c444555
feat(dd-apm): Added datadog apm integration to core python services
Jun 7, 2021
acb4e8f
fix expected rendered fence k8s yaml
themarcelor Jun 8, 2021
1c1234c
remove duplicate label
themarcelor Jun 8, 2021
8de850e
fix labels
themarcelor Jun 8, 2021
e6d97d3
fix labels again
themarcelor Jun 9, 2021
907c4a5
fix expected file for sheepdog yaml
themarcelor Jun 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions Docker/python-nginx/python3.6-alpine3.7/dockerrun.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,9 +86,17 @@ if [ -f ./wsgi.py ] && [ "$GEN3_DEBUG" = "True" ]; then
echo -e "\napplication.debug=True\n" >> ./wsgi.py
fi

if [[ -z $DD_ENABLED ]]; then
(
run uwsgi --ini /etc/uwsgi/uwsgi.ini
) &
else
pip install ddtrace
echo "import=ddtrace.bootstrap.sitecustomize" >> /etc/uwsgi/uwsgi.ini
(
ddtrace-run uwsgi --enable-threads --ini /etc/uwsgi/uwsgi.ini
) &
fi

if [[ $GEN3_DRYRUN == "False" ]]; then
(
Expand Down
6 changes: 6 additions & 0 deletions gen3/lib/g3k_manifest.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,13 @@ g3k_manifest_filter() {
# zsh friendly upper case
kvKey=$(echo "GEN3_${key}_IMAGE" | tr '[:lower:]' '[:upper:]')
kvList+=("$kvKey" "image: $value")
kvLabelKey=$(echo "GEN3_${key}_VERSION" | tr '[:lower:]' '[:upper:]')
version=$(echo $value | rev | cut -d ':' -f 1 | rev)
kvList+=("$kvLabelKey" "tags.datadoghq.com/version: '$version'")
done
environment="$(g3k_config_lookup ".global.environment" "$manifestPath")"
kvEnvKey=$(echo "GEN3_ENV_LABEL" | tr '[:lower:]' '[:upper:]')
kvList+=("$kvEnvKey" "tags.datadoghq.com/env: $environment")
for key in $(g3k_config_lookup '. | keys[]' "$manifestPath"); do
gen3_log_debug "harvesting key $key"
for key2 in $(g3k_config_lookup ".[\"${key}\"] "' | to_entries | map(select((.value|type != "array") and (.value|type != "object"))) | map(.key)[]' "$manifestPath" | grep '^[a-zA-Z]'); do
Expand Down
33 changes: 32 additions & 1 deletion gen3/lib/testData/test1.manifest.g3k/expectedFenceResult.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ spec:
netnolimit: "yes"
public: "yes"
userhelper: "yes"
tags.datadoghq.com/service: "fence"
tags.datadoghq.com/env: null
tags.datadoghq.com/version: 'master'
date: "1579711361"
spec:
affinity:
Expand Down Expand Up @@ -99,12 +102,40 @@ spec:
- name: fence
image: quay.io/cdis/fence:master
env:
- name: DD_ENABLED
valueFrom:
configMapKeyRef:
name: manifest-global
key: dd_enabled
optional: true
- name: DD_ENV
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/env']
- name: DD_SERVICE
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/service']
- name: DD_VERSION
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/version']
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_TRACE_SAMPLE_RATE
value: "1"
- name: GEN3_UWSGI_TIMEOUT
valueFrom:
configMapKeyRef:
name: manifest-global
key: uwsgi-timeout
optional: true
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
- name: PYTHONPATH
Expand Down Expand Up @@ -211,7 +242,7 @@ spec:
- |
echo "${FENCE_PUBLIC_CONFIG:-""}" > "/var/www/fence/fence-config-public.yaml"
python /var/www/fence/yaml_merge.py /var/www/fence/fence-config-public.yaml /var/www/fence/fence-config-secret.yaml > /var/www/fence/fence-config.yaml
bash /fence/dockerrun.bash && if [[ -f /dockerrun.sh ]]; then /dockerrun.sh; fi
bash /fence/dockerrun.bash && if [[ -f /dockerrun.sh ]]; then bash /dockerrun.sh; fi
- name: nginx-prometheus-exporter-wrapper
image: quay.io/cdis/nginx-prometheus-exporter-wrapper:pybase3-1.4.0
command: ["/bin/bash"]
Expand Down
33 changes: 32 additions & 1 deletion gen3/lib/testData/test1.manifest.g3k/expectedSheepdogResult.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ spec:
public: "yes"
# to download dictionary
s3: "yes"
tags.datadoghq.com/service: "sheepdog"
tags.datadoghq.com/env: null
tags.datadoghq.com/version: 'master'
date: "1522344212"
spec:
affinity:
Expand Down Expand Up @@ -74,6 +77,34 @@ spec:
- containerPort: 80
- containerPort: 443
env:
- name: DD_ENABLED
valueFrom:
configMapKeyRef:
name: manifest-global
key: dd_enabled
optional: true
- name: DD_ENV
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/env']
- name: DD_SERVICE
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/service']
- name: DD_VERSION
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/version']
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_TRACE_SAMPLE_RATE
value: "1"
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: DICTIONARY_URL
valueFrom:
configMapKeyRef:
Expand Down Expand Up @@ -152,4 +183,4 @@ spec:
memory: 1024Mi
limits:
cpu: 2
memory: 2048Mi
memory: 2048Mi
33 changes: 32 additions & 1 deletion kube/services/fence/fence-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ spec:
netnolimit: "yes"
public: "yes"
userhelper: "yes"
tags.datadoghq.com/service: "fence"
GEN3_ENV_LABEL
GEN3_FENCE_VERSION
GEN3_DATE_LABEL
spec:
affinity:
Expand Down Expand Up @@ -99,12 +102,40 @@ spec:
- name: fence
GEN3_FENCE_IMAGE
env:
- name: DD_ENABLED
valueFrom:
configMapKeyRef:
name: manifest-global
key: dd_enabled
optional: true
- name: DD_ENV
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/env']
- name: DD_SERVICE
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/service']
- name: DD_VERSION
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/version']
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_TRACE_SAMPLE_RATE
value: "1"
Comment on lines +105 to +128
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish we had proper templating using helm or kustomize so we could omit these whenever DD is not enabled.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, true. But it's really just unused env vars at that point right 🤷.

- name: GEN3_UWSGI_TIMEOUT
valueFrom:
configMapKeyRef:
name: manifest-global
key: uwsgi-timeout
optional: true
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
- name: PYTHONPATH
Expand Down Expand Up @@ -211,7 +242,7 @@ spec:
- |
echo "${FENCE_PUBLIC_CONFIG:-""}" > "/var/www/fence/fence-config-public.yaml"
python /var/www/fence/yaml_merge.py /var/www/fence/fence-config-public.yaml /var/www/fence/fence-config-secret.yaml > /var/www/fence/fence-config.yaml
bash /fence/dockerrun.bash && if [[ -f /dockerrun.sh ]]; then /dockerrun.sh; fi
bash /fence/dockerrun.bash && if [[ -f /dockerrun.sh ]]; then bash /dockerrun.sh; fi
- name: nginx-prometheus-exporter-wrapper
GEN3_NGINX_PROMETHEUS_EXPORTER_WRAPPER_IMAGE|-image: quay.io/cdis/nginx-prometheus-exporter-wrapper:pybase3-1.4.0-|
command: ["/bin/bash"]
Expand Down
31 changes: 31 additions & 0 deletions kube/services/indexd/indexd-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ spec:
app: indexd
release: production
public: "yes"
tags.datadoghq.com/service: "indexd"
GEN3_ENV_LABEL
GEN3_INDEXD_VERSION
GEN3_DATE_LABEL
spec:
affinity:
Expand Down Expand Up @@ -63,6 +66,34 @@ spec:
- name: indexd
GEN3_INDEXD_IMAGE
env:
- name: DD_ENABLED
valueFrom:
configMapKeyRef:
name: manifest-global
key: dd_enabled
optional: true
- name: DD_ENV
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/env']
- name: DD_SERVICE
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/service']
- name: DD_VERSION
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/version']
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_TRACE_SAMPLE_RATE
value: "1"
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: GEN3_DEBUG
GEN3_DEBUG_FLAG|-value: "False"-|
- name: DIST
Expand Down
31 changes: 31 additions & 0 deletions kube/services/peregrine/peregrine-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ spec:
public: "yes"
# to download dictionary
s3: "yes"
tags.datadoghq.com/service: "peregrine"
GEN3_ENV_LABEL
GEN3_PEREGRINE_VERSION
GEN3_DATE_LABEL
spec:
affinity:
Expand Down Expand Up @@ -64,6 +67,34 @@ spec:
- containerPort: 80
- containerPort: 443
env:
- name: DD_ENABLED
valueFrom:
configMapKeyRef:
name: manifest-global
key: dd_enabled
optional: true
- name: DD_ENV
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/env']
- name: DD_SERVICE
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/service']
- name: DD_VERSION
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/version']
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_TRACE_SAMPLE_RATE
value: "1"
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: DICTIONARY_URL
valueFrom:
configMapKeyRef:
Expand Down
31 changes: 31 additions & 0 deletions kube/services/pidgin/pidgin-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ spec:
labels:
app: pidgin
public: "yes"
tags.datadoghq.com/service: "pidgin"
GEN3_ENV_LABEL
GEN3_PIDGIN_VERSION
GEN3_DATE_LABEL
spec:
affinity:
Expand All @@ -44,6 +47,34 @@ spec:
- name: pidgin
GEN3_PIDGIN_IMAGE
env:
- name: DD_ENABLED
valueFrom:
configMapKeyRef:
name: manifest-global
key: dd_enabled
optional: true
- name: DD_ENV
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/env']
- name: DD_SERVICE
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/service']
- name: DD_VERSION
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/version']
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_TRACE_SAMPLE_RATE
value: "1"
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: GEN3_DEBUG
GEN3_DEBUG_FLAG|-value: "False"-|
livenessProbe:
Expand Down
33 changes: 32 additions & 1 deletion kube/services/presigned-url-fence/presigned-url-fence-deploy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ spec:
netnolimit: "yes"
public: "yes"
userhelper: "yes"
tags.datadoghq.com/service: "presigned-url-fence"
GEN3_ENV_LABEL
GEN3_FENCE_VERSION
GEN3_DATE_LABEL
spec:
affinity:
Expand Down Expand Up @@ -99,6 +102,34 @@ spec:
- name: fence
GEN3_FENCE_IMAGE
env:
- name: DD_ENABLED
valueFrom:
configMapKeyRef:
name: manifest-global
key: dd_enabled
optional: true
- name: DD_ENV
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/env']
- name: DD_SERVICE
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/service']
- name: DD_VERSION
valueFrom:
fieldRef:
fieldPath: metadata.labels['tags.datadoghq.com/version']
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_PROFILING_ENABLED
value: "true"
- name: DD_TRACE_SAMPLE_RATE
value: "1"
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: NGINX_RATE_LIMIT
value: "6"
- name: PYTHONPATH
Expand Down Expand Up @@ -213,7 +244,7 @@ spec:
nginx_limit=$(cat /fence/fence/config-default.yaml | sed -n -e 's/^.*OVERRIDE_NGINX_RATE_LIMIT: //p')
fi
export OVERRIDE_NGINX_RATE_LIMIT="$nginx_limit"
([[ ! -f /entrypoint.sh ]] || bash /entrypoint.sh) && /fence/dockerrun.bash && if [[ -f /dockerrun.sh ]]; then /dockerrun.sh; fi
([[ ! -f /entrypoint.sh ]] || bash /entrypoint.sh) && /fence/dockerrun.bash && if [[ -f /dockerrun.sh ]]; then bash /dockerrun.sh; fi
- name: nginx-prometheus-exporter-wrapper
GEN3_NGINX_PROMETHEUS_EXPORTER_WRAPPER_IMAGE|-image: quay.io/cdis/nginx-prometheus-exporter-wrapper:pybase3-1.4.0-|
command: ["/bin/bash"]
Expand Down
Loading