-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all: secrets cannot contain $$
or ${SOMETEXT}
#188377
Comments
Pinging @elastic/elastic-agent (Team:Elastic-Agent) |
Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane) |
A simpler reproduction using just the output section of the policy: outputs:
default:
type: elasticsearch
hosts: [127.0.0.1:9200]
api_key: "example-key"
#username: "elastic"
#password: "changeme"
preset: balanced
dollar_signs: $$$$ Gets transformed into the following in the output of outputs:
default:
api_key: example-key
hosts:
- 127.0.0.1:9200
preset: balanced
dollar_signs: $$
type: elasticsearch Edit: switched use of |
I had thought the transpiler and variable substation was to blame here but I think these are getting eaten by go-ucfg. |
Reproduction isolated to a unit test: diff --git a/internal/pkg/config/config_test.go b/internal/pkg/config/config_test.go
index 5d393d3486..0c02ea9079 100644
--- a/internal/pkg/config/config_test.go
+++ b/internal/pkg/config/config_test.go
@@ -20,6 +20,19 @@ func TestConfig(t *testing.T) {
testLoadFiles(t)
}
+func TestToMapStrWithDollarSigns(t *testing.T) {
+ in := map[string]interface{}{
+ "hello": map[string]interface{}{
+ "what": "$$$$",
+ },
+ }
+
+ c := MustNewConfigFrom(in)
+ out, err := c.ToMapStr()
+ assert.NoError(t, err)
+ assert.Equal(t, in, out)
+}
+
func TestInputsResolveNOOP(t *testing.T) {
contents := map[string]interface{}{
"outputs": map[string]interface{}{
(END) Output:
|
Caused by go-ucfg variable expansion I believe, the following change makes the test above pass: index b9524eafd1..85020664c1 100644
--- a/internal/pkg/config/config.go
+++ b/internal/pkg/config/config.go
@@ -36,7 +36,7 @@ func VarSkipKeys(keys ...string) Option {
var DefaultOptions = []interface{}{
ucfg.PathSep("."),
ucfg.ResolveEnv,
- ucfg.VarExp,
+ // ucfg.VarExp,
VarSkipKeys("inputs"),
ucfg.IgnoreCommas,
}
(END) |
This PR in go-ucfg seems to have introduced this behavior. https://github.com/elastic/go-ucfg/pull/120/files I suspect this is a bug in the go-ucfg lexer which appears to be hand written. |
I see a test case in https://github.com/elastic/go-ucfg/blob/5edd2c2d96ddaa8e5ba1b5fd48fd1ebe6ce25432/variables_test.go#L27-L46 that would break if we simply removed this. {"string with escaped var", "escaped $${var}", str("escaped ${var}")}, I don't want to over focus on Really what needs to happen is all secret values need to be treated as literal strings with none of go-ucfg's processing applied. |
Something previously proposed is a dedicated secrets provider so the values only show up after variable substitution: outputs:
default:
type: elasticsearch
api_key: ${secrets.<uuid>}
providers:
secrets:
<uuid>: <value>
inputs:
- id: 'asdf'
type: httpjson
headers:
Authorization: Bearer ${secrets.<uuid>} Having a secret provider like this would be a way around this, where we need a way to escape |
To summarize the problem:
We need to add proper escape sequences for each special character in our configuration parser and variable substitution to allow the second case. |
If the solution does turn out to be adding an escape syntax, older agent versions aren't going to support it and we need to account for that in the implementation. There will need to be a companion issue for Fleet to implement escaping (or a secrets provider per above, or whatever else we may decide on). |
This is already supported on all versions of Elastic Agent (I bet you didn't know that??). Its just not called secrets, its called
|
I added a test case to see what happens if we nested a value that contains I think the agent is not unpacking/rendering that part of the input with the |
I think this may be caused by the fleet-ui. If I create an example policy with only the custom logs integration (collecting custom.key: $$$$ The policy will render the input as inputs:
- id: logfile-logs-802a2527-fc49-496e-b7ba-349c9024e5a4
name: log-test
revision: 1
type: logfile
use_output: default
meta:
package:
name: log
version: 2.3.1
data_stream:
namespace: default
package_policy_id: 802a2527-fc49-496e-b7ba-349c9024e5a4
streams:
- id: logfile-log.logs-802a2527-fc49-496e-b7ba-349c9024e5a4
data_stream:
dataset: generic
paths:
- /var/log/example
ignore_older: 72h
custom.key: $$ cc @kpollich |
The API request to create the policy contains the raw When the code gets into kibana/x-pack/plugins/fleet/server/services/epm/agent/agent.ts Lines 19 to 48 in b3dcc54
Diving deeper into the code it looks like the culprit is the
This method seems to escape I checked first if the
|
This can probably move over to the UI team and we can pick it up from here if that makes sense to you @michel-laterman cc @ycombinator |
$$
or ${SOMETEXT}
$$
or ${SOMETEXT}
Thanks @michel-laterman and @kpollich for your investigations. Transferred issue to Kibana repo. |
Yes makes sense, do the same methods get called when a secret is specified as well? |
$$
or ${SOMETEXT}
$$
or ${SOMETEXT}
@kpollich I'm re-assigning this issue to you for the moment so you can re-assign to the appropriate UI engineer. |
is safeload called anywhere else that might cause similar issues? |
Yes the same code is called whenever the inputs for a policy are compiled, so if secrets are present they're being escaped.
It's possible, but it wasn't obvious to me on first glance. I'm going to pull this into next sprint to investigate. I think this will be a prerequisite for the secrets propagation project. |
Pinging @elastic/fleet (Team:Fleet) |
Closing as PR above landed. |
@kpollich @michel-laterman The PR I did fix a bug with the $$ in fleet yaml variable, I think there is still an issue on how secrets are handled by Fleet server and elastic-agent no? |
I think agent works as expected so long as the escaping issue in Fleet UI is fixed, e.g. #188377 (comment) |
Do we have tests for this also? I feel that it may be something that would rear its ugly head every now and then |
We have coverage in agent as unit tests for policy reading and computing to a component model. I don't believe we have an end to end integration test. |
And there's a unit test in the UI code as well: kibana/x-pack/plugins/fleet/server/services/epm/agent/agent.test.ts Lines 330 to 347 in 201c9d3
I created an issue for adding an E2E test to Agent: elastic/elastic-agent#5320 |
I added the test case in this PR. I tried to replicate the steps taken by @efd6 and came to the same conclusion. Just to be safe, I tried using the Below is the test output
cc: @ycombinator |
Relates:
This was identified while working on a customer issue with the ti_threatconnect integration. In that issue we were seeing 400 status codes coming back from API queries. After tracking down the cause, it is seen to come from variable expansion when taking the provided config and rendering it for beats. The secret in question had an instance of the substring
$$
which was being interpreted as a $-escaped$
.This causes confusion for users and is not ideal.
To reproduce this.
Start a stack with
elastic-package
.Add the ThreatConnect integration with the Elastic-Agent (elastic-package) as the agent with the following config.

Save and deploy.
Collect diagnostics from the agent and examine the pre-config.yaml file for the package installation.
Examine the corresponding components/cel-default/beat-rendered-config.yml file.
Note that the secret_key has had each of the
$$
resolved to$
. This results in failure to authenticate to the API. This is confusing for customers who have values that include these sequences.The change is a result of variable expansion when the beats config is rendered and would also be expected to result in changes if the secret (or other field) included syntax that would be expected to expand, for example



${NOT_A_VAR}
.This results in no error, but also no component being run
and no components in the diags.
The input shows up in the pre-config.yaml, but nothing is present in the computed-config.yaml.
Related issues: elastic/elastic-agent#2177 elastic/elastic-agent#2261 elastic/beats#35260
The text was updated successfully, but these errors were encountered: