Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Elastic-Agent] Modify output to be insecure if flag is provided #28007

Merged
merged 12 commits into from
Oct 13, 2021
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ func newFleetServerBootstrap(
agentInfo,
router,
&pipeline.ConfigModifiers{
Filters: []pipeline.FilterFunc{filters.StreamChecker, modifiers.InjectFleet(rawConfig, sysInfo.Info(), agentInfo)},
Filters: []pipeline.FilterFunc{filters.StreamChecker, modifiers.InjectInsecureOutput(cfg.Fleet), modifiers.InjectFleet(rawConfig, sysInfo.Info(), agentInfo)},
},
)
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion x-pack/elastic-agent/pkg/agent/application/managed_mode.go
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ func newManaged(
router,
&pipeline.ConfigModifiers{
Decorators: []pipeline.DecoratorFunc{modifiers.InjectMonitoring},
Filters: []pipeline.FilterFunc{filters.StreamChecker, modifiers.InjectFleet(rawConfig, sysInfo.Info(), agentInfo)},
Filters: []pipeline.FilterFunc{filters.StreamChecker, modifiers.InjectInsecureOutput(cfg.Fleet), modifiers.InjectFleet(rawConfig, sysInfo.Info(), agentInfo)},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it will affect more than just Fleet Server running under Elastic Agent, it will also affect all the other beats, correct?

If this does affect the other beats, I don't think we want this, because how will this work when it comes to multiple outputs? I believe it will have the effect that if --insecure is used all outputs will then become insecure.

Copy link
Contributor Author

@michalpristas michalpristas Sep 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes that's the purpose. if we wont pass, events wont get consumed
this is to ease up on initial experience. you wont use insecure in prod. at least i hope nobody will

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They would get consumed if the policy is configured correctly. I don't think it should be Elastic Agent's job to override the output settings from Kibana.

If we had proper health reporting for outputs, it would be clear there is an issue. The real issue is that the Elastic Agent reports healthy when it is not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if value is provided by kibana it should not be overriden, i'll add testcase to make it more visible

Copy link
Contributor Author

@michalpristas michalpristas Sep 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kibana on the other hand would needs to have the info that hey this cert we're using is self signed and may no be properly configured to support some config automatically (providing values to out config option).
insecure is our concept and i think it's up to us to help user using this with FRE.

},
caps,
monitor,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package modifiers

import (
"github.com/elastic/beats/v7/libbeat/common/transport/tlscommon"
"github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/configuration"
"github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/transpiler"
"github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/core/logger"
)

const (
sslKey = "ssl"
verificationModeKey = "verification_mode"
sslVerificationKey = "ssl.verification_mode"
)

// InjectInsecureOutput injects a verification none into output configuration.
func InjectInsecureOutput(fleetConfig *configuration.FleetAgentConfig) func(*logger.Logger, *transpiler.AST) error {
return func(_ *logger.Logger, rootAst *transpiler.AST) error {
// if verification mode is not set abort
if fleetConfig == nil ||
fleetConfig.Server == nil ||
fleetConfig.Server.TLS == nil ||
fleetConfig.Server.TLS.VerificationMode == tlscommon.VerifyFull {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does seem to have the affect that it will only do something if the Elastic Agent is running Fleet Server, but it will affect all beats.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right. changed to use client config instead

// no change
return nil
}

// look for outputs
// inject verification mode to each output
outputsNode, ok := transpiler.Lookup(rootAst, outputsKey)
if !ok {
// no outputs from configuration; skip
return nil
}

outputsList, ok := outputsNode.Value().(*transpiler.Dict)
if !ok {
return nil
}

outputsNodeCollection, ok := outputsList.Value().([]transpiler.Node)
if !ok {
return nil
}

modeString := fleetConfig.Server.TLS.VerificationMode.String()

for _, outputNode := range outputsNodeCollection {
outputKV, ok := outputNode.(*transpiler.Key)
if !ok {
continue
}

output, ok := outputKV.Value().(*transpiler.Dict)
if !ok {
continue
}

// do not overwrite already specified config
_, found := output.Find(sslVerificationKey)
if found {
continue
}

// it may be broken down
if sslNode, found := output.Find(sslKey); found {
if _, found := sslNode.Find(verificationModeKey); found {
continue
}
}

output.Insert(
transpiler.NewKey(sslVerificationKey, transpiler.NewStrVal(modeString)),
)

}

return nil
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package modifiers

import (
"testing"

"github.com/google/go-cmp/cmp"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"

"github.com/elastic/beats/v7/libbeat/common/transport/tlscommon"
"github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/configuration"
"github.com/elastic/beats/v7/x-pack/elastic-agent/pkg/agent/transpiler"
)

func TestInjectInsecure(t *testing.T) {
cases := []struct {
Name string
Config map[string]interface{}
Expected map[string]interface{}
VerificationMode tlscommon.TLSVerificationMode
}{
{
Name: "no change on default",
Config: map[string]interface{}{
"outputs": map[string]interface{}{
"elasticsearch": map[string]interface{}{
"key": "value",
},
},
},
Expected: map[string]interface{}{
"outputs": map[string]interface{}{
"elasticsearch": map[string]interface{}{
"key": "value",
},
},
},
VerificationMode: tlscommon.VerifyFull, // default
},
{
Name: "inject none",
Config: map[string]interface{}{
"outputs": map[string]interface{}{
"elasticsearch": map[string]interface{}{
"key": "value",
},
},
},
Expected: map[string]interface{}{
"outputs": map[string]interface{}{
"elasticsearch": map[string]interface{}{
"key": "value",
"ssl.verification_mode": "none",
},
},
},
VerificationMode: tlscommon.VerifyNone,
},
}

for _, tc := range cases {
t.Run(tc.Name, func(t *testing.T) {
fn := InjectInsecureOutput(&configuration.FleetAgentConfig{
Server: &configuration.FleetServerConfig{
TLS: &tlscommon.Config{
VerificationMode: tc.VerificationMode,
},
},
})

ast, err := transpiler.NewAST(tc.Config)
require.NoError(t, err)
expectedAST, err := transpiler.NewAST(tc.Expected)
require.NoError(t, err)

require.NoError(t, fn(nil, ast))

visitor := &transpiler.MapVisitor{}
expectedVisitor := &transpiler.MapVisitor{}

ast.Accept(visitor)
expectedAST.Accept(expectedVisitor)

if !assert.True(t, cmp.Equal(expectedVisitor.Content, visitor.Content)) {
diff := cmp.Diff(expectedVisitor.Content, visitor.Content)
if diff != "" {
t.Errorf("mismatch (-want +got):\n%s", diff)
}
}
})
}

}
19 changes: 15 additions & 4 deletions x-pack/elastic-agent/pkg/agent/cmd/enroll_cmd.go
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,7 @@ func (c *enrollCmd) fleetServerBootstrap(ctx context.Context) (string, error) {
c.options.ProxyURL,
c.options.ProxyDisabled,
c.options.ProxyHeaders,
c.options.Insecure,
)
if err != nil {
return "", err
Expand Down Expand Up @@ -497,7 +498,9 @@ func (c *enrollCmd) enroll(ctx context.Context, persistentConfig map[string]inte
c.options.FleetServer.Host, c.options.FleetServer.Port,
c.options.FleetServer.Cert, c.options.FleetServer.CertKey, c.options.FleetServer.ElasticsearchCA,
c.options.FleetServer.Headers,
c.options.ProxyURL, c.options.ProxyDisabled, c.options.ProxyHeaders)
c.options.ProxyURL, c.options.ProxyDisabled, c.options.ProxyHeaders,
c.options.Insecure,
)
if err != nil {
return err
}
Expand Down Expand Up @@ -806,16 +809,21 @@ func createFleetServerBootstrapConfig(
proxyURL string,
proxyDisabled bool,
proxyHeaders map[string]string,
insecure bool,
) (*configuration.FleetAgentConfig, error) {
localFleetServer := connStr != ""

es, err := configuration.ElasticsearchFromConnStr(connStr, serviceToken)
es, err := configuration.ElasticsearchFromConnStr(connStr, serviceToken, insecure)
if err != nil {
return nil, err
}
if esCA != "" {
es.TLS = &tlscommon.Config{
CAs: []string{esCA},
if es.TLS == nil {
es.TLS = &tlscommon.Config{
CAs: []string{esCA},
}
} else {
es.TLS.CAs = []string{esCA}
}
}
if host == "" {
Expand Down Expand Up @@ -857,6 +865,9 @@ func createFleetServerBootstrapConfig(
Key: key,
},
}
if insecure {
cfg.Server.TLS.VerificationMode = tlscommon.VerifyNone
}
}

if localFleetServer {
Expand Down
8 changes: 7 additions & 1 deletion x-pack/elastic-agent/pkg/agent/cmd/inspect.go
Original file line number Diff line number Diff line change
Expand Up @@ -259,10 +259,16 @@ func getProgramsFromConfig(log *logger.Logger, agentInfo *info.AgentInfo, cfg *c
if err != nil {
return nil, err
}

configuration, err := configuration.NewFromConfig(cfg)
if err != nil {
return nil, err
}

composableWaiter := newWaitForCompose(composableCtrl)
configModifiers := &pipeline.ConfigModifiers{
Decorators: []pipeline.DecoratorFunc{modifiers.InjectMonitoring},
Filters: []pipeline.FilterFunc{filters.StreamChecker},
Filters: []pipeline.FilterFunc{filters.StreamChecker, modifiers.InjectInsecureOutput(configuration.Fleet)},
}

if !isStandalone {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ type Elasticsearch struct {
}

// ElasticsearchFromConnStr returns an Elasticsearch configuration from the connection string.
func ElasticsearchFromConnStr(conn string, serviceToken string) (Elasticsearch, error) {
func ElasticsearchFromConnStr(conn string, serviceToken string, insecure bool) (Elasticsearch, error) {
u, err := url.Parse(conn)
if err != nil {
return Elasticsearch{}, err
Expand All @@ -64,6 +64,11 @@ func ElasticsearchFromConnStr(conn string, serviceToken string) (Elasticsearch,
Path: u.Path,
TLS: nil,
}
if insecure {
cfg.TLS = &tlscommon.Config{
VerificationMode: tlscommon.VerifyNone,
}
}
if serviceToken != "" {
cfg.ServiceToken = serviceToken
return cfg, nil
Expand Down
5 changes: 5 additions & 0 deletions x-pack/elastic-agent/pkg/agent/transpiler/ast.go
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,11 @@ func (d *Dict) Find(key string) (Node, bool) {
return nil, false
}

// Insert inserts a value into a collection.
func (d *Dict) Insert(node Node) {
d.value = append(d.value, node)
}

func (d *Dict) String() string {
var sb strings.Builder
for i := 0; i < len(d.value); i++ {
Expand Down