Add a guide to using the API for auto-discovery

[ptgott]

See #19716

Adding resources is a popular use of Teleport's API, so I have added a guide to using the API for syncing resources to a service discovery API using an example of a local Docker setup.

[zmb3]

This example might be simpler if you registered dynamic apps rather than started new app services and generated new join tokens.

[ptgott]

I wanted to demonstrate the token method in this guide somehow so users could see an example, but it makes sense to advise using a dynamic approach for simplicity. I've moved the Details box about the dynamic approach higher in this section and changed the first paragraph after the code snippet to make the dynamic approach more prominent.

[ptgott]

@alexfornuto This is ready for copy review

[alexfornuto]

@ptgott do you wanna resolve the conflict with introduction.mdx before I review that page, in case the resolution changes the copy?

[alexfornuto]

Would this line replace func (t *tokenDemoApp) createAppToken(ctx context.Context) (string, error) { ?

[ptgott]

The current app creates a token and launches an App Service instance with it. If you use CreateApp, you don't need to create a token or launch a new App Service instance, but you do need to have an App Service instance running already.

[alexfornuto]

By "client application must have" are you referring to the spec of the register-apps role?

[ptgott]

I've changed this to "Your API client application's Teleport user must also have the following permissions:"

[alexfornuto]

Would Docker not pull the image as needed? Below when we spin up containers based on the rabbitmq:3-management image it does. Should we pull that image here as well to get it out of the way?

[ptgott]

It didn't when I ran the demo app. I think the docker CLI bundles some actions together that the API client library separates. docker run pulls a container, creates it, and executes it, whereas these are separate Docker API client function calls. I thought it would make sense to pull the container manually, since it's a quick step that you only need to do once.

[alexfornuto]

This is where my testing fails. I'm on macOS, and running the rabbit containers:

❯ docker container ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS                                                                  NAMES
e20bbe8e9dea   rabbitmq:3-management   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes   4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 15691-15692/tcp, 25672/tcp   naughty_shtern
fbca4f93ba67   rabbitmq:3-management   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes   4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 15691-15692/tcp, 25672/tcp   affectionate_dubinsky
0f3a961dd928   rabbitmq:3-management   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes   4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 15691-15692/tcp, 25672/tcp   cranky_wright

But the go program is not responding:

❯ go run main.go
Starting the application
Connected to Teleport
Connected to the Docker daemon

And tsh apps ls doesn't list any new applications. I tried restarting the go client while the containers were still running, and then restarting the containers while the go client was running. I've also waited up to five minutes after creating the containers for the client app to do something.

Here's my completed main.go, to check for PEBKAC errors:

main.go

package main

import (
	"context"
	"crypto/rand"
	"encoding/hex"
	"fmt"
	"net"
	"net/url"
	"strings"
	"time"

	dtypes "github.com/docker/docker/api/types"
	"github.com/docker/docker/api/types/container"
	"github.com/docker/docker/api/types/filters"
	"github.com/docker/docker/api/types/strslice"
	docker "github.com/docker/docker/client"
	teleport "github.com/gravitational/teleport/api/client"
	"github.com/gravitational/teleport/api/client/proto"
	"github.com/gravitational/teleport/api/types"
	"github.com/gravitational/trace"
	"google.golang.org/grpc"
)

const (
	// Assign proxyAddr to the host and port of your Teleport Proxy Service instance
	proxyAddr      string = "REDACTED:443"
	teleportImage  string = "public.ecr.aws/gravitational/teleport:12.1.1"
	initTimeout           = time.Duration(30) * time.Second
	updateInterval        = time.Duration(5) * time.Second
	tokenTTL              = time.Duration(5) * time.Minute
	networkName    string = "bridge"
	managementPort string = "15672"
	tokenLenBytes         = 16
	rabbitMQImage  string = "rabbitmq:3-management"
)

type tokenDemoApp struct {
	dockerClient   *docker.Client
	teleportClient *teleport.Client
}

func (t *tokenDemoApp) listRegisteredAppURLs(ctx context.Context) (map[string]types.AppServer, error) {
	m := make(map[string]types.AppServer)

	for {
		req := proto.ListResourcesRequest{
			ResourceType: "app_server",
			Limit:        10,
		}
		resp, err := t.teleportClient.ListResources(
			ctx,
			req,
		)

		if err != nil {
			return nil, trace.Wrap(err)
		}

		for _, r := range resp.Resources {
			if p, ok := r.(types.AppServer); ok {
				m[p.GetApp().GetURI()] = p
			}
		}

		// No more pages to request
		if resp.NextKey == "" {
			break
		}

		req.StartKey = resp.NextKey
	}

	return m, nil
}

func (t *tokenDemoApp) listAppContainerURLs(ctx context.Context, image string) (map[string]struct{}, error) {
	c, err := t.dockerClient.ContainerList(ctx, dtypes.ContainerListOptions{
		Filters: filters.NewArgs(filters.KeyValuePair{
			Key:   "ancestor",
			Value: image,
		}),
	})
	if err != nil {
		return nil, trace.Wrap(err)
	}

	l := make(map[string]struct{})

	for _, r := range c {
		b, ok := r.NetworkSettings.Networks[networkName]
		// Not connected to the chosen network, so skip it
		if !ok {
			continue
		}

		u, err := url.Parse("http://" + net.JoinHostPort(
			b.IPAddress,
			managementPort,
		))

		if err != nil {
			return nil, trace.Wrap(err)
		}

		l[u.String()] = struct{}{}
	}

	return l, nil
}

func cryptoRandomHex(len int) (string, error) {
	randomBytes := make([]byte, len)
	if _, err := rand.Read(randomBytes); err != nil {
		return "", trace.Wrap(err)
	}
	return hex.EncodeToString(randomBytes), nil
}

func (t *tokenDemoApp) createAppToken(ctx context.Context) (string, error) {
	n, err := cryptoRandomHex(tokenLenBytes)
	if err != nil {
		return "", trace.Wrap(err)
	}

	tok, err := types.NewProvisionTokenFromSpec(
		n,
		time.Now().Add(tokenTTL),
		types.ProvisionTokenSpecV2{
			Roles: types.SystemRoles{types.RoleApp},
		})

	if err := t.teleportClient.CreateToken(ctx, tok); err != nil {
		return "", trace.Wrap(err)
	}
	return n, nil
}

func (t *tokenDemoApp) startApplicationServiceContainer(
	ctx context.Context,
	token string,
	u url.URL,
) error {

	name := strings.ReplaceAll(u.Hostname(), ".", "-")
	resp, err := t.dockerClient.ContainerCreate(
		ctx,
		&container.Config{
			Image: teleportImage,
			Entrypoint: strslice.StrSlice{
				"/usr/bin/dumb-init",
				"teleport",
				"start",
				"--roles=app",
				"--auth-server=" + proxyAddr,
				"--token=" + token,
				"--app-name=rabbitmq-" + name,
				"--app-uri=" + u.String(),
			},
		},
		nil,
		nil,
		nil,
		"",
	)
	if err != nil {
		return trace.Wrap(err)
	}

	err = t.dockerClient.ContainerStart(
		ctx,
		resp.ID,
		dtypes.ContainerStartOptions{},
	)
	if err != nil {
		return trace.Wrap(err)
	}

	return nil
}

func (t *tokenDemoApp) pruneAppServiceInstance(ctx context.Context, p types.AppServer) error {
	host := p.GetHostname()

	if err := t.teleportClient.DeleteApplicationServer(
		ctx,
		p.GetNamespace(),
		p.GetHostID(),
		p.GetName(),
	); err != nil {
		return trace.Wrap(err)
	}

	fmt.Println("Deleted unnecessary Application Service record:", p.GetName())

	// Don't check errors when removing the container, since it may already
	// have been removed.
	t.dockerClient.ContainerStop(ctx, host, container.StopOptions{})
	t.dockerClient.ContainerRemove(ctx, host, dtypes.ContainerRemoveOptions{})

	fmt.Println("Deleted unnecessary Application Service container:", host)
	return nil
}

func (t *tokenDemoApp) reconcileApps() error {
	ctx := context.Background()
	apps, err := t.listRegisteredAppURLs(ctx)
	if err != nil {
		return trace.Wrap(err)
	}

	urls, err := t.listAppContainerURLs(ctx, rabbitMQImage)
	if err != nil {
		return trace.Wrap(err)
	}

	for u, _ := range urls {
		if _, ok := apps[u]; ok {
			continue
		}
		tok, err := t.createAppToken(ctx)
		if err != nil {
			return trace.Wrap(err)
		}
		fmt.Println("Created a new application token for URL: " + u)

		r, err := url.Parse(u)
		if err != nil {
			return trace.Wrap(err)
		}

		err = t.startApplicationServiceContainer(ctx, tok, *r)
		if err != nil {
			return trace.Wrap(err)
		}
		fmt.Println("Started an Application Service container to proxy URL: " + u)
	}

	for a, p := range apps {
		_, ok := urls[a]
		if ok {
			continue
		}

		if err := t.pruneAppServiceInstance(ctx, p); err != nil {
			return trace.Wrap(err)
		}
	}
	return nil
}

func newTokenDemoApp() *tokenDemoApp {
	ctx := context.Background()
	ctx, cancel := context.WithTimeout(ctx, initTimeout)
	defer cancel()
	creds := teleport.LoadIdentityFile("auth.pem")

	t, err := teleport.New(ctx, teleport.Config{
		Addrs:       []string{proxyAddr},
		Credentials: []teleport.Credentials{creds},
		DialOpts: []grpc.DialOption{
			grpc.WithReturnConnectionError(),
		},
	})
	if err != nil {
		panic(err)
	}
	fmt.Println("Connected to Teleport")

	d, err := docker.NewClientWithOpts(
		docker.WithAPIVersionNegotiation(),
	)
	if err != nil {
		panic(err)
	}
	fmt.Println("Connected to the Docker daemon")

	return &tokenDemoApp{
		teleportClient: t,
		dockerClient:   d,
	}

}

func main() {
	fmt.Println("Starting the application")
	app := newTokenDemoApp()

	k := time.NewTicker(updateInterval)
	defer k.Stop()
	for {
		<-k.C
		if err := app.reconcileApps(); err != nil {
			panic(err)
		}
	}
}

[ptgott]

I'm going to push a commit that includes the whole program in the examples directory so there's no risk of accidentally pasting the wrong thing (I don't think that that's the problem, but just in case...).

It works on my machine™, so there must be some difference between our environments that's coming into play.

One thing you could do is add some Println calls to *tokenDemoApp.reconcileApps to see if the app is listing containers/registered apps as expected:

func (t *tokenDemoApp) reconcileApps() error {
	ctx := context.Background()
	apps, err := t.listRegisteredAppURLs(ctx)
	if err != nil {
		return trace.Wrap(err)
	}

+     fmt.Println("registered apps", apps)

	urls, err := t.listAppContainerURLs(ctx, rabbitMQImage)
	if err != nil {
		return trace.Wrap(err)
	}

+      fmt.Println("app container URLs:", urls)

	for u, _ := range urls {
		if _, ok := apps[u]; ok {
			continue
		}
		tok, err := t.createAppToken(ctx)
		if err != nil {
			return trace.Wrap(err)
		}
		fmt.Println("Created a new application token for URL: " + u)

		r, err := url.Parse(u)
		if err != nil {
			return trace.Wrap(err)
		}

		err = t.startApplicationServiceContainer(ctx, tok, *r)
		if err != nil {
			return trace.Wrap(err)
		}
		fmt.Println("Started an Application Service container to proxy URL: " + u)
	}

	for a, p := range apps {
		_, ok := urls[a]
		if ok {
			continue
		}

		if err := t.pruneAppServiceInstance(ctx, p); err != nil {
			return trace.Wrap(err)
		}
	}
	return nil
}

Are the containers connected to the default bridge network?

docker network inspect bridge | jq '.[0].Containers'

If not, the app also won't be able to find them, since it only keeps track of container URLs on the default bridge network.

In any case, thanks for testing this out!

[alexfornuto]

Some additional copy edits.

[alexfornuto]

To match tense:

Suggested change

	- Writing a program/bot to manage Access Requests automatically, based on your
	- Write a program/bot to manage Access Requests automatically, based on your

[ptgott]

👍
I've also deleted this particular line (from the original page), since we don't recommend approving/denying Access Requests automatically (and recommend displaying links to the Teleport Web UI to do this).

[public-teleport-github-review-bot]

@ptgott - this PR will require admin approval to merge due to its size. Consider breaking it up into a series smaller changes.

[ptgott]

@alexfornuto Thanks for the review! I've responded to your feedback and also pushed a commit to incorporate some feedback from Zac in this PR that also applies to this one.

[alexfornuto]

Just a couple more suggestions.