Marketplace support (LeanerCloud#466)

Adds support for releasing stable builds on the AWS Marketplace

CUSTOM_BUILDS.md

@@ -1,13 +1,134 @@
 # AutoSpotting Setup #
 
-It's usually recommended to use the provided binaries, but in some cases you may
-need to customize AutoSpotting for your own environment.
+It's usually recommended to use the provided binaries available as Docker
+images, but in some cases you may need to customize AutoSpotting for your own
+environment.
 
-You'll need to set up a local environment able to compile Go code, compile the
-binaries locally, upload them to an S3 bucket in your AWS account and update
-your CloudFormation or Terraform stack to use those new binaries.
+## Docker ##
 
-## Dependencies ##
+Pre-built Docker images for the latest evaluation builds are also available on
+the Docker Hub at
+[AutoSpotting/AutoSpotting](https://hub.docker.com/r/AutoSpotting/AutoSpotting/)
+
+``` shell
+docker run autospotting/autospotting
+```
+
+They might be useful for quick tests, otherwise you might need to build your own
+docker images.
+
+The repository contains a `Dockerfile` and `docker-compose` configuration that
+allows you to build AutoSpotting Docker container images and run them
+conveniently on your local machine without installing the Go build environment
+usually required for local development(which is also documented below).
+
+This can be useful for trying it out locally or even for running it on a
+container hosting solution such as Kubernetes. They won't support the full
+functionality that relies on CloudWatch Events but it's probably enough for some
+people.
+
+If you have `docker` and `docker-compose` installed, it's as simple as running
+
+``` shell
+docker-compose run autospotting
+```
+
+ This also accepts all the AutoSpotting command-line arguments, including
+`-help` which explains all the other available options.
+
+The usual AWS credential environment variables listed in the
+`docker-compose.yaml` configuration file are passed to the running container and
+will need to be set for it to actually work.
+
+## Using your own Docker images in AWS Lambda ##
+
+AutoSpotting uses a Lambda function configured to use a Docker image. Such a
+configuration [currently](https://github.com/aws/containers-roadmap/issues/1281)
+requires the Docker image to be stored in an ECR from your own account.
+
+Also, in order for the generated Docker image to be compatible with the Lambda
+runtime, the image must currently be built on a x86_64 host. Unfortunately ARM
+hosts such as Apple M1 Macbook aren't currently supported because of QEMU
+emulator crashes when performing the cross-compilation using `docker buildx`.
+
+In order to support the AWS Marketplace setup, which relies on an ECR repository
+hosted in another AWS-managed account, the current CloudFormation template uses
+a custom resource that copies the Docker image from a source ECR (by default the
+Marketplace ECR) into an ECR created inside the CloudFormation stack. This adds
+some complexity but has the nice side effect of being able to push the image to
+any arbitrary ECR in another account/region, offering more flexibility for
+customers who may want to manage custom deployments at scale.
+
+You'll therefore need to build an x86_64 Docker image, upload it to an ECR
+repository in your AWS account and configure your CloudFormation or Terraform
+stack to use this new image as a source image.
+
+1. Set up an ECR repository in your AWS account that will host your custom
+   Docker images.
+
+1. The build system can use a `DOCKER_IMAGE` variable that tells it where to
+   upload the image. Set it into your environment to the name of your ECR
+   repository. When unset you'll attempt to push to the Marketplace ECR and
+   you'll receive permission errors.
+
+   ``` shell
+   export DOCKER_IMAGE=1234567890123.dkr.ecr.<region>.amazonaws.com/<my-ecr-name>
+   export DOCKER_IMAGE_VERSION=1.0.2 # it's strongly recommended versioning images
+   ```
+
+1. Define some AWS credentials or profile information into your
+   [environment](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-environment).
+
+1. Authenticate to your ECR repository
+
+   ```shell
+   make docker-login
+   ```
+
+1. Build and upload your Docker image to your ECR and configure a CloudFormation
+   template to use your ECR
+
+   ``` shell
+   make docker-push-artifacts
+   ```
+
+1. Use the CloudFormation template from the `build` directory to create the
+   resources. Make sure you set the parameters `SourceECR` and `SourceImage` to
+   point to your ECR repository (`SourceECR` should be set to contain the
+   hostname part of your ECR repository, before the first `/` character and
+   `SourceImage` should contain the rest). The version number will be set
+   automatically based on the value you defined earlier.
+
+   AutoSpotting should now be running against the binaries you built locally and
+   uploaded to your own ECR repository.
+
+   The same process can be used for updating AutoSpotting to a newer version.
+
+## Maintaining your own fork ##
+
+It is recommended to contribute your changes into the mainline version of the
+project whenever possible, so that others can benefit from your enhancements and
+bug fixes, but for some reasons you may still want to run your own fork.
+
+Unfortunately the golang import paths can make it tricky, but there is a nice
+[article](http://code.openark.org/blog/development/forking-golang-repositories-on-github-and-managing-the-import-path)
+which documents the problem in detail and gives a couple of possible
+workarounds.
+
+## Make directives ##
+
+The Makefile from the root of the git repository contains a number of useful
+directives, they're not documented here as they might change over time, so you
+may want to have a look at it.
+
+## Local Development setup ##
+
+AutoSpotting is written in Go so for local development you need a Go toolchain.
+You can probably also use docker-compose for this to avoid it as mentioned above
+but I prefer the native Go tooling which offers faster feedback for local
+development.
+
+### Dependencies ##
 
 1. Install [Go](https://golang.org/dl/), [git](https://git-scm.com/downloads),
    [Docker](https://www.docker.com/) and the [AWS command-line
@@ -24,14 +145,14 @@ your CloudFormation or Terraform stack to use those new binaries.
 
    `aws --version`
 
-## Compiling the binaries locally ##
+### Compiling the binaries locally ##
 
-1. Set up a directory for your Go development. I'm using `godev` in my home
+1. Set up a directory for your Go development. I'm using `go` in my home
    directory for this example.
 
-1. Set the `GOPATH` environment variable to point at your `godev` directory:
+1. Set the `GOPATH` environment variable to point at your `go` directory:
 
-   `export GOPATH=$HOME/godev`
+   `export GOPATH=$HOME/go`
 
    Optionally add this line to your .bash_profile to persist across console
    sessions.
@@ -60,111 +181,22 @@ your CloudFormation or Terraform stack to use those new binaries.
 
    `make test`
 
-   Below you can see more details on the other available make directives.
+1. Build the code again:
 
-1. Build the code:
+   `make build`
 
-   Run `make build` to generate a binary at `./AutoSpotting`. The resulting
-   binary is suitable for running on AWS Lambda. If you want a binary for
-   running locally and your local environment is not linux/amd64, try running
-   the following:
-
-   `GOOS=$(go env GOOS) GOARCH=$(go env GOARCH) make build`
-
-## Running locally ##
+### Running locally ###
 
 1. Run the code, assuming you have AWS credentials defined in your environment
    or in the default AWS credentials profile:
 
    `./AutoSpotting`
 
-   You may also pass some command line flags, see the --help for more
+   You may also pass some command line flags, see the `--help` output for more
    information on the available options.
 
    When you are happy with how your custom build behaves, you can generate a
-   build for AWS Lambda.
-
-## Using your own binaries in AWS Lambda ##
-
-1. Set up an S3 bucket in your AWS account that will host your custom binaries.
-
-1. The build system can use a `BUCKET_NAME` variable that tells it where to
-   upload new binaries. Set it into your environment to the name of your S3
-   bucket.
-   `export BUCKET_NAME=my-bucket`
-
-1. Define some AWS credentials or profile information into your
-   [environment](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-environment).
-
-1. (Optional) You can also create the lambda deployment package using the below
-   command, the zip file is generated in the `build` directory.
-   `make archive`
-
-1. Build and upload your binaries to the S3 bucket.
-   `make upload`
-
-1. If you're already set up to use the tool with the official binaries, update
-   your existing CloudFormation stack, and change the `LambdaS3Bucket` field to
-   your S3 bucket name on the Stack Parameters section of the stack
-   configuration.
-
-   ![LambdaS3Bucket
-   Configuration](https://mcristi.files.wordpress.com/2016/04/installationcloudformation2.png)
-
-1. Save the CloudFormation configuration and let it create/update the resources.
-   The tool should now be running against the binaries you built locally and
-   uploaded to your own S3 bucket.
-
-## Maintaining your own fork ##
-
-It is recommended to contribute your changes into the mainline version of the
-project whenever possible, so that others can benefit from your enhancements and
-bug fixes, but for some reasons you may still want to run your own fork.
-
-Unfortunately the golang import paths can make it tricky, but there is a nice
-[article](http://code.openark.org/blog/development/forking-golang-repositories-on-github-and-managing-the-import-path)
-which documents the problem in detail and gives a couple of possible
-workarounds.
-
-## Make directives ##
-
-Use these directives defined in the `Makefile` to build, release, and test the
-tool:
-
-* **all (default, can be ommitted)**
-  * Verifies that the necessary dependencies are installed.
-  * Runs `go build` to compile the project for local development.
-
-* **upload**
-  * Prepares a special build designed to run in AWS Lambda.
-  * Uploads the generated binaries from `build/s3` to the specified S3 bucket.
-
-* **test**
-  * Runs the test suite.
-
-* **build_local**
-  * Compiles the project for local execution.
-
-## Docker builds ##
-
-The repository also contains a `Dockerfile` and `docker-compose` configuration
-that allows you to build AutoSpotting Docker containers and run them
-conveniently without installing a Go build environment.
-
-If you have `docker` and `docker-compose` installed, it's as simple as running
-`docker-compose up`
-
-All the supported environment variables listed in the `docker-compose.yaml`
-configuration file are passed to the running container. You will at least need
-to have defined some AWS credential environment variables.
-
-You can also invoke the container entrypoint directly using
-    `docker-compose run autospotting`
-
-This also accepts all the AutoSpotting command-line arguments, including `-help`
-which explains all the available options.
 
-Pre-built Docker images for the latest builds are also available on Dockerhub at
-[AutoSpotting/AutoSpotting](https://hub.docker.com/r/autospotting/autospotting/)
+   build for AWS Lambda using the Docker method documented above.
 
 [Back to the main Readme](./README.md)

CloudFormation_and_StackSets.md

@@ -5,7 +5,7 @@ AWS accounts and also to multiple regions within a single account.
 
 AutoSpotting supports being deployed using StackSets across multiple accounts,
 and also leverages them internally to deploy some of its components across
-multiple regions.
+multiple regions within each account.
 
 This document explains the way the current CloudFormation deployment method of
 AutoSpotting uses CloudFormation StackSets internally, what consequences it has

Dockerfile

@@ -2,7 +2,7 @@ FROM golang:1.16-alpine as golang
 RUN apk add -U --no-cache ca-certificates git make
 COPY . /src
 WORKDIR /src
-RUN FLAVOR=nightly CGO_ENABLED=0 make
+RUN FLAVOR=nightly CGO_ENABLED=0 GOPROXY=direct make
 
 FROM scratch
 COPY LICENSE BINARY_LICENSE THIRDPARTY /

Dockerfile.build

@@ -0,0 +1,10 @@
+FROM golang:1.16-alpine
+
+ARG flavor
+
+RUN apk add -U --no-cache ca-certificates git make
+
+COPY . /src
+WORKDIR /src
+
+RUN CGO_ENABLED=0 FLAVOR="$flavor" make ci

Dockerfile.marketplace

@@ -0,0 +1,11 @@
+FROM golang:1.16-alpine as golang
+RUN apk add -U --no-cache ca-certificates git make
+COPY . /src
+WORKDIR /src
+RUN FLAVOR=stable CGO_ENABLED=0 GOPROXY=direct make
+
+FROM alpine:3.14.1
+COPY LICENSE BINARY_LICENSE THIRDPARTY /
+COPY --from=golang /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
+COPY --from=golang /src/AutoSpotting .
+ENTRYPOINT ["./AutoSpotting"]