Add a detailed README and clean up the configuration file (#94)

* Add a more complete project readme.
* Include design goals and architecture details.
* Include a section on client development.
* Include contributing and development sections.
* Add a more reference like config.
* Document all fields in the config file.
* Move some general parameters under the server object.
* Add issue and PR templates.
* Clarify where queue defaults came from.
* Clarify whether events are delayed or discarded.

Co-authored-by: Denis Rechkunov <denis@rdner.de>

.github/ISSUE_TEMPLATE/bug-report.md

@@ -0,0 +1,20 @@
+---
+name: Bug
+about: "Report confirmed bugs. For unconfirmed bugs please visit https://discuss.elastic.co/c/elastic-stack/elastic-agent"
+
+---
+
+Please post all questions and issues on https://discuss.elastic.co/c/elastic-stack/elastic-agent
+before opening a Github Issue. Your questions will reach a wider audience there,
+and if we confirm that there is a bug, then you can open a new issue.
+
+For security vulnerabilities please only send reports to security@elastic.co.
+See https://www.elastic.co/community/security for more information.
+
+Please include configurations and logs if available.
+
+For confirmed bugs, please report:
+- Version:
+- Operating System:
+- Discuss Forum URL
+- Steps to Reproduce:

.github/ISSUE_TEMPLATE/feature-request.md

@@ -0,0 +1,10 @@
+---
+name: Enhancement request
+about: The Elastic agent data shipper can't do all the things, but maybe it can do your things.
+
+---
+
+**Describe the enhancement:**
+
+**Describe a specific use case for the enhancement or feature:**
+

.github/ISSUE_TEMPLATE/flaky-test.md

@@ -0,0 +1,19 @@
+---
+name: Flaky Test
+about: Report a flaky test (one that doesn't pass consistently)
+labels: flaky-test
+---
+
+## Flaky Test
+
+* **Test Name:** Name of the failing test.
+* **Link:** Link to file/line number in github.
+* **Branch:** Git branch the test was seen in. If a PR, the branch the PR was based off.
+* **Artifact Link:** If available, attach the generated zip artifact associated with the stack trace for this failure.
+* **Notes:** Additional details about the test. e.g. theory as to failure cause
+
+### Stack Trace
+
+```
+paste stack trace here
+```

.github/ISSUE_TEMPLATE/question.md

@@ -0,0 +1,23 @@
+---
+name: Question
+about: Who, what, when, where, and how?
+
+---
+
+Hey, stop right there!
+
+We use GitHub to track feature requests and bug reports. Please do not
+submit issues for questions about how to use features of the Elastic agent, how to
+set the Elastic agent up, best practices, or development related help.
+
+However, we do want to help! Head on over to our official Elastic agent forums and ask
+your questions there. In additional to awesome, knowledgeable community
+contributors, core Elastic agent developers are on the forums every single day to help
+you out.
+
+The forums are here: https://discuss.elastic.co/c/elastic-stack/elastic-agent
+
+We can't stop you from opening an issue here, but it will likely
+linger without a response for days or weeks before it is closed and we
+ask you to join us on the forums instead. Save yourself the time, and
+ask on the forums today.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,81 @@
+<!-- Type of change
+Please label this PR with one of the following labels, depending on the scope of your change:
+- Bug
+- Enhancement
+- Breaking change
+- Deprecation
+- Cleanup
+- Docs
+-->
+
+## What does this PR do?
+
+<!-- Mandatory
+Explain here the changes you made on the PR. Please explain the WHAT: patterns used, algorithms implemented, design architecture, message processing, etc.
+-->
+
+## Why is it important?
+
+<!-- Mandatory
+Explain here the WHY, or the rationale/motivation for the changes.
+-->
+
+## Checklist
+
+<!-- Mandatory
+Add a checklist of things that are required to be reviewed in order to have the PR approved
+
+List here all the items you have verified BEFORE sending this PR. Please DO NOT remove any item, striking through those that do not apply. (Just in case, strikethrough uses two tildes. ~~Scratch this.~~)
+-->
+
+- [ ] My code follows the style guidelines of this project
+- [ ] I have commented my code, particularly in hard-to-understand areas
+- [ ] I have made corresponding changes to the documentation
+- [ ] I have made corresponding change to the default configuration files
+- [ ] I have added tests that prove my fix is effective or that my feature works
+- [ ] I have added an entry in `CHANGELOG.md` or `CHANGELOG-developer.md`.
+
+## Author's Checklist
+
+<!-- Recommended
+Add a checklist of things that are required to be reviewed in order to have the PR approved
+-->
+- [ ]
+
+## How to test this PR locally
+
+<!-- Recommended
+Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.
+-->
+
+## Related issues
+
+<!-- Recommended
+Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it.
+
+- Closes #123
+- Relates #123
+- Requires #123
+- Superseds #123
+-->
+- 
+
+## Use cases
+
+<!-- Recommended
+Explain here the different behaviors that this PR introduces or modifies in this project, user roles, environment configuration, etc.
+
+If you are familiar with Gherkin test scenarios, we recommend its usage: https://cucumber.io/docs/gherkin/reference/
+-->
+
+## Screenshots
+
+<!-- Optional
+Add here screenshots about how the project will be changed after the PR is applied. They could be related to web pages, terminal, etc, or any other image you consider important to be shared with the team.
+-->
+
+## Logs
+
+<!-- Recommended
+Paste here output logs discovered while creating this PR, such as stack traces or integration logs, or any other output you consider important to be shared with the team.
+-->

README.md

@@ -1,2 +1,66 @@
 # elastic-agent-shipper
-Data shipper for the Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
+
+> :warning: The Elastic agent data shipper is under active development, and will be available as an opt-in beta feature in an upcoming Elastic agent release.
+
+Data shipper for the Elastic Agent - a single, unified way to add monitoring for logs, metrics, and
+other types of data to a host.
+
+The data shipper is a new process in the Elastic agent system designed to centralize local data
+processing, queueuing, and publishing to the target output (Elasticsearch, Logstash, etc.).
+
+The data shipper is a part of a larger effort to rearchitect the Elastic agent. In the initial Elastic
+agent architecture each underlying data collector (e.g. Filebeat) was required to implement its own
+processing, queueing, and output connection(s) for each supported output type. The data shipper
+simplifies this architecture by allowing data collectors to implement a single gRPC client to
+process, queue, and publish data. The initial design goals of the data shipper are to:
+
+- Remove the need for processing, queueing, and output protocols to be reimplemented in each input.
+- Minimize the number of output connections required in Elastic agent deployments.
+- Simplify configuration and performance tuning of Elastic agent data pipelines.
+- Make Elastic agent data pipelines more observable and easier to debug.
+- Improve on or maintain the performance of the existing Beats outputs.
+- Define the event publishing interface all current and future Elastic agent data inputs will use.
+
+Each output in an agent policy will map to a separate instance of the shipper process:
+![Elastic Agent Data Shipper](docs/elastic-agent-shipper-arch.png)
+
+## Client Development
+
+Data shipper clients must implement the shipper [gRPC API](https://github.com/elastic/elastic-agent-shipper-client/tree/main/api).
+The reference client is the [Beats shipper output](https://github.com/elastic/beats/tree/main/libbeat/outputs/shipper), which is
+used by Beats like Filebeat and Metricbeat when they are started by Elastic agent integrations. 
+
+Data shipper support in the Elastic Agent is under active development. The shipper currently depends on a running Elastic agent instance to
+start but it will soon be possible to run the shipper in standalone mode for development: https://github.com/elastic/elastic-agent-shipper/issues/83
+
+The [reference shipper configuration file](https://github.com/elastic/elastic-agent-shipper/blob/main/elastic-agent-shipper.yml) defines the available
+configuration options.
+
+## Contributing
+
+The process for contributing to any of the Elastic repositories is similar.
+
+1. Please make sure you have signed our [Contributor License Agreement](https://www.elastic.co/contributor-agreement/). 
+We are not asking you to assign copyright to us, but to give us the right to distribute your code
+without restriction. We ask this of all contributors in order to assure our users of the origin and
+continuing existence of the code. You only need to sign the CLA once.
+
+2. Send a pull request! Push your changes to your fork of the repository and [submit a pull
+request](https://help.github.com/articles/using-pull-requests). New PRs go to the main branch. The
+development team will backport your PR to previous release branches if necessary. In the pull request, describe what
+your changes do and mention any bugs/issues related to the pull request. 
+
+### Developing
+The data shipper is developed in [Go](http://golang.org/) so install the [version](https://github.com/elastic/elastic-agent-shipper/blob/main/.go-version) 
+which is being used for shipper development. One deterministic manner to install the proper Go version to work with the shipper is to use the
+[GVM](https://github.com/andrewkroh/gvm) Go version manager.
+
+The data shipper primarily uses the [mage](https://magefile.org/) build system. The list of supported mage commands can be obtained
+by running `mage -l` from the root of the repository. The most commonly used commands are:
+
+* `mage build` to build the data shipper binary.
+* `mage check` to check license files and dependencies.
+* `mage lint` to lint the source code using [golangci-lint](https://golangci-lint.run/).
+* `go test ./...` to run all tests.
+
+Run mage commands with the `-v` flag for more detailed output, for example `mage -v check`.

config/config.go

@@ -23,7 +23,6 @@ import (
 
 const (
 	defaultConfigName = "elastic-agent-shipper.yml"
-	defaultPort       = 50051
 )
 
 var (
@@ -41,10 +40,6 @@ func init() {
 //ShipperConfig defines the options present in the config file
 type ShipperConfig struct {
 	Log     logp.Config       `config:"logging"`
-	TLS     bool              `config:"tls"`
-	Cert    string            `config:"cert"`       //TLS cert file, if TLS is enabled
-	Key     string            `config:"key"`        //TLS Keyfile, if specified
-	Port    int               `config:"port"`       //Port to listen on
 	Monitor monitoring.Config `config:"monitoring"` //Queue monitoring settings
 	Queue   queue.Config      `config:"queue"`      //Queue settings
 	Server  server.Config     `config:"server"`     //gRPC Server settings
@@ -65,7 +60,6 @@ func ReadConfig() (ShipperConfig, error) {
 	}
 	// systemd environment will send us to stdout environment, which we want
 	config := ShipperConfig{
-		Port:    defaultPort,
 		Log:     logp.DefaultConfig(logp.SystemdEnvironment),
 		Monitor: monitoring.DefaultConfig(),
 		Queue:   queue.DefaultConfig(),
@@ -101,7 +95,6 @@ func ReadConfigFromJSON(raw string) (ShipperConfig, error) {
 		return ShipperConfig{}, fmt.Errorf("error parsing string config: %w", err)
 	}
 	shipperConfig := ShipperConfig{
-		Port:    defaultPort,
 		Log:     logp.DefaultConfig(logp.SystemdEnvironment),
 		Monitor: monitoring.DefaultConfig(),
 		Queue:   queue.DefaultConfig(),

controller/run.go

@@ -84,7 +84,7 @@ func (c *clientHandler) Run(cfg config.ShipperConfig, unit *client.Unit) error {
 	out := output.NewConsole(queue)
 	out.Start()
 
-	lis, err := net.Listen("tcp", fmt.Sprintf("localhost:%d", cfg.Port))
+	lis, err := net.Listen("tcp", fmt.Sprintf("localhost:%d", cfg.Server.Port))
 	if err != nil {
 		return fmt.Errorf("failed to listen: %w", err)
 	}
@@ -97,8 +97,8 @@ func (c *clientHandler) Run(cfg config.ShipperConfig, unit *client.Unit) error {
 	_ = unit.UpdateState(client.UnitStateConfiguring, "starting shipper server", nil)
 
 	var opts []grpc.ServerOption
-	if cfg.TLS {
-		creds, err := credentials.NewServerTLSFromFile(cfg.Cert, cfg.Key)
+	if cfg.Server.TLS {
+		creds, err := credentials.NewServerTLSFromFile(cfg.Server.Cert, cfg.Server.Key)
 		if err != nil {
 			return fmt.Errorf("failed to generate credentials %w", err)
 		}
@@ -122,7 +122,7 @@ func (c *clientHandler) Run(cfg config.ShipperConfig, unit *client.Unit) error {
 		shipperServer.Close()
 	}
 	handleShutdown(shutdownFunc, c.shutdownInit)
-	log.Debugf("gRPC server is listening on port %d", cfg.Port)
+	log.Debugf("gRPC server is listening on port %d", cfg.Server.Port)
 	_ = unit.UpdateState(client.UnitStateHealthy, "Shipper Running", nil)
 
 	// This will get sent after the server has shutdown, signaling to the runloop that it can stop.