Skip to content

Commit

Permalink
chore: link to docs (#2124)
Browse files Browse the repository at this point in the history
* chore: link to docs

* fix

* fix
  • Loading branch information
raulb authored Feb 5, 2025
1 parent 9496002 commit 432528b
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 109 deletions.
41 changes: 4 additions & 37 deletions docs/connector_discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,43 +2,10 @@

## Connectors' location

Conduit loads standalone connectors at startup. The connector binaries need to be placed in the `connectors` directory
relative to the Conduit binary so Conduit can find them. Alternatively, the path to the standalone connectors can be
adjusted using the CLI flag `-connectors.path`, for example:

```shell
./conduit run -connectors.path=/path/to/connectors/
```

Names of the connector binaries are not important, since Conduit is getting the information about connectors from
connectors themselves (using their gRPC API).
> [!IMPORTANT]
> The content of this page has been moved to [Conduit's documentation website](https://conduit.io/docs/using/connectors/installing/).
## Referencing connectors

The name used to reference a connector in API requests (e.g. to create a new connector) comes in the following format:

`[PLUGIN-TYPE:]PLUGIN-NAME[@VERSION]`

- `PLUGIN-TYPE` (`builtin`, `standalone` or `any`)
- Defines if the specified plugin should be built-in or standalone.
- If `any`, Conduit will use a standalone plugin if it exists and fall back to a built-in plugin.
- Default is `any`.
- `PLUGIN-NAME`
- Defines the name of the plugin as specified in the plugin specifications, it has to be an exact match.
- `VERSION`
- Defines the plugin version as specified in the plugin specifications, it has to be an exact match.
- If `latest`, Conduit will use the latest semantic version.
- Default is `latest`.

Examples:

- `postgres`
- will use the **latest** **standalone** **postgres** plugin
- will fallback to the **latest** **builtin** **postgres** plugin if standalone wasn't found
- `postgres@v0.2.0`
- will use the **standalone** **postgres** plugin with version **v0.2.0**
- will fallback to a **builtin** **postgres** plugin with version **v0.2.0** if standalone wasn't found
- `builtin:postgres`
- will use the **latest** **builtin** **postgres** plugin
- `standalone:postgres@v0.3.0`
- will use the **standalone** **postgres** plugin with version **v0.3.0** (no fallback to builtin)
> [!IMPORTANT]
> The content of this page has been moved to [Conduit's documentation website](https://conduit.io/docs/using/connectors/referencing).
83 changes: 11 additions & 72 deletions docs/connectors.md
Original file line number Diff line number Diff line change
@@ -1,91 +1,30 @@
# Conduit Connectors

Connectors are an integral part of Conduit. Conduit ships with a couple of
[connectors that are directly built into](#built-in-connectors) the service,
but it can also be expanded with additional
[standalone connectors](#standalone-connectors).

One of the main differences between Conduit connectors and those that you might find from other services is
that all Conduit connectors are Change Data Capture-first (CDC).

CDC allows your pipeline to only get the changes that have happened over time
instead of pulling down an entire upstream data store and then tracking diffs
between some period of time.
This is critical for building real-time, event-driven pipelines and applications.
But, we'll note where connectors do or do not have CDC capabilities.
> [!IMPORTANT]
> The content of this page has been moved to [Conduit's documentation website](https://conduit.io/docs/core-concepts#connector).
## Roadmap & Feedback

If you need support for a particular data store that doesn't exist on the connector list, check out the list of
requested [source connectors](https://github.com/ConduitIO/conduit/issues?q=is%3Aissue+label%3Aconnector%3Asource+is%3Aopen)
and the list of requested [destination connectors](https://github.com/ConduitIO/conduit/issues?q=is%3Aissue+label%3Aconnector%3Adestination+is%3Aopen).
Give the issue a `+1` if you really need that connector. The upvote will help the team understand demand for any
particular connector. If you find that an issue hasn't been created for your data store, please create a new issue in
the Conduit repo.
> [!IMPORTANT]
> The content of this page has been moved to [Conduit's documentation website](https://conduit.io/docs/future/roadmap).
## More about Connectors

### Built-in connectors

To help developers bootstrap pipelines much more quickly,
Conduit ships with several built-in connectors by default.
This includes the Postgres, File, Random Data Generator, Kafka and Amazon S3 connectors.

If you are creating and distributing customized, pre-built Conduit binaries yourself,
you may want to modify the list of built-in connectors, too.
By modifying the list of built-in connectors,
your compiled binary will include a set of pre-installed connectors specific to your end users' needs,
and it can be run by others, as is, without them having to follow any additional installation instructions.

If you want to create your own Conduit binary with a different set of built-in connectors,
you can build-in an [existing connector](#the-list).
Alternatively, you can also create your own
[using the Conduit connector template or the Conduit Connector SDK](https://conduit.io/docs/developing/connectors).

Once you have chosen a connector to be built-in, you can:

- Download the new package and its dependencies: `go get "github.com/foo/conduit-connector-new"`
- Import the Go module defining the connector
into the [builtin registry](https://github.com/ConduitIO/conduit/blob/main/pkg/plugin/connector/builtin/registry.go)
and add a new key to `DefaultDispenserFactories`:

```diff
package builtin

import (
// ...
file "github.com/conduitio/conduit-connector-file"
// ...
+ myNewConnector "github.com/foo/conduit-connector-new"
)

var (
// DefaultDispenserFactories contains default dispenser factories for
// built-in plugins. The key of the map is the import path of the module
// containing the connector implementation.
DefaultDispenserFactories = map[string]DispenserFactory{
"github.com/conduitio/conduit-connector-file": sdkDispenserFactory(file.Connector),
// ...
+ "github.com/foo/conduit-connector-new": sdkDispenserFactory(myNewConnector.Connector),
}
)
```

- Run `make`
- You now have a binary with your own, custom set of built-in connectors! 🎉
> [!IMPORTANT]
> The content of this page has been moved to Conduit's documentation website
> [here](https://conduit.io/docs/core-concepts#built-in-connector), and [here](/docs/using/connectors/additional-built-in-plugins).
### Standalone connectors

In addition to built-in connectors, Conduit can be used together with standalone connectors
to enable even more data streaming use cases.
The Conduit team and other community developers create and maintain standalone connectors.

Learn more about how you can install [standalone connectors to Conduit here](https://conduit.io/docs/using/connectors/installing).
> [!IMPORTANT]
> The content of this page has been moved to [Conduit's documentation website](https://conduit.io/docs/core-concepts#standalone-connector).
### Source & Destination

Source means the connector has the ability to get data from an upstream data store. Destination means the connector can
to write to a downstream data store.
> [!IMPORTANT]
> The content of this page has been moved to [Conduit's documentation website](https://conduit.io/docs/core-concepts#source-and-destination).
### The List

Expand Down
Empty file removed docs/storage.md
Empty file.

0 comments on commit 432528b

Please sign in to comment.