specgen docs

docs/1-using/4-connectors/6-configuration-parameters/2-schema-extraction.mdx

@@ -26,9 +26,14 @@ connectors):
 
 
 :::caution
-`sdk.schema.extract.payload.enabled` and `sdk.schema.extract.key.enabled` should be set to `false` when producing raw (not structured) data, as shown in the example below. 
-
-If you are developing a connector, you can disable this automatically by updating the connector's default middleware. For more information about `NewSource()` when developing a source connector, see [here](/docs/developing/connectors/developing-source-connectors/#newsource).
+`sdk.schema.extract.payload.enabled` and `sdk.schema.extract.key.enabled` should
+be set to `false` when producing raw (not structured) data, as shown in the
+example below.
+
+If you are developing a connector, you can disable this automatically by
+updating the connector's default middleware. For more information about
+`NewSource()` when developing a source connector,
+see [here](/docs/developing/connectors/developing-source-connectors/#newsource).
 :::
 
 ## Example
@@ -66,7 +71,8 @@ pipelines:
           path: /tmp/file-destination.txt
 ```
 
-When the pipeline is run, `/tmp/file-destination.txt` will contain output similar to this:
+When the pipeline is run, `/tmp/file-destination.txt` will contain an output
+similar to this:
 
 ```json
 {

docs/2-developing/0-connectors/2-connector-specification.mdx

@@ -2,25 +2,160 @@
 title: "Connector Specification"
 ---
 
-[Specification](https://pkg.go.dev/github.com/conduitio/conduit-connector-sdk#Specification) contains general information regarding the plugin like its name and what it does.
+A connector's specification is used to present information about the connector
+and validate connector parameters. The specification can be found in the
+`connectors.yaml` file, that's found in the root of a connector's repository.
+The following is an example of
+the [file connector's](https://github.com/ConduitIO/conduit-connector-file)
+`connector.yaml`:
 
-### `spec.go`
+```yaml
+version: "1.0"
+specification:
+  name: file
+  summary: A file source and destination plugin for Conduit.
+  description: |-
+    The file source allows you to listen to a local file and
+    detect any changes happening to it. Each change will create a new record. The
+    destination allows you to write record payloads to a destination file, each new
+    record payload is appended to the file in a new line.
+  version: v0.10.0
+  author: Meroxa, Inc.
+  source:
+    parameters:
+      - name: path
+        description: Path is the file path used by the connector to read/write records.
+        type: string
+        default: ""
+        validations:
+          - type: required
+            value: ""
+    # other parameters
+```
+
+The contents of this file are embedded into a connector's binary and read by the
+connector binary itself. `connector.yaml` is also being used by other tools that
+enrich Conduit's functionality, such as the CLI, the documentation website, etc.
+
+## Contents of `connector.yaml`
+
+A part of `connector.yaml` is written by a developer (manually) and the other
+part is generated by a tool that Conduit provides, [
+`specgen`](https://github.com/ConduitIO/conduit-connector-sdk/tree/main/specgen).
+
+A developer is writing the general information about the connector, which can be
+seen in this part from the example above:
+
+```yaml
+specification:
+  name: file
+  summary: A file source and destination plugin for Conduit.
+  description: |-
+    The file source allows you to listen to a local file and
+    detect any changes happening to it. Each change will create a new record. The
+    destination allows you to write record payloads to a destination file, each new
+    record payload is appended to the file in a new line.
+  version: v0.10.0
+  author: Meroxa, Inc.
+```
+
+Additional information may be added outside the `specification` key, for
+example:
+
+```yaml
+version: "1.0"
+specification:
+  # specification section
+# developer-info is just an example, any key can be used
+developer-info:
+  email: contact@example.com
+  website: https://example.com
+```
+
+The information about a connector's source and destination parameters is
+generated using a tool called [
+`specgen`](https://github.com/ConduitIO/conduit-connector-sdk/tree/main/specgen).
+`specgen` uses the configuration structs in a connector's code to generate the
+list of parameters, the validations for those, etc. `specgen` is described in
+more details in the following section.
+
+## `specgen`
+
+`specgen` works by inspecting the configuration structs that the source and
+destination connectors return in the `Config()` method. Those configuration
+structs will be transformed into a map of parameter descriptions that's written
+into `connector.yaml`. More information about source and destination
+configuration can be found
+in [Developing a Source Connector](/docs/developing/connectors/developing-source-connectors)
+and [Developing a Destination Connector](/docs/developing/connectors/developing-destination-connectors).
+
+If you're using
+the [Conduit Connector Template](/docs/developing/connectors/conduit-connector-template),
+then the `specgen` tool is already listed in `tools.go` and can be installed
+with `make install-tools`.
+
+`specgen` is run as part of `go generate`. It also needs access to the
+`sdk.Connector` variable that holds references to the constructor functions for the
+source and the destination, so it's best to place it in the `connector.go` file.
+The following is an example from the Kafka connector:
+
+```go
+//go:generate specgen
+
+// Package kafka contains implementations for Kafka source and destination
+// connectors for Conduit.
+package kafka
+
+import (
+	_ "embed"
+
+	sdk "github.com/conduitio/conduit-connector-sdk"
+)
+
+//go:embed connector.yaml
+var specs string
+
+var Connector = sdk.Connector{
+	NewSpecification: sdk.YAMLSpecification(specs),
+	NewSource:        NewSource,
+	NewDestination:   NewDestination,
+}
+```
 
-This file defines the public-facing characteristics of the connector such as its name and capabilities. Begin by updating placeholder values with the specific details of your connector.
+If you run `go generate ./...` or just use `make generate` (that's provided by
+the connector template), you'll see that the `connector.yaml` is updated with
+the source and/or destination parameters.
+
+What you also see in the example above is how to embed the `connector.yaml`
+file (with `//go:embed connector.yaml`) and then extract the specification from
+it (`NewSpecification: sdk.YAMLSpecification(specs)`).
+
+## Default values
+
+Default values are taken from the following sources (higher to lower precedence):
+
+1. A source or destination connector's configuration (returned by a connector's
+   `Config()` method).
+2. The `default` tag on fields in a configuration struct.
+
+For example:
 
 ```go
-var version = "v0.1.0"
-
-// Specification returns the connector's specification.
-func Specification() sdk.Specification {
- return sdk.Specification{
-  Name:        "file-sync",
-  Summary:     "<describe your connector>",
-  Description: "<describe your connector in detail>",
-  Version:     version,
-  Author:      "<your name>",
- }
+type SourceConfig struct {
+	URL string `default:"http://localhost:8080"`
+}
+
+func NewSource() sdk.Source {
+    return sdk.SourceWithMiddleware(&Source{
+        config: source.Config{
+            URL: "https://example.com"
+        },
+    })
 }
 ```
 
-![scarf pixel conduit-site-docs-developing-connectors](https://static.scarf.sh/a.png?x-pxid=3ada0949-fa61-40d6-a44a-76447ea4e39f)
+In this example, the default value in the generated specification will be
+`"https://example.com"`, i.e. if a user doesn't specify a URL, its value will be
+`"https://example.com"`.
+
+![scarf pixel conduit-site-docs-developing-connectors](https://static.scarf.sh/a.png?x-pxid=3ada0949-fa61-40d6-a44a-76447ea4e39f)