feat: merge connector and operator repos into this repo (#72)

Because
- As our project expands to include various component types such as
generic, iterator, and AI components, managing separate repositories for
each becomes less practical. Although the initial plan aimed to
integrate them directly into the component repository, the introduction
of the new AI Component poses challenges. Given that the AI Component
relies on the implementation of all connectors in the connector
repository, including it in the component repository would lead to
cyclic dependencies. Therefore, the optimal solution is to consolidate
all components into a single repository.

This commit

- Merges code from [connector](https://github.com/instill-ai/connector)
and [operator](https://github.com/instill-ai/operator) into this
repository.
- Removes unused `objectmapper`.

---------

Co-authored-by: Juan Vallés <3977183+jvallesm@users.noreply.github.com>

.github/CONTRIBUTING.md

@@ -42,7 +42,7 @@ There are different types of component:
 
 **Connector**
 
-- **Connectors** are used for connecting the pipeline to a vendor service. They are defined and initialized in the [connector](https://github.com/instill-ai/connector/) repository.
+- **Connectors** are used for connecting the pipeline to a vendor service. They are defined and initialized in the [connector](../pkg/connector) package.
 - A connector **resource** needs to be set up first to configure the connection.
 - Setup a Connector
 ```mermaid
@@ -55,7 +55,7 @@ sequenceDiagram
 
 **Operator**
 
-- An operator is used for data operations inside the pipeline. They are defined and initialized in the [operator](https://github.com/instill-ai/operator/) repository.
+- An operator is used for data operations inside the pipeline. They are defined and initialized in the [operator](../pkg/operator) package.
 
 The key difference between `connector` and `operator` is that the former will connect to an external service, so it's **I/O bound** while the latter is **CPU bound**. Connectors don't process but transfer data.
 
@@ -122,7 +122,7 @@ When you want to contribute with a new connector or operator, you need to create
 2 configuration files define the behaviour of the component:
 
 - `definitions.json`
-    - You can refer to [OpenAI connector](https://github.com/instill-ai/connector/blob/main/pkg/openai/v0/config/definitions.json) as an example.
+    - You can refer to [OpenAI connector](../pkg/connector/openai/v0/config/definitions.json) as an example.
     - We define the id, uid, vendor info and other metadata in this file.
       - `uid` MUST be a unique UUID. Once it is set, it MUST NOT change.
       - `version` MUST be a [SemVer](https://semver.org/) string.
@@ -134,7 +134,7 @@ When you want to contribute with a new connector or operator, you need to create
         This will showcase the upcoming component at [instill.tech](https://instill.tech).
     - We define the `resource_configuration` in this file, which defines the connector resource setup.
 - `tasks.json`
-    - You can refer to [OpenAI connector](https://github.com/instill-ai/connector/blob/main/pkg/openai/v0/config/tasks.json) as an example.
+    - You can refer to [OpenAI connector](../pkg/connector/openai/v0/config/tasks.json) as an example.
     - A component can have multiple tasks.
     - The input and output schema of each task is defined in this file.
 
@@ -175,7 +175,7 @@ type IExecution interface {
 
 <!--
 TODO:
- 1. explain how we import the connectors or operators like [here](https://github.com/instill-ai/connector/blob/main/pkg/main.go)
+ 1. explain how we import the connectors or operators like [here](../pkg/connector/main.go)
  2. Add a step by step example to implement a new connector or operator.
 -->
 
@@ -216,12 +216,6 @@ The typical version and release stage evolution of a component might look like t
 | 0.4.0 | `RELEASE_STAGE_BETA` |
 | 1.0.0 | `RELEASE_STAGE_GA` |
 
-#### Repositories
-
-Currently, we maintain four repositories for component implementations
-- [Connector](https://github.com/instill-ai/connector): collect all connector implementations
-- [Operator](https://github.com/instill-ai/operator): collect all operator implementations
-
 ### Sending PRs
 
 Please take these general guidelines into consideration when you are sending a PR:

.github/workflows/coverage.yml

@@ -10,6 +10,9 @@ jobs:
       contents: read
       pull-requests: write
     steps:
+      - name: Install operator dependencies
+        run: sudo apt-get update -y && sudo apt-get install poppler-utils wv unrtf tidy tesseract-ocr libtesseract-dev -y
+
       - uses: actions/checkout@v3
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
@@ -29,7 +32,7 @@ jobs:
       - name: Generate coverage report
         run: |
           go mod tidy
-          go test -race ./... -coverprofile=coverage.txt -covermode=atomic
+          go test -tags ocr -race ./... -coverprofile=coverage.txt -covermode=atomic
 
       - name: Upload coverage report
         uses: codecov/codecov-action@v2

go.mod

@@ -3,40 +3,134 @@ module github.com/instill-ai/component
 go 1.21
 
 require (
+	cloud.google.com/go/bigquery v1.57.1
+	cloud.google.com/go/iam v1.1.5
+	cloud.google.com/go/storage v1.30.1
+	code.sajari.com/docconv v1.3.8
+	github.com/JohannesKaufmann/html-to-markdown v1.5.0
+	github.com/PuerkitoBio/goquery v1.9.1
+	github.com/allegro/bigcache v1.2.1
+	github.com/docker/docker v25.0.4+incompatible
+	github.com/fogleman/gg v1.3.0
 	github.com/frankban/quicktest v1.14.6
 	github.com/gabriel-vasile/mimetype v1.4.3
+	github.com/go-resty/resty/v2 v2.12.0
+	github.com/gocolly/colly/v2 v2.1.0
 	github.com/gofrs/uuid v4.4.0+incompatible
+	github.com/h2non/filetype v1.1.3
 	github.com/instill-ai/protogen-go v0.3.3-alpha.0.20240306151355-4398dad0ba73
+	github.com/instill-ai/x v0.4.0-alpha
+	github.com/itchyny/gojq v0.12.14
 	github.com/lestrrat-go/jspointer v0.0.0-20181205001929-82fadba7561c
 	github.com/lestrrat-go/jsref v0.0.0-20211028120858-c0bcbb5abf20
 	github.com/lestrrat-go/option v1.0.0
 	github.com/lestrrat-go/pdebug v0.0.0-20210111095411-35b07dbf089b
 	github.com/lestrrat-go/structinfo v0.0.0-20210312050401-7f8bd69d6acb
 	github.com/pkg/errors v0.9.1
+	github.com/pkoukk/tiktoken-go v0.1.6
+	github.com/redis/go-redis/v9 v9.5.1
 	github.com/santhosh-tekuri/jsonschema/v5 v5.3.0
-	github.com/stretchr/testify v1.8.0
+	github.com/stretchr/testify v1.8.4
 	go.uber.org/zap v1.24.0
-	golang.org/x/text v0.13.0
+	golang.org/x/image v0.15.0
+	golang.org/x/text v0.14.0
+	google.golang.org/api v0.149.0
+	google.golang.org/grpc v1.61.1
 	google.golang.org/protobuf v1.33.0
 )
 
 require (
-	cloud.google.com/go/longrunning v0.5.1 // indirect
+	cloud.google.com/go v0.111.0 // indirect
+	cloud.google.com/go/compute v1.23.3 // indirect
+	cloud.google.com/go/compute/metadata v0.2.3 // indirect
+	cloud.google.com/go/longrunning v0.5.4 // indirect
+	github.com/JalfResi/justext v0.0.0-20170829062021-c0282dea7198 // indirect
+	github.com/Microsoft/go-winio v0.4.14 // indirect
+	github.com/advancedlogic/GoOse v0.0.0-20191112112754-e742535969c1 // indirect
+	github.com/andybalholm/brotli v1.0.4 // indirect
+	github.com/andybalholm/cascadia v1.3.2 // indirect
+	github.com/antchfx/htmlquery v1.2.3 // indirect
+	github.com/antchfx/xmlquery v1.2.4 // indirect
+	github.com/antchfx/xpath v1.1.8 // indirect
+	github.com/apache/arrow/go/v12 v12.0.0 // indirect
+	github.com/apache/thrift v0.16.0 // indirect
+	github.com/araddon/dateparse v0.0.0-20200409225146-d820a6159ab1 // indirect
+	github.com/cespare/xxhash/v2 v2.2.0 // indirect
+	github.com/containerd/log v0.1.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
+	github.com/distribution/reference v0.5.0 // indirect
+	github.com/dlclark/regexp2 v1.10.0 // indirect
+	github.com/docker/go-connections v0.5.0 // indirect
+	github.com/docker/go-units v0.5.0 // indirect
+	github.com/fatih/set v0.2.1 // indirect
+	github.com/felixge/httpsnoop v1.0.4 // indirect
+	github.com/gigawattio/window v0.0.0-20180317192513-0f5467e35573 // indirect
+	github.com/go-logr/logr v1.4.1 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
+	github.com/gobwas/glob v0.2.3 // indirect
+	github.com/goccy/go-json v0.9.11 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
+	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
 	github.com/golang/protobuf v1.5.3 // indirect
-	github.com/google/go-cmp v0.5.9 // indirect
-	github.com/grpc-ecosystem/grpc-gateway/v2 v2.7.2 // indirect
+	github.com/golang/snappy v0.0.4 // indirect
+	github.com/google/flatbuffers v2.0.8+incompatible // indirect
+	github.com/google/go-cmp v0.6.0 // indirect
+	github.com/google/s2a-go v0.1.7 // indirect
+	github.com/google/uuid v1.4.0 // indirect
+	github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
+	github.com/googleapis/gax-go/v2 v2.12.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
+	github.com/itchyny/timefmt-go v0.1.5 // indirect
+	github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7 // indirect
+	github.com/kennygrant/sanitize v1.2.4 // indirect
+	github.com/klauspost/asmfmt v1.3.2 // indirect
+	github.com/klauspost/compress v1.15.9 // indirect
+	github.com/klauspost/cpuid/v2 v2.0.9 // indirect
 	github.com/kr/pretty v0.3.1 // indirect
 	github.com/kr/text v0.2.0 // indirect
+	github.com/levigross/exp-html v0.0.0-20120902181939-8df60c69a8f5 // indirect
+	github.com/mattn/go-runewidth v0.0.15 // indirect
+	github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
+	github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
+	github.com/moby/term v0.5.0 // indirect
+	github.com/morikuni/aec v1.0.0 // indirect
+	github.com/olekukonko/tablewriter v0.0.4 // indirect
+	github.com/opencontainers/go-digest v1.0.0 // indirect
+	github.com/opencontainers/image-spec v1.1.0 // indirect
+	github.com/otiai10/gosseract/v2 v2.2.4 // indirect
+	github.com/pierrec/lz4/v4 v4.1.15 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/richardlehane/mscfb v1.0.3 // indirect
+	github.com/richardlehane/msoleps v1.0.3 // indirect
+	github.com/rivo/uniseg v0.4.4 // indirect
 	github.com/rogpeppe/go-internal v1.9.0 // indirect
-	go.uber.org/atomic v1.7.0 // indirect
+	github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca // indirect
+	github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf // indirect
+	github.com/temoto/robotstxt v1.1.1 // indirect
+	github.com/zeebo/xxh3 v1.0.2 // indirect
+	go.opencensus.io v0.24.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
+	go.opentelemetry.io/otel v1.24.0 // indirect
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.24.0 // indirect
+	go.opentelemetry.io/otel/metric v1.24.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.24.0 // indirect
+	go.opentelemetry.io/otel/trace v1.24.0 // indirect
+	go.uber.org/atomic v1.9.0 // indirect
 	go.uber.org/multierr v1.6.0 // indirect
-	golang.org/x/net v0.17.0 // indirect
-	golang.org/x/sys v0.13.0 // indirect
-	google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc // indirect
-	google.golang.org/genproto/googleapis/api v0.0.0-20230530153820-e85fd2cbaebc // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
-	google.golang.org/grpc v1.56.3 // indirect
+	golang.org/x/crypto v0.21.0 // indirect
+	golang.org/x/mod v0.10.0 // indirect
+	golang.org/x/net v0.22.0 // indirect
+	golang.org/x/oauth2 v0.15.0 // indirect
+	golang.org/x/sync v0.5.0 // indirect
+	golang.org/x/sys v0.18.0 // indirect
+	golang.org/x/tools v0.9.1 // indirect
+	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
+	google.golang.org/appengine v1.6.8 // indirect
+	google.golang.org/genproto v0.0.0-20231212172506-995d672761c0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20240102182953-50ed04b92917 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240102182953-50ed04b92917 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
+	gotest.tools/v3 v3.5.1 // indirect
 )