Skip to content
This repository has been archived by the owner on Mar 25, 2023. It is now read-only.

Commit

Permalink
Update docs to include new FDW features + elasticsearch FDW
Browse files Browse the repository at this point in the history
  • Loading branch information
mildbyte committed Sep 15, 2020
1 parent 61ae642 commit 6705e2d
Show file tree
Hide file tree
Showing 6 changed files with 77 additions and 11 deletions.
2 changes: 1 addition & 1 deletion content/docs/0000_getting-started/0100_introduction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ You do not need to download the full Splitgraph image to query it. Instead, you

Splitgraph does not limit your data sources to Postgres databases. It includes first-class support for importing and querying data from other databases using Postgres [foreign data wrappers](../ingesting-data/foreign-data-wrappers/introduction). You can create Splitgraph images or query data in [MongoDB](../ingesting-data/foreign-data-wrappers/load-mongo-collections),
[MySQL](../ingesting-data/foreign-data-wrappers/load-mysql-tables),
[CSV files](../ingesting-data/load-csv-files) or [other Postgres databases](../ingesting-data/foreign-data-wrappers/load-postgres-tables) using the same interface.
[CSV files](../ingesting-data/load-csv-files), [other Postgres databases](../ingesting-data/foreign-data-wrappers/load-postgres-tables) or even [Elasticsearch clusters](../ingesting-data/foreign-data-wrappers/load-elasticsearch-index) using the same interface.

Finally, Splitgraph is peer-to-peer. You can push and pull data images between other Splitgraph installations and use it as a standalone tool to supercharge your data workflows.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ which will let you ingest data from existing databases without installing Splitg

### Does my data have to be in PostgreSQL to use Splitgraph?

With [mounting](../concepts/mounting), you can query data in other databases (including MongoDB, MySQL or PostgreSQL) directly through Spligraph with
With [mounting](../concepts/mounting), you can query data in other databases (including MongoDB, MySQL, PostgreSQL or Elasticsearch) directly through Spligraph with
[any PostgreSQL client](../integrating-splitgraph/other-clients). You do not need to copy your data into PostgreSQL to use Splitgraph.

### Why PostgreSQL? Why not write your own database?
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ Foreign tables act similarly to normal tables and can accept the same SQL querie

In Splitgraph terminology, this is called "mounting", kind of like mounting a filesystem to mount points in the directory tree. You can snapshot these mounted tables, turning them into Splitgraph images, with the [`sgr import`](../../sgr/image-management-creation/import) command.

The Splitgraph engine ships with several open-source foreign data wrappers: [`postgres_fdw`](https://www.postgresql.org/docs/12/postgres-fdw.html), [`mongo_fdw`](https://github.com/EnterpriseDB/mongo_fdw) and [`mysql_fdw`](https://github.com/EnterpriseDB/mysql_fdw).
The Splitgraph engine ships with several open-source foreign data wrappers: [`postgres_fdw`](https://www.postgresql.org/docs/12/postgres-fdw.html), [`mongo_fdw`](https://github.com/EnterpriseDB/mongo_fdw), [`mysql_fdw`](https://github.com/EnterpriseDB/mysql_fdw) and [`pg_es_fdw`](https://github.com/splitgraph/postgres-elasticsearch-fdw).
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,29 @@ export const meta = {
To use [`postgres_fdw`](https://www.postgresql.org/docs/12/postgres-fdw.html) and mount remote PostgreSQL databases, run:

```
$ sgr mount postgres_fdw \\
-c username:password@server:port \\
-o '{"dbname": remote_db_name, "remote_schema": remote_schema_name, \\
"tables_to_mount": ["table_1", "table_2"]}' \\
local_schema
$ sgr mount postgres_fdw local_schema -c username:password@server:port -o@- <<EOF
{
"dbname": remote_db_name,
"remote_schema": remote_schema_name,
"tables": ["table_1", "table_2"],
"extra_server_args": [optional, e.g. {"use_remote_estimate": "true"}]
}
EOF
```

`"tables_to_mount"` can be omitted (in which case all tables in `remote_schema` will be mounted into `local_schema`).
`"tables"` can be omitted (in which case all tables in `remote_schema` will be mounted into `local_schema`).

You can also pass a dictionary of tables and their schema as `tables`. This will override the schema that's inferred by `IMPORT FOREIGN SCHEMA`. For example:

```json
{
"tables": {
"table_1": {
"col_1": "integer",
"col_2": "text"
}
}
}
```

To snapshot the data, making it part of an actual Splitgraph image, use the [`sgr import`](../../sgr/image-management-creation/import) command.
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,31 @@ export const meta = {
To mount a MySQL database into the Splitgraph engine using [`mysql_fdw`](https://github.com/EnterpriseDB/mysql_fdw), run:

```
$ sgr mount mysql_fdw -c username:password@host:port \\
-o '{"remote_schema": "remote_schema"}' schema_name
$ sgr mount mysql_fdw local_schema -c username:password@host:port -o@- <<EOF
{
"remote_schema": "remote_schema",
"tables": ["table_1"],
}
```

This will mount a MySQL schema `remote_schema` into a local schema `schema_name` on the
Splitgraph engine. Note that if your MySQL server is running on localhost and you wish to
access it over a TCP connection, you should use `127.0.0.1` instead of `localhost` as the `host`.

`"tables"` can be omitted (in which case all tables in `remote_schema` will be mounted into `local_schema`).

You can also pass a dictionary of tables and their schema as `tables`. This will override the schema that's inferred by `IMPORT FOREIGN SCHEMA`. For example:

```json
{
"tables": {
"table_1": {
"col_1": "integer",
"col_2": "text"
}
}
}
```

To snapshot the table, making it part of an actual Splitgraph image, use the [`sgr import`](../../sgr/image-management-creation/import) command.
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
export const meta = {
id: "load-elasticsearch-index",
title: "Loading Elasticsearch indexes",
};

To mount a remote Elasticsearch index into the Splitgraph engine using Splitgraph's fork of [`pg_es_fdw`](https://github.com/splitgraph/postgres-elasticsearch-fdw), run:

```
$ sgr mount elasticsearch -c elasticsearch:9200 -o@- <<EOF
{
"table_spec": {
"table_1": {
"schema": {
"id": "text",
"@timestamp": "timestamp",
"query": "text",
"col_1": "text",
"col_2": "boolean",
},
"index": "index-pattern*",
"rowid_column": "id",
"query_column": "query",
}
}
}
EOF
```

The FDW translates local queries into the [Elasticsearch Query DSL](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html). You can also pass Lucene query strings as `query_column` equality clauses. For example, if `query_column` is set to `query`, you can use `WHERE query='text_body:chess'`).

To snapshot the table, making it part of an actual Splitgraph image, use the [`sgr import`](../../sgr/image-management-creation/import) command.

0 comments on commit 6705e2d

Please sign in to comment.