Skip to content

Commit

Permalink
DOC: Add new backends and geo spatial documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
xmnlab committed Dec 6, 2019
1 parent 631a8e9 commit 7441683
Show file tree
Hide file tree
Showing 7 changed files with 200 additions and 34 deletions.
171 changes: 171 additions & 0 deletions docs/source/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,66 @@ Use ``ibis.sqlite.connect`` to create a SQLite client.
SQLiteClient.list_tables
SQLiteClient.table

.. _api.mysql:

MySQL client (Experimental)
---------------------------
.. currentmodule:: ibis.sql.mysql.api

The MySQL client is accessible through the ``ibis.mysql`` namespace.

Use ``ibis.mysql.connect`` with a SQLAlchemy-compatible connection string to
create a client.

.. autosummary::
:toctree: generated/

connect
MySQLClient.database
MySQLClient.list_databases
MySQLClient.list_tables
MySQLClient.table

.. _api.omniscidb:

OmniSciDB client (Experimental)
-------------------------------
.. currentmodule:: ibis.omniscidb.api

The OmniSciDB client is accessible through the ``ibis.omniscidb`` namespace.

Use ``ibis.omniscidb.connect`` to create a client.

.. autosummary::
:toctree: generated/

compile
connect
verify
OmniSciDBClient.alter_user
OmniSciDBClient.close
OmniSciDBClient.create_database
OmniSciDBClient.create_table
OmniSciDBClient.create_user
OmniSciDBClient.create_view
OmniSciDBClient.database
OmniSciDBClient.describe_formatted
OmniSciDBClient.drop_database
OmniSciDBClient.drop_table
OmniSciDBClient.drop_table_or_view
OmniSciDBClient.drop_user
OmniSciDBClient.drop_view
OmniSciDBClient.exists_table
OmniSciDBClient.get_schema
OmniSciDBClient.list_tables
OmniSciDBClient.load_data
OmniSciDBClient.log
OmniSciDBClient.set_database
OmniSciDBClient.sql
OmniSciDBClient.table
OmniSciDBClient.truncate_table
OmniSciDBClient.version

.. _api.hdfs:

HDFS
Expand All @@ -199,9 +259,49 @@ HDFS.
HDFS.size
HDFS.status

.. _api.spark:

SparkSQL client (Experimental)
------------------------------
.. currentmodule:: ibis.spark.api

The Spark SQL client is accessible through the ``ibis.spark`` namespace.

Use ``ibis.spark.connect`` to create a client.

.. autosummary::
:toctree: generated/

connect
SparkClient.database
SparkClient.list_databases
SparkClient.list_tables
SparkClient.table

.. _api.pyspark:

PySpark client (Experimental)
-----------------------------
.. currentmodule:: ibis.pyspark.api

The PySpark client is accessible through the ``ibis.pyspark`` namespace.

Use ``ibis.pyspark.connect`` to create a client.

.. autosummary::
:toctree: generated/

connect
PySparkClient.database
PySparkClient.list_databases
PySparkClient.list_tables
PySparkClient.table

Top-level expression APIs
-------------------------

.. currentmodule:: ibis

These methods are available directly in the ``ibis`` module namespace.

.. autosummary::
Expand Down Expand Up @@ -606,3 +706,74 @@ Decimal methods

DecimalValue.precision
DecimalValue.scale

.. _api.geospatial:

Geospatial methods
-------------------

Scalar or column methods
~~~~~~~~~~~~~~~~~~~~~~~~

.. autosummary::
:toctree: generated/

GeoSpatialValue.area
GeoSpatialValue.as_binary
GeoSpatialValue.as_ewkb
GeoSpatialValue.as_ewkt
GeoSpatialValue.as_text
GeoSpatialValue.azimuth
GeoSpatialValue.buffer
GeoSpatialValue.centroid
GeoSpatialValue.contains
GeoSpatialValue.contains_properly
GeoSpatialValue.covers
GeoSpatialValue.covered_by
GeoSpatialValue.crosses
GeoSpatialValue.d_fully_within
GeoSpatialValue.d_within
GeoSpatialValue.difference
GeoSpatialValue.disjoint
GeoSpatialValue.distance
GeoSpatialValue.end_point
GeoSpatialValue.envelope
GeoSpatialValue.equals
GeoSpatialValue.geometry_n
GeoSpatialValue.geometry_type
GeoSpatialValue.intersection
GeoSpatialValue.intersects
GeoSpatialValue.is_valid
GeoSpatialValue.line_locate_point
GeoSpatialValue.line_merge
GeoSpatialValue.line_substring
GeoSpatialValue.length
GeoSpatialValue.max_distance
GeoSpatialValue.n_points
GeoSpatialValue.n_rings
GeoSpatialValue.ordering_equals
GeoSpatialValue.overlaps
GeoSpatialValue.perimeter
GeoSpatialValue.point_n
GeoSpatialValue.set_srid
GeoSpatialValue.simplify
GeoSpatialValue.srid
GeoSpatialValue.start_point
GeoSpatialValue.touches
GeoSpatialValue.transform
GeoSpatialValue.union
GeoSpatialValue.within
GeoSpatialValue.x
GeoSpatialValue.x_max
GeoSpatialValue.x_min
GeoSpatialValue.y
GeoSpatialValue.y_max
GeoSpatialValue.y_min

Column methods
~~~~~~~~~~~~~~

.. autosummary::
:toctree: generated/

GeoSpatialColumn.unary_union
38 changes: 11 additions & 27 deletions docs/source/backends.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,13 @@ Generally speaking these backends also need to handle their own execution.
They work by translating each node into a string, and passing the generated
string to the database through a driver API.

Impala
******

TODO

Clickhouse
**********

TODO

BigQuery
********

TODO
- Impala
- Clickhouse
- BigQuery
- HDFS
- OmniSciDB (Experimental)
- PySpark (Experimental)
- SparkSQL (Experimental)

.. _expression_generating_backends:

Expand All @@ -56,15 +49,9 @@ an expression. These backends tend to execute their expressions directly
through the driver APIs provided by SQLAlchemy (or one of its transitive
dependencies).

SQLite
******

TODO

PostgreSQL
**********

TODO
- PostgreSQL
- SQLite
- MySQL (Experimental)

.. _direct_execution_backends:

Expand All @@ -75,7 +62,4 @@ The only existing backend that directly executes ibis expressions is the pandas
backend. A full description of the implementation can be found in the module
docstring of the pandas backend located in ``ibis/pandas/execution/core.py``.

Pandas
******

TODO
- Pandas
5 changes: 4 additions & 1 deletion docs/source/contributing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -135,11 +135,14 @@ will have to generate an application-specific password by following this
You will then use that generated password on the command line for the ``-P``
argument.

Access the `Ibis "Merging PRs" wiki <https://github.com/ibis-project/ibis/wiki/Merging-PRs>`_ page
for more information.

Releasing
~~~~~~~~~

TODO
Access the `Ibis "Releasing" wiki <https://github.com/ibis-project/ibis/wiki/Releasing-Ibis>`_ page
for more information.


***************
Expand Down
5 changes: 4 additions & 1 deletion docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,16 @@ At this time, Ibis offers some level of support for the following systems:

- `Apache Impala <https://impala.apache.org/>`_
- `Apache Kudu <https://kudu.apache.org/>`_
- `Hadoop Distributed File System (HDFS) <https://hadoop.apache.org/>`_
- `PostgreSQL <https://www.postgresql.org/>`_
- `MySQL <https://www.mysql.com/>`_ (Experimental)
- `SQLite <https://www.sqlite.org/>`_
- `Google BigQuery <https://cloud.google.com/bigquery/>`_
- `Yandex Clickhouse <https://clickhouse.yandex/>`_
- Direct execution of ibis expressions against `Pandas
<http://pandas.pydata.org/>`_ objects
- `OmniSciDB <https://www.omnisci.com/>`_ (Experimental)
- `PySpark/Spark SQL <https://spark.apache.org/sql/>`_ (Experimental)

Coming from SQL? Check out :ref:`Ibis for SQL Programmers <sql>`.

Expand All @@ -69,7 +73,6 @@ SQL engine support needing code contributors:

- `Redshift <https://aws.amazon.com/redshift/>`_
- `Vertica <https://www.vertica.com/>`_
- `Spark SQL <https://spark.apache.org/sql/>`_
- `Presto <https://prestosql.io/>`_
- `Hive <https://hive.apache.org/>`_

Expand Down
3 changes: 2 additions & 1 deletion docs/source/release.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ Release Notes
These release notes are for versions of ibis **1.0 and later**. Release
notes for pre-1.0 versions of ibis can be found at :doc:`/release-pre-1.0`

* :release:`1.2.1 <pending>`
* :release:`1.3.0 <pending>`
* :support:`2034` Add initial documentation for OmniSciDB, MySQL, PySpark and SparkSQL backends, add initial documentation for geospatial methods and add links to Ibis wiki page
* :support:`2037` Drop support for Python 3.5
* :bug:`2023` HTML escape column names and types in png repr.
* :support:`2031` Change omniscidb image tag from v4.7.0 to v5.0.0 on docker-compose recipe
Expand Down
6 changes: 4 additions & 2 deletions ibis/pyspark/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,10 @@

def connect(session):
"""
Create a `SparkClient` for use with Ibis. Pipes **kwargs into SparkClient,
which pipes them into SparkContext. See documentation for SparkContext:
Create a `SparkClient` for use with Ibis.
Pipes `**kwargs` into SparkClient, which pipes them into SparkContext.
See documentation for SparkContext:
https://spark.apache.org/docs/latest/api/python/_modules/pyspark/context.html#SparkContext
"""
client = PySparkClient(session)
Expand Down
6 changes: 4 additions & 2 deletions ibis/spark/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,10 @@ def verify(expr, params=None):

def connect(spark_session):
"""
Create a `SparkClient` for use with Ibis. Pipes **kwargs into SparkClient,
which pipes them into SparkContext. See documentation for SparkContext:
Create a `SparkClient` for use with Ibis.
Pipes `**kwargs` into SparkClient, which pipes them into SparkContext.
See documentation for SparkContext:
https://spark.apache.org/docs/latest/api/python/_modules/pyspark/context.html#SparkContext
"""
client = SparkClient(spark_session)
Expand Down

0 comments on commit 7441683

Please sign in to comment.