Skip to content

Commit

Permalink
PowerBI certification changes (#349)
Browse files Browse the repository at this point in the history
* Added Tableau Connector to OpenSearch SQL

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Added CEILING and FLOOR functions

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Added IFNULL function

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Added Kyle's fix for ADDDATE and SUBDATE with resulting 00:00:00 being unexpectedly null

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Added MIN and MAX for two string arguments

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Fixed return type for MID function redefenitions, part of AOS-202

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added CAST functions to convert to int or string as part of AOS-202

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added HEXBINX and HEXBINY functions from the templace without any modifications

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Fixed cast to use the OSSQL type

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added parenthesis to avoid /issues/293

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Modified Timestamp conversion formula

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Added comment to TimestampType

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Updated Company Name accroding to the customer's request

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Updated Vendor Name as well

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added few small changes to simplify connector developing and debugging

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Typo fix

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added a simple test from TDVT: case bool0 from test calcs_data (an expression test)
Some mofidication were made in the test framework to make the output more readable

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Modified the driver to recognize its support for conversion of types

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Removed changes from a different branch

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Removed changes from a different branch

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Made the connector convert keyword columns to string columns

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Fix requested by PR review

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Modified integration tests to reflect support for conversion types

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Removed connector changes

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Change values to bitmasks

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Add timestamp as a type for the driver

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Added timestamp to SQLGetTypeInfo for the ODBC Driver

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Reverted some previous changes

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Removed generated test output file

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Renamed SqlOdbcPBIConnector to OpenSearch

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* According to AOS-248, we have to build the server address string properly:
- add http[s]:// prefix
- add port
In scope of this task 2 new fields were added to the connection dialog (port, useSSL) and the corresponding handling.

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added version and disable trace output as per Microsoft's feedback

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Added documentation for the OpenSearch connector

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Renamed connector markdown file to follow naming conventions

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Changed image file extention as it was not being detected by GitHub

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Updating docs according to PR #16 comments.

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added GitHub actions script to build Tableau and Power BI connector. The PBI connector is being built twice with different names accroding to the Amazon's request.

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* A typo fix in yml

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* A dummy commit to make GitHub Actions start the desired workflow

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Workflow fix - pack Tableau ODBC connector instead of JDBC one

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Fix line endings in the workflow file

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Another dummy commit to start the desired workflow

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Fix paths in the GitHub workflow

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Modified workflow to trigger on changes of itself

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Path fix for preparation step

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added some changes to the second PBI connector - they should differ more than in filename to make PBI distuingish them

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Renamed PBI connector files

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Moved Tableau JDBC connector to bi-connectors

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Added packing Tableau JDBC connector
For all connectors added check for successful packing before publishing

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Renamed Power BI connector as "OpenSearch Project"

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Renamed output artifact

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Update SqlOdbcPBIConnector.pq according to PR review notes

Co-authored-by: Kyle Porter <kylep@bitquilltech.com>

* Update SqlOdbcPBIConnector.pq according to PR review notes

Co-authored-by: Kyle Porter <kylep@bitquilltech.com>

* Modified code that builds the server string

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Fixed grammatical errors, added a docs directory, and updated the connection string options image

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Removed md file as it was moved to the docs directory

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Fixed grammatical errors and updated the old power bi support document

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Removed extra line

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

* Disabled building Tableau ODBC connector, because it is not ready for shipping

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Updated workflow step description

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Removed packing ODBC driver for Tableau (re: 4cadfd1).
Removed Tableau ODBC driver source files since the driver is not supposed to be.

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Updated workflow, because connector source files were renamed after merge

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Revert "Add GitHub workflow to pack BI connectors"

* Changed string replacement in the main connector file

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Script was modified according to PR comments

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Removed Tableau ODBC connector

Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Merged OpenSearch-sql main to this branch and updated links as per the PR comment

Signed-off-by: Guian Gumpac <guiang@bitquilltech.com>

Co-authored-by: Yury Fridlyand <yuryf@bitquilltech.com>
Co-authored-by: Yury-Fridlyand <88679692+Yury-Fridlyand@users.noreply.github.com>
Co-authored-by: Kyle Porter <kylep@bitquilltech.com>
  • Loading branch information
4 people authored Jan 5, 2022
1 parent c0085be commit 3ffa1ca
Show file tree
Hide file tree
Showing 52 changed files with 259 additions and 634 deletions.
60 changes: 60 additions & 0 deletions .github/workflows/bi-connectors.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Build connectors for BI tools

on:
push:
paths:
- 'bi-connectors/PowerBIConnector/**'
- 'bi-connectors/TableauConnector/**'
- '.github/workflows/bi-connectors.yml'

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Pack Tableau JDBC connector
id: pack-tableau-jdbc
run: |
zip -r opensearch_sql_jdbc.taco . -x *.taco
working-directory: bi-connectors/TableauConnector/opensearch_sql_jdbc
- name: Prepare Power BI ODBC connector
run: |
cp OpenSearch.pq OpenSearch.m
working-directory: bi-connectors/PowerBIConnector
- name: Pack Power BI ODBC connector
id: pack-powerbi-odbc-os-proj
run: |
zip OpenSearchProject.mez *.png *.m *.resx *.pqm
working-directory: bi-connectors/PowerBIConnector
- name: Prepare Power BI ODBC connector for 'Amazon OpenSearch Service'
id: prep-powerbi-odbc-amz-os-svc
run: |
sed -i 's/<value>OpenSearch Project<\/value>/<value>Amazon OpenSearch Service<\/value>/g' resources.resx
sed -i 's/Documentation.Name = "OpenSearch Project"/Documentation.Name = "Amazon OpenSearch Service"/g' OpenSearch.m
sed -i 's/section OpenSearch;/section AmazonOpenSearchConnector;/' OpenSearch.m
sed -i 's/\[DataSource.Kind="OpenSearch", Publish="OpenSearch.Publish"\]/[DataSource.Kind="AmazonOpenSearchConnector", Publish="AmazonOpenSearchConnector.Publish"]/g' OpenSearch.m
sed -i 's/OpenSearch.Contents/AmazonOpenSearchConnector.Contents/g' OpenSearch.m
sed -i 's/OpenSearch.Publish/AmazonOpenSearchConnector.Publish/g' OpenSearch.m
sed -i 's/OpenSearch.Icons/AmazonOpenSearchConnector.Icons/g' OpenSearch.m
sed -i 's/OpenSearchImpl/AmazonOpenSearchConnectorImpl/g' OpenSearch.m
sed -i 's/OpenSearchType/AmazonOpenSearchConnectorType/g' OpenSearch.m
sed -i 's/OpenSearch = \[/AmazonOpenSearchConnector = [/' OpenSearch.m
# sed -i 's/"opensearch-sql"/"Amazon-OpenSearch-Service"/g' OpenSearch.m
working-directory: bi-connectors/PowerBIConnector
- name: Pack Power BI ODBC connector for 'Amazon OpenSearch Service'
id: pack-powerbi-odbc-amz-os-svc
run: |
zip AmazonOpenSearchService.mez *.png *.m *.resx *.pqm
working-directory: bi-connectors/PowerBIConnector
- name: Upload Tableau JDBC connector
if: steps.pack-tableau-jdbc.outcome == 'success'
uses: actions/upload-artifact@v2
with:
name: TableauConnectors
path: bi-connectors/TableauConnector/opensearch_sql_jdbc/opensearch_sql_jdbc.taco
- name: Upload Power BI ODBC connectors
if: steps.pack-powerbi-odbc-os-proj.outcome == 'success' || (steps.prep-powerbi-odbc-amz-os-svc.outcome == 'success' && steps.pack-powerbi-odbc-amz-os-svc.outcome == 'success')
uses: actions/upload-artifact@v2
with:
name: PBIConnectors
path: 'bi-connectors/PowerBIConnector/*.mez'
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
<SuppressCellErrors>False</SuppressCellErrors>
<MaxRows>1000</MaxRows>
<ExtensionProject>Yes</ExtensionProject>
<Name>SqlOdbcPBIConnector</Name>
<Name>OpenSearch</Name>
</PropertyGroup>
<PropertyGroup Condition=" '$(Configuration)' == 'Debug' ">
<DebugSymbols>false</DebugSymbols>
Expand All @@ -36,31 +36,31 @@
<Reference Include="System.Xml" />
</ItemGroup>
<ItemGroup>
<Compile Include="SqlOdbcPBIConnector.pq">
<Compile Include="OpenSearch.pq">
<SubType>Code</SubType>
</Compile>
<Compile Include="SqlOdbcPBIConnector16.png">
<Compile Include="OpenSearch16.png">
<SubType>Code</SubType>
</Compile>
<Compile Include="SqlOdbcPBIConnector20.png">
<Compile Include="OpenSearch20.png">
<SubType>Code</SubType>
</Compile>
<Compile Include="SqlOdbcPBIConnector24.png">
<Compile Include="OpenSearch24.png">
<SubType>Code</SubType>
</Compile>
<Compile Include="SqlOdbcPBIConnector32.png">
<Compile Include="OpenSearch32.png">
<SubType>Code</SubType>
</Compile>
<Compile Include="SqlOdbcPBIConnector40.png">
<Compile Include="OpenSearch40.png">
<SubType>Code</SubType>
</Compile>
<Compile Include="SqlOdbcPBIConnector48.png">
<Compile Include="OpenSearch48.png">
<SubType>Code</SubType>
</Compile>
<Compile Include="SqlOdbcPBIConnector64.png">
<Compile Include="OpenSearch64.png">
<SubType>Code</SubType>
</Compile>
<Compile Include="SqlOdbcPBIConnector80.png">
<Compile Include="OpenSearch80.png">
<SubType>Code</SubType>
</Compile>
<Compile Include="resources.resx">
Expand All @@ -72,7 +72,7 @@
<Compile Include="OdbcConstants.pqm">
<SubType>Content</SubType>
</Compile>
<Content Include="SqlOdbcPBIConnector.query.pq">
<Content Include="OpenSearch.query.pq">
<SubType>Code</SubType>
</Content>
</ItemGroup>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,28 +1,39 @@
// This file contains Data Connector logic
section SqlOdbcPBIConnector;
[Version = "1.0.0"]
section OpenSearch;

// When set to true, additional trace information will be written out to the User log.
// This should be set to false before release. Tracing is done through a call to
// Diagnostics.LogValue(). When EnableTraceOutput is set to false, the call becomes a
// no-op and simply returns the original value.
EnableTraceOutput = true;
EnableTraceOutput = false;

[DataSource.Kind="SqlOdbcPBIConnector", Publish="SqlOdbcPBIConnector.Publish"]
shared SqlOdbcPBIConnector.Contents = Value.ReplaceType(SqlOdbcPBIConnectorImpl, SqlOdbcPBIConnectorType);
[DataSource.Kind="OpenSearch", Publish="OpenSearch.Publish"]
shared OpenSearch.Contents = Value.ReplaceType(OpenSearchImpl, OpenSearchType);

// Wrapper function to provide additional UI customization.
SqlOdbcPBIConnectorType = type function (
OpenSearchType = type function (
Server as (type text meta [
Documentation.FieldCaption = "Server",
Documentation.FieldDescription = "The hostname of the OpenSearch server.",
Documentation.SampleValues = { "localhost:9200" }
Documentation.SampleValues = { "localhost" }
]),
Port as (type number meta [
Documentation.FieldCaption = "Port",
Documentation.FieldDescription = "Port which OpenSearch server listens on.",
Documentation.SampleValues = { 9200 }
]),
UseSSL as (type logical meta [
Documentation.FieldCaption = "Use SSL",
Documentation.FieldDescription = "Use SSL",
Documentation.AllowedValues = { true, false }
])
)
as table meta [
Documentation.Name = "OpenSearch"
];

SqlOdbcPBIConnectorImpl = (Server as text) as table =>
OpenSearchImpl = (Server as text, Port as number, UseSSL as logical) as table =>
let
Credential = Extension.CurrentCredential(),
AuthenticationMode = Credential[AuthenticationKind],
Expand Down Expand Up @@ -56,9 +67,16 @@ SqlOdbcPBIConnectorImpl = (Server as text) as table =>
UseSSL = 0
],

// Subtract the server from the user input in case it's entered like 'http://localhost' or 'https://srv.com:100500' or 'localhost:0'
// And build the proper string on our own
FinalServerString = if UseSSL then
"https://" & Uri.Parts(Server)[Host] & ":" & Text.From(Port)
else
"http://" & Uri.Parts(Server)[Host] & ":" & Text.From(Port),

ConnectionString = [
Driver = "OpenSearch SQL ODBC Driver",
Host = Server
Host = FinalServerString
],

SQLGetInfo = Diagnostics.LogValue("SQLGetInfo_Options", [
Expand Down Expand Up @@ -160,14 +178,16 @@ OnOdbcError = (errorRecord as record) =>
error errorRecord;

// Data Source Kind description
SqlOdbcPBIConnector = [
OpenSearch = [
// Required for use with Power BI Service.
TestConnection = (dataSourcePath) =>
let
json = Json.Document(dataSourcePath),
Server = json[Server]
Server = json[Server],
Port = json[Port],
UseSSL = json[UseSSL]
in
{ "SqlOdbcPBIConnector.Contents", Server },
{ "OpenSearch.Contents", Server, Port, UseSSL },

// Authentication modes
Authentication = [
Expand All @@ -185,11 +205,11 @@ SqlOdbcPBIConnector = [

// PBIDS Handler
DSRHandlers = [
#"sqlodbc" = [
GetDSR = (server, schema, object, optional options) => [ protocol = "sqlodbc", address = [ server = server ] ],
GetFormula = (dsr, optional options) => () =>
#"opensearch-sql" = [
GetDSR = (server, schema, object, optional options) => [ protocol = "opensearch-sql", address = [ server = server, port = schema, useSSL = object ] ],
GetFormula = (dsr, optional options) => () =>
let
db = SqlOdbcPBIConnector.Contents(dsr[address][server])
db = OpenSearch.Contents(dsr[address][server], dsr[address][port], dsr[address][useSSL])
in
db,
GetFriendlyName = (dsr) => "OpenSearch SQL ODBC"
Expand All @@ -203,7 +223,7 @@ SqlOdbcPBIConnector = [
];

// Data Source UI publishing description
SqlOdbcPBIConnector.Publish = [
OpenSearch.Publish = [
Beta = true,
Category = "Other",
ButtonText = { Extension.LoadString("ButtonTitle"), Extension.LoadString("ButtonHelp") },
Expand All @@ -212,13 +232,13 @@ SqlOdbcPBIConnector.Publish = [
// Disabling direct query due to limited SQL query support
SupportsDirectQuery = true,

SourceImage = SqlOdbcPBIConnector.Icons,
SourceTypeImage = SqlOdbcPBIConnector.Icons
SourceImage = OpenSearch.Icons,
SourceTypeImage = OpenSearch.Icons
];

SqlOdbcPBIConnector.Icons = [
Icon16 = { Extension.Contents("SqlOdbcPBIConnector16.png"), Extension.Contents("SqlOdbcPBIConnector20.png"), Extension.Contents("SqlOdbcPBIConnector24.png"), Extension.Contents("SqlOdbcPBIConnector32.png") },
Icon32 = { Extension.Contents("SqlOdbcPBIConnector32.png"), Extension.Contents("SqlOdbcPBIConnector40.png"), Extension.Contents("SqlOdbcPBIConnector48.png"), Extension.Contents("SqlOdbcPBIConnector64.png") }
OpenSearch.Icons = [
Icon16 = { Extension.Contents("OpenSearch16.png"), Extension.Contents("OpenSearch20.png"), Extension.Contents("OpenSearch24.png"), Extension.Contents("OpenSearch32.png") },
Icon32 = { Extension.Contents("OpenSearch32.png"), Extension.Contents("OpenSearch40.png"), Extension.Contents("OpenSearch48.png"), Extension.Contents("OpenSearch64.png") }
];

// Load common library functions
Expand Down
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
// This file contains queries to test your data connector
section SqlOdbcPBIConnector.UnitTests;
section OpenSearch.UnitTests;

shared MyExtension.UnitTest =
[
// Common variables for all tests
Host = "localhost:9200",
Host = "localhost",
Port = 9200,
UseSSL = false,

facts =
{
Fact("Connection Test",
7,
let
Source = SqlOdbcPBIConnector.Contents(Host),
Source = OpenSearch.Contents(Host, Port, UseSSL),
no_of_columns = Table.ColumnCount(Source)
in
no_of_columns
Expand All @@ -20,7 +22,7 @@ shared MyExtension.UnitTest =
#table(type table [bool0 = logical],
{ {null}, {false}, {true} }),
let
Source = SqlOdbcPBIConnector.Contents(Host),
Source = OpenSearch.Contents(Host, Port, UseSSL),
calcs_null_null = Source{[Item="calcs",Schema=null,Catalog=null]}[Data],
grouped = Table.Group(calcs_null_null, {"bool0"}, {})
in
Expand Down
Binary file not shown.
66 changes: 66 additions & 0 deletions bi-connectors/PowerBIConnector/docs/AmazonOpenSearchService.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Connecting Amazon OpenSearch Service to Microsoft Power BI Desktop

>[!Note]
>The following connector article is provided by Amazon, the owner of this connector and a member of the Microsoft Power Query Connector Certification Program. If you have questions regarding the content of this article or have changes you would like to see made to this article, visit the OpenSearch website and use the support channels there.
## Summary
| Item | Description |
| ---- | ----------- |
| Release State | General Availability |
| Products | Power BI Desktop |
| Authentication Types Supported | Basic |

## Prerequisites
* Microsoft Power BI Desktop
* [OpenSearch](https://opensearch.org/docs/latest/opensearch/install/index/)
* [OpenSearch SQL ODBC driver](https://opensearch.org/docs/latest/search-plugins/sql/odbc/)

## Capabilities supported
* Import
* DirectQuery

## Connect to Amazon OpenSearch Service
1. Open Power BI Desktop.

2. Click on **Home** > **Get Data** > **More** > **Other**. Select **Amazon OpenSearch Service**. Click on **Connect**.

<img src="img/pbi_select_connector_amz.png" width="500">

3. You will get a warning for using a third-party service. Click on **Continue**.

<img src="img/pbi_third_party_warning.png" width="500">

4. Enter host and port values and select your preferred SSL option. Click on **OK**.

<img src="img/pbi_connection_string_options.png" width="500">

5. Select authentication option. Enter credentials if required and click on **Connect**.

<img src="img/pbi_auth.png" width="500">

6. Select required table. Data preview will be loaded.

<img src="img/pbi_data_preview.png">

7. Click on **Load**.

8. Select required columns for creating a graph.

<img src="img/pbi_simple_graph.png">


## Troubleshooting

* If you get the following error, please install the [OpenSearch SQL ODBC Driver](https://docs-beta.opensearch.org/search-plugins/sql/odbc/).

<img src="img/pbi_error_driver_not_installed.png" width="350">

* If you get the following error,

<img src="img/pbi_error_conn.png" width="350">

1. Check if host and port values are correct.
2. Check if auth credentials are correct.
3. Check if server is running.

Loading

0 comments on commit 3ffa1ca

Please sign in to comment.