From 9c5086387433285946ac1be9bd55b7c6139d25b1 Mon Sep 17 00:00:00 2001 From: PENG Zhengshuai Date: Wed, 18 May 2022 15:02:45 +0800 Subject: [PATCH] refine readme (#1785) --- README.md | 227 +++++++++++++++++------------------------------------- 1 file changed, 71 insertions(+), 156 deletions(-) diff --git a/README.md b/README.md index 2ca57ccd8..9ed6012e5 100644 --- a/README.md +++ b/README.md @@ -4,29 +4,8 @@ drawing

-## TOC - * [Byzer-Lang](#byzer-lang) - * [Byzer Code Example](#byzer-code-example) - * [Byzer Architecture](#byzer-aritchitechture) - * [Official WebSite](#official-website) - * [VSCode Extension（MacOS、Linux）](#vscode-extensionmacoslinux) - * [Docker Sandbox](#docker-sandbox) - * [Pulling Sandbox Docker Image](#pulling-sandbox-docker-image) - * [Start Container](#start-container) - * [Download Byzer](#download-byzer) - * [Building a Distribution](#building-a-distribution) - * [Prerequisites](#prerequisites) - * [Downloading Source Code](#downloading-source-code) - * [Building Spark 2.4.3 Bundle](#building-spark-243-bundle) - * [Building Spark 3.1.1 Bundle](#building-spark-311-bundle) - * [Building without Chinese Analyzer](#building-without-chinese-analyzer) - * [Building with Aliyun OSS Support](#building-with-aliyun-oss-support) - * [Deploying](#deploying) - * [How to contribute to Byzer-Lang](#how-to-contribute-to-byzer-lang) - * [Contributors](#contributors) - * [WeChat Group](#wechat-group) - -## Byzer-Lang + +## Byzer-lang **Byzer** (former MLSQL) is a low-code, open-sourced and distributed programming language for data pipeline, analytics and AI in cloud native way. @@ -38,175 +17,110 @@ We believe that everything is a table, a simple and powerful SQL-like language c ![Byzer-lang Arch](images/Byzer-arch.png) -### Byzer Code Example - - -```sql -load hive.`raw.stripe_discounts` as discounts; -load hive.`raw.stripe_invoice_items` as invoice_items; - -select - invoice_items.*, - case - when discounts.discount_type = 'percent' - then amount * (1.0 - discounts.discount_value::float / 100) - else amount - discounts.discount_value - end as discounted_amount +You can build a data product based on Byzer engine & Byzer-lang without interacting with computing framework directly like Spark in your data APP. Thus will simplify your data app significantly. - from invoice_items +For example, Byzer org contributes a data app [Byzer Notebook](https://github.com/byzer-org/byzer-notebook), which provides notebook interaction & workflow GUI interaction. - left outer join discounts - on invoice_items.customer_id = discounts.customer_id - and invoice_items.invoice_date > discounts.discount_start - and (invoice_items.invoice_date < discounts.discount_end - or discounts.discount_end is null) -as joined; +### BIP (Byzer Improvement Proposal) +Byzer project uses the [BIP](https://github.com/byzer-org/byzer-lang/wiki) for the community collaboration, you can checkout the feature design or architecture design in BIP. +### Online Trial -select +You can access the official website [https://www.byzer.org/](https://www.byzer.org/) and try Byzer-lang & Byzer Notebook online. - id, - invoice_id, - customer_id, - coalesce(discounted_amount, amount) as discounted_amount, - currency, - description, - created_at, - deleted_at - from joined -as final; +### Download +You can download Byzer engine via: +- [https://download.byzer.org/](https://download.byzer.org/) +- [Byzer Docker Hub](https://hub.docker.com/u/byzer) +- [Github Release](https://github.com/byzer-org/byzer-lang/releases) +For more details, please refer to the [docs](https://docs.byzer.org/#/byzer-lang/zh-cn/installation/README) -set allColumns = "all,wow"; +### Install -!if ''' split(:allColumns,",")[0] == "all" '''; - select * from final as final2; -!else; - select id,invoice from final as final2; -!fi; +For **dev/test** purpose, you can download [Byzer All In One Package](https://docs.byzer.org/#/byzer-lang/zh-cn/installation/server/byzer-all-in-one-deployment), extract and then execute the command below -select * from final2 as output; +``` +$ cd {BYZER_HOME} +$ ./bin/byzer.sh start ``` -## Official WebSite +And for **production** purpose, we recommend to use [Byzer Server Pacakge](https://docs.byzer.org/#/byzer-lang/zh-cn/installation/server/binary-installation) and deploy it on Hadoop. -[https://www.byzer.org](https://www.byzer.org) -## Notebook Support +You can also install [Byzer VSCode Extension](https://docs.byzer.org/#/byzer-lang/zh-cn/installation/vscode/byzer-vscode-extension-installation) to use Byzer-lang. -[byzer-notebook](https://github.com/byzer-org/byzer-notebook) +For the Docker Image or , please refer to the [docs](https://docs.byzer.org/#/byzer-lang/zh-cn/installation/README) -## VSCode Extension（MacOS、Linux、Windows） -[VSCode IDE Extension](https://github.com/byzer-org/byzer-desktop) +### Byzer Code Example -[More document about byzer-lang vscode extension（Chinese version）](https://docs.byzer.org/#/byzer-lang/zh-cn/installation/desktop-installation) +Below list an example that how to process Github API as a table to get the information of Byzer Org -## Docker Sandbox (With Notebook) +```sql +-- Get Github Organization Info -``` -export MYSQL_PASSWORD=${1:-root} -export SPARK_VERSION=${SPARK_VERSION:-3.1.1} -export MLSQL_VERSION=${MLSQL_VERSION:-2.2.0-SNAPSHOT} - -docker run -d \ --p 3306:3306 \ --p 9002:9002 \ --p 9003:9003 \ --e MYSQL_ROOT_HOST=% \ --e MYSQL_ROOT_PASSWORD="${MYSQL_PASSWORD}" \ ---name mlsql-sandbox-${SPARK_VERSION}-${MLSQL_VERSION} \ -mlsql-sandbox:${SPARK_VERSION}-${MLSQL_VERSION} -``` +-- set API URL and params +SET org_name="byzer-org"; +SET GITHUB_ORGANIZATION_URL="https://api.github.com/orgs/${org_name}"; -Then you can visit `http://127.0.0.1:9002` . +-- Load Github Organization API as table +LOAD Rest.`$GITHUB_ORGANIZATION_URL` + where `config.connect-timeout`="10s" + and `config.method`="GET" + and `header.accept`="application/vnd.github.v3+json" +as github_org; -## Download Byzer +-- decode API response from binary to a json string +select string(content) as content from github_org as response_content; -* The latest stable version is release-2.2.0 -* You can download from [Byzer Download Website](https://download.byzer.org) -* Spark 2.4.3/3.1.1 have been tested +-- expand the json string +run response_content as JsonExpandExt.`` where inputCol="content" and structColumn="true" as github_org; -***Naming Convention*** +-- retrieve user infomation and process as a table +select content.* from github_org as org_info; -mlsql-engine_${spark_major_version}-${mlsql_version}.tgz -```shell -## Pre-built for Spark 2.4.3 -byzer-lang-2.4.3-2.1.0.tar.gz +-- save the table to delta lake +save overwrite org_info as delta.`github_info_db.byzer_org`; +``` -## Pre-built for Spark 3.1.1 -byzer-lang-3.1.1-2.1.0.tar.gz -``` -## Building a Distribution -### Prerequisites -- JDK 8+ -- Maven -- Linux or MacOS +For more details about the Byzer-lang grammer, please refer to the user manaual [Byzer-lang Grammer](https://docs.byzer.org/#/byzer-lang/zh-cn/grammar/outline) -### Downloading Source Code -```shell -## Clone the code base -git clone https://github.com/byzer-org/byzer-lang.git -cd byzer-lang -``` +### Development -### Building Spark 2.4.3 Bundle -```shell -export MLSQL_SPARK_VERSION=2.4 -./dev/make-distribution.sh -``` +1. Fork the repository and clone -### Building Spark 3.1.1 Bundle -```shell -export MLSQL_SPARK_VERSION=3.0 -./dev/make-distribution.sh ``` -### Building without Chinese Analyzer -```shell -## Chinese analyzer is enabled by default. -export ENABLE_CHINESE_ANALYZER=false -./dev/make-distribution.sh +git clone https://github.com/{YOUR_GITHUB}/byzer-lang.git ``` -## Deploying -1. [Download](#Download) or [build a distribution](#Build) -2. Install Spark and set environment variable SPARK_HOME, make sure Spark version matches that of MLSQL -3. Deploy tgz -- Set environment variable MLSQL_HOME -- Copy distribution tar ball over and untar it - -4.Start Byzer in local mode -```shell -cd $MLSQL_HOME -## Run process in background -nohup ./bin/start-local.sh 2>&1 > ./local_mlsql.log & -``` -5. Open a browser and type in http://localhost:9003, have fun. - -Directory structure -```shell -|-- mlsql - |-- bin - |-- conf - |-- data - |-- examples - |-- libs - |-- main - |-- README.md - |-- LICENSE - |-- RELEASE -``` +2. Use Intellj Idea to open the project, choose the scala version `2.12.10` + +3. In Intellj Idea Maven Setting, check the profile below + - gpg + - local + - scala-2.12 + - spark-3.0.0 + - streamingpro-spark-3.0.0-adaptor -## How to contribute to Byzer-Lang +4. Click Maven Refresh and wait for Idea load finished -If you are planning to contribute to this repository, please create an issue at [our Issue page](https://github.com/byzer-org/byzer-lang/issues) +5. Find the class `tech.mlsql.example.app.LocalSparkServiceApp`, click Debug button then Byzer Engine will be started, then you can access the Byzer Web Console in [http://localhost:9003/](http://localhost:9003/#/) + +### Build + +You can refer to the project [byzer-org/byzer-build](https://github.com/byzer-org/byzer-build) to check how to build the Byzer engine binary pacakges and images + +### How to contribute to Byzer-Lang + +If you are planning to contribute to this project, please create an issue at [our Issue page](https://github.com/byzer-org/byzer-lang/issues) even if the topic is not related to source code itself (e.g., documentation, new idea and proposal). This is an active open source project for everyone, @@ -215,7 +129,7 @@ and we are always open to people who want to use this system or contribute to it For more details about how to contribute to the Byzer Org, please refer to [How to Contribute](https://docs.byzer.org/#/byzer-lang/zh-cn/appendix/contribute) -## Contributors +### Contributors @@ -223,9 +137,10 @@ For more details about how to contribute to the Byzer Org, please refer to [How Made with [contrib.rocks](https://contrib.rocks). -## WeChat Group +### Community + +- **Slack**: [byzer-org.slack.com](https://byzer-org.slack.com) +- **Wechat Official Account:** Byzer Community -扫码添加K小助微信号，添加成功后，发送 mlsql 这5个英文字母进群。 -![](https://github.com/allwefantasy/mlsql/blob/master/images/dc0f4493-570f-4660-ab41-0e487b17a517.png)