Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

537 use confluent cloud apache flink to handle avro flight consolidator app #572

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,9 @@ Explore the repo, run the Docker containers (compatible with both Mac M chips an
+ [2.2.1 Run from the cloud](#221-run-from-the-cloud)
- [2.3 Visualizing the Terraform Configuration](#23-visualizing-the-terraform-configuration)
+ [3.0 Hands-On Kickoff: Practical Examples for Rapid Learning](#30-hands-on-kickoff-practical-examples-for-rapid-learning)
- [3.1 Flink Applications Powered by Java](#31-flink-applications-powered-by-java)
- [3.2 Flink Applications Powered by Python](#32-flink-applications-powered-by-python)
- [3.1 Flink Applications Powered by Java on a locally running Apache Flink Cluster in Docker](#31-flink-applications-powered-by-java-on-a-locally-running-apache-flink-cluster-in-docker)
- [3.2 Flink Applications Powered by Python on a locally running Apache Flink Cluster in Docker](#32-flink-applications-powered-by-python-on-a-locally-running-apache-flink-cluster-in-docker)
- [3.3 Flink Applications Powered by Python on Confluent Cloud Apache Flink](#33-flink-applications-powered-by-python-on-confluent-cloud-apache-flink)
+ [4.0 Powering Analytics with Apache Iceberg tables in Snowflake](#40-powering-analytics-with-apache-iceberg-tables-in-snowflake)
+ [5.0 Resources](#50-resources)
+ [6.0 Important Note(s)](#60-important-notes)
Expand Down Expand Up @@ -167,12 +168,15 @@ terraform graph | dot -Tpng > .blog/images/terraform-visualization.png

## 3.0 Hands-On Kickoff: Practical Examples for Rapid Learning

### 3.1 Flink Applications Powered by Java
### 3.1 Flink Applications Powered by Java on a locally running Apache Flink Cluster in Docker
[Let's go!](java/README.md)

### 3.2 Flink Applications Powered by Python
### 3.2 Flink Applications Powered by Python on a locally running Apache Flink Cluster in Docker
[Let's go!](python/README.md)

### 3.3 Flink Applications Powered by Python on Confluent Cloud Apache Flink
[Let's go!](python_ccaf/README.md)

## 4.0 Powering Analytics with Apache Iceberg tables in Snowflake
After running the Flink applications, you can now leverage the data stored in the Apache Iceberg tables in Snowflake for analytics. Click [here](.blog/using-non-managed-apache-iceberg-tables-in-snowflake.md) to learn more about using **Non-Managed Apache Iceberg tables** in Snowflake. This blog post will help you understand the trade-offs, operational characteristics, and integration patterns of using Apache Iceberg tables in Snowflake. These terraform configurations:

Expand Down
2 changes: 1 addition & 1 deletion java/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Flink Applications Powered by Java
# Flink Applications Powered by Java on a locally running Apache Flink Cluster in Docker
Discover how Apache Flink® can transform your data pipelines! Explore hands-on examples of Flink applications using the [DataStream API](https://nightlies.apache.org/flink/flink-docs-master/docs/learn-flink/datastream_api/) and [Table API](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/overview/) in Java. You'll see how these technologies integrate seamlessly with AWS Services, Apache Kafka, and Apache Iceberg.

Curious about the differences between the DataStream API and Table API? Click [here](../.blog/datastream-vs-table-api.md) to learn more and find the best fit for your next project.
Expand Down
2 changes: 1 addition & 1 deletion python/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Flink Applications Powered by Python
# Flink Applications Powered by Python on a locally running Apache Flink Cluster in Docker
Discover how Apache Flink® can transform your data pipelines! Explore hands-on examples of Flink applications using the [DataStream API](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/python/datastream/intro_to_datastream_api/), [Table API](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/python/table/intro_to_table_api/), and [Flink SQL](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/overview/#sql)—all built in Python with PyFlink, which compiles these apps to Java. You'll see how these technologies integrate seamlessly with AWS Services, Apache Kafka, and Apache Iceberg.

Curious about the differences between the DataStream API and Table API? Click [here](../.blog/datastream-vs-table-api.md) to learn more and find the best fit for your next project.
Expand Down
2 changes: 1 addition & 1 deletion python/pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[project]
name = "kickstarter"
version = "0.54.00.000"
version = "0.55.00.000"
description = "Apache Flink Kickstarter"
readme = "README.md"
requires-python = "~=3.11.9"
Expand Down
6 changes: 6 additions & 0 deletions python_ccaf/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.venv/
.DS_Store
__pycache__
.pytest_cache
python_files.zip
tf_snowflake_user
1 change: 1 addition & 0 deletions python_ccaf/.python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.11.9
1 change: 1 addition & 0 deletions python_ccaf/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# Flink Applications Powered by Python on Confluent Cloud Apache Flink
7 changes: 7 additions & 0 deletions python_ccaf/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[project]
name = "kickstarter"
version = "0.56.00.000"
description = "Confluent Cloud Apache Flink Kickstarter"
readme = "README.md"
requires-python = "~=3.11.9"
dependencies = []
Empty file.
Empty file.
Empty file.
Empty file.