-
Notifications
You must be signed in to change notification settings - Fork 1k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: Refactor feature server helm charts to allow passing feature_st…
…ore.yaml in environment variables (#3113) * feat: Refactor feature server helm charts to allow passing feature_store.yaml in environment variables Signed-off-by: Danny Chiao <danny@tecton.ai> * lint Signed-off-by: Danny Chiao <danny@tecton.ai> * lint Signed-off-by: Danny Chiao <danny@tecton.ai> * add docs Signed-off-by: Danny Chiao <danny@tecton.ai> * lint Signed-off-by: Danny Chiao <danny@tecton.ai> * revert bad helm docs Signed-off-by: Danny Chiao <danny@tecton.ai> * add to bump files Signed-off-by: Danny Chiao <danny@tecton.ai> * add to bump files Signed-off-by: Danny Chiao <danny@tecton.ai> * add to release Signed-off-by: Danny Chiao <danny@tecton.ai> * fix readme for feast-feature-server Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Danny Chiao <danny@tecton.ai>
- Loading branch information
Showing
24 changed files
with
393 additions
and
100 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
|
|
||
| # Running Feast Python / Go Feature Server with Redis on Kubernetes | ||
|
|
||
| For this tutorial, we set up Feast with Redis. | ||
|
|
||
| We use the Feast CLI to register and materialize features, and then retrieving via a Feast Python feature server deployed in Kubernetes | ||
|
|
||
| ## First, let's set up a Redis cluster | ||
| 1. Start minikube (`minikube start`) | ||
| 2. Use helm to install a default Redis cluster | ||
| ```bash | ||
| helm repo add bitnami https://charts.bitnami.com/bitnami | ||
| helm repo update | ||
| helm install my-redis bitnami/redis | ||
| ``` | ||
|  | ||
| 3. Port forward Redis so we can materialize features to it | ||
|
|
||
| ```bash | ||
| kubectl port-forward --namespace default svc/my-redis-master 6379:6379 | ||
| ``` | ||
| 4. Get your Redis password using the command (pasted below for convenience). We'll need this to tell Feast how to communicate with the cluster. | ||
| ```bash | ||
| export REDIS_PASSWORD=$(kubectl get secret --namespace default my-redis -o jsonpath="{.data.redis-password}" | base64 --decode) | ||
| echo $REDIS_PASSWORD | ||
| ``` | ||
| ## Next, we setup a local Feast repo | ||
| 1. Install Feast with Redis dependencies `pip install "feast[redis]"` | ||
| 2. Make a bucket in GCS (or S3) | ||
| 3. The feature repo is already setup here, so you just need to swap in your GCS bucket and Redis credentials. | ||
| We need to modify the `feature_store.yaml`, which has two fields for you to replace: | ||
| ```yaml | ||
| registry: gs://[YOUR GCS BUCKET]/demo-repo/registry.db | ||
| project: feast_python_demo | ||
| provider: gcp | ||
| online_store: | ||
| type: redis | ||
| # Note: this would normally be using instance URL's to access Redis | ||
| connection_string: localhost:6379,password=[YOUR PASSWORD] | ||
| offline_store: | ||
| type: file | ||
| entity_key_serialization_version: 2 | ||
| ``` | ||
| 4. Run `feast apply` from within the `feature_repo` directory to apply your local features to the remote registry | ||
| - Note: you may need to authenticate to gcloud first with `gcloud auth login` | ||
| 5. Materialize features to the online store: | ||
| ```bash | ||
| CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S") | ||
| feast materialize-incremental $CURRENT_TIME | ||
| ``` | ||
|
|
||
| ## Now let's setup the Feast Server | ||
| 1. Add the gcp-auth addon to mount GCP credentials: | ||
| ```bash | ||
| minikube addons enable gcp-auth | ||
| ``` | ||
| 2. Add Feast's Python/Go feature server chart repo | ||
| ```bash | ||
| helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com | ||
| helm repo update | ||
| ``` | ||
| 3. For this tutorial, because we don't have a direct hosted endpoint into Redis, we need to change `feature_store.yaml` to talk to the Kubernetes Redis service | ||
| ```bash | ||
| sed -i '' 's/localhost:6379/my-redis-master:6379/g' feature_store.yaml | ||
| ``` | ||
| 4. Install the Feast helm chart: `helm install feast-release feast-charts/feast-feature-server --set feature_store_yaml_base64=$(base64 feature_store.yaml)` | ||
| > **Dev instructions**: if you're changing the java logic or chart, you can do | ||
| 1. `eval $(minikube docker-env)` | ||
| 2. `make build-feature-server-dev` | ||
| 3. `helm install feast-release ../../../infra/charts/feast-feature-server --set image.tag=dev --set feature_store_yaml_base64=$(base64 feature_store.yaml)` | ||
| 5. (Optional): check logs of the server to make sure it’s working | ||
| ```bash | ||
| kubectl logs svc/feast-feature-server | ||
| ``` | ||
| 6. Port forward to expose the grpc endpoint: | ||
| ```bash | ||
| kubectl port-forward svc/feast-feature-server 6566:80 | ||
| ``` | ||
| 7. Run test fetches for online features:8. | ||
| - First: change back the Redis connection string to allow localhost connections to Redis | ||
| ```bash | ||
| sed -i '' 's/my-redis-master:6379/localhost:6379/g' feature_store.yaml | ||
| ``` | ||
| - Then run the included fetch script, which fetches both via the HTTP endpoint and for comparison, via the Python SDK | ||
| ```bash | ||
| python test_python_fetch.py | ||
| ``` |
Empty file.
Binary file added
BIN
+34.5 KB
examples/python-helm-demo/feature_repo/data/driver_stats_with_string.parquet
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| from datetime import timedelta | ||
|
|
||
| import pandas as pd | ||
|
|
||
| from feast.data_source import RequestSource | ||
| from feast.on_demand_feature_view import on_demand_feature_view | ||
| from feast.types import Float32, Float64, Int64, String | ||
| from feast.field import Field | ||
|
|
||
| from feast import Entity, FileSource, FeatureView | ||
|
|
||
| driver_hourly_stats = FileSource( | ||
| path="data/driver_stats_with_string.parquet", | ||
| timestamp_field="event_timestamp", | ||
| created_timestamp_column="created", | ||
| ) | ||
| driver = Entity(name="driver_id", description="driver id",) | ||
| driver_hourly_stats_view = FeatureView( | ||
| name="driver_hourly_stats", | ||
| entities=[driver], | ||
| ttl=timedelta(days=365), | ||
| schema=[ | ||
| Field(name="conv_rate", dtype=Float32), | ||
| Field(name="acc_rate", dtype=Float32), | ||
| Field(name="avg_daily_trips", dtype=Int64), | ||
| Field(name="string_feature", dtype=String), | ||
| ], | ||
| online=True, | ||
| source=driver_hourly_stats, | ||
| tags={}, | ||
| ) | ||
|
|
||
| # Define a request data source which encodes features / information only | ||
| # available at request time (e.g. part of the user initiated HTTP request) | ||
| input_request = RequestSource( | ||
| name="vals_to_add", | ||
| schema=[ | ||
| Field(name="val_to_add", dtype=Int64), | ||
| Field(name="val_to_add_2", dtype=Int64), | ||
| ], | ||
| ) | ||
|
|
||
|
|
||
| # Define an on demand feature view which can generate new features based on | ||
| # existing feature views and RequestSource features | ||
| @on_demand_feature_view( | ||
| sources=[ | ||
| driver_hourly_stats_view, | ||
| input_request, | ||
| ], | ||
| schema=[ | ||
| Field(name="conv_rate_plus_val1", dtype=Float64), | ||
| Field(name="conv_rate_plus_val2", dtype=Float64), | ||
| ], | ||
| ) | ||
| def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame: | ||
| df = pd.DataFrame() | ||
| df["conv_rate_plus_val1"] = inputs["conv_rate"] + inputs["val_to_add"] | ||
| df["conv_rate_plus_val2"] = inputs["conv_rate"] + inputs["val_to_add_2"] | ||
| return df | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| registry: gs://[YOUR GCS BUCKET]/demo-repo/registry.db | ||
| project: feast_python_demo | ||
| provider: gcp | ||
| online_store: | ||
| type: redis | ||
| # Note: this would normally be using instance URL's to access Redis | ||
| connection_string: localhost:6379,password=[YOUR PASSWORD] | ||
| offline_store: | ||
| type: file | ||
| entity_key_serialization_version: 2 |
Oops, something went wrong.