Merge branch 'develop' into bugfix/fix-sdk-docs-client-rendering

README.md

@@ -1,7 +1,7 @@
 <div align="center">
   <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=0fcbab94-8fbe-4a38-93e8-c2348450a42e" />
-  <h1 align="center">Connecting data science teams seamlessly to cloud infrastructure.
-</h1>
+  <h1 align="center">Beyond The Demo: Production-Grade AI Systems</h1>
+  <h3 align="center">ZenML brings battle-tested MLOps practices to your AI applications, handling evaluation, monitoring, and deployment at scale</h3>
 </div>
 
 <!-- PROJECT SHIELDS -->
@@ -75,6 +75,8 @@
 
 ---
 
+Need help with documentation? Visit our [docs site](https://docs.zenml.io) for comprehensive guides and tutorials, or browse the [SDK reference](https://sdkdocs.zenml.io/) to find specific functions and classes.
+
 ## ⭐️ Show Your Support
 
 If you find ZenML helpful or interesting, please consider giving us a star on GitHub. Your support helps promote the project and lets others know that it's worth checking out. 
@@ -98,40 +100,44 @@ Take a tour with the guided quickstart by running:
 zenml go
 ```
 
-## 🪄 Simple, integrated, End-to-end MLOps 
+## 🪄 From Prototype to Production: AI Made Simple
 
-### Create machine learning pipelines with minimal code changes
+### Create AI pipelines with minimal code changes
 
-ZenML is a MLOps framework intended for data scientists or ML engineers looking to standardize machine learning practices. Just add `@step` and `@pipeline` to your existing Python functions to get going. Here is a toy example:
+ZenML is an open-source framework that handles MLOps and LLMOps for engineers scaling AI beyond prototypes. Automate evaluation loops, track performance, and deploy updates across 100s of pipelines—all while your RAG apps run like clockwork.
 
 ```python
 from zenml import pipeline, step
 
-@step  # Just add this decorator
-def load_data() -> dict:
-    training_data = [[1, 2], [3, 4], [5, 6]]
-    labels = [0, 1, 0]
-    return {'features': training_data, 'labels': labels}
+@step
+def load_rag_documents() -> dict:
+    # Load and chunk documents for RAG pipeline
+    documents = extract_web_content(url="https://www.zenml.io/")
+    return {"chunks": chunk_documents(documents)}
 
 @step
-def train_model(data: dict) -> None:
-    total_features = sum(map(sum, data['features']))
-    total_labels = sum(data['labels'])
-
-    print(f"Trained model using {len(data['features'])} data points. "
-          f"Feature sum is {total_features}, label sum is {total_labels}")
+def generate_embeddings(data: dict) -> None:
+    # Generate embeddings for RAG pipeline
+    embeddings = embed_documents(data['chunks'])
+    return {"embeddings": embeddings}
 
-@pipeline  # This function combines steps together 
-def simple_ml_pipeline():
-    dataset = load_data()
-    train_model(dataset)
+@step
+def index_generator(
+    embeddings: dict,
+) -> str:
+    # Generate index for RAG pipeline
+    index = create_index(embeddings)
+    return index.id
+
 
-if __name__ == "__main__":
-    run = simple_ml_pipeline()  # call this to run the pipeline
-
+@pipeline
+def rag_pipeline() -> str:
+    documents = load_rag_documents()
+    embeddings = generate_embeddings(documents)
+    index = index_generator(embeddings)
+    return index
 ```
-
-![Running a ZenML pipeline](/docs/book/.gitbook/assets/readme_basic_pipeline.gif)
+![Running a ZenML pipeline](/docs/book/.gitbook/assets/readme_simple_pipeline.gif)
 
 ### Easily provision an MLOps stack or reuse your existing infrastructure
 
@@ -183,18 +189,47 @@ def training(...):
 
 Create a complete lineage of who, where, and what data and models are produced.
 
-You’ll be able to find out who produced which model, at what time, with which data, and on which version of the code. This guarantees full reproducibility and auditability.
+You'll be able to find out who produced which model, at what time, with which data, and on which version of the code. This guarantees full reproducibility and auditability.
 
 ```python
 from zenml import Model
 
-@step(model=Model(name="classification"))
-def trainer(training_df: pd.DataFrame) -> Annotated["model", torch.nn.Module]:
-	...
+@step(model=Model(name="rag_llm", tags=["staging"]))
+def deploy_rag(index_id: str) -> str:
+    deployment_id = deploy_to_endpoint(index_id)
+    return deployment_id
 ```
 
 ![Exploring ZenML Models](/docs/book/.gitbook/assets/readme_mcp.gif)
 
+## 🚀 Key LLMOps Capabilities
+
+### Continual RAG Improvement
+**Build production-ready retrieval systems**  
+
+<div align="center">
+  <img src="/docs/book/.gitbook/assets/rag_zenml_home.png" width="800" alt="RAG Pipeline">
+</div>
+
+ZenML tracks document ingestion, embedding versions, and query patterns. Implement feedback loops and:
+- Fix your RAG logic based on production logs
+- Automatically re-ingest updated documents
+- A/B test different embedding models
+- Monitor retrieval quality metrics
+
+### Reproducible Model Fine-Tuning
+**Confidence in model updates**
+
+<div align="center">
+  <img src="/docs/book/.gitbook/assets/finetune_zenml_home.png" width="800" alt="Finetuning Pipeline">
+</div>
+
+Maintain full lineage of SLM/LLM training runs:
+- Version training data and hyperparameters
+- Track performance across iterations
+- Automatically promote validated models
+- Roll back to previous versions if needed
+
 ### Purpose built for machine learning with integrations to your favorite tools
 
 While ZenML brings a lot of value out of the box, it also integrates into your existing tooling and infrastructure without you having to be locked in.
@@ -211,6 +246,14 @@ def train_and_deploy(training_df: pd.DataFrame) -> bento.Bento
 
 ![Exploring ZenML Integrations](/docs/book/.gitbook/assets/readme_integrations.gif)
 
+## 🔄 Your LLM Framework Isn't Enough for Production
+
+While tools like LangChain and LlamaIndex help you **build** LLM workflows, ZenML helps you **productionize** them by adding:
+
+✅ **Artifact Tracking** - Every vector store index, fine-tuned model, and evaluation result versioned automatically  
+✅ **Pipeline History** - See exactly what code/data produced each version of your RAG system  
+✅ **Stage Promotion** - Move validated pipelines from staging → production with one click  
+
 ## 🖼️ Learning
 
 The best way to learn about ZenML is the [docs](https://docs.zenml.io/). We recommend beginning with the [Starter Guide](https://docs.zenml.io/user-guide/starter-guide) to get up and running quickly.
@@ -295,13 +338,23 @@ Or, if you
 prefer, [open an issue](https://github.com/zenml-io/zenml/issues/new/choose) on
 our GitHub repo.
 
-## ⭐️ Show Your Support
+## 📚 LLM-focused Learning Resources
 
-If you find ZenML helpful or interesting, please consider giving us a star on GitHub. Your support helps promote the project and lets others know that it's worth checking out. 
+1. [LL Complete Guide - Full RAG Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide) - Document ingestion, embedding management, and query serving
+2. [LLM Fine-Tuning Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/llm-finetuning) - From data prep to deployed model
+3. [LLM Agents Example](https://github.com/zenml-io/zenml-projects/tree/main/llm-agents) - Track conversation quality and tool usage
 
-Thank you for your support! 🌟
+## 🤖 AI-Friendly Documentation with llms.txt
 
-[![Star this project](https://img.shields.io/github/stars/zenml-io/zenml?style=social)](https://github.com/zenml-io/zenml/stargazers)
+ZenML implements the llms.txt standard to make our documentation more accessible to AI assistants and LLMs. Our implementation includes:
+
+- Base documentation at [zenml.io/llms.txt](https://zenml.io/llms.txt) with core user guides
+- Specialized files for different documentation aspects:
+  - [Component guides](https://zenml.io/component-guide.txt) for integration details
+  - [How-to guides](https://zenml.io/how-to-guides.txt) for practical implementations
+  - [Complete documentation corpus](https://zenml.io/llms-full.txt) for comprehensive access
+
+This structured approach helps AI tools better understand and utilize ZenML's documentation, enabling more accurate code suggestions and improved documentation search.
 
 ## 📜 License
 

docs/book/component-guide/image-builders/kaniko.md

@@ -143,7 +143,7 @@ List of some possible additional flags:
 
 * `--cache`: Set to `false` to disable caching. Defaults to `true`.
 * `--cache-dir`: Set the directory where to store cached layers. Defaults to `/cache`.
-* `--cache-repo`: Set the repository where to store cached layers. Defaults to `gcr.io/kaniko-project/executor`.
+* `--cache-repo`: Set the repository where to store cached layers.
 * `--cache-ttl`: Set the cache expiration time. Defaults to `24h`.
 * `--cleanup`: Set to `false` to disable cleanup of the working directory. Defaults to `true`.
 * `--compressed-caching`: Set to `false` to disable compressed caching. Defaults to `true`.

docs/book/component-guide/orchestrators/sagemaker.md

@@ -167,7 +167,7 @@ Additional configuration for the Sagemaker orchestrator can be passed via `Sagem
 * `sagemaker_session`
 * `entrypoint`
 * `base_job_name`
-* `env`
+* `environment`
 
 For example, settings can be provided and applied in the following way:
 
@@ -180,6 +180,7 @@ from zenml.integrations.aws.flavors.sagemaker_orchestrator_flavor import (
 sagemaker_orchestrator_settings = SagemakerOrchestratorSettings(
     instance_type="ml.m5.large",
     volume_size_in_gb=30,
+    environment={"MY_ENV_VAR": "my_value"}
 )
 
 

docs/book/user-guide/production-guide/README.md

@@ -22,4 +22,8 @@ Like in the starter guide, make sure you have a Python environment ready and `vi
 
 By the end, you will have completed an [end-to-end](end-to-end.md) MLOps project that you can use as inspiration for your own work. Let's get right into it!
 
+{% hint style="info" %}
+Throughout this guide, we will be referencing internal ZenML functions and classes, which are more easily discoverable in the [SDK Docs](https://sdkdocs.zenml.io/). Consult the SDK docs if you're ever stuck!
+{% endhint %}
+
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

docs/book/user-guide/production-guide/ci-cd.md

@@ -142,7 +142,7 @@ When you push to a branch now, that is within a Pull Request, this action will r
 ### (Optional) Comment Metrics onto the PR
 
 Finally you can configure your github action workflow to leave a report based on the pipeline that was run. 
-Check out the template for this [here](https://github.com/zenml-io/zenml-gitflow/blob/main/.github/workflows/pipeline_run.yaml#L87-L99.
+Check out the template for this [here](https://github.com/zenml-io/zenml-gitflow/blob/main/.github/workflows/pipeline_run.yaml#L87-L99).
 
 ![Comment left on Pull Request](../../.gitbook/assets/github-action-pr-comment.png)
 

docs/book/user-guide/production-guide/understand-stacks.md

@@ -221,4 +221,8 @@ To run a pipeline using the new stack:
 
 Keep this code handy as we'll be using it in the next chapters!
 
+{% hint style="info" %}
+If you ever want to learn more about individual ZenML functions or classes, check out the [SDK Docs](https://sdkdocs.zenml.io/)
+{% endhint %}
+
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

docs/book/user-guide/starter-guide/README.md

@@ -20,4 +20,8 @@ Before jumping in, make sure you have a Python environment ready and `virtualenv
 
 Let this guide be not only your introduction to ZenML but also a foundational asset in your MLOps toolkit. Prepare your development environment, and let's get started!
 
+{% hint style="info" %}
+Throughout this guide, we will be referencing internal ZenML functions and classes, which are more easily discoverable in the [SDK Docs](https://sdkdocs.zenml.io/). Consult the SDK docs if you're ever stuck!
+{% endhint %}
+
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

docs/book/user-guide/starter-guide/create-an-ml-pipeline.md

@@ -273,6 +273,10 @@ training_pipeline.write_run_configuration_template(path='/local/path/to/config.y
 
 Check out [this section](../../how-to/pipeline-development/use-configuration-files/README.md) for advanced configuration options.
 
+{% hint style="info" %}
+If you ever want to learn more about individual ZenML functions or classes, check out the [SDK Docs](https://sdkdocs.zenml.io/)
+{% endhint %}
+
 ## Full Code Example
 
 This section combines all the code from this section into one simple script that you can use to run easily:

pyproject.toml

@@ -155,11 +155,12 @@ pytest-rerunfailures = { version = ">=13.0", optional = true }
 pytest-split = { version = "^0.8.1", optional = true }
 
 # mkdocs including plugins
-mkdocs = { version = "^1.6.1", optional = true }
-mkdocs-material = { version = "9.6.5", optional = true }
-mkdocs-awesome-pages-plugin = { version = "2.10.1", optional = true }
-mkdocstrings = { extras = ["python"], version = "^0.28.1", optional = true }
-mike = { version = "2.1.3", optional = true }
+mkdocs = { version = "^1.6.1,<2.0.0", optional = true }
+mkdocs-material = { version = ">=9.6.5,<10.0.0", optional = true }
+mkdocs-awesome-pages-plugin = { version = ">=2.10.1,<3.0.0", optional = true }
+mkdocstrings = { extras = ["python"], version = "^0.28.1,<1.0.0", optional = true }
+mkdocs-autorefs = { version = ">=0.4.1,<1.0.0", optional = true }
+mike = { version = ">=1.1.2,<2.0.0", optional = true }
 
 # mypy type stubs
 types-certifi = { version = "^2021.10.8.0", optional = true }
@@ -255,6 +256,8 @@ dev = [
     "mkdocs-material",
     "mkdocs-awesome-pages-plugin",
     "mkdocstrings",
+    "mkdocstrings-python",
+    "mkdocs-autorefs",
     "mike",
     "maison",
     "types-certifi",
@@ -406,6 +409,7 @@ module = [
     "google.*",
     "transformers.*", # https://github.com/huggingface/transformers/issues/13390
     "datasets.*",
+    "langchain_community.*",
     "IPython.core.*",
 ]
 follow_imports = "skip"
@@ -481,7 +485,6 @@ module = [
     "prodigy.components.*",
     "prodigy.components.db.*",
     "transformers.*",
-    "langchain_community.*",
     "vllm.*",
     "numba.*",
     "uvloop.*",

src/zenml/integrations/aws/flavors/sagemaker_orchestrator_flavor.py

@@ -71,6 +71,7 @@ class SagemakerOrchestratorSettings(BaseSettings):
             For processor_args.instance_type, check
             https://docs.aws.amazon.com/sagemaker/latest/dg/notebooks-available-instance-types.html
             for a list of available instance types.
+        environment: Environment variables to pass to the container.
         estimator_args: Arguments that are directly passed to the SageMaker
             Estimator for a specific step, allowing for overriding the default
             settings provided when configuring the component. See
@@ -116,6 +117,7 @@ class SagemakerOrchestratorSettings(BaseSettings):
 
     processor_args: Dict[str, Any] = {}
     estimator_args: Dict[str, Any] = {}
+    environment: Dict[str, str] = {}
 
     input_data_s3_mode: str = "File"
     input_data_s3_uri: Optional[Union[str, Dict[str, str]]] = Field(

src/zenml/integrations/aws/flavors/sagemaker_step_operator_flavor.py

@@ -55,6 +55,7 @@ class SagemakerStepOperatorSettings(BaseSettings):
             For estimator_args.instance_type, check
             https://docs.aws.amazon.com/sagemaker/latest/dg/notebooks-available-instance-types.html
             for a list of available instance types.
+        environment: Environment variables to pass to the container.
 
     """
 
@@ -64,6 +65,7 @@ class SagemakerStepOperatorSettings(BaseSettings):
         default=None, union_mode="left_to_right"
     )
     estimator_args: Dict[str, Any] = {}
+    environment: Dict[str, str] = {}
 
     _deprecation_validator = deprecation_utils.deprecate_pydantic_attributes(
         "instance_type"

src/zenml/integrations/aws/orchestrators/sagemaker_orchestrator.py

@@ -323,6 +323,19 @@ def prepare_or_run_pipeline(
                 ExecutionVariables.PIPELINE_EXECUTION_ARN
             )
 
+            if step_settings.environment:
+                step_environment = step_settings.environment.copy()
+                # Sagemaker does not allow environment variables longer than 256
+                # characters to be passed to Processor steps. If an environment variable
+                # is longer than 256 characters, we split it into multiple environment
+                # variables (chunks) and re-construct it on the other side using the
+                # custom entrypoint configuration.
+                split_environment_variables(
+                    size_limit=SAGEMAKER_PROCESSOR_STEP_ENV_VAR_SIZE_LIMIT,
+                    env=step_environment,
+                )
+                environment.update(step_environment)
+
             use_training_step = (
                 step_settings.use_training_step
                 if step_settings.use_training_step is not None
@@ -457,6 +470,11 @@ def prepare_or_run_pipeline(
                         )
                     )
 
+            # Convert environment to a dict of strings
+            environment = {
+                key: str(value) for key, value in environment.items()
+            }
+
             if use_training_step:
                 # Create Estimator and TrainingStep
                 estimator = sagemaker.estimator.Estimator(

src/zenml/integrations/aws/step_operators/sagemaker_step_operator.py

@@ -181,6 +181,11 @@ def launch(
                 self.name,
             )
 
+        settings = cast(SagemakerStepOperatorSettings, self.get_settings(info))
+
+        if settings.environment:
+            environment.update(settings.environment)
+
         # Sagemaker does not allow environment variables longer than 512
         # characters to be passed to Estimator steps. If an environment variable
         # is longer than 512 characters, we split it into multiple environment
@@ -194,8 +199,6 @@ def launch(
         image_name = info.get_image(key=SAGEMAKER_DOCKER_IMAGE_KEY)
         environment[_ENTRYPOINT_ENV_VARIABLE] = " ".join(entrypoint_command)
 
-        settings = cast(SagemakerStepOperatorSettings, self.get_settings(info))
-
         # Get and default fill SageMaker estimator arguments for full ZenML support
         estimator_args = settings.estimator_args
 
@@ -221,6 +224,9 @@ def launch(
             "instance_type", settings.instance_type or "ml.m5.large"
         )
 
+        # Convert environment to a dict of strings
+        environment = {key: str(value) for key, value in environment.items()}
+
         estimator_args["environment"] = environment
         estimator_args["instance_count"] = 1
         estimator_args["sagemaker_session"] = session