Skip to content

Contexter/Ensemble-Service

BranchesTags

Repository files navigation

FountainAI Ensemble Service Development Plan (Dockerized Workflow)

Introduction

This guide outlines a specification-driven development strategy for implementing the FountainAI Ensemble Service as a FastAPI application within a fully Dockerized local development environment. The FountainAI Ensemble Service is a core component of the FountainAI ecosystem, facilitating structured interactions between users, the OpenAI Assistant SDK (The Assistant), and various FountainAI services. It dynamically generates system prompts based on the OpenAPI specifications of each integrated FountainAI service.

Read its OpenAPI Specification.

Project Organization and Script Management

To ensure a consistent and unambiguous project structure, the repository follows a standardized folder layout with a clear root directory and explicitly defined paths. The root directory of the project is /service.

Project Structure

/service
├── README.md
├── openapi3_1.yml
├── run_setup.sh  # Shell script to automate Docker build and setup
├── scripts/
│   ├── generate_dockerfile.py          # Script to generate the Dockerfile
│   ├── generate_docker_compose.py      # Script to generate docker-compose.yml
│   ├── create_directory_structure.py   # Script to create project directories and initial files
│   ├── generate_minimal_main.py        # Script to create a minimal main.py for FastAPI
│   ├── generate_openapi_parser.py      # Script to create an OpenAPI parser module
│   ├── generate_authentication.py      # Script to set up authentication dependencies
│   ├── generate_schemas.py             # Script to generate Pydantic schemas from OpenAPI spec
│   ├── generate_models.py              # Script to generate SQLAlchemy models from OpenAPI spec
│   ├── generate_alembic_migration.py   # Script to set up Alembic and create migrations
│   ├── generate_crud.py                # Script to generate CRUD operations for models
│   ├── generate_api_routes.py          # Script to create FastAPI route files from OpenAPI spec
│   ├── generate_typesense_sync.py      # Script to set up Typesense synchronization
│   ├── generate_logging.py             # Script to configure logging and middleware
│   ├── generate_comprehensive_main.py  # Script to enhance main.py with routers and dependencies
│   ├── validate_openapi_schema.py      # Script to validate FastAPI OpenAPI schema against original spec
│   ├── generate_tests.py               # Script to generate pytest test cases from OpenAPI spec
│   ├── setup_fountainai_ensemble.py    # Master script to orchestrate setup by running all other scripts
│   └── ...                             # Placeholder for additional scripts
└── app/
    ├── api/
    │   └── routes/                      # FastAPI route files
    ├── auth/                            # Authentication components and dependencies
    ├── core/
    │   ├── logging.py                   # Logging configuration and setup
    │   ├── middleware.py                # Middleware configurations
    │   └── openapi_parser.py            # OpenAPI parser module
    ├── crud/                            # CRUD operation functions for models
    ├── models/                          # SQLAlchemy models representing database tables
    ├── schemas/                         # Pydantic schemas for request and response validation
    ├── typesense/
    │   └── client.py                    # Typesense client configuration and synchronization functions
    ├── main.py                          # FastAPI main application file
    ├── tests/
    │   └── test_api.py                  # Pytest test cases for API endpoints
    └── ...                              # Placeholder for additional application modules

Implementation Using Modular Scripts

Overview

The development workflow utilizes modular Python scripts placed within the /service/scripts/ directory. Each script performs a specific task, contributing to the overall setup of the FountainAI Ensemble Service. The setup_fountainai_ensemble.py script orchestrates the execution of all these scripts in the correct order.

Below is a detailed description of each script, including its purpose and the prompt that defines its functionality.

1. generate_dockerfile.py

Description:

This script generates a Dockerfile in the /service root directory, setting up the Docker environment with the necessary configurations for the FastAPI application. In addition, it integrates migration management to ensure the database schema aligns with any updates when containers are started or rebuilt. The Dockerfile should include commands to apply migrations as part of the container’s startup routine.

Prompt:

Generate a script named generate_dockerfile.py that will be placed in /service/scripts/. This script creates a Dockerfile in the /service root directory that uses an official Python base image (e.g., python:3.11-slim). It should set the working directory to /service to prevent recursive directory structures, install required system packages, copy the application code and scripts into the container ensuring files are placed correctly, install Python dependencies using pip, set environment variables as needed, and specify the command to run the FastAPI application using Uvicorn. Additionally, include the necessary tools for database migrations and add a command to apply migrations (e.g., alembic upgrade head) as part of the container startup routine.

2. generate_docker_compose.py

Description:

This script generates a docker-compose.yml file in the /service root directory, defining the services required for the FastAPI application and Typesense, along with necessary configurations. Docker Compose should define persistent volumes for the database to ensure that schema changes persist across container restarts.

Prompt:

Generate a script named generate_docker_compose.py that will be placed in /service/scripts/. This script creates a docker-compose.yml file in the /service root directory that defines services for the FastAPI application and Typesense. It should mount volumes to allow code changes to be reflected without rebuilding the image, expose necessary ports, set up network configurations, include environment variables for development settings, and define persistent volumes for the database to ensure schema changes persist across container restarts.

3. create_directory_structure.py

Description:

This script creates the necessary directories and initial files for the FountainAI Ensemble Service relative to the /service root directory, ensuring a minimal but functional FastAPI application setup.

Prompt:

Generate a script named create_directory_structure.py that will be placed in /service/scripts/. This script creates the necessary directories and initial files for the FountainAI Ensemble Service relative to the /service root directory. It should create the following directories: /service/app/, /service/app/api/routes/, /service/app/models/, /service/app/schemas/, /service/app/crud/, /service/app/core/, /service/app/auth/, /service/app/typesense/, and /service/app/tests/. Additionally, it should create the initial files: /service/app/main.py with a basic FastAPI app instance, /service/app/api/routes/__init__.py, /service/app/api/routes/root.py with a root endpoint, and __init__.py files in each package as needed. The script should ensure that after execution, there is a minimal but functional FastAPI application that can be run and will respond to requests. It should print a confirmation message for each directory and file created. Ensure that the script is compatible with execution inside a Docker container and that all paths are explicitly defined relative to /service.

4. generate_minimal_main.py

Description:

This script generates a minimal main.py in /service/app/ to ensure that the FastAPI application can run independently. This minimal setup includes the FastAPI instance and a simple root endpoint.

Prompt:

Generate a script named generate_minimal_main.py that will be placed in /service/scripts/. This script creates a minimal /service/app/main.py with a basic FastAPI app instance and a simple root endpoint (/). The minimal main.py should allow the FastAPI application to start and respond to requests without dependencies on other components. It should overwrite any existing main.py file, provide a confirmation message upon completion, and ensure compatibility with Docker by explicitly defining all paths relative to /service.

5. generate_openapi_parser.py

Description:

This script creates an OpenAPI parser module in /service/app/core/openapi_parser.py that can be imported by other scripts to facilitate the parsing of the OpenAPI specification. This parser centralizes the logic for extracting necessary information from the OpenAPI file, promoting code reusability and consistency across scripts.

Prompt:

Generate a script named generate_openapi_parser.py that will be placed in /service/scripts/. This script creates /service/app/core/openapi_parser.py, which contains a reusable OpenAPI parser class or functions to parse the OpenAPI specification located at /service/openapi3_1.yml. The parser should provide methods to extract schemas, security schemes, paths, parameters (categorized by path, query, header, and cookie), responses, request bodies, tags, callbacks, examples, links, headers, content types, operation IDs, server information, and security requirements needed by other scripts. It should also handle resolving $ref references, including both internal and external references, and provide validation of the loaded specification against the OpenAPI 3.1 schema. It should overwrite any existing parser file, provide a confirmation message upon completion, and ensure compatibility with Docker by explicitly defining all paths relative to /service.

6. generate_authentication.py

Description:

This script creates authentication dependencies in /service/app/auth/dependencies.py based on the OpenAPI specification, implementing API key authentication mechanisms.

Prompt:

Generate a script named generate_authentication.py that will be placed in /service/scripts/. This script creates /service/app/auth/dependencies.py with authentication dependencies as per the OpenAPI specification located at /service/openapi3_1.yml. It should implement the API key authentication mechanisms for both apiKeyAuth and adminApiKeyAuth, ensuring that the dependencies validate API keys correctly and raise appropriate HTTP exceptions. The script should overwrite any existing file and confirm upon completion. Ensure compatibility with Docker and that all paths are explicitly defined relative to /service.

7. generate_schemas.py

Description:

This script creates Pydantic models in /service/app/schemas/ based on the schemas defined in the OpenAPI specification, ensuring they match exactly with field types and validations. It utilizes the OpenAPI parser to extract necessary information.

Prompt:

Generate a script named generate_schemas.py that will be placed in /service/scripts/. This script creates Pydantic models in /service/app/schemas/ corresponding to the schemas defined in the OpenAPI specification located at /service/openapi3_1.yml. It should use the OpenAPI parser created in /service/app/core/openapi_parser.py to extract the relevant information from the specification, ensuring that the models match the specification exactly, including field types and validations. The script should generate separate files for each model as appropriate, overwrite any existing files, and provide a confirmation message upon completion. Ensure compatibility with Docker by explicitly defining all paths relative to /service.

8. generate_models.py

Description:

This script generates SQLAlchemy models in /service/app/models/ based on the OpenAPI schemas, ensuring accurate representation for database use. It utilizes the OpenAPI parser to extract necessary information.

Prompt:

Generate a script named generate_models.py that will be placed in /service/scripts/. This script will create SQLAlchemy models in /service/app/models/ corresponding to the database schemas defined in the OpenAPI specification located at /service/openapi3_1.yml. It should use the OpenAPI parser created in /service/app/core/openapi_parser.py to extract the relevant information from the specification, including necessary relationships, field types, primary keys, and constraints. The script should validate that each schema is suitable for generating a model, such as having valid class names. It should write each model into a separate Python file and create backups of existing files. Ensure compatibility with Docker by explicitly defining all paths relative to /service. Additionally, the script should log progress and errors, and report tables created successfully in the database.

9. generate_alembic_migration.py

Description:

This script sets up Alembic for the project and creates new migrations based on the current SQLAlchemy models, facilitating database table creation and updates.

Prompt:

Generate a script named generate_alembic_migration.py that will be placed in /service/scripts/. This script should set up Alembic for the project if it is not already set up. It should create a new Alembic migration based on the current state of the SQLAlchemy models in /service/app/models/ and place the migration files in /service/migrations/. Ensure that all paths are explicitly defined relative to /service and that the script is compatible with Docker. Additionally, it should log the success or failure of migration creation and any encountered issues.

10. generate_crud.py

Description:

This script generates CRUD operation functions in /service/app/crud/ for the models created by the generate_models.py script, ensuring proper session handling and exception management.

Prompt:

Generate a script named generate_crud.py that will be placed in /service/scripts/. This script will create CRUD operation functions in /service/app/crud/ for the models generated by the generate_models.py script. It should define functions for creating, reading, updating, and deleting records, ensuring that each function properly handles sessions and exceptions. The script should write each model’s CRUD operations into a separate Python file, ensure that all paths are explicitly defined relative to /service, and are compatible with Docker. Additionally, it should include error handling, logging, and validation to ensure robustness.

11. generate_api_routes.py

Description:

This script creates FastAPI route files in /service/app/api/routes/ based on the OpenAPI specification, including route decorators and security dependencies.

Prompt:

Generate a script named generate_api_routes.py that will be placed in /service/scripts/. This script creates route files in /service/app/api/routes/ with FastAPI endpoints as per the OpenAPI specification located at /service/openapi3_1.yml. It should use the OpenAPI parser created in /service/app/core/openapi_parser.py to extract the relevant information from the specification for each endpoint, using the exact path, HTTP method, parameters, and response models defined. The script should add route decorators with summary, description, and operation_id to enhance the OpenAPI documentation, apply the appropriate security dependencies (get_api_key, get_admin_api_key), and include error handling and response models as specified. It should overwrite any existing route files and confirm upon completion. Ensure that the script is compatible with the Dockerized environment and that all paths are explicitly defined relative to /service.

12. generate_typesense_sync.py

Description:

This script sets up Typesense client configuration and synchronization functions in /service/app/typesense/, ensuring data consistency with Typesense.

Prompt:

Generate a script named generate_typesense_sync.py that will be placed in /service/scripts/. This script should create /service/app/typesense/client.py with the Typesense client configuration and synchronization functions in /service/app/typesense/ to keep the data in sync with Typesense. It should use the OpenAPI parser created in /service/app/core/openapi_parser.py to extract relevant information from the OpenAPI specification located at /service/openapi3_1.yml. The synchronization logic should include error handling and retries as per the specification. The script should overwrite existing files and confirm upon completion. It should be designed to run within Docker, with all paths explicitly defined relative to /service.

13. generate_logging.py

Description:

This script sets up logging configuration and middleware in /service/app/core/logging.py to capture and log interactions as specified.

Prompt:

Generate a script named generate_logging.py that will be placed in /service/scripts/. This script should create logging configuration in /service/app/core/logging.py and set up middleware or dependency injection for logging requests and responses. It should ensure that the logging setup captures interactions as specified in the OpenAPI specification located at /service/openapi3_1.yml. The script should overwrite existing files and confirm upon completion. Ensure compatibility with Docker and that all paths are explicitly defined relative to /service.

14. generate_comprehensive_main.py

Description:

This script enhances the main.py file in /service/app/ to include routers, middleware, event handlers, and any global dependencies now that all dependencies and components are in place. This ensures the application is fully configured and ready to run with Uvicorn.

Prompt:

Generate a script named generate_comprehensive_main.py that will be placed in /service/scripts/. This script updates /service/app/main.py to include routers from /service/app/api/routes/, add middleware, event handlers, and any global dependencies as needed based on the components generated by other scripts. It should ensure the application is ready to run with Uvicorn, overwrite the minimal main.py file created by generate_minimal_main.py, and confirm upon completion. Ensure compatibility with Docker and that all paths are explicitly defined relative to /service.

15. validate_openapi_schema.py

Description:

This script validates the generated FastAPI OpenAPI schema against the original specification, ensuring consistency and accuracy.

Prompt:

Generate a script named validate_openapi_schema.py that will be placed in /service/scripts/. This script should start the FastAPI application within the script to generate the OpenAPI schema, load the original OpenAPI specification from /service/openapi3_1.yml, compare the generated schema with the original specification, and report any discrepancies in a clear and actionable manner. It should ensure the script is compatible with execution inside the Docker container and that all paths are explicitly defined relative to /service.

16. generate_tests.py

Description:

This script generates pytest test cases in /service/app/tests/test_api.py based on the OpenAPI specification, covering valid requests, authentication, and error handling.

Prompt:

Generate a script named generate_tests.py that will be placed in /service/scripts/. This script should read the OpenAPI specification from /service/openapi3_1.yml and, for each endpoint defined in the specification, generate a test case using pytest that sends a request to the endpoint with valid parameters, checks that the response status code and body match the expected outputs defined in the specification, includes tests for authentication mechanisms validating that unauthorized requests are handled correctly, and handles error cases as defined in the specification. It should output the test cases to /service/app/tests/test_api.py, ensure the script is compatible with execution inside the Docker container, and that all paths are explicitly defined relative to /service.

17. setup_fountainai_ensemble.py

Description:

This master script executes all the previously generated Python scripts in the correct order to set up the FountainAI Ensemble Service project.

Prompt:

Generate a script named setup_fountainai_ensemble.py that will be placed in /service/scripts/. This script should execute all the previously generated Python scripts in the correct order to set up the FountainAI Ensemble Service project. The execution order should ensure that dependencies are met before dependent scripts run. It should execute each Python script and check for successful completion, provide a final confirmation message upon successful setup, and ensure that the script is designed to run within the Docker container with all paths explicitly defined relative to /service.

18. run_setup.sh

Description:

This shell script automates the Docker build and setup process by building and starting the Docker containers and running the setup Python script within the container.

Prompt:

Generate a shell script named run_setup.sh to be placed in /service/. This script should build and start the Docker containers using docker-compose up --build -d, wait for the containers to be ready, and then execute python scripts/setup_fountainai_ensemble.py inside the FastAPI application container. It should provide appropriate echo statements to inform the user of each step and ensure the script is executable and compatible with Unix-based systems.

Execution Order

To accommodate the creation of a minimal FastAPI application, establish the OpenAPI parser early, and set up authentication mechanisms, the execution order of the scripts has been adjusted as follows:

  1. generate_dockerfile.py
  2. generate_docker_compose.py
  3. create_directory_structure.py
  4. generate_minimal_main.py
  5. generate_openapi_parser.py
  6. generate_authentication.py
  7. generate_schemas.py
  8. generate_models.py
  9. generate_alembic_migration.py
  10. generate_crud.py
  11. generate_api_routes.py
  12. generate_typesense_sync.py
  13. generate_logging.py
  14. generate_comprehensive_main.py
  15. validate_openapi_schema.py
  16. generate_tests.py
  17. setup_fountainai_ensemble.py
  18. run_setup.sh

This order ensures that the OpenAPI parser is established early since it is a dependency for the authentication setup and other subsequent scripts.

Notes on Workflow

Prerequisites

  • Docker and Docker Compose must be installed on your local machine.
  • Refer to the Docker installation guide for detailed instructions.

Clone the Repository

Clone the repository and navigate to the /service directory:

git clone https://github.com/Contexter/Ensemble-Service.git
cd Ensemble-Service/service

Configure the Docker Environment

Use the generate_dockerfile.py and generate_docker_compose.py scripts to create the Dockerfile and docker-compose.yml in the /service root directory.

python scripts/generate_dockerfile.py
python scripts/generate_docker_compose.py

Ensure that the Docker configurations are set up to support a seamless local development workflow.

Build and Start the Docker Environment

Use the generated run_setup.sh script to automate the setup process:

chmod +x run_setup.sh
./run_setup.sh

This command will:

  1. Build the Docker image for the FastAPI application.
  2. Start the FastAPI application and Typesense services.
  3. Execute the setup_fountainai_ensemble.py script inside the Docker container to set up the project.

Development Steps Breakdown

  1. Generate Docker Configuration:

    • generate_dockerfile.py
    • generate_docker_compose.py

  2. Create Project Structure:

    • create_directory_structure.py

  3. Establish Minimal FastAPI App:

    • generate_minimal_main.py

  4. Create OpenAPI Parser:

    • generate_openapi_parser.py

  5. Set Up Authentication:

    • generate_authentication.py

  6. Generate Core Components:

    • generate_schemas.py
    • generate_models.py
    • generate_alembic_migration.py
    • generate_crud.py
    • generate_api_routes.py
    • generate_typesense_sync.py
    • generate_logging.py

  7. Enhance Application:

    • generate_comprehensive_main.py

  8. Validate and Test:

    • validate_openapi_schema.py
    • generate_tests.py

  9. Orchestrate Setup:

    • setup_fountainai_ensemble.py

  10. Automate with Shell Script:

    • run_setup.sh

Develop and Test

With the Docker environment running, you can engage in development and testing activities. Editing code locally will reflect changes inside the container due to volume mounting.

  • Access the application by visiting http://localhost:8000.
  • Run tests inside the container: 
    docker-compose exec app bash
pytest

The application should be fully functional after running the initial scripts, including the basic FastAPI app created by generate_minimal_main.py, the OpenAPI parser from generate_openapi_parser.py, and the authentication dependencies from generate_authentication.py.

Shut Down the Environment

When development is complete, you can stop the Docker containers with:

docker-compose down

Notes on Volume Mounting and Environment Variables

Volume Mounting

The docker-compose.yml file is configured to mount your local project directory into the Docker container. This setup allows you to make changes locally and have them take effect inside the container without the need to rebuild the Docker image each time.

Environment Variables

Manage environment variables using a .env file to maintain development settings. It is crucial to ensure that sensitive information is not committed to version control systems to maintain security.

Script Execution Details

setup_fountainai_ensemble.py

This master script ensures that all necessary scripts are executed in the correct sequence, handling dependencies appropriately.

Execution Order within setup_fountainai_ensemble.py:

  1. generate_dockerfile.py
  2. generate_docker_compose.py
  3. create_directory_structure.py
  4. generate_minimal_main.py
  5. generate_openapi_parser.py
  6. generate_authentication.py
  7. generate_schemas.py
  8. generate_models.py
  9. generate_alembic_migration.py
  10. generate_crud.py
  11. generate_api_routes.py
  12. generate_typesense_sync.py
  13. generate_logging.py
  14. generate_comprehensive_main.py
  15. validate_openapi_schema.py
  16. generate_tests.py

This sequencing ensures that the OpenAPI parser is established early, as it is a dependency for the authentication setup and other subsequent scripts.

run_setup.sh

This shell script automates the Docker build and setup process, ensuring that all steps are executed smoothly.

Script Steps:

  1. Build and Start Docker Containers:

    docker-compose up --build -d

  2. Wait for Services to Initialize:

    Incorporate a delay or health checks to ensure services like the FastAPI app and Typesense are up and running.

  3. Execute Setup Script Inside the FastAPI Container:

    docker-compose exec app python scripts/setup_fountainai_ensemble.py

  4. Provide Echo Statements:

    Inform the user about each step's progress and completion status.

Sample run_setup.sh:

#!/bin/bash

echo "Starting Docker containers..."
docker-compose up --build -d

echo "Waiting for services to initialize..."
sleep 10  # Adjust the sleep time as needed or implement health checks

echo "Executing setup script inside the FastAPI container..."
docker-compose exec app python scripts/setup_fountainai_ensemble.py

echo "Setup completed successfully!"

Ensure the script is executable:

chmod +x run_setup.sh

Summary

By restructuring the development plan to first establish a minimal FastAPI application, then introducing an OpenAPI parser, and setting up authentication dependencies, we ensure that the service is operational early in the setup process and that dependencies are properly managed. Subsequent scripts then incrementally enhance the application by adding schemas, models, migrations, CRUD operations, routes, synchronization with Typesense, logging, and comprehensive configurations. This approach promotes a stable foundation, allowing for easier debugging and validation at each step.

Following this guide will help you set up a fully Dockerized development environment for the FountainAI Ensemble Service, leveraging modular scripts to automate the creation of the application structure, configurations, and components based on the provided OpenAPI specification. This method ensures consistency, scalability, and ease of maintenance, aligning with best practices in modern software development.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages