diff --git a/README.md b/README.md index 0379ba1..6eb72b3 100644 --- a/README.md +++ b/README.md @@ -11,22 +11,33 @@ It allows a very easy way to define texts for documentation and extra parameters for each task. - License: BSD 3 Clause -- Documentation: https://osl-incubator.github.io/makim +- Documentation: [Makim Docs](https://osl-incubator.github.io/makim) + +--- ## Features -- Help text as first-class in the `.makim.yaml` specification. It can be used by - tasks and arguments. -- Tasks have an option for arguments. -- Tasks have an option for dependencies. -- Dependencies can call a task with specific arguments. -- Dependencies can have a conditional control flow (`if`). -- Allow the creation of groups, so the tasks can be organized by topics. -- Tasks and groups have an option for user defined variables and/or environment - variables. -- Access arguments, variables or environment variables via template (using - Jinja2). -- Option for using dot environment files using `env-file` key. +- **Help Text as First-Class:** Add detailed `help` text to tasks and arguments + for clear documentation. +- **Task Arguments:** Define and manage arguments with types, defaults, and + descriptions. +- **Dependencies with Conditional Control:** Set task dependencies with `if` + conditionals to manage execution dynamically. +- **Environment Variables:** Scope variables globally, by group, or by task to + reduce redundancy and maintain modularity. +- **Jinja2 Templating:** Access arguments, variables, or environment variables + via Jinja2 templates. +- **Matrix Configuration:** Automate tasks across multiple parameter + combinations (ideal for CI/CD workflows). +- **Hooks:** Use `pre-run` and `post-run` hooks to customize task lifecycles. +- **Scheduler:** Cron-like scheduling with APScheduler integration for periodic + tasks. +- **Remote Execution:** Execute tasks on remote servers via SSH with flexible + configurations. +- **Validation:** Ensures `.makim.yaml` configurations are correct with schema + validation. + +--- ## How to use it @@ -34,74 +45,69 @@ First you need to place the config file `.makim.yaml` in the root of your project. This is an example of a configuration file: ```yaml -version: 1.0.0 groups: - default: - env-file: .env + build: + env: + GROUP_ENV: group_value tasks: clean: - help: Use this task to clean up temporary files + help: Clean build artifacts args: - all: + cache: type: bool action: store_true - help: Remove all files that are tracked by git + help: Remove all cache files run: | - echo "remove file X" - build: - help: Build the program - args: - clean: - type: bool - action: store_true - help: if not set, the clean dependency will not be triggered. - dependencies: - - task: clean - if: {% raw %}${{ args.clean == true }}{% endraw %} + echo "Cleaning build directory..." + rm -rf build/ + compile: + help: Compile the project + hooks: + pre-run: + - task: clean # Run 'clean' before 'compile' run: | - echo "build file x" - echo "build file y" - echo "build file z" + echo "Compiling the project..." + +# Scheduler for automated tasks +scheduler: + daily-clean: + task: build.clean + schedule: "0 0 * * *" # Every day at midnight ``` Some examples of how to use it: -- run the `build` task: `makim build` - -- run the `clean` task: `makim clean` +- run the `compile` task: `makim build.compile` -- run the `build` task with the `clean` flag: `makim build --clean` +- run the `clean` task with argument: `makim build.clean --cache` The help menu for the `.makim.yaml` file would looks like this: ``` $ makim --help -usage: MakIm [--help] [--version] [--config-file MAKIM_FILE] [task] - -MakIm is a tool that helps you to organize and simplify your helper commands. - -positional arguments: - task - Specify the task command to be performed. Options are: - - default: - -------- - default.clean => Use this task to clean up temporary files - ARGS: - --all: (bool) Remove all files that are tracked by git - default.build => Build the program - ARGS: - --clean: (bool) if not set, the clean dependency will not be triggered. - -options: - --help, -h - Show the help menu - --version - Show the version of the installed MakIm tool. - --config-file MAKIM_FILE - Specify a custom location for the config file. - -If you have any problem, open an issue at: https://github.com/osl-incubator/makim + + Usage: makim [OPTIONS] COMMAND [ARGS]... + + Makim is a tool that helps you to organize and simplify your helper commands. + +╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│ --version -v Show the version and exit │ +│ --file TEXT Makim config file [default: .makim.yaml] │ +│ --dry-run Execute the command in dry mode │ +│ --verbose Execute the command in verbose mode │ +│ --install-completion Install completion for the current shell. │ +│ --show-completion Show completion for the current shell, to copy it or customize the installation. │ +│ --help Show this message and exit. │ +╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ build ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│ build.clean Clean build artifacts │ +│ build.compile Compile the project │ +╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ +╭─ Extensions ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ +│ cron Tasks Scheduler │ +╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ + + If you have any problem, open an issue at: https://github.com/osl-incubator/makim ``` As you can see, the help menu automatically adds information defined by all the diff --git a/docs/features.md b/docs/features.md index 0691302..544b39d 100644 --- a/docs/features.md +++ b/docs/features.md @@ -1,86 +1,278 @@ -# Features +# Overview -## Attribute: dir +Makim is a task automation tool that enhances the way tasks and dependencies are +defined using YAML instead of Makefile syntax. This document explains Makim's +features in detail, providing use cases and benefits for developers and teams +looking to simplify and automate workflows efficiently. -The dir feature in Makim allows users to define the directory from which -commands associated with specific tasks or groups are executed. This provides -greater flexibility and control over the execution environment. +--- -The `dir` attribute can be specified at three different scopes: global, group, -and task. It allows users to set the working directory for a specific task, a -group of tasks, or globally. +# Features -### Syntax and Scopes +## 1. Help Text -The dir attribute can be applied to three different scopes: +### What It Does -- #### **Global Scope** +Makim allows you to add detailed `help` text to tasks and their arguments. This +improves documentation and makes it easier for users to understand how to use +each task. - Setting the global working directory impacts all tasks and groups in the Makim - configuration. +### Use Case - ```yaml - version: 1.0 - dir: /path/to/global/directory - # ... other configuration ... - ``` +Developers often forget how a certain script works. With Makim, they can simply +run: -- #### Group Scope +```sh +makim --help +``` - Setting the working directory at the group scope affects all tasks within that - group. +And get an automatically generated help menu with clear descriptions for tasks +and arguments. - ```yaml - version: 1.0 +--- - groups: - my-group: - dir: /path/to/group/directory - tasks: - task-1: - run: | - # This task will run with the working directory set to - # /path/to/group/directory - ``` +## 2. Task Arguments -- #### Task Scope +### What It Does - Setting the working directory at the task scope allows for fine grained - control over individual tasks. +Makim allows tasks to accept arguments with predefined types, default values, +and descriptions. - ```yaml - version: 1.0 - groups: - my-group: - tasks: - my-task: - dir: /path/to/task/directory - run: | - # This task will run with the working directory set to - # /path/to/task/directory - ``` +### Use Case -## Example +Suppose you have a task that clears the cache but also wants an option to remove +all build artifacts: ```yaml -version: 1.0 -dir: /project-root +tasks: + clean: + help: Clean build artifacts + args: + cache: + type: bool + action: store_true + help: Remove all cache files + run: rm -rf build/ +``` + +You can run: + +```sh +makim clean --cache +``` + +### Benefit + +- Prevents hardcoded parameters in scripts. +- Offers a flexible way to pass arguments dynamically. + +--- + +### Benefit + +- Reduces redundant execution. +- Ensures dependencies are met before running a task. +--- + +## 3. Environment Variables + +### What It Does + +Makim allows scoping environment variables globally, at the group level, or at +the task level. + +### Use Case + +If different tasks need different environment variables, you can define: + +```yaml groups: - backend: - dir: backend - tasks: - build: - help: Build the backend services - dir: services - run: | - echo "Building backend services..." - # Additional build commands specific to the backend - - test: - help: Run backend tests - dir: tests - run: | - echo "Running backend tests..." - # Additional test commands specific to the backend + build: + env: + GROUP_ENV: group_value +``` + +### Benefit + +- Avoids redundant variable definitions. +- Increases modularity in task execution. + +--- + +## 4. Jinja2 Templating + +### What It Does + +Makim integrates with Jinja2 templating, allowing dynamic access to arguments +and environment variables. + +### Use Case + +You can use Jinja2 to dynamically insert values: + +```yaml +tasks: + greet: + help: Print a greeting + run: echo "Hello, ${{ args.name }}!" +``` + +Running: + +```sh +makim greet --name John +``` + +Outputs: + +``` +Hello, John! +``` + +### Benefit + +- Adds flexibility in command execution. +- Reduces the need for complex shell scripting. + +--- + +## 5. Matrix Configuration + +### What It Does + +Makim allows running tasks across multiple parameter combinations, making it +useful for CI/CD workflows. + +### Use Case + +If you need to test across multiple environments: + +```yaml +tasks: + test: + matrix: + python_version: + - 3.8 + - 3.9 + - 3.10 + os: + - ubuntu + - macos + run: + echo "Running test on Python ${{ matrix.python_version }} and OS ${{ + matrix.os }}" ``` + +Makim automatically expands this into multiple runs for each combination. + +### Benefit + +- Automates testing across configurations. +- Reduces manual setup in CI/CD pipelines. + +--- + +## 6. Hooks + +### What It Does + +Makim provides `pre-run` and `post-run` hooks to execute tasks before or after +another task runs. + +### Use Case + +If you need to clean before compiling: + +```yaml +tasks: + compile: + hooks: + pre-run: + - task: clean +``` + +### Benefit + +- Automates task chaining. +- Reduces duplication of logic. + +--- + +## 7. Scheduler (Cron-like Automation) + +### What It Does + +Makim integrates with APScheduler to allow periodic execution of tasks using +cron syntax. + +### Use Case + +If you need to run a cleanup task daily at midnight: + +```yaml +scheduler: + daily-clean: + task: build.clean + schedule: "0 0 * * *" +``` + +### Benefit + +- Automates repetitive tasks. +- Reduces manual intervention. + +--- + +## 8. Remote Execution + +### What It Does + +Makim allows running tasks on remote servers via SSH with flexible +configurations. + +### Use Case + +If you need to deploy a project to a remote server: + +```yaml +tasks: + deploy: + remote: my_server + run: echo "Deploying application..." +``` + +### Benefit + +- Automates remote deployments. +- Eliminates the need for manual SSH logins. + +--- + +## 9. Validation + +### What It Does + +Makim ensures that `.makim.yaml` configurations are correctly formatted using +schema validation. + +### Use Case + +Before running any tasks, Makim checks for syntax errors and missing fields, +preventing misconfigurations. + +### Benefit + +- Reduces runtime errors. +- Improves reliability of task execution. + +--- + +## Conclusion + +Makim is an all-in-one task automation tool that simplifies workflows, enhances +documentation, and integrates seamlessly with modern development practices. By +using YAML instead of Makefiles, it makes task definition more readable and +maintainable, improving developer productivity and CI/CD efficiency. + +Start using Makim today to streamline your project automation! diff --git a/docs/spec.md b/docs/spec.md new file mode 100644 index 0000000..c9f105a --- /dev/null +++ b/docs/spec.md @@ -0,0 +1,264 @@ +# Makim Specification + +## Overview + +Makim uses a YAML-based configuration file (`.makim.yaml`) to define tasks, +dependencies, and execution environments. This document provides a detailed +specification of all components available in a Makim configuration file. + +## 1. File Structure + +A `.makim.yaml` file consists of the following top-level sections: + +- `groups` +- `scheduler` +- `hosts` +- `vars` +- `env` +- `env-file` + +Each section is explained below in detail. + +--- + +## 2. Groups + +### Description + +Defines collections of related tasks. Each group contains tasks and environment +variables specific to that group. + +### Structure + +```yaml +groups: + : + env: + : + tasks: + : +``` + +### Example + +```yaml +groups: + build: + env: + BUILD_DIR: dist + tasks: + clean: + help: Clean build artifacts + run: rm -rf ${{ env.BUILD_DIR }} + compile: + help: Compile the project + run: echo "Compiling..." +``` + +--- + +## 3. Tasks + +### Description + +Tasks define executable commands with optional arguments, dependencies, and +hooks. + +### Structure + +```yaml +tasks: + : + help: + args: + env: + hooks: 
+    matrix: 
+    run: 
+```
+
+### Example
+
+```yaml
+tasks:
+  test:
+    help: Run tests
+    args:
+      verbose:
+        type: bool
+        action: store_true
+    run: pytest --verbose=${{ args.verbose }}
+```
+
+---
+
+## 4. Arguments
+
+### Description
+
+Defines arguments that tasks can accept with types, defaults, and help
+descriptions.
+
+### Structure
+
+```yaml
+args:
+  :
+    type: 
+    default: 
+    interactive: 
+    help: 
+```
+
+### Example
+
+```yaml
+args:
+  env:
+    type: str
+    default: "dev"
+    help: Environment setting
+```
+
+---
+
+## 5. Hooks
+
+### Description
+
+Hooks define tasks that run before (`pre-run`) or after (`post-run`) a task
+executes.
+
+### Structure
+
+```yaml
+hooks:
+  pre-run:
+    - task: 
+  post-run:
+    - task: 
+```
+
+### Example
+
+```yaml
+hooks:
+  pre-run:
+    - task: clean
+```
+
+---
+
+## 6. Environment Variables
+
+### Description
+
+Defines global, group, or task-specific environment variables.
+
+### Structure
+
+```yaml
+env:
+  : 
+```
+
+### Example
+
+```yaml
+env:
+  API_KEY: abc123
+```
+
+---
+
+## 7. Matrix Configuration
+
+### Description
+
+Defines multiple combinations of parameters for running a task.
+
+### Structure
+
+```yaml
+matrix:
+  :
+    - 
+    - 
+```
+
+### Example
+
+```yaml
+matrix:
+  python_version:
+    - 3.8
+    - 3.9
+    - 3.10
+  os:
+    - ubuntu
+    - macos]
+```
+
+---
+
+## 8. Scheduler
+
+### Description
+
+Defines scheduled tasks using cron syntax.
+
+### Structure
+
+```yaml
+scheduler:
+  :
+    task: 
+    schedule: 
+```
+
+### Example
+
+```yaml
+scheduler:
+  daily-clean:
+    task: build.clean
+    schedule: "0 0 * * *"
+```
+
+---
+
+## 9. Remote Hosts
+
+### Description
+
+Defines SSH configuration for executing tasks on remote servers.
+
+### Structure
+
+```yaml
+hosts:
+  :
+    username: 
+    host: 
+    port: 
+    password: 
+```
+
+### Example
+
+```yaml
+hosts:
+  my_server:
+    username: user
+    host: example.com
+    port: 22
+```
+
+---
+
+## Conclusion
+
+This document outlines the complete specification for `.makim.yaml`, covering
+all components, their syntax, and usage examples. Understanding these elements
+enables users to define efficient and scalable task automation workflows using
+Makim.
diff --git a/docs/tutorials/features/.makim.yaml b/docs/tutorials/features/.makim.yaml
new file mode 100644
index 0000000..7a4f129
--- /dev/null
+++ b/docs/tutorials/features/.makim.yaml
@@ -0,0 +1,12 @@
+hosts:
+  my_server:
+    user: deploy_user
+    host: example.com
+    port: "22"
+
+groups:
+  deploy:
+    tasks:
+      deploy:
+        remote: my_server
+        run: echo "Deploying application..."
diff --git a/docs/tutorials/features/makim-features.ipynb b/docs/tutorials/features/makim-features.ipynb
new file mode 100644
index 0000000..8490ade
--- /dev/null
+++ b/docs/tutorials/features/makim-features.ipynb
@@ -0,0 +1,207 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Makim Feature Tutorials\n",
+    "\n",
+    "This Jupyter Notebook provides in-depth tutorials for key features of Makim:\n",
+    "1. **Scheduler**: Automating tasks at specified intervals.\n",
+    "2. **SSH Execution**: Running tasks on remote servers securely.\n",
+    "\n",
+    "Each section includes explanations, YAML configuration examples, and how to run these features."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 52,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1. Scheduler: Automating Tasks\n",
+    "\n",
+    "### What is the Scheduler?\n",
+    "The Makim scheduler allows you to define recurring tasks using cron expressions.\n",
+    "\n",
+    "Writing the Makim Configuration File"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 53,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting .makim.yaml\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile .makim.yaml\n",
+    "scheduler:\n",
+    "  daily-clean:\n",
+    "    task: build.clean\n",
+    "    schedule: \"0 0 * * *\"  # Runs every day at midnight\n",
+    "\n",
+    "groups:\n",
+    "  build:\n",
+    "    tasks:\n",
+    "      clean:\n",
+    "        help: Clean build artifacts\n",
+    "        run: echo \"Cleaning build artifacts...\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 54,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Makim file: .makim.yaml\n",
+      "┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
+      "┃\u001b[1;35m \u001b[0m\u001b[1;35mName       \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mTask       \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mSchedule \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mStatus  \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mNext Run     \u001b[0m\u001b[1;35m \u001b[0m┃\n",
+      "┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
+      "│\u001b[36m \u001b[0m\u001b[36mdaily-clean\u001b[0m\u001b[36m \u001b[0m│\u001b[34m \u001b[0m\u001b[34mbuild.clean\u001b[0m\u001b[34m \u001b[0m│\u001b[33m \u001b[0m\u001b[33m0 0 * * *\u001b[0m\u001b[33m \u001b[0m│\u001b[32m \u001b[0m\u001b[32mInactive\u001b[0m\u001b[32m \u001b[0m│\u001b[35m \u001b[0m\u001b[35mNot scheduled\u001b[0m\u001b[35m \u001b[0m│\n",
+      "└─────────────┴─────────────┴───────────┴──────────┴───────────────┘\n"
+     ]
+    }
+   ],
+   "source": [
+    "!makim cron list"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 55,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Makim file: .makim.yaml\n",
+      "Successfully started schedule 'daily-clean'\n"
+     ]
+    }
+   ],
+   "source": [
+    "!makim cron start daily-clean"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 56,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Makim file: .makim.yaml\n",
+      "Successfully stopped schedule 'daily-clean'\n"
+     ]
+    }
+   ],
+   "source": [
+    "!makim cron stop daily-clean"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "### 2. SSH Execution: Running Remote Tasks\n",
+    "\n",
+    "### What is SSH Execution?\n",
+    "SSH execution allows tasks to be run on remote machines securely.\n",
+    "\n",
+    "Writing the SSH Configuration"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 57,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting .makim.yaml\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile .makim.yaml\n",
+    "hosts:\n",
+    "  my_server:\n",
+    "    user: deploy_user\n",
+    "    host: example.com\n",
+    "    port: '22'\n",
+    "\n",
+    "groups:\n",
+    "  deploy:\n",
+    "    tasks:\n",
+    "      deploy:\n",
+    "        remote: my_server\n",
+    "        run: echo \"Deploying application...\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 58,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# !makim deploy.deploy"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Summary\n",
+    "- **Scheduler** automates recurring tasks using cron syntax.\n",
+    "- **SSH Execution** enables remote task execution securely.\n",
+    "\n",
+    "Start using Makim to enhance your workflow!"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "makim",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}