Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds validate task #35

Merged
merged 2 commits into from
Jul 25, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
Gemfile.lock
pkg
.bundle
1 change: 1 addition & 0 deletions .ruby-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
2.5.3
35 changes: 35 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,41 @@ runs faster.

Reads the Puppetfile in the current directory and installs them under the `path` provided as an argument.

### r10k:validate[path]
The validate rake task will determine if the url is a valid url by connecting
to the repository and verififying it actually exists and can be accessed.
Additional if a branch, tag, or ref is specified in the Puppetfile the validate
task will also verify that that branch/tag/ref exists in the remote repository.

If you have ever deployed r10k to production only to find out a tag or branch is
missing this validate task will catch that issue.

A exit status of 0 is returned if there are no faults, while a 1 is returned if
any module has a bad status.

Status emojis can be customized by setting the following environment variables.

Example

* `GOOD_EMOJI='👍'`
* `BAD_EMOJI='😨'`


```
NAME | URL | REF | STATUS
---------|-----------------------------------------------|--------------------------------|-------
splunk | https://github.com/cudgel/splunk.git | prod | 👍
r10k | https://github.com/acidprime/r10k | v3.1.1 | 👍
gms | https://github.com/npwalker/abrader-gms | gitlab_disable_ssl_verify_s... | 👍
rbac | https://github.com/puppetlabs/pltraining-rbac | 2f60e1789a721ce83f8df061e13... | 👍
acl | https://github.com/dobbymoodge/puppet-acl.git | master | 👍
deploy | https://github.com/cudgel/deploy.git | master | 👍
dotfiles | https://github.com/cudgel/puppet-dotfiles.git | master | 👍
gitlab | https://github.com/vshn/puppet-gitlab | 00397b86dfb3487d9df768cbd36... | 👍

👍👍 Puppetfile looks good.👍👍
```

#### Limitations

* It works only with modules from the [Forge](https://forge.puppetlabs.com), and Git.
Expand Down
8 changes: 7 additions & 1 deletion Rakefile
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,14 @@ require 'rake/clean'
require 'rubygems'
require 'bundler/gem_tasks'
require 'fileutils'
require 'rspec/core'
require 'rspec/core/rake_task'

CLEAN.include("pkg/", "tmp/")
CLOBBER.include("Gemfile.lock")

task :default => [:clean, :build]
task :default => [:spec]

RSpec::Core::RakeTask.new(:spec) do |spec|
spec.pattern = FileList['spec/**/*_spec.rb']
end
5 changes: 4 additions & 1 deletion lib/ra10ke.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,17 @@
require 'ra10ke/syntax'
require 'ra10ke/dependencies'
require 'ra10ke/install'
require 'ra10ke/validate'
require 'git'
require 'semverse'

require 'r10k/puppetfile'
module Ra10ke
class RakeTask < ::Rake::TaskLib
include Ra10ke::Solve
include Ra10ke::Syntax
include Ra10ke::Dependencies
include Ra10ke::Install
include Ra10ke::Validate

attr_accessor :basedir, :moduledir, :puppetfile_path, :puppetfile_name, :force, :purge

Expand All @@ -32,6 +34,7 @@ def initialize(*args)
define_task_syntax(*args)
define_task_dependencies(*args)
define_task_install(*args)
define_task_validate(*args)
end
end

Expand Down
1 change: 0 additions & 1 deletion lib/ra10ke/dependencies.rb
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,3 @@ def define_task_dependencies(*_args)

end
end

28 changes: 28 additions & 0 deletions lib/ra10ke/monkey_patches.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# frozen_string_literal: true

class String
# colorization
def colorize(color_code)
"\e[#{color_code}m#{self}\e[0m"
end

def red
colorize(31)
end

def green
colorize(32)
end

def yellow
colorize(33)
end

# removes specified markes from string.
# @return [String] - the string with markers removed
def strip_comment(markers = ['#', "\n"])
re = Regexp.union(markers)
index = (self =~ re)
index.nil? ? rstrip : self[0, index].rstrip
end
end
83 changes: 83 additions & 0 deletions lib/ra10ke/puppetfile_parser.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# it might be desirable to parse the Puppetfile as a string instead of evaling it.
# this module allows you to do just that.
require 'ra10ke/monkey_patches'

module Ra10ke
module PuppetfileParser

# @return [Array] - returns a array of hashes that contain modules with a git source
def git_modules(file = puppetfile)
modules(file).find_all do |mod|
mod[:args].key?(:git)
end
end

# @param puppetfile [String] - the absolute path to the puppetfile
# @return [Array] - returns an array of module hashes that represent the puppetfile
# @example
# [{:namespace=>"puppetlabs", :name=>"stdlib", :args=>[]},
# {:namespace=>"petems", :name=>"swap_file", :args=>["'4.0.0'"]}]
def modules(puppetfile)
@modules ||= begin
return [] unless File.exist?(puppetfile)

all_lines = File.read(puppetfile).lines.map(&:strip_comment)
# remove comments from all the lines
lines_without_comments = all_lines.reject { |line| line.match(/#.*\n/) }.join("\n").delete("\n")
lines_without_comments.split('mod').map do |line|
next nil if line =~ /^forge/
next nil if line.empty?

parse_module_args(line)
end.compact.uniq
end
end

# @param data [String] - the string to parse the puppetfile args out of
# @return [Array] - an array of arguments in hash form
# @example
# {:namespace=>"puppetlabs", :name=>"stdlib", :args=>[]}
# {:namespace=>"petems", :name=>"swap_file", :args=>["'4.0.0'"]}
def parse_module_args(data)
return {} if data.empty?
args = data.split(',').map(&:strip)
# we can't guarantee that there will be a namespace when git is used
# remove quotes and dash and slash
namespace, name = args.shift.gsub(/'|"/, '').split(%r{-|/})
name ||= namespace
namespace = nil if namespace == name
{
namespace: namespace,
name: name,
args: process_args(args)
}
end

# @return [Array] - returns an array of hashes with the args in key value pairs
# @param [Array] - the arguments processed from each entry in the puppetfile
# @example
# [{:args=>[], :name=>"razor", :namespace=>"puppetlabs"},
# {:args=>[{:version=>"0.0.3"}], :name=>"ntp", :namespace=>"puppetlabs"},
# {:args=>[], :name=>"inifile", :namespace=>"puppetlabs"},
# {:args=>
# [{:git=>"https://github.com/nwops/reportslack.git"}, {:ref=>"1.0.20"}],
# :name=>"reportslack",
# :namespace=>"nwops"},
# {:args=>{:git=>"git://github.com/puppetlabs/puppetlabs-apt.git"},
# :name=>"apt",
# :namespace=>nil}
# ]
def process_args(args)
results = {}
args.each do |arg|
a = arg.gsub(/'|"/, '').split(/\A\:|\:\s|\=\>/).map(&:strip).reject(&:empty?)
if a.count < 2
results[:version] = a.first
else
results[a.first.to_sym] = a.last
end
end
results
end
end
end
126 changes: 126 additions & 0 deletions lib/ra10ke/validate.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# frozen_string_literal: true

require 'ra10ke/monkey_patches'
require 'tempfile'
require 'table_print'
require 'ra10ke/puppetfile_parser'
require 'English'

module Ra10ke
module Validate

GOOD_EMOJI = ENV['GOOD_EMOJI'] || '👍'
logicminds marked this conversation as resolved.
Show resolved Hide resolved
BAD_EMOJI = ENV['BAD_EMOJI'] || '😨'

# Validate the git urls and refs
def define_task_validate(*args)
desc 'Validate the git urls and branches, refs, or tags'
task :validate do
gitvalididation = Ra10ke::Validate::Validation.new(get_puppetfile.puppetfile_path)
exit_code = 0
if gitvalididation.bad_mods?
exit_code = 1
message = BAD_EMOJI + ' Not all modules in the Puppetfile are valid. '.red + BAD_EMOJI
else
message = GOOD_EMOJI + ' Puppetfile looks good. '.green + GOOD_EMOJI
end
tp gitvalididation.sorted_mods, :name, { url: { width: 50 } }, :ref, :status
abort(message) if exit_code.positive?
puts message
end
end

class Validation
include Ra10ke::PuppetfileParser

attr_reader :puppetfile

def initialize(file)
file ||= './Puppetfile'
@puppetfile = File.expand_path(file)
abort("Puppetfile does not exist at #{puppetfile}") unless File.readable?(puppetfile)
end

# @return [Boolean] - return true if the ref is valid
# @param url [String] - the git string either https or ssh url
# @param ref [String] - the ref object, branch name, tag name, or commit sha, defaults to HEAD
def valid_ref?(url, ref = 'HEAD')
raise ArgumentError unless ref
found = all_refs(url).find do |sha, data |
# we don't need to bother with these types
next if data[:type] == :pull || data[:type] == :merge_request
# is the name equal to the tag or branch? Is the commit sha equal?
data[:name].eql?(ref) || sha.slice(0,8).eql?(ref.slice(0,8))
end
!found.nil?
end

# @return [Hash] - a hash of all the refs associated with the remote repository
# @param url [String] - the git string either https or ssh url
# @example
# {"0ec707e431367bbe2752966be8ab915b6f0da754"=>{:ref=>"refs/heads/74110ac", :type=>:branch, :subtype=>nil, :name=>"74110ac"},
# "07bb5d2d94db222dca5860eb29c184e8970f36f4"=>{:ref=>"refs/pull/74/head", :type=>:pull, :subtype=>:head, :name=>"74"},
# "156ca9a8ea69e056e86355b27d944e59d1b3a1e1"=>{:ref=>"refs/heads/master", :type=>:branch, :subtype=>nil, :name=>"master"},
# "fcc0532bbc5a5b65f3941738339e9cc7e3d767ce"=>{:ref=>"refs/pull/249/head", :type=>:pull, :subtype=>:head, :name=>"249"},
# "8d54891fa5df75890ee15d53080c2a81b4960f92"=>{:ref=>"refs/pull/267/head", :type=>:pull, :subtype=>:head, :name=>"267"} }
def all_refs(url)
logicminds marked this conversation as resolved.
Show resolved Hide resolved
data = `git ls-remote --symref #{url}`
raise "Error downloading #{url}" unless $CHILD_STATUS.success?
data.lines.reduce({}) do |refs, line|
sha, ref = line.split("\t")
next refs if sha.eql?('ref: refs/heads/master')
_, type, name, subtype = ref.chomp.split('/')
next refs unless name
type = :tag if type.eql?('tags')
type = type.to_sym
subtype = subtype.to_sym if subtype
type = :branch if type.eql?(:heads)
refs[sha] = {ref: ref.chomp, type: type, subtype: subtype, name: name }
refs
end
end

# @return [Boolean] - return true if the commit sha is valid
# @param url [String] - the git string either https or ssh url
# @param ref [String] - the sha id
def valid_commit?(url, sha)
return false if sha.nil? || sha.empty?
return true if valid_ref?(url, sha)
Dir.mktmpdir do |dir|
`git clone --no-tags #{url} #{dir} 2>&1 > /dev/null`
Dir.chdir(dir) do
`git show #{sha} 2>&1 > /dev/null`
$CHILD_STATUS.success?
end
end
end

# @return [Array[Hash]] array of module information and git status
def all_modules
begin
git_modules(puppetfile).map do |mod|
ref = mod[:args][:ref] || mod[:args][:tag] || mod[:args][:branch]
valid_ref = valid_ref?(mod[:args][:git], ref) || valid_commit?(mod[:args][:git], mod[:args][:ref])
{
name: mod[:name],
url: mod[:args][:git],
ref: ref,
valid_ref?: valid_ref,
status: valid_ref ? Ra10ke::Validate::GOOD_EMOJI : Ra10ke::Validate::BAD_EMOJI
}
end
end
end

# @return [Boolean] - true if there are any bad mods
def bad_mods?
all_modules.find_all { |mod| !mod[:valid_ref?] }.count > 0
end

# @return [Hash] - sorts the mods based on good/bad
def sorted_mods
all_modules.sort_by { |a| a[:valid_ref?] ? 1 : 0 }
end
end
end
end
2 changes: 2 additions & 0 deletions ra10ke.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,6 @@ Gem::Specification.new do |spec|
spec.add_dependency "git"
spec.add_dependency "solve"
spec.add_dependency 'semverse', '~> 2.0'
spec.add_dependency 'table_print', '~> 1.5.6'
spec.add_development_dependency 'rspec', '~> 3.6'
end
43 changes: 43 additions & 0 deletions spec/fixtures/Puppetfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# frozen_string_literal: true

forge 'http://forge.puppetlabs.com'

mod 'puppetlabs/inifile', '2.2.0'
mod 'puppetlabs/stdlib', '4.24.0'
mod 'puppetlabs/concat', '4.0.0'
mod 'puppetlabs/ntp', '6.4.1'

# introduced for tomcat module collaboration with uws
mod 'puppet-archive', '3.1.1'

mod 'gitlab',
git: 'https://github.com/vshn/puppet-gitlab',
ref: '00397b86dfb3487d9df768cbd3698d362132b5bf' # master

mod 'r10k',
git: 'https://github.com/acidprime/r10k',
tag: 'v3.1.1'

mod 'gms',
git: 'https://github.com/npwalker/abrader-gms',
branch: 'gitlab_disable_ssl_verify_support'

mod 'pltraining-rbac',
git: 'https://github.com/puppetlabs/pltraining-rbac',
ref: '2f60e1789a721ce83f8df061e13f8bf81cd4e4ce'

mod 'puppet-acl',
git: 'https://github.com/dobbymoodge/puppet-acl.git',
branch: 'master'

mod 'deploy',
git: 'https://github.com/cudgel/deploy.git',
branch: 'master'

mod 'dotfiles',
git: 'https://github.com/cudgel/puppet-dotfiles.git',
branch: 'master'

mod 'splunk',
git: 'https://github.com/cudgel/splunk.git',
branch: 'prod'
Loading