Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python bandit checks. #316

Merged
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
f9de057
add linters
dvkruchinin Jun 29, 2021
efe68bf
Updated linters check
dvkruchinin Jun 29, 2021
5b0af21
Fix linters errors
dvkruchinin Jun 29, 2021
8232c9a
Fix remark checks
dvkruchinin Jun 29, 2021
27ff2ac
Remark linter fix errors
dvkruchinin Jun 29, 2021
b1c4abd
Update remark checks
dvkruchinin Jun 29, 2021
53d46c1
Simple update
dvkruchinin Jun 29, 2021
5fd0c38
Merge branch 'develop' of https://github.com/openvinotoolkit/datumaro…
dvkruchinin Jun 29, 2021
1e1c408
Apply comments
dvkruchinin Jun 29, 2021
4fe6b78
Exclude "docs", "ci" folders from setup.py
dvkruchinin Jun 29, 2021
a92c4cb
Linters output to console
dvkruchinin Jun 29, 2021
8153455
Remark report to console
dvkruchinin Jun 29, 2021
c6fac80
added --frail to remark-cli for exit with 1 on warnings
dvkruchinin Jun 29, 2021
1fcc16d
Remove json_to_html.py
dvkruchinin Jun 29, 2021
3185f09
check removed "pip install --user -U pip wheel setuptools"
dvkruchinin Jun 29, 2021
d4990b3
Merge branch 'develop' of https://github.com/openvinotoolkit/datumaro…
dvkruchinin Jul 1, 2021
0c6e016
Removed pylint checking. Due to it in another PR
dvkruchinin Jul 1, 2021
2ae2805
Revert setup.py for develop
dvkruchinin Jul 1, 2021
682cb20
Reqork pip install.
dvkruchinin Jul 1, 2021
163b507
disable some bandit warnings
Jul 2, 2021
930c193
Merge branch 'develop' of https://github.com/openvinotoolkit/datumaro…
dvkruchinin Jul 2, 2021
da22177
Merge remote-tracking branch 'origin/dkru/add-linters-check' into dkr…
dvkruchinin Jul 2, 2021
37b98c3
Update datumaro/components/operations.py
Jul 5, 2021
7d4abc6
Merge branch 'develop' into dkru/add-linters-check
Jul 5, 2021
2b0f147
Update datumaro/util/tf_util.py
Jul 5, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions .github/workflows/bandit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Linter
on: pull_request
jobs:
Bandit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Run checks
run: |
URL="https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files"
PR_FILES=$(curl -s -X GET -G $URL | jq -r '.[] | select(.status != "removed") | .filename')
for files in $PR_FILES; do
extension="${files##*.}"
if [[ $extension == 'py' ]]; then
changed_files_bandit+=" ${files}"
fi
done

if [[ ! -z ${changed_files_bandit} ]]; then
sudo apt-get --no-install-recommends install -y build-essential curl python3-dev python3-pip python3-venv
python3 -m venv .env
. .env/bin/activate
pip install -U pip wheel setuptools
pip install $(egrep "bandit.*" ./requirements.txt)
mkdir -p bandit_report

echo "Bandit version: "`bandit --version | head -1`
echo "The files will be checked: "`echo ${changed_files_bandit}`
bandit ${changed_files_bandit} --exclude '**/tests/**' -a file --ini ./.bandit -f html -o ./bandit_report/bandit_checks.html
deactivate
else
echo "No files with the \"py\" extension found"
fi

- name: Upload artifacts
if: failure()
uses: actions/upload-artifact@v2
with:
name: bandit_report
path: bandit_report
44 changes: 44 additions & 0 deletions .github/workflows/pylint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: Linter
IRDonch marked this conversation as resolved.
Show resolved Hide resolved
on: pull_request
jobs:
PyLint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Run checks
run: |
URL="https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files"
PR_FILES=$(curl -s -X GET -G $URL | jq -r '.[] | select(.status != "removed") | .filename')
for files in $PR_FILES; do
extension="${files##*.}"
if [[ $extension == 'py' ]]; then
changed_files_pylint+=" ${files}"
fi
done

if [[ ! -z ${changed_files_pylint} ]]; then
sudo apt-get --no-install-recommends install -y build-essential curl python3-dev python3-pip python3-venv
zhiltsov-max marked this conversation as resolved.
Show resolved Hide resolved
python3 -m venv .env
. .env/bin/activate
pip install -U pip wheel setuptools
pip install pylint-json2html
pip install $(egrep "pylint.*" ./requirements.txt)
mkdir -p pylint_report

echo "Pylint version: "`pylint --version | head -1`
echo "The files will be checked: "`echo ${changed_files_pylint}`
pylint ${changed_files_pylint} --output-format=json > ./pylint_report/pylint_checks.json || exit_code=`echo $?` || true
pylint-json2html -o ./pylint_report/pylint_checks.html ./pylint_report/pylint_checks.json
zhiltsov-max marked this conversation as resolved.
Show resolved Hide resolved
deactivate
exit ${exit_code}
else
echo "No files with the \"py\" extension found"
fi

- name: Upload artifacts
if: failure()
uses: actions/upload-artifact@v2
with:
name: pylint_report
path: pylint_report
31 changes: 31 additions & 0 deletions .github/workflows/remark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Linter
on: pull_request
jobs:
Remark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
with:
node-version: 12

- name: Run checks
run: |
npm ci
mkdir -p remark_report

echo "Remark version: "`npx remark --version`
npx remark --quiet --report json --no-stdout . 2> ./remark_report/remark_report.json
get_report=`cat ./remark_report/remark_report.json | jq -r '.[] | select(.messages | length > 0)'`
if [[ ! -z ${get_report} ]]; then
pip install json2html
python ./json_to_html.py ./remark_report/remark_report.json
exit 1
fi

- name: Upload artifacts
if: failure()
uses: actions/upload-artifact@v2
with:
name: remark_report
path: remark_report
18 changes: 18 additions & 0 deletions .remarkrc.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
exports.settings = { bullet: '*', paddedTable: false };

exports.plugins = [
'remark-frontmatter',
'remark-gfm',
'remark-preset-lint-recommended',
'remark-preset-lint-consistent',
['remark-lint-list-item-indent', 'space'],
['remark-lint-no-dead-urls', false], // Does not work because of github protection system
['remark-lint-maximum-line-length', 120],
zhiltsov-max marked this conversation as resolved.
Show resolved Hide resolved
['remark-lint-maximum-heading-length', 120],
['remark-lint-strong-marker', '*'],
['remark-lint-emphasis-marker', '_'],
['remark-lint-unordered-list-marker-style', '-'],
['remark-lint-ordered-list-marker-style', '.'],
['remark-lint-no-file-name-irregular-characters', false],
['remark-lint-list-item-spacing', false],
];
22 changes: 22 additions & 0 deletions json_to_html.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env python
zhiltsov-max marked this conversation as resolved.
Show resolved Hide resolved

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

import sys
import os
import json
from json2html import *

def json_to_html(path_to_json):
with open(path_to_json) as json_file:
data = json.load(json_file)
html_report = json2html.convert(json = data)

with open(os.path.splitext(path_to_json)[0] + '.html', 'w') as html_file:
html_file.write(html_report)


if __name__ == '__main__':
json_to_html(sys.argv[1])
Loading