Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[build] Add safety report to docker image #1186

Merged
merged 96 commits into from
Sep 26, 2021
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
ab8487a
inital commit
tejaschumbalkar Jun 23, 2021
d385f59
Add safety report to docker image
tejaschumbalkar Jun 25, 2021
e38004f
adapt build logic
tejaschumbalkar Jun 25, 2021
e286853
fix safety dir
tejaschumbalkar Jun 25, 2021
f3df287
address file strcuture
tejaschumbalkar Jun 25, 2021
8400170
add copy cmd
tejaschumbalkar Jun 25, 2021
821a4e2
add bash option
tejaschumbalkar Jun 25, 2021
f610d45
fix file name
tejaschumbalkar Jun 25, 2021
af82f00
add quotes
tejaschumbalkar Jun 25, 2021
ea87368
install jq
tejaschumbalkar Jun 28, 2021
e20b379
remove safety pacakge
tejaschumbalkar Jun 28, 2021
67b969a
modify buildspec
tejaschumbalkar Jun 28, 2021
6dd087c
Merge branch 'master' into safety_report
tejaschumbalkar Jun 28, 2021
7143bd8
add safety for TF2.5
tejaschumbalkar Jun 29, 2021
2f23d6d
support safety for TF2.5
tejaschumbalkar Jun 29, 2021
f1f36c6
adapt the new safety workflow on the sanity test
tejaschumbalkar Jun 29, 2021
3ca89e5
run sanity test
tejaschumbalkar Jun 29, 2021
7603d30
run safety on PR
tejaschumbalkar Jun 29, 2021
d887fb1
revert unrelated change
tejaschumbalkar Jun 29, 2021
8d6982e
revert temp changes
tejaschumbalkar Jun 29, 2021
cb6c347
multi stage docker build
tejaschumbalkar Jul 27, 2021
36fb18f
Merge branch 'master' into safety_report
tejaschumbalkar Jul 27, 2021
61641ec
Fixed a few issues. Edited Dockerfiles to allow running the script fa…
shantanutrip Aug 12, 2021
c5a20a2
Made Context to properly run conclude stage Dockerfile
shantanutrip Aug 12, 2021
a5d4fb7
Renaming to Conclusion_Stage
shantanutrip Aug 13, 2021
b091ea0
Created Safety Scan Reports
shantanutrip Aug 16, 2021
9674ebc
Added the generate_safety_report_for_image function in utils.py
shantanutrip Aug 16, 2021
2c43fb1
Added the report generation functionality in image class
shantanutrip Aug 17, 2021
a83067d
Added ConclusionStage class
shantanutrip Aug 17, 2021
bbf0060
Added logic to first genereate conclusion and then example. Also adde…
shantanutrip Aug 18, 2021
9c168d5
Added logic to build conclusion images at the very end
shantanutrip Aug 18, 2021
7eeac6f
Fixed the pre build step
shantanutrip Aug 18, 2021
e4ddf98
Refactored First Stage to Initial Stage
shantanutrip Aug 18, 2021
096ff19
Added the image push logic
shantanutrip Aug 18, 2021
c629e9b
Added the working for upload metrics
shantanutrip Aug 19, 2021
549d1da
Added the set_test_env portion
shantanutrip Aug 19, 2021
4313a84
Resolved the same name tarfile conflict
shantanutrip Aug 19, 2021
372bfae
Added the safety tests
shantanutrip Aug 20, 2021
cc79be7
Added reason to ignore field in safety_check_v2
shantanutrip Aug 20, 2021
36d822e
Added the new safety_check_v2 script
shantanutrip Aug 20, 2021
4c6d379
Modified the tests to run on new safety report
shantanutrip Aug 20, 2021
89666a6
Added the ignore dict logic with the data
shantanutrip Aug 21, 2021
800a5b8
Refactored PYTHONPATH to ROOT_FOLDER_PATH
shantanutrip Aug 21, 2021
a8cea00
Refactored comments in safety_check_v2.py
shantanutrip Aug 21, 2021
db5f9a4
Deleted src/test_and_generate_python_safety_report.py
shantanutrip Aug 21, 2021
50adb38
Resolved mxnet Dockerfile cpu conflict
shantanutrip Aug 21, 2021
ef62fa9
Resolved mxnet buildpsec
shantanutrip Aug 21, 2021
41edfbd
Reverted test_safety_check.py to master
shantanutrip Aug 21, 2021
8d2c4e3
Deleted safety_report.py
shantanutrip Aug 21, 2021
94c0733
Added container removal logic
shantanutrip Aug 23, 2021
dd5c6bd
Added safety report generator class
shantanutrip Aug 25, 2021
5dce6c7
Changes the utils generate_safety_report_for_image function
shantanutrip Aug 25, 2021
ed24724
Added Formatter messages
shantanutrip Aug 25, 2021
8181a38
Add null entrypoint while running docker images
shantanutrip Aug 25, 2021
71dce0e
Added extended timeouts
shantanutrip Aug 25, 2021
a1fb7e4
Added debug print lines
shantanutrip Aug 25, 2021
73bd4a6
Pushing images sequentially
shantanutrip Aug 25, 2021
1f48f08
Added client timeout in constants with parallelization
shantanutrip Aug 26, 2021
49a6e2a
Changed the test. Removed the safety_check_v2.py
shantanutrip Aug 26, 2021
6286f8e
Reverted print and debug statements
shantanutrip Aug 26, 2021
9cdfaaa
Restricted worker count for pushing images
shantanutrip Aug 26, 2021
d576ae9
Reverted modified Dockerfile from pull request
shantanutrip Aug 26, 2021
3923a68
removed Safety Data Json
shantanutrip Aug 26, 2021
63d31d1
Merge remote-tracking branch 'alt/master'(AWS-DLC repo) into safety-r…
shantanutrip Aug 26, 2021
301c663
Made changes to build images in non Codebuild Context
shantanutrip Aug 30, 2021
5ec5776
Changed the test files to run on desired contexts
shantanutrip Aug 30, 2021
ddd25aa
Changing dummy client
shantanutrip Sep 2, 2021
d9db790
Handling context deletion
shantanutrip Sep 2, 2021
9a4af96
Changed the context remove
shantanutrip Sep 2, 2021
a5ca249
Refactoring image, conclusion, misc_docker
shantanutrip Sep 2, 2021
a15e23c
Refactoring Conclusion to Common and Initial to Pre_push
shantanutrip Sep 2, 2021
c71628c
Added the get_root_folder_path method
shantanutrip Sep 3, 2021
8146137
Changed safety report generator
shantanutrip Sep 3, 2021
f093cde
Changing the key in test file
shantanutrip Sep 3, 2021
2bb790a
Added new logging logic
shantanutrip Sep 3, 2021
3eb309f
Removed extra vars, Added docstring
shantanutrip Sep 3, 2021
e3fccd3
Changed test file
shantanutrip Sep 4, 2021
f62553e
Renamed test file. Added buildspec
shantanutrip Sep 5, 2021
699caca
Added logging method to image
shantanutrip Sep 5, 2021
3d0cde0
Ran black -l 120 (https://github.com/psf/black) for following files:
shantanutrip Sep 5, 2021
9b6ea43
Merge remote-tracking branch 'alt/master'(AWS-DLC repo) into safety-r…
shantanutrip Sep 6, 2021
664e946
Minor changed
shantanutrip Sep 7, 2021
0a0aeeb
Add multistage.common tag
shantanutrip Sep 8, 2021
d9d09b8
Merging from alt master after DIY changes
shantanutrip Sep 21, 2021
4bf75d9
Added the process_image function to build and push images. Changed ta…
shantanutrip Sep 21, 2021
742d1c9
Made the get_safety_ignore_dict method bit generic
shantanutrip Sep 22, 2021
2b3d735
Merge remote-tracking branch 'alt/master' into safety-report-tejas
shantanutrip Sep 23, 2021
dd1c08c
Removed the if condition for context
shantanutrip Sep 23, 2021
cae71c0
Addressed the nit changes
shantanutrip Sep 24, 2021
76335c1
Merge remote-tracking branch 'alt/master' into safety-report-tejas
shantanutrip Sep 24, 2021
0f7416c
Ignoring TF2.6 vulnerability
shantanutrip Sep 24, 2021
feed6bd
Added the multi-tagging mechanism
shantanutrip Sep 24, 2021
9c192c9
Configured test env file to dump no appended tags. Other loggin relat…
shantanutrip Sep 24, 2021
bde69df
Merge remote-tracking branch 'alt/master' into safety-report-tejas
shantanutrip Sep 25, 2021
0996ac0
Disabling Common Stage
shantanutrip Sep 25, 2021
e90439e
Fix
shantanutrip Sep 26, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/Dockerfile.multipart
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Use the Deep Learning Container as a base Image
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rename this file as Dockerfile.safetyreport or something, because multipart is much too generic, and could confuse readers about its purpose, especially with other PRs such as #1278.

Also, consider moving this dockerfile into another folder (that is not src/) which could contain many different kinds of multipart dockerfiles meant to add specific layers to images on which they are used. For example, we might want to include the dependency_check report, or the ECR Scan report, or pull the OSSCompliance layer out from all images into a dockerfile similar to this.

It doesn't make sense to have this in src/ because src/ contains "source code" used to build images, and adding more of these multipart dockerfiles could bloat the src/ folder.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current design decision is to use 1 single docker file to host stuff that are applicable to all containers. So something like Dockerfile.common will work.

Having multi-staged capability will be definitely better so we can have Dockerfile.oss_compliance, Dockerfile.dependency_check etc. And a DLC build process can pick and choose which ones to run. But we are not there yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This dockerfile is meant to be built as a final stage in the build process. Any functionality including ECR sans, OSS compliance script which is common to all the DLC should be placed in this final stage Dockerfile.

I agree that the file name can be renamed to Dockerfile.common or Dockerfile.finalstage

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to Docker.common

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I will change the name to Dockerfile.common and make a different folder called miscellaneous_dockerfiles to consist of all dockerfiles that will be used for embellishing base images.

ARG FIRST_STAGE_IMAGE=""

FROM $FIRST_STAGE_IMAGE

# Add any script or repo as required
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Comment: Copy safety report generated from INITIAL_STAGE_IMAGE to docker image?

COPY safety_report.json /var/safety_report.json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why /var/ instead of anywhere else?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the location for the file should be /opt/aws/dlc/info

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saimidu there was no specific reason for putting in var. I could follow @junpuf 's recommendation and store it in /opt/aws/dlc/info

4 changes: 4 additions & 0 deletions src/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,10 @@
# Left and right padding between text and margins in output
PADDING = 1

#Docker build stages
FIRST_STAGE="first"
SECOND_STAGE="second"

# Docker connections
DOCKER_URL = "unix://var/run/docker.sock"

Expand Down
120 changes: 76 additions & 44 deletions src/image.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ class DockerImage:
"""

def __init__(
self, info, dockerfile, repository, tag, to_build, context=None,
self, info, dockerfile, repository, tag, to_build, stage, context=None,
):

# Meta-data about the image should go to info.
Expand All @@ -36,6 +36,7 @@ def __init__(
self.summary = {}
self.build_args = {}
self.labels = {}
self.stage = stage

self.dockerfile = dockerfile
self.context = context
Expand Down Expand Up @@ -69,11 +70,7 @@ def collect_installed_packages_information(self):
docker_client.containers.prune()
return command_responses

def build(self):
"""
The build function builds the specified docker image
"""
self.summary["start_time"] = datetime.now()
def pre_build_configuration(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please change this function name to a verb, like "update_pre_build_configuration", because it does not have a return value, and it would be unclear exactly why this function is invoked in the code where it is used.

Also, please add a docstring to describe why this function exists, and what it does.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


if not self.to_build:
self.log = ["Not built"]
Expand All @@ -84,19 +81,42 @@ def build(self):
if self.info.get("base_image_uri"):
self.build_args["BASE_IMAGE"] = self.info["base_image_uri"]

if self.ecr_url:
self.build_args["FIRST_STAGE_IMAGE"] = self.ecr_url

if self.info.get("extra_build_args"):
self.build_args.update(self.info.get("extra_build_args"))

if self.info.get("labels"):
self.labels.update(self.info.get("labels"))

print(f"self.build_args {self.build_args}")
print(f"self.labels {self.labels}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these print statements permanent additions? If so, could you change this to be printed with better readability?

Suggested change
print(f"self.build_args {self.build_args}")
print(f"self.labels {self.labels}")
print(f"self.build_args = {json.dumps(self.build_args, indent=4)}")
print(f"self.labels = {json.dumps(self.labels, indent=4)}")

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will remove this.


with open(self.context.context_path, "rb") as context_file:
response = []

for line in self.client.build(
fileobj=context_file,
def build(self):
"""
The build function builds the specified docker image
"""
self.summary["start_time"] = datetime.now()
self.pre_build_configuration()
print(f"self.context {self.context}")
if self.context:
with open(self.context.context_path, "rb") as context_file:
print("within context")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

print statement can be removed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

self.docker_build(fileobj=context_file, custom_context=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the build function has been split into multiple functions with build as the parent function, there is no longer a single function that changes self.build_status. This means that both self.docker_build as well as self.image_size_check change self.build_status.

If self.docker_build sets build status to constants.FAIL, it is possible for self.image_size_check to throw a hard-to-debug exception, or worse, pass-through and pretend the build was successful if there already exists an image with the expected target image URI.

Please check the return value of self.docker_build (or just self.build_status) after this if-else block to return the build status immediately in case the status is not SUCCESS.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, will do.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

self.context.remove()
else:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a case where the self.docker_build() is invoked?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't seen any yet. It was already existing in the code, so I thought of not playing with it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Existing code as in on the PR branch or the master, if PR branch then lets just remove it, probably it might have been added for experimental purposes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I will remove it.

print("out of context")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

self.docker_build()
#check the size after image is built.
self.image_size_check()

def docker_build(self, fileobj=None, custom_context=False):
response = []
for line in self.client.build(
fileobj=fileobj,
path=self.dockerfile,
custom_context=True,
custom_context=custom_context,
rm=True,
decode=True,
tag=self.ecr_url,
Expand All @@ -121,39 +141,51 @@ def build(self):
else:
response.append(str(line))

self.context.remove()
self.log = response
print(f"self.log {self.log}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this, or insert improvements upon the way that it is printed. The logs printed in the build job take up too much space on the logs, and affect readability on the job.

An example of the logs is:

self.logs ['Step 1/37 : FROM ubuntu:18.04', '\n', 'Pulling from library/ubuntu', 'Digest: sha256:7bd7a9ca99f868bf69c4b6212f64f2af8e243f97ba13abb3e641e03a7ceb59e8', ... ]

for thousands of lines.

If we really want to see the build logs after each build (do we need this for every build, or for only those builds that fail?), perhaps they could be uploaded somewhere with better formatting.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have removed this log display line. Also, I have changed the way in which logs are now displayed and made it such that it doesn't take up a lot of space.

self.build_status = constants.SUCCESS
#TODO: return required?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: comment can be deleted

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

return self.build_status

self.summary["image_size"] = int(

def image_size_check(self):
response = []
self.summary["image_size"] = int(
self.client.inspect_image(self.ecr_url)["Size"]
) / (1024 * 1024)
if self.summary["image_size"] > self.info["image_size_baseline"] * 1.20:
response.append("Image size baseline exceeded")
response.append(f"{self.summary['image_size']} > 1.2 * {self.info['image_size_baseline']}")
response += self.collect_installed_packages_information()
self.build_status = constants.FAIL_IMAGE_SIZE_LIMIT
else:
self.build_status = constants.SUCCESS

for line in self.client.push(
self.repository, self.tag, stream=True, decode=True
):
if line.get("error") is not None:
response.append(line["error"])

self.log = response
self.build_status = constants.FAIL
self.summary["status"] = constants.STATUS_MESSAGE[self.build_status]
self.summary["end_time"] = datetime.now()

return self.build_status
if line.get("stream") is not None:
response.append(line["stream"])
else:
response.append(str(line))
if self.summary["image_size"] > self.info["image_size_baseline"] * 1.20:
response.append("Image size baseline exceeded")
response.append(f"{self.summary['image_size']} > 1.2 * {self.info['image_size_baseline']}")
response += self.collect_installed_packages_information()
self.build_status = constants.FAIL_IMAGE_SIZE_LIMIT
else:
self.build_status = constants.SUCCESS
self.log = response
print(f"self.log {self.log}")
#TODO: return required?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: comment can be deleted

return self.build_status

def push_image(self):

for line in self.client.push(self.repository, self.tag, stream=True, decode=True):
response = []
if line.get("error") is not None:
response.append(line["error"])

self.summary["status"] = constants.STATUS_MESSAGE[self.build_status]
self.summary["end_time"] = datetime.now()
self.summary["ecr_url"] = self.ecr_url
self.log = response
self.log = response
self.build_status = constants.FAIL
self.summary["status"] = constants.STATUS_MESSAGE[self.build_status]
self.summary["end_time"] = datetime.now()

return self.build_status
return self.build_status
if line.get("stream") is not None:
response.append(line["stream"])
else:
response.append(str(line))

self.summary["status"] = constants.STATUS_MESSAGE[self.build_status]
self.summary["end_time"] = datetime.now()
self.summary["ecr_url"] = self.ecr_url
self.log = response
#TODO: return required?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: comment can be deleted

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

return self.build_status
Loading