Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix order of file copies in entrypoint.sh #935

Merged
merged 8 commits into from
May 2, 2020

Conversation

dthorsen
Copy link
Contributor

@dthorsen dthorsen commented Apr 30, 2020

Fixes this error when you upgrade the CNI because the CNI starts prior to portmap
is copied and portmap is then sometimes in use by the time the cp commands execute

Issue #, if available:

Description of changes:
This fixes the following error from causing the CNI pod to enter crashloopbackoff:
cp: cannot create regular file '/host/opt/cni/bin/portmap': Text file busy

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Fixes this error when you upgrade the CNI because the CNI starts prior to portmap
is copied and portmap is then sometimes in use by the time the cp commands execute
@mogren
Copy link
Contributor

mogren commented Apr 30, 2020

Hi @dthorsen, good catch finding that cp error message! In this case, we would like to only copy the portmap and not the CNI binary and config before ipamd starts, since that might make a new node ready even if it cannot start. Could you update the PR to only copy the portmap early?

That said, I'm working on a PR to add an init container instead, to make the setup part more explicit. (Issue #158)

@mogren
Copy link
Contributor

mogren commented Apr 30, 2020

Taking a second look at this, I wonder if we should check if the file exists, if so rename it to portmap.old, copy in the (potentially new) portmap then delete portmap.old. That seems like a safer way to do this update in case some process is currently executing. Thanks again for noticing this, it has been hard to track down.

@dthorsen
Copy link
Contributor Author

Thanks @mogren! That makes sense I think. I moved only the portmap binary to copy early.

@dthorsen
Copy link
Contributor Author

Oh that might make sense. We can totally do that.

@dthorsen
Copy link
Contributor Author

I think this does what you are asking. The timing difference here is probably milliseconds though. I am not certain it buys much over simply copying over the file. I suspect that if something was actually executing portmap it would be cached in RAM anyway and I don't think those few extra milliseconds will protect from much.

scripts/entrypoint.sh Outdated Show resolved Hide resolved
Copy link
Contributor

@mogren mogren left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dthorsen, looks good to me. I wonder if we might have to do the same thing for the aws-cni as well, but that would be a separate PR. I'll let someone else take a quick look at this as well before merging. Thanks a lot for finding and fixing this issue!

Comment on lines 63 to 71
HOST_PORTMAP="$HOST_CNI_BIN_PATH/portmap"
if [[ -f "$HOST_PORTMAP" ]]
then
mv "$HOST_PORTMAP" "$HOST_PORTMAP.old"
cp portmap "$HOST_CNI_BIN_PATH"
rm "$HOST_PORTMAP.old"
else
cp portmap "$HOST_CNI_BIN_PATH"
fi
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having taken another look, I think this could be simplified even more. What do you think about:

HOST_PORTMAP="$HOST_CNI_BIN_PATH/portmap"
if [[ -f "$HOST_PORTMAP" ]]; then
    rm "$HOST_PORTMAP"
fi
cp portmap "$HOST_CNI_BIN_PATH"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed @mogren. ++

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplification engaged

Copy link
Contributor

@jaypipes jaypipes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++

@mogren mogren merged commit 2487803 into aws:master May 2, 2020
mogren pushed a commit to mogren/amazon-vpc-cni-k8s that referenced this pull request May 19, 2020
* copy only portmap early before cni starts in entrypoint.sh

Fixes this error when you upgrade the CNI because the CNI starts prior to portmap
is copied, and portmap is then sometimes in use by the time the cp commands execute

Co-authored-by: Dane Thorsen <dthorsen@newrelic.com>
Co-authored-by: Claes Mogren <mogren@amazon.com>
(cherry picked from commit 2487803)
mogren pushed a commit that referenced this pull request May 19, 2020
* copy only portmap early before cni starts in entrypoint.sh

Fixes this error when you upgrade the CNI because the CNI starts prior to portmap
is copied, and portmap is then sometimes in use by the time the cp commands execute

Co-authored-by: Dane Thorsen <dthorsen@newrelic.com>
Co-authored-by: Claes Mogren <mogren@amazon.com>
(cherry picked from commit 2487803)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants