Unable to cf push large jar file with mcafee antiviurs software

[randomtask1155]

when pushing a jar file that has 50k small files cf cli has to extract the jar file, perform resource matching, and zip up for upload.

After running perf when the cf push was generating the resource_match request we observed that cf was only consuming about 3 % cpu and the kernel was consuming about 25%. Further investigation found that the kernel usage was a result of antivirus software that is scanning on reads and writes which means the 50k files get scanned during extraction and during chksum. This is the cause for the delay.

this is impacting a CI pipeline because it takes more than 1 hour to push a single app like this witch does not allow us to perform constant builds

some workarounds we considered:

• Disabling the antivirus allows the app to be pushed in just couple minutes.
  • this is not a valid workaround as per our security policy
• Excluding the $TMPDIR form the antivirus scan could help
  • This will be difficult to implement because we have to modify a lot of existing CI pipeline to use these methods and a lot of jobs are run as all different users.
• Deploy a local maven repository to store all the application dependencies and exclude dependencies from the jar file
  • This is possible however that would mean changing of internal development processes and updating existing applications. This would take a lot of time to implement

Would it be possible to have an option to skip resource matching and unpacking of the jar file?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/134717897

The labels on this github issue will be updated when the story is started.

[dkoper]

Hi @randomtask1155

Thanks for those details.
We are about to refactor our push command, and explore what it's doing exactly and why.
As part of that we can look at addressing this issue.

Cheers,
Dies Koper
CF CLI PM

[mgadiya]

We have a similar issue with our node app which has lot of dependencies ( 17K files together). On some environments, CF push fails at the resource matching step. So having a CLI flag to disable resource matching would be helpful to ensure our node app can be successfully pushed to all CF environments.

[dkoper]

@mgadiya what do you mean it's failing at the resource matching step? If resource matching is failing, we should fix it- not introduce a flag to skip it.
Maybe you can open a new issue about that with the CF trace of the failing response and matching request?

[mgadiya]

@dkoper apologies for the confusion. I meant to say that cf push timed out when it called CC's /resource_match endpoint because it is trying to do the resource match for all the files of our app. We discovered it by looking at CF Trace output.

[dkoper]

@mgadiya Please submit a new issue and include that error response (and request, although you can omit the lengthy body). The solution we're exploring for the issue in the issue description will not resolve the issue you raised. Many thanks.

[mgadiya]

@dkoper Thanks for your response. I created a new issue here - #1042

[shouah]

Is there an update on this specific issue? The last update on this issue was 11/23/2016 by Dies. It looks like issue #1042 is fixed in the current CLI version 6.23.1, according to that one's latest update but am wondering about this one specifically and whether the previously mentioned 'push command refactor' effort addressed it or not (since the case is still in 'open' status).

[dkoper]

@shouah We have started the refactor of push by refactoring the cf app and cf start commands that push orchestrates. push is our biggest command so will take many weeks to complete.

At the same time, we've done some explorations to reduce file I/O when pushing an app, which I believe will resolve this issue. Those will be implemented and evaluated during the refactor of the app upload part of push.
I will update this issue as soon as we have something, even experimental, for you to try and see if it helps with your app in your environment.

[dkoper]

@shouah @mgadiya @randomtask1155
We are refactoring the push command to greatly reduce file I/O when resource matching and uploading the app. Although not ready to replace the current push implementation yet, maybe you can give it a go and report here whether it improves your push performance and reduces timeouts?

Use a recent release (e.g. cf CLI 6.30.0) - it has an experimental v2-push command that is not included in cf help output.
This command does not yet support manifest inheritance, promoted ("global") variables and route declarations in the app manifest. Other than that it should be equivalent in usage compared to current push.

Cheers,
Dies Koper
CF CLI PM

[shouah]

Here is the java executable file stats:

• 56,786,944 bytes with 2,217 directories and 40,580 files
• issue is around the 'Binding...' step and 'Uploading...' step taking huge amount of time.

Using 64-bits CLI 6.30.0 on Windows 7 Enterprise 64-bits to push the jar file with --no-start option:

1. cf push with no AV exclusion = ~21 min
2. cf v2-push with no AV exclusion = ~4 min
3. cf push with AV exclusion filter enabled = ~9 min
4. cf v2-push with AV exclusion filter enabled = ~5 min

It looks like nearly a 700% improvement with standard AV setup between push and v2-push.

There doesn't seem to be a major difference in v2-push without AV exclusion or with AV exclusion because it seems that the majority of the time is taken by the network upload operation itself so subject to traffic load. However it's worth noticing that v2-push doesn't seem to be sensitive to AV setup so it performs about the same with or with AV exclusion filter.

[dkoper]

Thanks @shouah - great feedback!

[dkoper]

This is now incorporated in push in cf CLI 6.34.0.

Regards,
Dies Koper
CF CLI PM