CF push fails with "Error processing app files: remove <TEMP_FOLDER>\unzipped-app160441603\BOOT-INF\lib: The directory is not empty"

[faisaltheparttimecoder]

Command

cf push <app-name> -b java_buildpack

What occurred

When pushing a java app from a windows machine once in a while we are treated with error after the upload of file is complete

FAILED 
Error processing app files: remove c:\User\<user-name>\AppData\Local\Temp\unzipped-app160441603\BOOT-INF\lib: The directory is not empty

This is similar to when someone has opened the folder and from other windows someone tries to run rmdir /s <folder>. The issue doesn't popup always but sometimes (i.e intermittent), as part of the troubleshooting I happen to check on the CF CLI code at "https://github.com/cloudfoundry/cli/blob/master/cf/actors/push.go". The function "ProcessPath" doesn't do anything fancy, it just extracts all the files (jar) to a temp location defined by %TEMP% or %TMP% and then uploads to the foundation after which it removes the temp file using "os.removeAll" function (which is where the error is thrown).

What you expected to occur

CF CLI should not fail since it can't clean up temp folder, as it's a minor issue to exit the push, it should ignore and continue with the remaining task.

CLI Version

CF CLI Version: cf version 6.29.2+c66d0f3.2017-08-25

CC API Endpoint Version

v2 (2.94.0)

CF Trace

Not needed in this case.

Platform & Shell Details

Windows 8.1 64-bit with PowerShell

Any other relevant information

Here are some troubleshooting steps that I have taken

1. My initial thought was that something ( like antivirus ) is opening the file which is causing the "os.removeAll" to fail, since the machine was running McAfee antivirus, I created a folder under user home, excluded the scan on McAfee virus exclusion folder tab to that folder, changed the %TEMP% & %TMP% and ran cf push, but it didn't work and failed at the same point in removing the temp directory via cf push.

2. Found some online document that there is a known issue with windows.

https://www.repairwin.com/fix-cannot-remove-folder-directory-is-not-empty-solved/
https://answers.microsoft.com/en-us/windows/forum/windows_7-windows_programs/error-0x80070091-the-directory-is-not-empty-while/8c58e1d9-b6f2-4164-8fa6-19f35c09ea3d?auth=1

So as a test I wrote this simple script and started to loop multiple time

package  main

import (
    "io/ioutil"
    "fmt"
    "os"
)

func main() {

    folderPath := "/lib/bin/text.file"
    tempDir, err := ioutil.TempDir("", "unzipped-app")
    if err != nil {
        panic(err)
    }

    os.MkdirAll(tempDir + folderPath, os.ModePerm)

    fmt.Println(tempDir)

    fmt.Println("Location 01: Just Remove")
    err = os.Remove(tempDir)
    if err != nil {
        fmt.Print(fmt.Sprintf("This will error out with the error: %s", err))
    }

    fmt.Println("\nLocation 02: Remove All")
    err = os.RemoveAll(tempDir)
    if err != nil {
        fmt.Print(fmt.Sprintf("This will not reach here unless any issues, if it does the error is : %s", err))
    }

}

but still, no luck in reproducing the issue, treating this is not a windows issue seems more to be a cf CLI problem.

3. Checked on the folder permission and also ran rmdir (as command line) as soon as the cf failed, but os never had any issue in cleaning up the folder, again pointing to its only the cf that has an issue.

4. I turned on resmon.exe and started to find for the handlers that are currently locking the folder, to surprise its only cf.exe that has the file opened and no third party software had any handler to it, so I believe during deleting files which multiple folders there seems to be a race condition with multiple handlers started during "os.removeAll" or during "cf push" not sure from where...

Attached is the screenshot that I see on the resmon.exe, this was collected as soon as the cf push failed and refresh of the resmon.exe, as you can see cf.exe even though it failed and exited it still had a handler to those folders, seconds later it cleaned up the process and the lock was released.

Suggestion fix

The easiest fix to this issue would be instead of returning error

	err = os.RemoveAll(tempDir)
	if err != nil {
		return err
	}

and error out the progress of push, it would be easier to just print a warning and continue as its anyways a temp file which would be cleaned up DiskCleanup later by the user, it should not be a cause of failure.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/150931625

The labels on this github issue will be updated when the story is started.

[dkoper]

Impressive how you tried to get to the bottom of the issue (and shared it with us). I am looking forward to more such bug reports!

We occasionally receive similar issues and are in the process of rewriting the push command to not extract files anymore. We do still need to create one temporary file so I'll check with the team if we display a warning instead of fail with an error if that fails.
We hope to include the new push in a release early next month.

[faisaltheparttimecoder]

Thanks @dkoper for the update. Moreover, i think this is more of the GoLang kernel issue with os.removeall package..

Attaching golang bug for reference: golang/go#20841

[ssc-hrep3]

Is there some quick workaround for now? I'm running into this issue all the time with the latest release of the cli on Windows.

[dkoper]

Use an older version, if you're seeing it only with the latest?
Otherwise, if you're OK with the limitations and (undocumented) caveats of using an experimental command, you can try v2-push. Note that this command is still a work in progress and does not have feature parity with push. (In particular, route related flags and app manifest attributes are mostly not implemented yet).

[sumitAtDigital]

Hi All,
From today I started facing below issue while using "cf push" command.

Done uploading
OK
FAILED
Error processing app files: remove C:\Users\xxxxxx\AppData\Local\Temp\unzipped-app54043\BOOT-INF\lib: The directory is not empty.

note: Tried with older versions of CLI too, but no success.

Please suggest if this can be resolved.

[ssc-hrep3]

push +1 the issue is still occurring and is still a huge problem for our setup (deploying works only every ~7th time on a Windows machine)

[dkoper]

This is now resolved in cf CLI 6.34.0.

Regards,
Dies Koper
CF CLI PM

[tanhung3101]

Hi,
I'm using CLI v6.35.2 and I have the same issue. Do we have any solution to work around?

Thanks,
Hung

[jbpivotal]

@tanhung3101 Sorry to hear that. Please open a new github issue providing all the details of your situation.

[trongvm2]

Now the issue is remain. please help to resolve ?