'cf push myapp -i n' exit with code 0 before all instances up and running (v6.38)

[shiyan0109]

Command

"cf push myapp -i 2"

What occurred

'Cf push ' command exit with 0 before both 2 of my requested instances started and running (only 1 has run)

“cf push mytestapp --no-manifest -p python.zip -i 2 -m 128M -k 256M -b python_buildpack
Pushing app mytestapp to org myorg / space dev as xxx...
Getting app info...
Creating app with these attributes...

• name: mytestapp
  path: /opt/python.zip
  buildpacks:
• python_buildpack
• disk quota: 256M
• instances: 2
• memory: 128M
  routes:
• mytestapp.xxxx

Creating app mytestapp...
Mapping routes...
Comparing local files to remote cache...
Packaging files to upload...
Uploading files...

0 B / 16.60 KiB 0.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00%
16.60 KiB / 16.60 KiB 100.00% 4s

Waiting for API to complete processing files...

Staging app and tracing logs...
Downloading python_buildpack...
Downloaded python_buildpack
Cell diego-cell-0 creating container for instance 0de0028f-35e2-43b1-a903-9f3db7546ebc
Cell diego-cell-0 successfully created container for instance 0de0028f-35e2-43b1-a903-9f3db7546ebc
Downloading app package...
Downloaded app package (16.6K)
-----> Python Buildpack version 1.6.11
-----> Supplying Python
-----> Installing python 2.7.14
Download [https://buildpacks.cloudfoundry.org/dependencies/python/python-2.7.14-linux-x64-30d9c08f.tgz]
-----> Installing setuptools 38.5.2
Download [https://buildpacks.cloudfoundry.org/dependencies/setuptools/setuptools-38.5.2-8246123e.zip]
-----> Installing pip 9.0.1
Download [https://buildpacks.cloudfoundry.org/dependencies/manual-binaries/pip/pip-9.0.1-35f01da3.tar.gz]
-----> Installing pip-pop 0.1.1
Download [https://buildpacks.cloudfoundry.org/dependencies/manual-binaries/pip-pop/pip-pop-0.1.1-d410583a.tar.gz]
-----> Installing pipenv 11.1.4
Download [https://buildpacks.cloudfoundry.org/dependencies/pipenv/pipenv-v11.1.4-7394418e.tgz]
-----> Running Pip Install
You must give at least one requirement to install (see "pip help install")
You are using pip version 9.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Exit status 0
Uploading droplet, build artifacts cache...
Uploading build artifacts cache...
Uploading droplet...
Uploaded build artifacts cache (37.1M)
Uploaded droplet (39.9M)
Uploading complete
Cell diego-cell-0 stopping instance 0de0028f-35e2-43b1-a903-9f3db7546ebc
Cell diego-cell-0 destroying container for instance 0de0028f-35e2-43b1-a903-9f3db7546ebc
Cell diego-cell-0 successfully destroyed container for instance 0de0028f-35e2-43b1-a903-9f3db7546ebc

Waiting for app to start...

name: mytestapp
requested state: started
routes: mytestapp.
last uploaded: Mon 13 Aug 02:59:22 UTC 2018
stack: cflinuxfs2
buildpacks: python

type: web
instances: 1/2
memory usage: 128M
start command: python server.py
state since cpu memory disk
#0 starting 2018-08-13T02:59:44Z 0.0% 0 of 128M 0 of 256M
#1 running 2018-08-13T02:59:57Z 0.0% 0 of 128M 0 of 256M

Std Error:

Process Exit Value: 0

What you expected to occur

Before v6.38, the behavior is 'cf push' exit with code 0 only when both 2 of 2 instances has been up and running

CLI Version

6.38.0+7ddf0aadd.2018-08-07

CC API Endpoint Version

Cloudcontroller_api: v2.106.0

CF Trace

Platform & Shell Details

Ubuntu 16.06

Any other relevant information

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/159722828

The labels on this github issue will be updated when the story is started.

[XenoPhex]

Hey there @shiyan0109!

The intended behavior of this code is that it polls the CF API until any instance reports running, then exits 0. It's possible that it might occasionally report that more than one instance is already running, assuming they started up at almost exactly the same time, but it does exit as soon as at least one instance is running.

If you want to verify that multiple instances are running, you can always run cf app <APPNAME> after cf push exits.

From what we can tell from history - this code has not been modified for multiple released versions of the CF CLI, so this behavior is not new. We'd recommend adjusting your script to be aware of this possible situation as we are unlikely to change it.

[shiyan0109]

Thanks for reply, but is a little hard to explain.

We had such error right after we upgraded version to 6.38, and the error went away once we downupgraded the version to v6.37 again. No change at all can result in such behavior?

[XenoPhex]

There are three pieces to the "error" that you're perceiving that should be pointed out and explained:

1. The push command is not blocking until all the app instances are up to consider the app running.
2. The app display of the push command is displaying an number of instances as starting not running.
3. The push command is exiting 0 even though there's some instances in the starting state.

For 1 and 3, the code I pointed out above shows that that the CF CLI has always exhibited the "exit 0 when at least 1 instance is up" behavior.

As for Point 2, these values for this display is obtained by various CC calls, depending on the timing of things, the state of any instance beyond the first running one could have always been in the starting state. This can be seen by our [restart integration tests[https://github.com/cloudfoundry/cli/blame/master/integration/isolated/restart_command_test.go#L144-L147] which use the exact same app display code as the other commands. This code has not been touched in at least a year. So this is not new behavior in v6.38.0.

What is new in v6.38.0 is which API calls the CF CLI uses for the app display. The new API calls are faster/lighter-weight than the old API calls - meaning it's more likely to catch the app in a state where at least some of the app instances are in the starting state.