rename preCommand to pythonPath #3410

acured · 2021-03-02T06:48:05Z

No description provided.

J-shang · 2021-03-03T01:19:00Z

docs/en_US/Release.rst

@@ -110,7 +110,7 @@ Training service
 """"""""""""""""


-* Support configuring python environment using "preCommand" in remote mode (#2875)
+* Support configuring python environment using "pythonPath" in remote mode (#2875)


This is the release doc of v1.9, please revert it.

J-shang · 2021-03-03T01:23:45Z

nni/experiment/config/convert.py

@@ -105,7 +105,7 @@ def _handle_training_service(ts, data):
                'gpuIndices': _convert_gpu_indices(machine.get('gpuIndices')),
                'maxTrialNumPerGpu': machine.get('maxTrialNumPerGpu'),
                'useActiveGpu': machine.get('useActiveGpu'),
-                'preCommand': machine.get('trialPrepareCommand')
+                'pythonPath': machine.get('trialPrepareCommand')


trialPrepareCommand -> trialPythonPath
trial_prepare_command -> trial_python_path

also in other files, like experiment_config.rst.

J-shang · 2021-03-03T04:19:05Z

docs/en_US/reference/experiment_config.rst

@@ -538,7 +538,7 @@ If `trialGpuNumber`_ is less than the length of this value, only a subset will b
 This will be used as ``CUDA_VISIBLE_DEVICES`` environment variable.


-trialPrepareCommand
+trialPythonPath
 -------------------

 Command(s) to run before launching each trial.


The path will insert to PATH before launching each trial.

QuanluZhang · 2021-03-03T05:54:08Z

docs/en_US/reference/experiment_config.rst

@@ -538,10 +538,10 @@ If `trialGpuNumber`_ is less than the length of this value, only a subset will b
 This will be used as ``CUDA_VISIBLE_DEVICES`` environment variable.


-trialPrepareCommand
+trialPythonPath


what is the difference between pythonPath and trialPythonPath?

pythonPath is config v1
trialPythonPath is config v2

why use different names?

Just follow the previous naming way: preCommand is config v1, trialPrepareCommand is config v2. Or use pythonPath is enough? @liuzhe-lz

I think pythonPath is enough.
It was named trialPrepareCommand because the command executes once per trial. preCommand made me guess it would execute once per machine.

could someone explain when to use pythonPath and when trialPythonPath should be used?

SparkSnail · 2021-03-03T05:56:48Z

docs/en_US/TrainingService/RemoteMachineMode.rst

@@ -144,7 +144,7 @@ Files in ``codeDir`` will be uploaded to remote machines automatically. You can
 Configure python environment
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-By default, commands and scripts will be executed in the default environment in remote machine. If there are multiple python virtual environments in your remote machine, and you want to run experiments in a specific environment, then use **preCommand** to specify a python environment on your remote machine. 
+By default, commands and scripts will be executed in the default environment in remote machine. If there are multiple python virtual environments in your remote machine, and you want to run experiments in a specific environment, then use **pythonPath** to specify a python path environment on your remote machine. 


python path evnironment seems strange, python environment is enough.

QuanluZhang · 2021-03-03T06:03:26Z

examples/trials/mnist-tfv2/config_remote.yml

-    # preCommand: source ${replace_to_absolute_path_recommended_here}/bin/activate
-    # preCommand: source ${replace_to_conda_path}/bin/activate ${replace_to_conda_env_name}
-    preCommand: export PATH=${replace_to_python_environment_path_in_your_remote_machine}:$PATH
+    pythonPath: ${replace_to_python_environment_path_in_your_remote_machine}


better to add comments to show concrete path examples here

The doc said "The path will insert to PATH before launching each trial". Is /usr/bin/python3.7 supposed to be a directory?

It is a directory, "python3.7" is a folder name..I will rename it to "yourPythonPath" to avoid the misunderstanding.

Then I suggest /opt/python3.7/bin for Linux example.

QuanluZhang · 2021-03-03T06:08:39Z

@acured , how to set pythonPath when using natively installed python, anaconda, and python virtual env?

acured · 2021-03-03T06:12:58Z

@acured , how to set pythonPath when using natively installed python, anaconda, and python virtual env?

Set the natively python path(located) directly.

SparkSnail · 2021-03-03T06:15:00Z

@acured , how to set pythonPath when using natively installed python, anaconda, and python virtual env?

Set the natively python path(located) directly.

did you tested using export PATH=${pythonPath}:$PATH && ${command} works for both anaconda and python virtual env? In previous, seems it use source ${replace_to_absolute_path_recommended_here}/bin/activate to activate python.

acured · 2021-03-03T06:28:54Z

@acured , how to set pythonPath when using natively installed python, anaconda, and python virtual env?

Set the natively python path(located) directly.

did you tested using export PATH=${pythonPath}:$PATH && ${command} works for both anaconda and python virtual env? In previous, seems it use source ${replace_to_absolute_path_recommended_here}/bin/activate to activate python.

For "pip install" I am going to test, for "python" it is working fine.

liuzhe-lz · 2021-03-03T06:37:16Z

docs/en_US/reference/experiment_config.rst

 -------------------

-Command(s) to run before launching each trial.
+The path will insert to PATH before launching each trial.


Please make it clear whether it's inserted at front or at tail.

acured · 2021-03-03T09:20:14Z

@acured , how to set pythonPath when using natively installed python, anaconda, and python virtual env?

Set the natively python path(located) directly.

did you tested using export PATH=${pythonPath}:$PATH && ${command} works for both anaconda and python virtual env? In previous, seems it use source ${replace_to_absolute_path_recommended_here}/bin/activate to activate python.

For "pip install" I am going to test, for "python" it is working fine.

After a test, It will cause an issue when working on windows anaconda if only set the native python path without active the env. To fix this issue, user have to set other two path: "../script" and "../Library/bin", I already add it in config_remote.yml in this PR.

There is no issue on linux.

QuanluZhang · 2021-03-03T11:39:40Z

docs/en_US/reference/experiment_config.rst

 -------------------

-Command(s) to run before launching each trial.
+The path will insert at the front of PATH.


suggest to add detailed description here, for example, what is this field used for? how to set the path for different python environments and OS, add examples like the ones you wrote in the example yaml file below

ts/nni_manager/training_service/remote_machine/test/shellExecutor.test.ts

ultmaster · 2021-03-04T06:02:26Z

docs/en_US/TrainingService/RemoteMachineMode.rst

@@ -144,7 +144,7 @@ Files in ``codeDir`` will be uploaded to remote machines automatically. You can
 Configure python environment
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-By default, commands and scripts will be executed in the default environment in remote machine. If there are multiple python virtual environments in your remote machine, and you want to run experiments in a specific environment, then use **preCommand** to specify a python environment on your remote machine. 
+By default, commands and scripts will be executed in the default environment in remote machine. If there are multiple python virtual environments in your remote machine, and you want to run experiments in a specific environment, then use **pythonPath** to specify a python environment on your remote machine. 


How does it work? If I execute a command line tool that uses python, for example, nnictl, lightgbm, horovodrun, will it work?

When you set this pythonPath, it will be added to the front of env PATH in current terminal, which means when you run any tool that calls python or run python directly from the terminal, this python your provided the path will be chosen first.
But it only works for current terminal which your set the pythonPath at first.

rename preCommand to pythonPath

1f6f6c1

J-shang reviewed Mar 3, 2021

View reviewed changes

J-shang requested a review from QuanluZhang March 3, 2021 01:29

rename trialPrepareCommand to trialPythonPath..

ff8d9be

J-shang approved these changes Mar 3, 2021

View reviewed changes

modify the description of trialPythonPath

9bbb6f0

J-shang approved these changes Mar 3, 2021

View reviewed changes

QuanluZhang reviewed Mar 3, 2021

View reviewed changes

QuanluZhang requested a review from SparkSnail March 3, 2021 05:54

SparkSnail reviewed Mar 3, 2021

View reviewed changes

QuanluZhang requested review from liuzhe-lz and ultmaster March 3, 2021 05:58

remove path in *pythonPath* description

f805c55

QuanluZhang reviewed Mar 3, 2021

View reviewed changes

add examples on yaml

f6855db

SparkSnail approved these changes Mar 3, 2021

View reviewed changes

liuzhe-lz reviewed Mar 3, 2021

View reviewed changes

Hao Ni added 2 commits March 3, 2021 15:21

keep the two version key words consistent

22e8b0c

modify example for windows anaconda user.

f6937a0

J-shang closed this Mar 3, 2021

J-shang reopened this Mar 3, 2021

QuanluZhang reviewed Mar 3, 2021

View reviewed changes

ts/nni_manager/training_service/remote_machine/test/shellExecutor.test.ts Show resolved Hide resolved

add description on pythonPath

8c65e65

QuanluZhang approved these changes Mar 4, 2021

View reviewed changes

ultmaster reviewed Mar 4, 2021

View reviewed changes

ultmaster approved these changes Mar 5, 2021

View reviewed changes

J-shang closed this Mar 5, 2021

J-shang reopened this Mar 5, 2021

J-shang merged commit f897829 into microsoft:master Mar 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rename preCommand to pythonPath #3410

rename preCommand to pythonPath #3410

acured commented Mar 2, 2021

J-shang Mar 3, 2021

acured Mar 3, 2021

J-shang Mar 3, 2021

acured Mar 3, 2021

J-shang Mar 3, 2021

acured Mar 3, 2021

QuanluZhang Mar 3, 2021

J-shang Mar 3, 2021

QuanluZhang Mar 3, 2021

J-shang Mar 3, 2021

liuzhe-lz Mar 3, 2021 •

edited

Loading

QuanluZhang Mar 3, 2021

SparkSnail Mar 3, 2021

acured Mar 3, 2021

QuanluZhang Mar 3, 2021

acured Mar 3, 2021

liuzhe-lz Mar 3, 2021

acured Mar 3, 2021

liuzhe-lz Mar 3, 2021 •

edited

Loading

acured Mar 4, 2021

QuanluZhang commented Mar 3, 2021

acured commented Mar 3, 2021

SparkSnail commented Mar 3, 2021 •

edited

Loading

acured commented Mar 3, 2021

liuzhe-lz Mar 3, 2021

acured Mar 3, 2021

acured commented Mar 3, 2021

QuanluZhang Mar 3, 2021

acured Mar 4, 2021

ultmaster Mar 4, 2021

acured Mar 4, 2021

rename preCommand to pythonPath #3410

rename preCommand to pythonPath #3410

Conversation

acured commented Mar 2, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuzhe-lz Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuzhe-lz Mar 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuanluZhang commented Mar 3, 2021

acured commented Mar 3, 2021

SparkSnail commented Mar 3, 2021 • edited Loading

acured commented Mar 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acured commented Mar 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuzhe-lz Mar 3, 2021 •

edited

Loading

liuzhe-lz Mar 3, 2021 •

edited

Loading

SparkSnail commented Mar 3, 2021 •

edited

Loading