Support Windows as remote node. #2431

squirrelsc · 2020-05-14T07:44:30Z

Add WindowsCommand.ts to support different shell commands in windows.
Update documents to explain how to enable remote node in Windows.

Fixed some bugs,

Fix folder support problem on tarAdd function in training_service/common/util.ts.
Fix Keras compatibility issue on val_acc or val_accuracy name in logs.
Fix killChildProcesses of linux to support kill all descendant processes, instead of only direct child processes.
Fix too muchGPU stats commands

Small improvements

Let nni_trial_tool.trial_keeper to save its process id itself, to simplify script, and be more reliable.
Migrate non-cross platform code into osCommands.ts.
Refine ExecutorManager and ShellExecutor, to let ShellExecutor manage MaxTrials, and make it 1 to reduce concurrency issues.
Move map of Trial to shellexecutor from RemoteMachineTrainingService to ExecutorManager. It let resource management be more convenient, and avoid null reference.
Add RandomInt in utils.ts.
Move gpu collector related files into experiment folder for multi-run.
Let OsCommands can normalize folder like / to \ in windows, and versus in Linux.
Extend timeout of setMetaddata to 30 seconds for high latency remote machines like in cloud.
Fix partial success of pipelines-it-remote-windows.yml
Remove gpuNum: 0 in test cases to use GPU.
Improved some logs.

try to fix a bug on not found executor.

increase shellexecutor count to reduce ssh connections.

to use default GPU if there is.

This reverts commit 9e240dc.

chicm-ms · 2020-05-19T10:57:25Z

src/nni_manager/training_service/remote_machine/remoteMachineData.ts

-     */
-    public getFirstExecutor(): ShellExecutor {
-        return this.executorArray[0];
+        // init a new executor if no free one.


This comments does not match the code

chicm-ms · 2020-05-19T10:59:22Z

src/nni_manager/training_service/remote_machine/remoteMachineData.ts

-        return this.executorArray[0];
+        // init a new executor if no free one.
+        if (executor === undefined) {
+            throw new Error("executor shouldn't be undefined before return!");


this block is duplicated with above?

chicm-ms · 2020-05-19T11:34:32Z

src/nni_manager/training_service/remote_machine/shellExecutor.ts

    public get getUsedConnectionNumber(): number {
-        return this.usedConnectionNumber;
+        return this.usedCount;
    }

    public addUsedConnectionNumber(): void {


should this function addUsedConnectionNumber be removed?

SparkSnail · 2020-05-19T12:50:46Z

src/nni_manager/training_service/remote_machine/remoteMachineTrainingService.ts

-        await executor.createFolder(trialWorkingFolder);
-        await executor.createFolder(unixPathJoin(trialWorkingFolder, '.nni'));
+        await executor.createFolder(trialJobDetail.workingDirectory);
+        await executor.createFolder(executor.joinPath(trialJobDetail.workingDirectory, '.nni'));


Why not use a function support create folder recursively?

SparkSnail · 2020-05-19T13:07:51Z

src/nni_manager/training_service/remote_machine/test/shellExecutor.test.ts

    }
 }

 async function getRemoteFileContentLoop(executor: ShellExecutor): Promise<void> {
-    for (let i: number = 0; i < 10; i++) {
+    const remoteFullName = executor.joinPath(executor.getTempPath(), REMOTEFILE);
+    for (let i: number = 0; i < 3; i++) {
        // console.log(i);


remove this line

SparkSnail · 2020-05-19T13:08:47Z

src/nni_manager/training_service/remote_machine/test/shellExecutor.test.ts

@@ -41,14 +44,16 @@ describe('ShellExecutor test', () => {
        rmMeta = JSON.parse(fs.readFileSync('../../.vscode/rminfo.json', 'utf8'));
        console.log(rmMeta);
    } catch (err) {
-        console.log(`Please configure rminfo.json to enable remote machine test.${err}`);
+        console.log(`Please configure rminfo.json to enable remote machine test. ${err}`);


remove console.log() in TS code, use this.log() instead.

tried, it doesn't work in test code.

SparkSnail · 2020-05-19T13:10:17Z

test/config/examples/cifar10-pytorch.yml

@@ -14,7 +14,6 @@ assessor:
 trial:
  codeDir: ../../../examples/trials/cifar10_pytorch
  command: python3 main.py --epochs 1 --batches 1
-  gpuNum: 1


Why remove gpuNum?

It looks the test environment is used by two test suites sometime. So if set it to 1, NNI will wait another finished, but it causes timeout error. Remove this setting, it lets test cases to use default GPU, not wait each other, and reduced failure chance.

We need to add examples to test gpuScheduler in NNI, if remove this configuration, we could not test gpu related functions in pipeline.

ok, I will add it back. But it's test is very limited.

Chi Song and others added 12 commits May 7, 2020 18:04

init code to support windows node

1f5dc47

fix several bugs on Windows node.

c04cb3f

Merge remote-tracking branch 'official/master' into dev-windows-node

ae82fc5

Merge remote-tracking branch 'official/master' into dev-windows-node

ae25ff6

remove useless doc comments

62a2cc7

Make GPU script cross platform

6b0b534

Support Windows init code

2a12147

fix bug on check running collecting process.

6ad17e3

add documents and corelate command logs.

c94a0c7

add windows ut

9cc302b

update document to install openssh

1a0db06

fix bugs and improve log

65aeea2

squirrelsc linked an issue May 14, 2020 that may be closed by this pull request

Support Windows as Remote Training Node #1973

Closed

2 tasks

squirrelsc added 2 commits May 14, 2020 16:33

move gpu scripts to experiment folder

21eb3f0

try to fix a bug on not found executor.

simple fix to avoid concurrency issues.

53be583

QuanluZhang requested review from chicm-ms, liuzhe-lz and SparkSnail May 15, 2020 08:57

squirrelsc and others added 12 commits May 18, 2020 09:00

Merge branch 'master' into dev-windows-node

11672fe

fix on partial success of windows pipeline

1e99dad

increase shellexecutor count to reduce ssh connections.

add debug flag, and fix typo on failOnStdErr

14fd6d3

fix position of failOnStdErr

f028538

fix unstable timeout on linux, due to get temp path.

e0b0bda

remove gpuNum:0 to imcrease IT speed

2bd1046

remove gpuNum:1 requirement,

350a51e

to use default GPU if there is.

pytorch to reuse downloaded data with same dir

9e240dc

Revert "pytorch to reuse downloaded data with same dir"

d6ae7e2

This reverts commit 9e240dc.

init remote linux to windows IT

1e3f135

fix test name

e8ff3b4

cd to right folder

afa62da

squirrelsc and others added 9 commits May 19, 2020 11:30

Update pipelines-it-remote-linux-to-windows.yml for Azure Pipelines

3ade482

Update pipelines-it-remote-linux-to-windows.yml for Azure Pipelines

1b3e99d

Merge remote-tracking branch 'official/master' into dev-windows-node

5d6baab

fix port detect check

733d8fc

merge remote storage changes

4bfd313

fix tslint error

c35ba62

Merge remote-tracking branch 'official/master' into dev-windows-node

22ee0a9

remove extra comment.

1acb871

reduce mnist-keras trial count

889f499

chicm-ms reviewed May 19, 2020

View reviewed changes

Chi Song added 2 commits May 19, 2020 20:33

address review comments.

f258dcf

Merge remote-tracking branch 'official/master' into dev-windows-node

18ba9a3

chicm-ms approved these changes May 19, 2020

View reviewed changes

SparkSnail reviewed May 19, 2020

View reviewed changes

merge createfolder commands.

1cb0049

SparkSnail reviewed May 19, 2020

View reviewed changes

remove useless code.

5ca1666

SparkSnail reviewed May 19, 2020

View reviewed changes

restore 1 GPU limitation for cifar10 case.

7eef3b1

SparkSnail approved these changes May 19, 2020

View reviewed changes

Chi Song added 2 commits May 19, 2020 21:51

fix code not to use GPU as before

4995a73

reverse cifar10 test config

f372f1c

SparkSnail merged commit 69cae21 into master May 19, 2020

squirrelsc deleted the dev-windows-node branch May 20, 2020 02:07

squirrelsc restored the dev-windows-node branch May 20, 2020 02:07

squirrelsc deleted the dev-windows-node branch May 20, 2020 02:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Windows as remote node. #2431

Support Windows as remote node. #2431

squirrelsc commented May 14, 2020 •

edited

Loading

chicm-ms May 19, 2020

squirrelsc May 19, 2020

chicm-ms May 19, 2020 •

edited

Loading

squirrelsc May 19, 2020

chicm-ms May 19, 2020

squirrelsc May 19, 2020

SparkSnail May 19, 2020

squirrelsc May 19, 2020

SparkSnail May 19, 2020

squirrelsc May 19, 2020

SparkSnail May 19, 2020

squirrelsc May 19, 2020

SparkSnail May 19, 2020

squirrelsc May 19, 2020

SparkSnail May 19, 2020

squirrelsc May 19, 2020

Support Windows as remote node. #2431

Support Windows as remote node. #2431

Conversation

squirrelsc commented May 14, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chicm-ms May 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

squirrelsc commented May 14, 2020 •

edited

Loading

chicm-ms May 19, 2020 •

edited

Loading