WebUI displayed with no content

[FalinLei]

My server is Ubuntu 16.04 , and I run a ngc tensorflow docker container on it.
After starting the experiment, it showed that the url of webui is http://10.10.62.8:8080 and http://127.0.0.1:8080. The first one had no response, so I mapped the port 8080 to my computer port 1018.

This is what the command "nnictl trial ls" shows, which means those trials all succeeded.

But there is no content in the webui.

[QuanluZhang]

In v0.2 version, nni requires two ports: 51188 (restserver) and 8080 (webui). Mapping 8080 to 1018 allows you to open webui, then webui needs to send requests to restserver through port 51188, so you also need to map container's 51188 port to host's 51188 port.
In our upcoming v0.3, restserver and webui are merged together, using only one port 8080 (configurable).

[FalinLei]

It works. Thank you!

[FalinLei]

I just run the example in mnist-annotation folder, after training all trials there is test accuracy information in the log file.

While in the tab "Trial Status" the "loss/accuracy" of trials seems to be blank,

and the Intermediate Result Graph is blank too.

"Optimization Process" and ""Hyper Parameter" tabs are blank and show "no data".

[QuanluZhang]

1. please provide nni's log by running nnictl log stderr, to see whether there is any errors or warnings.
2. run nnictl trial ls to check whether there is final result displayed. If there is, the problem comes from webui.
3. use for example chrome's inspect to check whether there is any error when loading the page.

[FalinLei]

[QuanluZhang]

there is no error in backend, then please try step 2 and 3 that I listed above.

[QuanluZhang]

Also, could you open http://yourIP:51188/api/v1/nni/trial-jobs, to see whether final result exists?

[FalinLei]

Thank you -_-

The command nnictl trial ls displays results of all trials.

And there is one error when loading the page.

http://yourIP:51188/api/v1/nni/trial-jobs also shows that final result exists.

[QuanluZhang]

Thanks. Could you also paste the content of ~/nni/experiments/<EXPERIMENT-ID>/log/nnimanager.log and ~/nni/experiments/<EXPERIMENT-ID>/log/dispatcher.log?

[QuanluZhang]

Thanks. This is not full log of ~/nni/experiments/<EXPERIMENT-ID>/log/nnimanager.log. Maybe github does not allow pasting so much data. Could you attach the log files by dropping?

[FalinLei]

nnimanager.log
nnimanager.log
dispatcher.log
dispatcher.log

[QuanluZhang]

You used the example mnist_annotation/, right? Did you modify the code? Could you try the unmodified example mnist/ to check whether it has the same problem?

[FalinLei]

Because the example mnist_annotation/ didn't work I changed the example to mnist/, I didn't modify the code.
I just tried both of them again, they had the same problem.

[QuanluZhang]

Could you check the files under ~/nni/experiments/<EXPERIMENT-ID>/trials/<TRIAL-ID>/.nni/ , there should be two files: metrics and state. Could you paste the content of them?

[FalinLei]

metrics file
ME000112{"sequence": 0, "value": 0.11349999904632568, "parameter_id": 3, "trial_job_id": "ftua9", "type": "PERIODICAL"}
ME000112{"sequence": 1, "value": 0.11349999904632568, "parameter_id": 3, "trial_job_id": "ftua9", "type": "PERIODICAL"}
ME000107{"sequence": 0, "value": 0.09799999743700027, "parameter_id": 3, "trial_job_id": "ftua9", "type": "FINAL"}

state file
0 1540372598378

[QuanluZhang]

Hi @FalinLei , could you tell us how to get the docker image that you used, so that we can reproduce the problem?

[FalinLei]

OK, the docker image that I used is nvcr.io/nvidia/tensorflow:18.06-py3. You can pull it from https://ngc.nvidia.com/registry/nvidia-tensorflow.

[QuanluZhang]

Hi @FalinLei , this problem is induced by that tail-stream package does not work in docker container. metrics file is changed, but tail-stream does not notice that. We will fix this problem very soon. Thanks for your assistance.

[QuanluZhang]

Hi @FalinLei , after double checking the problem, we suspect it is induced by host kernel version. Could you tell us the kernel version of your host machine? run uname -a, lsb_release -a, docker -v on that host machine.

[FalinLei]

uname -a
Linux sugon96 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
docker -v
Docker version 18.03.1-ce, build 9ee9f40

[QuanluZhang]

Hi @FalinLei , after further investigation, we found that this problem can only be reproduced on specific combination(s) of kernel version and docker version. We pushed a hot fix to v0.2, i.e., #273 , which can resolve the issue.