CIFAR10 EXAMPLE

[FabioNotaro2001]

Hi,
Is it possible to run the cifar10 example on a single host, without using the simulator and without having the GPU requirements?
In brief, I'd like to mantain the configuration with many different clients (also less than 8 would be ok), but I want them to be on a single host, that doesn't have a GPU or have a GPU with only 4GB.
In the github repo I read that this example requires 1 GPU with 8GB, but I'm running linux on a virtual machine in a laptop with only 4GB on a single GPU, is there a way to run this experiment on my hardware?
Thank you for the support

[holgerroth]

Hello @FabioNotaro2001, thank you for reaching out. You can run the "real-world" cifar-10 example on one host https://github.com/NVIDIA/NVFlare/tree/dev/examples/cifar10/cifar10-real-world

Just set export CUDA_VISIBLE_DEVICES= here, and the training should run on CPU only.

[FabioNotaro2001]

I re-tried to follow the instructions but maybe I miss something. I report the command that I insert and some screens, in such a way you can better help me. Thank you for your patience, but I'm trying to understand how it works. First of all I move myself into the cifar10 directory and here i inserted the following commands:
python3 ../pt/utils/cifar10_download_data.py
cd cifar10-real-world/
cd ./workspaces
nvflare provision -p ./secure_project.yml
cp -r ./workspace/secure_project/prod_00 ./secure_workspace
cd ..
./start_fl_secure.sh 8
And the I got the first screen

So I run ./submit_job.sh cifar10_fedavg_stream_tb 1.0 and got the second screen, that ends with "job submitted...success..."

But I think I missed something. Why I can't see the job that goes on, the epoch passing and so on? And why nothing changes in the workspace? I expected to find the trasìnsfered file and filo for tensoraboard and so on.
Thank you for you support, I remain availabla and I look forward to know where I go wrong.

[holgerroth]

It all looks good, but not sure why the job isn't being executed. Can you try logging into the admin console and type check_status server & check_status client? See the instructions here https://nvflare.readthedocs.io/en/main/real_world_fl/operation.html

You can also check the log.txt files in the job folders of client and server to see if there's an error message shown there under ./workspaces/secure_workspace.

[FabioNotaro2001]

I report what I've done in order to reply to your suggestion, in such a way you can correct me if I missed something. In order to log into the admin console I opened a terminal into the directory cifar10/cifar10-real-world/workspaces/secure_workspace/admin@nvidia.com/startup. I want you to remember that the other folders (local and transfer) are empty (I don't know why, I expected to find something there). Anyway, into the folder startup I run the command ./fl_admin.sh, I logged and inserted the command you cited me obtaining the result in the screen:
.
I also tried to check the log.txt files in the job folders of client and server.
All the client have the same 2 lines file that report: "2023-01-23 10:19:44,607 - FederatedClient - INFO - Got the new primary SP: localhost:8102
2023-01-23 10:19:47,397 - FederatedClient - INFO - Successfully registered client:site-3 for project secure_project. Token:1868f33e-b7b0-4b90-afc3-220e86593dd3 SSID:ebc6125d-0a56-4688-9b08-355fe9e4d61a", while the server (I opened the folder named localhost, I don't know if it is the server or you meant the overseer) contains a log.txt file with the content: "2023-01-23 10:19:44,607 - FederatedClient - INFO - Got the new primary SP: localhost:8102
2023-01-23 10:19:47,397 - FederatedClient - INFO - Successfully registered client:site-3 for project secure_project. Token:1868f33e-b7b0-4b90-afc3-220e86593dd3 SSID:ebc6125d-0a56-4688-9b08-355fe9e4d61a"
It all seems as I didn't run any job, but the screen in the first questions says that I did. What could it be? I precise that when I run the command /submit_job.sh cifar10_fedavg_stream_tb 1.0 I expected to wait, I expected a loading screen with epochs evolving, but I only get the write as you see in the first question after very few seconds. Thank you for you support, I'm waiting for your explanation and I remain available if you want other info.

[holgerroth]

Okay. So, no errors in the logs, correct? Please also send the output of the list_jobs command in the admin console. Need to check if the job was actually submitted. If no job is listed, please try submitting one again using the submit_job.sh script.

[FabioNotaro2001]

I retried another submission, so the id and time are different, but this is the output:

So, I suppose the job was submitted correctly, but as you can see the run duration is not available.

[holgerroth]

Great. So, the job is submitted but not being executed. Likely this is because some misconfiguration for the resource management. Can you check this part was executed correctly? https://github.com/NVIDIA/NVFlare/tree/dev/examples/cifar10/cifar10-real-world#33-multi-tasking-resource-management

You can also attach your resources.json here so I can take a look.

Likely, the resource requirements specified in meta.json are not met.

[YuanTingHsieh]

If you want to run without GPU, can follow Holger's comment below

[holgerroth]

I just realized that you want to run without GPU, correct? You can delete the "resource_spec" section of meta.json and resubmit. Then it should be able to run without the GPU requirement.

[FabioNotaro2001]

Yes, now it works. Thank you very much for your precious support! I have two last questions. What is secure workspace? Why we used it in this case? What are the technical differences between secure workspace and "default" workspace?

The second question regards the plot of loss/accuracy using tensorflow. After the training process started i wait 4 epochs and then I killed the process using CTRL+C because I wanted to plot the result. So i logged as admin and download the job and moved it into the admin/transfer folder. Then, following the documentation I change the line "download_dir" into the file figs/plot_tensorboard_events.py typing the same folder (admin/transfer). After that I tried many command in order to open the plot on the browser but maybe I miss something. I tried:
tensorboard --logdir=....../cifar10_real_world/workspaces/secure_workspace/admin@nvidia.com/transfer
tensorboard --logdir=....../cifar10_real_world/workspaces/secure_workspace/localhost/[JOB_ID]/tb_events
But then i open localhost:6006 on the browser and it says that No dashboards are active for the current data set
Which are my errors?
Anyway I wait for your response and thank you!

[FabioNotaro2001]

Thank you, and for what concern the tensorboard question? I also tried that when the job is running the command tensorboard --logdir=.../server/../tb_events while when the job is finished tensorboard --logdir=.../admin/<JOB_ID>/..tb_events but nothing, I miss something

[holgerroth]

The tensorboard files will only show while the job is running. After completion, the job results will be moved automatically to permanent storage. You can get them again using the download_job [JOB_ID] admin command. The download location should be shown when you run the command.

[FabioNotaro2001]

Yes, but which directory has to be pointed fromt the command tensorboard --logdir? Because I thought it was the tb_events directory into the server directory (in the case the job is still running), but it doesn't work

[holgerroth]

After download, the downloaded folder should contain the tensorboard event files. Then you should point the --logdir to that folder. If are no TB files, then something went wrong with the TB streaming.

[FabioNotaro2001]

I solved, it was a stupid error with the path. Now it works. Thank you very much for your support!