[Question] Understand the port connection in container environment

[Jeffwan]

I am trying to orchestrate everything in a Kubernetes environment. I use the real production mode for the testing.
I generate start kits from project.yml and bake everything in the container.

1. I create a Kubernetes service which points to server pod 8002,8003. In this case, clients can connect and register themselves in the cluster.
2. When I submit job to server, it doesn't work as expected. Client stuck here and I think it's because the port communication is blocked. However, I don't know what port I should open. Can someone give some clues?

2022-11-09 18:54:32,151 - PTLearner - INFO - [identity=site-2, run=732c54fc-d1dc-4d68-ac11-15455bf7fc5f, peer=example_project, peer_run=732c54fc-d1dc-4d68-ac11-15455bf7fc5f, task_name=train, task_id=1e9bf7aa-6072-4e67-97bf-bfecabefe48c]: Epoch: 0/5, Iteration: 0, Loss: 2.3987499376138053e-05

[YuanTingHsieh]

Hi @Jeffwan

thanks for your question

1. What is the job that you are using for your experiment, can you try our hello-XXX jobs first?
2. Can you attach your server log, and all of your clients log?

[Jeffwan]

1. I am using hello-pt-tb as an example.
2. Please check log attached. I test locally and it's done very fast, but the container based solution is kind of slow. It takes almost 3hrs to finish.

nvflare.log

I should clarify that client still works and it's just slow. Server side shows log below.

2022-11-09 22:50:44,432 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:44,802 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:45,977 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:46,326 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:46,868 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:47,510 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:48,421 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:49,040 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:49,950 - FederatedServer - INFO - getting AuxCommunicate request
2022-11-09 22:50:50,470 - FederatedServer - INFO - getting AuxCommunicate request

[chesterxgchen]

When using under K8s, there might be issues if you use it directly. The FLARE server and clients communicate via GRPC and authenticated SSL certificate. If the Server Pod is running and exposed the through some ingress service. How is the ingress controller is configured. what's the SSL certification termination point ? in other words, how does the client SSL pass through the ingestion controller service and reached to the FL server ?

[Jeffwan]

@chesterxgchen I didn't create ingress. Instead, I create a service and point to the server pod. Do you have any guidance running on Kubernetes? Trying to see whether I follow the best practice?

[chesterxgchen]

I think there are several challenges to deploy FLARE Server to the K8s env. Although few customers have done this successfully, but I would not suggest to try this at this point. We will make this process easier next year. FL Client on K8s should be ok.