prov/efa: Remove fi_sendmsg/recvmsg but still use efa_ep's qp and av

[sunkuamzn]

For SHM EP, use fi_sendv and fi_recvv
For EFA EP

• Allocate ibv_send_wr and ibv_recv_wr in rxr_pkt_entry
• Still use efa_ep->qp, efa_ep->av and efa_ep->xmit_more_wr*
• Directly call efa_post_flush or ibv_post_recv in rxr_pkt_entry.c

Signed-off-by: Sai Sunku sunkusa@amazon.com

[wzamazon]

With the change to efa_send_wr and efa_recv_wr you need to update how efa_ep->send_wr_bufpool and efa_ep->recv_wr_pool is created. Specifically the entry size should be just sizeof(efa_send_wr) and sizeof(efa_recv_wr)

[sunkuamzn]

With the change to efa_send_wr and efa_recv_wr you need to update how efa_ep->send_wr_bufpool and efa_ep->recv_wr_pool is created. Specifically the entry size should be just sizeof(efa_send_wr) and sizeof(efa_recv_wr)

But the DGRAM provider needs memory for the SGEs if a higher applications calls the DGRAM provider directly

[wzamazon]

But the DGRAM provider needs memory for the SGEs if a higher applications calls the DGRAM provider directly

I do not understand the comment. Current code is:

libfabric/prov/efa/src/efa_ep.c

Line 674 in 1eb9905

ret = ofi_bufpool_create(&ep->send_wr_pool,

ret = ofi_bufpool_create(&ep->send_wr_pool,
		sizeof(struct efa_send_wr) +
		user_info->tx_attr->iov_limit * sizeof(struct ibv_sge),
		16, 0, 1024, 0);

The reason to add user_info->tx_attr->iov_limit * sizeof(struct ibv_sge) is because original efa_send_wr does not allocate space for ibv_sge, so when the bufpool added some space to the end of efa_send_wr. Now that you declared 2 sge. There is no need to add that. (Maybe add an assertion about user_info->tx_attr->iov_limit <=2)

[sunkuamzn]

The reason to add user_info->tx_attr->iov_limit * sizeof(struct ibv_sge) is because original efa_send_wr does not allocate space for ibv_sge

OK I understand now. Fixed.