prov/rxm: Rework connection management

This patch greatly simplifies the management of msg
endpoints by removing the cmap indirection.  Unfortunately,
there's no straightforward way to replace cmap and all of
its complexities with set set of smaller, more manageable
changes.

The new scheme has the following behavior:

The AV manages 2 sets of addresses.  The first is an
array of addresses which were inserted into the AV.  This
is the standard AV implementation.  The second is an array
of known peer addresses (peer array).  That array includes
addresses where were inserted into the AV, plus the addresses of
peers to which there is an active connection.  Along with
these 2 arrays is an rbtree, which allows searching for
addresses that are in the peer array.

The peer array is implemented using the indexed buffer pool.

RxM ep's will maintain an array of connections to each peer
that it is communicating with.  Those connections will be
stored in an indexer.  The index that each connection uses
will match the index of the AV peer array corresponding to
the remote address.  So if address 10.1.2.3 is stored in the
AV's peer array at index 8, then the rxm ep will store its
connection to 10.1.2.3 in its indexer also at index 8.

This structure makes lookups consistent for all rxm ep's
and eliminiates the need for the complex cmap.

In order to remove the cmap, the entire CM state machine
needs to be rewritten.

Signed-off-by: Sean Hefty <sean.hefty@intel.com>

include/ofi_net.h

@@ -401,6 +401,21 @@ static inline int ofi_translate_addr_format(int family)
 	}
 }
 
+static inline size_t ofi_sizeof_addr_format(int format)
+{
+	switch (format) {
+	case FI_SOCKADDR_IN:
+		return sizeof(struct sockaddr_in);
+	case FI_SOCKADDR_IN6:
+		return sizeof(struct sockaddr_in6);
+	case FI_SOCKADDR_IB:
+		return sizeof(struct ofi_sockaddr_ib);
+	default:
+		FI_WARN(&core_prov, FI_LOG_CORE, "Unsupported address format\n");
+		return 0;
+	}
+}
+
 uint16_t ofi_get_sa_family(const struct fi_info *info);
 
 static inline bool ofi_sin_is_any_addr(const struct sockaddr *sa)

include/ofi_util.h

@@ -766,6 +766,8 @@ int ofi_ip_av_create_flags(struct fid_domain *domain_fid, struct fi_av_attr *att
 
 void *ofi_av_get_addr(struct util_av *av, fi_addr_t fi_addr);
 #define ofi_ip_av_get_addr ofi_av_get_addr
+void *ofi_av_addr_context(struct util_av *av, fi_addr_t fi_addr);
+
 fi_addr_t ofi_ip_av_get_fi_addr(struct util_av *av, const void *addr);
 
 int ofi_get_addr(uint32_t *addr_format, uint64_t flags,