Skip to content

Commit

Permalink
(cherry-pick) NFR Test Results for NGF version 1.6.0 (#3013) (#3019)
Browse files Browse the repository at this point in the history
Update with NFR test results for NGF version 1.6.0 ["oss","plus"]

Co-authored-by: nginx-bot <68849795+nginx-bot@users.noreply.github.com>
  • Loading branch information
bjee19 and nginx-bot authored Jan 15, 2025
1 parent 1712d40 commit b9aac16
Show file tree
Hide file tree
Showing 89 changed files with 1,958 additions and 0 deletions.
92 changes: 92 additions & 0 deletions tests/results/dp-perf/1.6.0/1.6.0-oss.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Results

## Test environment

NGINX Plus: false

NGINX Gateway Fabric:

- Commit: b61c61d3f9ca29c6eb93ce9b44e652c9a521b3a4
- Date: 2025-01-13T16:47:24Z
- Dirty: false

GKE Cluster:

- Node count: 12
- k8s version: v1.30.6-gke.1596000
- vCPUs per node: 16
- RAM per node: 65853984Ki
- Max pods per node: 110
- Zone: us-west1-b
- Instance Type: n2d-standard-16

## Summary:

- Performance stayed consistent with 1.5.0 results. Average latency slightly increased across all routing methods.
- Errors that occurred are consistent with errors that occurred in the previous results.

## Test1: Running latte path based routing

```text
Requests [total, rate, throughput] 30000, 1000.04, 998.38
Duration [total, attack, wait] 30s, 29.999s, 844.157µs
Latencies [min, mean, 50, 90, 95, 99, max] 309.6µs, 718.534µs, 681.308µs, 786.633µs, 827.114µs, 971.115µs, 18.386ms
Bytes In [total, mean] 4741568, 158.05
Bytes Out [total, mean] 0, 0.00
Success [ratio] 99.84%
Status Codes [code:count] 200:29951 503:49
Error Set:
503 Service Temporarily Unavailable
```

## Test2: Running coffee header based routing

```text
Requests [total, rate, throughput] 30000, 1000.01, 999.98
Duration [total, attack, wait] 30.001s, 30s, 708.18µs
Latencies [min, mean, 50, 90, 95, 99, max] 519.443µs, 728.205µs, 716.283µs, 820.709µs, 859.918µs, 962.843µs, 6.974ms
Bytes In [total, mean] 4770000, 159.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```

## Test3: Running coffee query based routing

```text
Requests [total, rate, throughput] 30000, 1000.02, 1000.00
Duration [total, attack, wait] 30s, 29.999s, 746.102µs
Latencies [min, mean, 50, 90, 95, 99, max] 533.22µs, 735.075µs, 722.549µs, 830.432µs, 871.714µs, 973.911µs, 6.9ms
Bytes In [total, mean] 5010000, 167.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```

## Test4: Running tea GET method based routing

```text
Requests [total, rate, throughput] 30000, 1000.02, 1000.00
Duration [total, attack, wait] 30s, 29.999s, 737.741µs
Latencies [min, mean, 50, 90, 95, 99, max] 528.445µs, 724.715µs, 711.435µs, 816.76µs, 859.214µs, 967.474µs, 11.985ms
Bytes In [total, mean] 4680000, 156.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```

## Test5: Running tea POST method based routing

```text
Requests [total, rate, throughput] 30000, 1000.03, 1000.01
Duration [total, attack, wait] 30s, 29.999s, 643.191µs
Latencies [min, mean, 50, 90, 95, 99, max] 538.368µs, 728.96µs, 714.974µs, 818.991µs, 860.142µs, 971.866µs, 11.543ms
Bytes In [total, mean] 4680000, 156.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```
90 changes: 90 additions & 0 deletions tests/results/dp-perf/1.6.0/1.6.0-plus.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Results

## Test environment

NGINX Plus: true

NGINX Gateway Fabric:

- Commit: b61c61d3f9ca29c6eb93ce9b44e652c9a521b3a4
- Date: 2025-01-13T16:47:24Z
- Dirty: false

GKE Cluster:

- Node count: 12
- k8s version: v1.30.6-gke.1596000
- vCPUs per node: 16
- RAM per node: 65853984Ki
- Max pods per node: 110
- Zone: us-west1-b
- Instance Type: n2d-standard-16

## Summary:

- Performance stayed consistent with 1.5.0 results. Average latency slightly increased across all routing methods.

## Test1: Running latte path based routing

```text
Requests [total, rate, throughput] 30000, 1000.03, 1000.00
Duration [total, attack, wait] 30s, 29.999s, 744.047µs
Latencies [min, mean, 50, 90, 95, 99, max] 535.49µs, 722.768µs, 702.708µs, 807.78µs, 849.575µs, 981.854µs, 21.041ms
Bytes In [total, mean] 4770000, 159.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```

## Test2: Running coffee header based routing

```text
Requests [total, rate, throughput] 30000, 1000.01, 999.98
Duration [total, attack, wait] 30s, 30s, 718.788µs
Latencies [min, mean, 50, 90, 95, 99, max] 558.587µs, 766.304µs, 750.921µs, 866.313µs, 907.422µs, 1.022ms, 10.872ms
Bytes In [total, mean] 4800000, 160.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```

## Test3: Running coffee query based routing

```text
Requests [total, rate, throughput] 30000, 1000.02, 999.99
Duration [total, attack, wait] 30s, 30s, 733.649µs
Latencies [min, mean, 50, 90, 95, 99, max] 572.624µs, 771.492µs, 758.449µs, 867.491µs, 907.997µs, 1.032ms, 11.906ms
Bytes In [total, mean] 5040000, 168.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```

## Test4: Running tea GET method based routing

```text
Requests [total, rate, throughput] 30000, 1000.03, 1000.00
Duration [total, attack, wait] 30s, 29.999s, 712.155µs
Latencies [min, mean, 50, 90, 95, 99, max] 549.224µs, 760.423µs, 746.75µs, 853.877µs, 894.554µs, 1.008ms, 8.12ms
Bytes In [total, mean] 4710000, 157.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```

## Test5: Running tea POST method based routing

```text
Requests [total, rate, throughput] 30000, 1000.00, 999.98
Duration [total, attack, wait] 30.001s, 30s, 778.666µs
Latencies [min, mean, 50, 90, 95, 99, max] 544.486µs, 762.077µs, 748.375µs, 852.722µs, 893.014µs, 1.009ms, 9.632ms
Bytes In [total, mean] 4710000, 157.00
Bytes Out [total, mean] 0, 0.00
Success [ratio] 100.00%
Status Codes [code:count] 200:30000
Error Set:
```
Binary file added tests/results/longevity/1.6.0/oss-cpu.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/results/longevity/1.6.0/oss-memory.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/results/longevity/1.6.0/oss-ngf-memory.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/results/longevity/1.6.0/oss-reloads.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
97 changes: 97 additions & 0 deletions tests/results/longevity/1.6.0/oss.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Results

## Test environment

NGINX Plus: false

NGINX Gateway Fabric:

- Commit: 8be03e1fc5161a2b1bc0962fb0d8732114a9093d
- Date: 2025-01-14T18:57:38Z
- Dirty: true

GKE Cluster:

- Node count: 3
- k8s version: v1.30.6-gke.1596000
- vCPUs per node: 2
- RAM per node: 4018128Ki
- Max pods per node: 110
- Zone: us-central1-c
- Instance Type: e2-medium

## Traffic

HTTP:

```text
Running 5760m test @ http://cafe.example.com/coffee
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 189.49ms 147.10ms 2.00s 78.44%
Req/Sec 293.54 193.84 1.95k 66.59%
198532845 requests in 5760.00m, 67.91GB read
Socket errors: connect 0, read 309899, write 63, timeout 2396
Requests/sec: 574.46
Transfer/sec: 206.05KB
```

HTTPS:

```text
Running 5760m test @ https://cafe.example.com/tea
2 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 179.59ms 121.50ms 1.99s 67.56%
Req/Sec 292.54 193.88 2.39k 66.47%
197890521 requests in 5760.00m, 66.57GB read
Socket errors: connect 176, read 303560, write 0, timeout 7
Requests/sec: 572.60
Transfer/sec: 201.98KB
```

### Logs

No error logs in nginx-gateway.

No error logs in nginx.


### Key Metrics

#### Containers memory

![oss-memory.png](oss-memory.png)

#### NGF Container Memory

![oss-ngf-memory.png](oss-ngf-memory.png)

### Containers CPU

![oss-cpu.png](oss-cpu.png)

### NGINX metrics

![oss-stub-status.png](oss-stub-status.png)

### Reloads

Rate of reloads - successful and errors:

![oss-reloads.png](oss-reloads.png)

Reload spikes correspond to 1 hour periods of backend re-rollouts.

No reloads finished with an error.

Reload time distribution - counts:

![oss-reload-time.png](oss-reload-time.png)


## Comparison with previous results

Graphs look similar to 1.5.0 results. There is a color change swap in a few graphs which is a little confusing.
NGINX container memory decreased dramatically. NGINX Stub Status graph is confusing to interpret, which can make it seem
quite different to the 1.5.0 results, but it is similar, only with an increase in requests.
Binary file added tests/results/longevity/1.6.0/plus-cpu.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/results/longevity/1.6.0/plus-memory.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/results/longevity/1.6.0/plus-ngf-memory.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/results/longevity/1.6.0/plus-reloads.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added tests/results/longevity/1.6.0/plus-status.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit b9aac16

Please sign in to comment.