Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

operator: Add support for configuring HTTP server timeouts #9405

Merged
merged 35 commits into from
May 22, 2023
Merged
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
70d3ea6
operator: Add LokiStack http server limits spec
periklis May 5, 2023
5932c37
Parse server config options from limits spec
periklis May 5, 2023
f5eb750
Add OpenShift route timeout annotation
periklis May 5, 2023
643fc08
Pass server options to loki config
periklis May 5, 2023
e9aaf56
Fix defaults
periklis May 5, 2023
c3a6557
Apply http server and upstream timeouts to the gateway
periklis May 5, 2023
268daaa
Add changelog entry
periklis May 5, 2023
d19eb9d
Add api docs
periklis May 5, 2023
89c7465
Merge branch 'main' into operator-fix-write-timeouts
periklis May 8, 2023
a37e1a2
Add operator-sdk API markers
periklis May 8, 2023
11c531e
Enhance error log message in lokistack handler
periklis May 8, 2023
1c661c5
Merge branch 'main' into operator-fix-write-timeouts
periklis May 8, 2023
08ae251
Merge branch 'main' into operator-fix-write-timeouts
periklis May 9, 2023
9079100
Apply code review suggestions
periklis May 15, 2023
2723949
Merge remote-tracking branch 'upstream' into operator-fix-write-timeouts
periklis May 15, 2023
146a034
Fix merge
periklis May 15, 2023
0bff0e9
Merge remote-tracking branch 'upstream' into operator-fix-write-timeouts
periklis May 17, 2023
10f668e
Regenerate
periklis May 17, 2023
d5e0bbb
Make defaults a global variable, remove pointer
xperimental May 17, 2023
d5d883d
Make defaults more visible in parse function
xperimental May 17, 2023
42e5b49
Unify defaults and simplify NewServerConfig
xperimental May 17, 2023
642dfba
Rename constants
xperimental May 17, 2023
094a59d
Use time.Duration in config template
xperimental May 17, 2023
cd9731d
Reduce indirection in config generator
xperimental May 17, 2023
a732ece
Separate config structs for Loki and gateway
xperimental May 17, 2023
9c4bb0b
Rename configuration struct
xperimental May 17, 2023
47d56cc
Merge remote-tracking branch 'upstream' into operator-fix-write-timeouts
periklis May 17, 2023
12e6080
Rename TestNewServerConfig to TestNewTimeoutConfig
periklis May 17, 2023
c441848
Raise error instead of using defaults
xperimental May 22, 2023
3382fac
Shorten gateway route timeout
xperimental May 22, 2023
4cb456c
Increase default query timeout to match existing default
xperimental May 22, 2023
11d0ebf
Regenerate docs
periklis May 22, 2023
43be4ef
Merge remote-tracking branch 'upstream' into operator-fix-write-timeouts
periklis May 22, 2023
38fc277
Fix failing tests
periklis May 22, 2023
7a2d0d2
Update import formatting to satisfy check
xperimental May 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions operator/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
## Main

- [9405](https://github.com/grafana/loki/pull/9405) **periklis**: Add support for configuring HTTP server timeouts
- [9378](https://github.com/grafana/loki/pull/9378) **aminesnow**: Add zone aware API spec validation
- [9408](https://github.com/grafana/loki/pull/9408) **JoaoBraveCoding**: Add PodAntiAffinity overwrites per component
- [9429](https://github.com/grafana/loki/pull/9429) **aminesnow**: Add default TopologySpreadContraints to Gateway
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing
certified: "false"
containerImage: docker.io/grafana/loki-operator:main-ac1c1fd
createdAt: "2023-05-11T08:04:29Z"
createdAt: "2023-05-17T13:41:47Z"
description: The Community Loki Operator provides Kubernetes native deployment
and management of Loki and related logging components.
operators.operatorframework.io/builder: operator-sdk-unknown
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing
certified: "false"
containerImage: docker.io/grafana/loki-operator:main-ac1c1fd
createdAt: "2023-05-11T08:04:26Z"
createdAt: "2023-05-17T13:41:44Z"
description: The Community Loki Operator provides Kubernetes native deployment
and management of Loki and related logging components.
operators.operatorframework.io/builder: operator-sdk-unknown
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing
certified: "false"
containerImage: quay.io/openshift-logging/loki-operator:v0.1.0
createdAt: "2023-05-11T08:04:32Z"
createdAt: "2023-05-17T13:41:50Z"
description: |
The Loki Operator for OCP provides a means for configuring and managing a Loki stack for cluster logging.
## Prerequisites and Requirements
Expand Down
18 changes: 17 additions & 1 deletion operator/docs/operator/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1302,6 +1302,21 @@ the component onto it.</p>
the component onto it.</p>
</td>
</tr>
<tr>
<td>
<code>podAntiAffinity</code><br/>
<em>
<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podantiaffinity-v1-core">
Kubernetes core/v1.PodAntiAffinity
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>PodAntiAffinity defines the pod anti affinity scheduling rules to schedule pods
of a component.</p>
</td>
</tr>
</tbody>
</table>

Expand Down Expand Up @@ -3329,7 +3344,8 @@ int32
</td>
<td>
<em>(Optional)</em>
<p>Zones defines an array of ZoneSpec that the scheduler will try to satisfy.</p>
<p>Zones defines an array of ZoneSpec that the scheduler will try to satisfy.
IMPORTANT: Make sure that the replication factor defined is less than or equal to the number of available zones.</p>
</td>
</tr>
</tbody>
Expand Down
6 changes: 6 additions & 0 deletions operator/internal/handlers/lokistack_create_or_update.go
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,11 @@ func CreateOrUpdateLokiStack(
certRotationRequiredAt = stack.Annotations[manifests.AnnotationCertRotationRequiredAt]
}

serverCfg, err := manifests.NewServerConfig(stack.Spec.Limits)
if err != nil {
ll.Error(err, "failed to parser server limits using fallback to defaults")
}

// Here we will translate the lokiv1.LokiStack options into manifest options
opts := manifests.Options{
Name: req.Name,
Expand All @@ -286,6 +291,7 @@ func CreateOrUpdateLokiStack(
Spec: rulerConfig,
Secret: rulerSecret,
},
Server: serverCfg,
Tenants: manifests.Tenants{
Secrets: tenantSecrets,
Configs: tenantConfigs,
Expand Down
21 changes: 21 additions & 0 deletions operator/internal/manifests/build_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@ import (
"github.com/stretchr/testify/require"
)

func defaultServerConfig() ServerConfig {
s, _ := NewServerConfig(nil)
return s
}

func TestApplyUserOptions_OverrideDefaults(t *testing.T) {
allSizes := []lokiv1.LokiStackSizeType{
lokiv1.SizeOneXDemo,
Expand All @@ -36,6 +41,7 @@ func TestApplyUserOptions_OverrideDefaults(t *testing.T) {
},
},
},
Server: defaultServerConfig(),
}
err := ApplyDefaultSettings(&opt)
defs := internal.StackSizeTable[size]
Expand Down Expand Up @@ -78,6 +84,7 @@ func TestApplyUserOptions_AlwaysSetCompactorReplicasToOne(t *testing.T) {
},
},
},
Server: defaultServerConfig(),
}
err := ApplyDefaultSettings(&opt)
defs := internal.StackSizeTable[size]
Expand Down Expand Up @@ -232,6 +239,7 @@ func TestBuildAll_WithFeatureGates_ServiceMonitors(t *testing.T) {
ServingCertsService: false,
},
},
Server: defaultServerConfig(),
},
},
{
Expand All @@ -250,6 +258,7 @@ func TestBuildAll_WithFeatureGates_ServiceMonitors(t *testing.T) {
ServingCertsService: false,
},
},
Server: defaultServerConfig(),
},
},
}
Expand Down Expand Up @@ -292,6 +301,7 @@ func TestBuildAll_WithFeatureGates_OpenShift_ServingCertsService(t *testing.T) {
ServingCertsService: false,
},
},
Server: defaultServerConfig(),
},
},
{
Expand All @@ -309,6 +319,7 @@ func TestBuildAll_WithFeatureGates_OpenShift_ServingCertsService(t *testing.T) {
ServingCertsService: true,
},
},
Server: defaultServerConfig(),
},
},
}
Expand Down Expand Up @@ -349,6 +360,7 @@ func TestBuildAll_WithFeatureGates_HTTPEncryption(t *testing.T) {
Gates: configv1.FeatureGates{
HTTPEncryption: true,
},
Server: defaultServerConfig(),
}

err := ApplyDefaultSettings(&opts)
Expand Down Expand Up @@ -422,6 +434,7 @@ func TestBuildAll_WithFeatureGates_ServiceMonitorTLSEndpoints(t *testing.T) {
HTTPEncryption: true,
ServiceMonitorTLSEndpoints: true,
},
Server: defaultServerConfig(),
}

err := ApplyDefaultSettings(&opts)
Expand Down Expand Up @@ -526,6 +539,7 @@ func TestBuildAll_WithFeatureGates_GRPCEncryption(t *testing.T) {
Gates: configv1.FeatureGates{
GRPCEncryption: false,
},
Server: defaultServerConfig(),
},
},
{
Expand Down Expand Up @@ -568,6 +582,7 @@ func TestBuildAll_WithFeatureGates_GRPCEncryption(t *testing.T) {
Gates: configv1.FeatureGates{
GRPCEncryption: true,
},
Server: defaultServerConfig(),
},
},
}
Expand Down Expand Up @@ -692,6 +707,7 @@ func TestBuildAll_WithFeatureGates_RuntimeSeccompProfile(t *testing.T) {
Gates: configv1.FeatureGates{
RuntimeSeccompProfile: false,
},
Server: defaultServerConfig(),
},
},
{
Expand Down Expand Up @@ -734,6 +750,7 @@ func TestBuildAll_WithFeatureGates_RuntimeSeccompProfile(t *testing.T) {
Gates: configv1.FeatureGates{
RuntimeSeccompProfile: true,
},
Server: defaultServerConfig(),
},
},
}
Expand Down Expand Up @@ -797,6 +814,7 @@ func TestBuildAll_WithFeatureGates_LokiStackGateway(t *testing.T) {
HTTPEncryption: true,
ServiceMonitorTLSEndpoints: false,
},
Server: defaultServerConfig(),
},
},
{
Expand Down Expand Up @@ -835,6 +853,7 @@ func TestBuildAll_WithFeatureGates_LokiStackGateway(t *testing.T) {
HTTPEncryption: true,
ServiceMonitorTLSEndpoints: true,
},
Server: defaultServerConfig(),
},
},
}
Expand Down Expand Up @@ -873,6 +892,7 @@ func TestBuildAll_WithFeatureGates_LokiStackAlerts(t *testing.T) {
ServiceMonitors: false,
LokiStackAlerts: false,
},
Server: defaultServerConfig(),
},
},
{
Expand All @@ -887,6 +907,7 @@ func TestBuildAll_WithFeatureGates_LokiStackAlerts(t *testing.T) {
ServiceMonitors: true,
LokiStackAlerts: true,
},
Server: defaultServerConfig(),
},
},
}
Expand Down
29 changes: 29 additions & 0 deletions operator/internal/manifests/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"crypto/sha1"
"fmt"
"strings"
"time"

lokiv1 "github.com/grafana/loki/operator/apis/loki/v1"
"github.com/grafana/loki/operator/internal/manifests/internal/config"
Expand Down Expand Up @@ -172,6 +173,7 @@ func ConfigOptions(opt Options) config.Options {
IngesterMemoryRequest: opt.ResourceRequirements.Ingester.Requests.Memory().Value(),
},
ObjectStorage: opt.ObjectStorage,
Server: serverConfig(opt.Server),
EnableRemoteReporting: opt.Gates.GrafanaLabsUsageReport,
Ruler: config.Ruler{
Enabled: rulerEnabled,
Expand All @@ -186,6 +188,33 @@ func ConfigOptions(opt Options) config.Options {
}
}

func serverConfig(cfg ServerConfig) config.ServerConfig {
oneMinute := 1 * time.Minute

idleTimeout := fmt.Sprintf("%.fm", cfg.HTTP.IdleTimeout.Minutes())
if cfg.HTTP.IdleTimeout < oneMinute {
idleTimeout = fmt.Sprintf("%.fs", cfg.HTTP.IdleTimeout.Seconds())
}

readTimeout := fmt.Sprintf("%.fm", cfg.HTTP.ReadTimeout.Minutes())
if cfg.HTTP.ReadTimeout < oneMinute {
readTimeout = fmt.Sprintf("%.fs", cfg.HTTP.ReadTimeout.Seconds())
}

writeTimeout := fmt.Sprintf("%.fm", cfg.HTTP.WriteTimeout.Minutes())
if cfg.HTTP.WriteTimeout < oneMinute {
writeTimeout = fmt.Sprintf("%.fs", cfg.HTTP.WriteTimeout.Seconds())
}

return config.ServerConfig{
HTTP: config.HTTPServerConfig{
IdleTimeout: idleTimeout,
ReadTimeout: readTimeout,
WriteTimeout: writeTimeout,
},
}
}

func alertManagerConfig(spec *lokiv1.AlertManagerSpec) *config.AlertManagerConfig {
if spec == nil {
return nil
Expand Down
Loading