Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[release-5.7] Backport PR grafana/loki#9405 #141

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions operator/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
## Main

- [9405](https://github.com/grafana/loki/pull/9405) **periklis**: Add support for configuring HTTP server timeouts
- [9346](https://github.com/grafana/loki/pull/9346) **periklis**: Enable Route by default on OpenShift clusters
- [9036](https://github.com/grafana/loki/pull/9036) **periklis**: Update Loki operand to v2.8.0
- [8978](https://github.com/grafana/loki/pull/8978) **aminesnow**: Add watch for the object storage secret
Expand Down
4 changes: 3 additions & 1 deletion operator/apis/loki/v1/lokistack_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -545,7 +545,7 @@ type QueryLimitSpec struct {
//
// +optional
// +kubebuilder:validation:Optional
// +kubebuilder:default:="1m"
// +kubebuilder:default:="3m"
// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Query Timeout"
QueryTimeout string `json:"queryTimeout,omitempty"`
}
Expand Down Expand Up @@ -854,6 +854,8 @@ const (
ReasonMissingGatewayOpenShiftBaseDomain LokiStackConditionReason = "MissingGatewayOpenShiftBaseDomain"
// ReasonFailedCertificateRotation when the reconciler cannot rotate any of the required TLS certificates.
ReasonFailedCertificateRotation LokiStackConditionReason = "FailedCertificateRotation"
// ReasonQueryTimeoutInvalid when the QueryTimeout can not be parsed.
ReasonQueryTimeoutInvalid LokiStackConditionReason = "ReasonQueryTimeoutInvalid"
)

// PodStatusMap defines the type for mapping pod status to pod name.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing
certified: "false"
containerImage: docker.io/grafana/loki-operator:main-99acb9b
createdAt: "2023-05-23T07:47:18Z"
createdAt: "2023-05-23T07:56:36Z"
description: The Community Loki Operator provides Kubernetes native deployment
and management of Loki and related logging components.
operators.operatorframework.io/builder: operator-sdk-unknown
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ spec:
format: int32
type: integer
queryTimeout:
default: 1m
default: 3m
description: Timeout when querying ingesters or storage
during the execution of a query request.
type: string
Expand Down Expand Up @@ -264,7 +264,7 @@ spec:
format: int32
type: integer
queryTimeout:
default: 1m
default: 3m
description: Timeout when querying ingesters or storage
during the execution of a query request.
type: string
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing
certified: "false"
containerImage: docker.io/grafana/loki-operator:main-99acb9b
createdAt: "2023-05-23T07:47:15Z"
createdAt: "2023-05-23T07:56:34Z"
description: The Community Loki Operator provides Kubernetes native deployment
and management of Loki and related logging components.
operators.operatorframework.io/builder: operator-sdk-unknown
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ spec:
format: int32
type: integer
queryTimeout:
default: 1m
default: 3m
description: Timeout when querying ingesters or storage
during the execution of a query request.
type: string
Expand Down Expand Up @@ -264,7 +264,7 @@ spec:
format: int32
type: integer
queryTimeout:
default: 1m
default: 3m
description: Timeout when querying ingesters or storage
during the execution of a query request.
type: string
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ metadata:
categories: OpenShift Optional, Logging & Tracing
certified: "false"
containerImage: quay.io/openshift-logging/loki-operator:v0.1.0
createdAt: "2023-05-23T07:47:20Z"
createdAt: "2023-05-23T07:56:39Z"
description: |
The Loki Operator for OCP provides a means for configuring and managing a Loki stack for cluster logging.
## Prerequisites and Requirements
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ spec:
format: int32
type: integer
queryTimeout:
default: 1m
default: 3m
description: Timeout when querying ingesters or storage
during the execution of a query request.
type: string
Expand Down Expand Up @@ -264,7 +264,7 @@ spec:
format: int32
type: integer
queryTimeout:
default: 1m
default: 3m
description: Timeout when querying ingesters or storage
during the execution of a query request.
type: string
Expand Down
4 changes: 2 additions & 2 deletions operator/config/crd/bases/loki.grafana.com_lokistacks.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ spec:
format: int32
type: integer
queryTimeout:
default: 1m
default: 3m
description: Timeout when querying ingesters or storage
during the execution of a query request.
type: string
Expand Down Expand Up @@ -247,7 +247,7 @@ spec:
format: int32
type: integer
queryTimeout:
default: 1m
default: 3m
description: Timeout when querying ingesters or storage
during the execution of a query request.
type: string
Expand Down
3 changes: 3 additions & 0 deletions operator/docs/operator/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1553,6 +1553,9 @@ for the ruler is missing.</p>
</tr><tr><td><p>&#34;PendingComponents&#34;</p></td>
<td><p>ReasonPendingComponents when all/some LokiStack components pending dependencies</p>
</td>
</tr><tr><td><p>&#34;ReasonQueryTimeoutInvalid&#34;</p></td>
<td><p>ReasonQueryTimeoutInvalid when the QueryTimeout can not be parsed.</p>
</td>
</tr><tr><td><p>&#34;ReadyComponents&#34;</p></td>
<td><p>ReasonReadyComponents when all LokiStack components are ready to serve traffic.</p>
</td>
Expand Down
11 changes: 11 additions & 0 deletions operator/internal/handlers/lokistack_create_or_update.go
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,16 @@ func CreateOrUpdateLokiStack(
certRotationRequiredAt = stack.Annotations[manifests.AnnotationCertRotationRequiredAt]
}

timeoutConfig, err := manifests.NewTimeoutConfig(stack.Spec.Limits)
if err != nil {
ll.Error(err, "failed to parse query timeout")
return &status.DegradedError{
Message: fmt.Sprintf("Error parsing query timeout: %s", err),
Reason: lokiv1.ReasonQueryTimeoutInvalid,
Requeue: false,
}
}

// Here we will translate the lokiv1.LokiStack options into manifest options
opts := manifests.Options{
Name: req.Name,
Expand All @@ -286,6 +296,7 @@ func CreateOrUpdateLokiStack(
Spec: rulerConfig,
Secret: rulerSecret,
},
Timeouts: timeoutConfig,
Tenants: manifests.Tenants{
Secrets: tenantSecrets,
Configs: tenantConfigs,
Expand Down
72 changes: 72 additions & 0 deletions operator/internal/handlers/lokistack_create_or_update_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1385,6 +1385,78 @@ func TestCreateOrUpdateLokiStack_MissingTenantsSpec_SetDegraded(t *testing.T) {
require.Equal(t, degradedErr, err)
}

func TestCreateOrUpdateLokiStack_WhenInvalidQueryTimeout_SetDegraded(t *testing.T) {
sw := &k8sfakes.FakeStatusWriter{}
k := &k8sfakes.FakeClient{}
r := ctrl.Request{
NamespacedName: types.NamespacedName{
Name: "my-stack",
Namespace: "some-ns",
},
}

degradedErr := &status.DegradedError{
Message: `Error parsing query timeout: time: invalid duration "invalid"`,
Reason: lokiv1.ReasonQueryTimeoutInvalid,
Requeue: false,
}

stack := &lokiv1.LokiStack{
TypeMeta: metav1.TypeMeta{
Kind: "LokiStack",
},
ObjectMeta: metav1.ObjectMeta{
Name: "my-stack",
Namespace: "some-ns",
UID: "b23f9a38-9672-499f-8c29-15ede74d3ece",
},
Spec: lokiv1.LokiStackSpec{
Size: lokiv1.SizeOneXExtraSmall,
Storage: lokiv1.ObjectStorageSpec{
Schemas: []lokiv1.ObjectStorageSchema{
{
Version: lokiv1.ObjectStorageSchemaV12,
EffectiveDate: "2023-05-22",
},
},
Secret: lokiv1.ObjectStorageSecretSpec{
Name: defaultSecret.Name,
Type: lokiv1.ObjectStorageSecretS3,
},
},
Tenants: &lokiv1.TenantsSpec{
Mode: "openshift",
},
Limits: &lokiv1.LimitsSpec{
Global: &lokiv1.LimitsTemplateSpec{
QueryLimits: &lokiv1.QueryLimitSpec{
QueryTimeout: "invalid",
},
},
},
},
}

// Create looks up the CR first, so we need to return our fake stack
k.GetStub = func(_ context.Context, name types.NamespacedName, object client.Object, _ ...client.GetOption) error {
if r.Name == name.Name && r.Namespace == name.Namespace {
k.SetClientObject(object, stack)
}
if defaultSecret.Name == name.Name {
k.SetClientObject(object, &defaultSecret)
}
return nil
}

k.StatusStub = func() client.StatusWriter { return sw }

err := handlers.CreateOrUpdateLokiStack(context.TODO(), logger, r, k, scheme, featureGates)

// make sure error is returned
require.Error(t, err)
require.Equal(t, degradedErr, err)
}

func TestCreateOrUpdateLokiStack_RemovesRulerResourcesWhenDisabled(t *testing.T) {
sw := &k8sfakes.FakeStatusWriter{}
k := &k8sfakes.FakeClient{}
Expand Down
16 changes: 16 additions & 0 deletions operator/internal/manifests/build_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ func TestApplyUserOptions_OverrideDefaults(t *testing.T) {
},
},
},
Timeouts: defaultTimeoutConfig,
}
err := ApplyDefaultSettings(&opt)
defs := internal.StackSizeTable[size]
Expand Down Expand Up @@ -75,6 +76,7 @@ func TestApplyUserOptions_AlwaysSetCompactorReplicasToOne(t *testing.T) {
},
},
},
Timeouts: defaultTimeoutConfig,
}
err := ApplyDefaultSettings(&opt)
defs := internal.StackSizeTable[size]
Expand Down Expand Up @@ -229,6 +231,7 @@ func TestBuildAll_WithFeatureGates_ServiceMonitors(t *testing.T) {
ServingCertsService: false,
},
},
Timeouts: defaultTimeoutConfig,
},
},
{
Expand All @@ -247,6 +250,7 @@ func TestBuildAll_WithFeatureGates_ServiceMonitors(t *testing.T) {
ServingCertsService: false,
},
},
Timeouts: defaultTimeoutConfig,
},
},
}
Expand Down Expand Up @@ -289,6 +293,7 @@ func TestBuildAll_WithFeatureGates_OpenShift_ServingCertsService(t *testing.T) {
ServingCertsService: false,
},
},
Timeouts: defaultTimeoutConfig,
},
},
{
Expand All @@ -306,6 +311,7 @@ func TestBuildAll_WithFeatureGates_OpenShift_ServingCertsService(t *testing.T) {
ServingCertsService: true,
},
},
Timeouts: defaultTimeoutConfig,
},
},
}
Expand Down Expand Up @@ -346,6 +352,7 @@ func TestBuildAll_WithFeatureGates_HTTPEncryption(t *testing.T) {
Gates: configv1.FeatureGates{
HTTPEncryption: true,
},
Timeouts: defaultTimeoutConfig,
}

err := ApplyDefaultSettings(&opts)
Expand Down Expand Up @@ -419,6 +426,7 @@ func TestBuildAll_WithFeatureGates_ServiceMonitorTLSEndpoints(t *testing.T) {
HTTPEncryption: true,
ServiceMonitorTLSEndpoints: true,
},
Timeouts: defaultTimeoutConfig,
}

err := ApplyDefaultSettings(&opts)
Expand Down Expand Up @@ -523,6 +531,7 @@ func TestBuildAll_WithFeatureGates_GRPCEncryption(t *testing.T) {
Gates: configv1.FeatureGates{
GRPCEncryption: false,
},
Timeouts: defaultTimeoutConfig,
},
},
{
Expand Down Expand Up @@ -565,6 +574,7 @@ func TestBuildAll_WithFeatureGates_GRPCEncryption(t *testing.T) {
Gates: configv1.FeatureGates{
GRPCEncryption: true,
},
Timeouts: defaultTimeoutConfig,
},
},
}
Expand Down Expand Up @@ -689,6 +699,7 @@ func TestBuildAll_WithFeatureGates_RuntimeSeccompProfile(t *testing.T) {
Gates: configv1.FeatureGates{
RuntimeSeccompProfile: false,
},
Timeouts: defaultTimeoutConfig,
},
},
{
Expand Down Expand Up @@ -731,6 +742,7 @@ func TestBuildAll_WithFeatureGates_RuntimeSeccompProfile(t *testing.T) {
Gates: configv1.FeatureGates{
RuntimeSeccompProfile: true,
},
Timeouts: defaultTimeoutConfig,
},
},
}
Expand Down Expand Up @@ -794,6 +806,7 @@ func TestBuildAll_WithFeatureGates_LokiStackGateway(t *testing.T) {
HTTPEncryption: true,
ServiceMonitorTLSEndpoints: false,
},
Timeouts: defaultTimeoutConfig,
},
},
{
Expand Down Expand Up @@ -832,6 +845,7 @@ func TestBuildAll_WithFeatureGates_LokiStackGateway(t *testing.T) {
HTTPEncryption: true,
ServiceMonitorTLSEndpoints: true,
},
Timeouts: defaultTimeoutConfig,
},
},
}
Expand Down Expand Up @@ -870,6 +884,7 @@ func TestBuildAll_WithFeatureGates_LokiStackAlerts(t *testing.T) {
ServiceMonitors: false,
LokiStackAlerts: false,
},
Timeouts: defaultTimeoutConfig,
},
},
{
Expand All @@ -884,6 +899,7 @@ func TestBuildAll_WithFeatureGates_LokiStackAlerts(t *testing.T) {
ServiceMonitors: true,
LokiStackAlerts: true,
},
Timeouts: defaultTimeoutConfig,
},
},
}
Expand Down
1 change: 1 addition & 0 deletions operator/internal/manifests/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,7 @@ func ConfigOptions(opt Options) config.Options {
IngesterMemoryRequest: opt.ResourceRequirements.Ingester.Requests.Memory().Value(),
},
ObjectStorage: opt.ObjectStorage,
HTTPTimeouts: opt.Timeouts.Loki,
EnableRemoteReporting: opt.Gates.GrafanaLabsUsageReport,
Ruler: config.Ruler{
Enabled: rulerEnabled,
Expand Down
Loading