Skip to content

Commit

Permalink
tetra/debug: Clone GetDebugResponse entries
Browse files Browse the repository at this point in the history
When running command "tetra debug dump processcache", we get sometimes
the following error:

Error: failed to get process dump debug info: rpc error: code = Internal
desc = grpc: error while marshaling: marshaling tetragon.GetDebugResponse:
size mismatch (see golang/protobuf#1609): calculated=1240,
measured=1269

By checking the link provided in the error message, we can see that the
most possible cause of the error is accidental sharing and concurrent mutation
of a Protobuf message or submessage.

To avoid that issue, this patch clones mutable submessages.

We managed to reproduce that in a server with heavy load. This patch
seems to solve that issue.

Signed-off-by: Anastasios Papagiannis <anastasios.papagiannis@isovalent.com>
  • Loading branch information
tpapagian committed Jan 28, 2025
1 parent 2c859b1 commit b76687b
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions pkg/process/cache.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package process

import (
"fmt"
"maps"
"path/filepath"
"sync/atomic"
"time"
Expand All @@ -15,6 +16,7 @@ import (
"github.com/cilium/tetragon/pkg/logger"
"github.com/cilium/tetragon/pkg/sensors/exec/execvemap"
lru "github.com/hashicorp/golang-lru/v2"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/known/wrapperspb"
)

Expand Down Expand Up @@ -221,9 +223,9 @@ func (pc *Cache) dump(opts *tetragon.DumpProcessCacheReqArgs) []*tetragon.Proces
}
}
processes = append(processes, &tetragon.ProcessInternal{
Process: v.process,
Process: proto.Clone(v.process).(*tetragon.Process),
Refcnt: &wrapperspb.UInt32Value{Value: v.refcnt},
RefcntOps: v.refcntOps,
RefcntOps: maps.Clone(v.refcntOps),
Color: colorStr[v.color],
})
}
Expand Down

0 comments on commit b76687b

Please sign in to comment.