Skip to content

Commit

Permalink
Invalid length of container termination message
Browse files Browse the repository at this point in the history
Tekton use container termination message of kubenetes to pass additional information to the controller(by Tekton Result).
Kubenates use `MaxContainerTerminationMessageLength` which is 4096 to limit the read size from disk, the redundant content will be discard.
  • Loading branch information
vincent-pli authored and tekton-robot committed May 28, 2020
1 parent 9fd0908 commit f60cff9
Show file tree
Hide file tree
Showing 4 changed files with 52 additions and 5 deletions.
4 changes: 4 additions & 0 deletions cmd/entrypoint/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (

"github.com/tektoncd/pipeline/pkg/credentials"
"github.com/tektoncd/pipeline/pkg/entrypoint"
"github.com/tektoncd/pipeline/pkg/termination"
)

var (
Expand Down Expand Up @@ -66,6 +67,9 @@ func main() {
case skipError:
log.Print("Skipping step because a previous step failed")
os.Exit(1)
case termination.MessageLengthError:
log.Print(err.Error())
os.Exit(1)
case *exec.ExitError:
// Copied from https://stackoverflow.com/questions/10385551/get-exit-code-go
// This works on both Unix and Windows. Although
Expand Down
10 changes: 5 additions & 5 deletions docs/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -423,16 +423,16 @@ spec:
The stored results can be used [at the `Task` level](./pipelines.md#configuring-execution-results-at-the-task-level)
or [at the `Pipeline` level](./pipelines.md#configuring-execution-results-at-the-pipeline-level).

**Note:** The maximum size of a `Task's` results is limited by the container termination log feature of Kubernetes,
**Note:** The maximum size of a `Task's` results is limited by the container termination message feature of Kubernetes,
as results are passed back to the controller via this mechanism. At present, the limit is
["2048 bytes or 80 lines, whichever is smaller."](https://kubernetes.io/docs/tasks/debug-application-cluster/determine-reason-pod-failure/#customizing-the-termination-message).
Results are written to the termination log encoded as JSON objects and Tekton uses those objects
["4096 bytes"](https://github.com/kubernetes/kubernetes/blob/96e13de777a9eb57f87889072b68ac40467209ac/pkg/kubelet/container/runtime.go#L632).
Results are written to the termination message encoded as JSON objects and Tekton uses those objects
to pass additional information to the controller. As such, `Task` results are best suited for holding
small amounts of data, such as commit SHAs, branch names, ephemeral namespaces, and so on.

If your `Task` writes a large number of small results, you can work around this limitation
by writing each result from a separate `Step` so that each `Step` has its own termination log.
However, for results larger than a kilobyte, use a [`Workspace`](#specifying-workspaces) to
by writing each result from a separate `Step` so that each `Step` has its own termination message.
About size limitation, there is validation for it, will raise exception: `Termination message is above max allowed size 4096, caused by large task result`. Since Tekton also uses the termination message for some internal information, so the real available size will less than 4096 bytes. For results larger than a kilobyte, use a [`Workspace`](#specifying-workspaces) to
shuttle data between `Tasks` within a `Pipeline`.

### Specifying `Volumes`
Expand Down
22 changes: 22 additions & 0 deletions pkg/termination/write.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,12 @@ import (
v1alpha1 "github.com/tektoncd/pipeline/pkg/apis/pipeline/v1alpha1"
)

const (
// MaxContainerTerminationMessageLength is the upper bound any one container may write to
// its termination message path. Contents above this length will cause a failure.
MaxContainerTerminationMessageLength = 1024 * 4
)

// WriteMessage writes the results to the termination message path.
func WriteMessage(path string, pro []v1alpha1.PipelineResourceResult) error {
// if the file at path exists, concatenate the new values otherwise create it
Expand All @@ -41,6 +47,11 @@ func WriteMessage(path string, pro []v1alpha1.PipelineResourceResult) error {
if err != nil {
return err
}

if len(jsonOutput) > MaxContainerTerminationMessageLength {
return aboveMax
}

f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE, 0666)
if err != nil {
return err
Expand All @@ -55,3 +66,14 @@ func WriteMessage(path string, pro []v1alpha1.PipelineResourceResult) error {
}
return nil
}

//MessageLengthError indicate the length of termination message of container is beyond 4096 which is the max length read by kubenates
type MessageLengthError string

const (
aboveMax MessageLengthError = "Termination message is above max allowed size 4096, caused by large task result."
)

func (e MessageLengthError) Error() string {
return string(e)
}
21 changes: 21 additions & 0 deletions pkg/termination/write_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,11 @@ limitations under the License.
package termination

import (
"errors"
"io/ioutil"
"log"
"os"
"strings"
"testing"

"github.com/google/go-cmp/cmp"
Expand Down Expand Up @@ -66,3 +68,22 @@ func TestExistingFile(t *testing.T) {
}
}
}

func TestMaxSizeFile(t *testing.T) {
value := strings.Repeat("a", 4096)
tmpFile, err := ioutil.TempFile(os.TempDir(), "tempFile")
if err != nil {
log.Fatal("Cannot create temporary file", err)
}
// Remember to clean up the file afterwards
defer os.Remove(tmpFile.Name())

output := []v1alpha1.PipelineResourceResult{{
Key: "key1",
Value: value,
}}

if err := WriteMessage(tmpFile.Name(), output); !errors.Is(err, aboveMax) {
t.Fatalf("Expected MessageLengthError, receved: %v", err)
}
}

0 comments on commit f60cff9

Please sign in to comment.