[Proposal] Enabling efficient fuzzing process of CNCF projects

[DavidKorczynski]

Description: what's your idea?

In short, this proposal is about making it easier to fuzz CNCF projects from a management perspective.

In the last year we (Ada Logics) have been doing a fair amount of work related to fuzzing CNCF projects. The goal is most often to integrate fuzzers into a given project and then have these fuzzers run by way of OSS-Fuzz, in order to ensure the projects are fuzzed continuously as opposed to us running them on an ad-hoc basis.

This has been very successful in terms of assisting with bug finding. Some CNCF projects that are being fuzzed are for example (notice some issues will be false positives, e.g. due to fuzzer implementation issues, and these are often labelled "WontFix" in the search URLs below):

• Fluent bit - more than 150 issues reported https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj%3Dfluent-bit%20Type%3DBug-Security%2CBug&can=1

• Vitess - more than 50 issues reported https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj%3Dvitess%20Type%3DBug-Security%2CBug&can=1

• Envoy - more than 800 issues reported https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj%3Denvoy%20Type%3DBug-Security%2CBug&can=1

• Kubernetes - more than 40 issues reported (this is with very limited fuzzing efforts) https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj%3Dkubernetes%20Type%3DBug-Security%2CBug&can=1

• Linkerd2-proxy - 15 issues reported https://bugs.chromium.org/p/oss-fuzz/issues/list?q=proj%3Dlinkerd2-proxy%20Type%3DBug-Security%2CBug&can=1

The security relevance varies amongst the issues reported above and many of them will be reliability issues. However, this is in many ways only the tip of the iceberg and there is room for a lot more work.

FYI you can see links to fuzzing reports here

However, we have also observed a few issues when doing this work, in particular issues with managing the fuzzers, e.g. support and engagement from maintainers. The problem is that there is often a long review time or simply lack of contacts to projects, e.g. Kubernetes, that can be on the receiving end of bug reports. For example,

• RunC integration has been stale for a few months: Fuzzing: Add new fuzzer opencontainers/runc#2848
• Distribution for a few months: Fuzzing: Add 3 fuzzers distribution/distribution#3458
• Kubernetes: Add parser fuzzers kubernetes/kubernetes#104772 Notice that Kubernetes was integrated into OSS-Fuzz two years ago (https://github.com/google/oss-fuzz/blob/master/projects/kubernetes/project.yaml) but has not really had much (any?) progress since then, and only a single email is listed as a recipient of bug reports - thus the fuzzers have now run continuously for two years but no one is responding to the reports.

In essence, the issue here is that a fair number of CNCF projects are not being fuzzed but there is interest in increasing fuzzing of these projects yet there are some blockers in getting the projects fuzzed.

Notice that I am not taking a jab at any of the projects with a long review time - I understand that many maintainers have a ton of focus elsewhere and that’s prioritised. In fact, one of the goals of our work is to make it easy for maintainers to use fuzzing, and having an approach to fuzzing CNCF projects with little overhead on the Maintainers’ availability is ideal. In addition to this, integrating fuzzing is a time-intensive task and can be a significant review effort for maintainers. However, it’s a shame that we can’t get the fuzzers to run for these projects.

The goal of the work we have done, which has been in collaboration with the Linux Foundation, is to ensure projects are being fuzzed continuously. However, for some projects it is difficult to do this due to non-technical reasons.

I am essentially putting this issue up to discuss options for an improved set up that allows easier fuzzing of CNCF projects. The current approach we have taken is to engage projects on an individual basis. However, there are limitations to this approach in terms of the fuzzer-development throughput and also lack of information about fuzzing from maintainers.

There are probably many reasonable ways to improve on this. One possible solution is to create a centralised CNCF-fuzzing repository where maintenance, documentation and fuzzer-development can take place. Such a repository can be used in various ways, e.g.

• Maintain and develop fuzzers for CNCF projects without needing to get them upstream. We can then have OSS-Fuzz download fuzzers from this repository for running the fuzzers continuously. Maintainers can migrate the fuzzers at their discrepancy but the fuzzers will be run continuously before they are migrated. The only thing needed from a developer’s perspective is an email that will receive the bug reports.
• Hold documentation and tutorials on fuzzing CNCF projects
• Provide fuzzing-related assistance through Github issues
• Track fuzzing of all CNCF projects
• Maintainers can ask for their projects to be fuzzed through this repo. This is currently happening sometimes within projects, e.g. Make fuzzing easy kubernetes-sigs/controller-runtime#456
• Centralise highlights from fuzzing CNCF projects as a way of engaging developers to use fuzzing
• Propose innovation in terms of fuzzing CNCF projects, e.g. we have recently worked on fuzzing custom Kubernetes controllers and work could be done there in terms of infrastructure to make this easier Fuzzing: Initial set up fluxcd/source-controller#443

Having a central repository for fuzzing does come with tradeoffs. For example, if the fuzzers developed treat the code they target in a wrong manner, then it may be that false positives will occur and developers will be flooded with issues that are irrelevant. This can be a discouragement and there should be significant efforts in developing accurate fuzzers.

This proposal was a dual effort with @AdamKorcz

---

Impact: Describe the customer impact of the problem. Who will this help? How will it help them?

Customers are CNCF projects. It will help these projects to become more security by way large-scale continuous fuzzing.

Scope: How much effort will this take? ok to provide a range of options if or "not yet determined" for initial proposals. Feel free to include proposed tasks below or link a Google doc

This will be a mix of ongoing efforts in terms of maintenance, and also initial efforts on a project-level basis.

TO DO

• Security TAG Leadership Representative:
• Project leader(s):
• Project Members:
• Fill in addition TODO items here so the project team and community can see progress!
• Scope
• Deliverable(s)
• Project Schedule
• Slack Channel (as needed)
• Meeting Time & Day:
• Meeting Notes (link)
• Meeting Details (zoom or hangouts link)
• Retrospective

[caniszczyk]

As an FYI @cncf/tag-security folks, CNCF is supporting fuzzing work for projects and we think that maybe TAG Security is a good home for some of this work + discussion.

[stale]

This issue has been automatically marked as inactive because it has not had recent activity.

[lumjjb]

We are planning to discuss this issue on the March 2nd meeting if you can be there, that would be helpful! @DavidKorczynski

[stale]

This issue has been automatically marked as inactive because it has not had recent activity.

[PushkarJ]

Related #1080 as the book will empower folks to adopt fuzzing

[mnm678]

Closing in favor of #1080 for now