Merge branch 'master' into mpuncel/secret-hc-sequence

* master: ci: use multiple stage (envoyproxy#13557) tls: update BoringSSL to 2192bbc8 (4240). (envoyproxy#13567) fix macos v8 build (envoyproxy#13572) Fixed Health Check Fuzz corpus syntax (envoyproxy#13576) ci: Remove shellcheck diff (envoyproxy#13560) ci: Increate brew retry interval (envoyproxy#13565) dependencies: fix some of the fallout from Wasm merge. (envoyproxy#13569) hds: add support for delta updates in specifier (envoyproxy#13067) ci: workaround for actions/runner-images#1811 (envoyproxy#13577) ratelimit: be able to disable x-envoy-ratelimited response header sent (envoyproxy#13270) Update opencensus library (envoyproxy#13549) ci: use azp for api and go-control-plane sync (envoyproxy#13550) docs: Remove/make generic lyft references in docs (envoyproxy#13559) check_format: adding 2 more release note checks (envoyproxy#13444) [Wasm] Add cluster metadata fallback and upstream host metadata (envoyproxy#13477) [fuzz] Added validation for secrets (envoyproxy#13543) Add Platform Specific Feature guidance to PR template (envoyproxy#13547) Signed-off-by: Michael Puncel <mpuncel@squareup.com>
mpuncel · Oct 15, 2020 · 76014af · 76014af
2 parents 2cd1719 + 73d78f8
commit 76014af
Show file tree

Hide file tree

Showing 55 changed files with 1,703 additions and 573 deletions.
diff --git a/.azure-pipelines/pipelines.yml b/.azure-pipelines/pipelines.yml
diff --git a/.circleci/config.yml b/.circleci/config.yml
diff --git a/PULL_REQUESTS.md b/PULL_REQUESTS.md
@@ -70,6 +70,16 @@ current version. Please include any relevant links. Each release note should be
 relevant subsystem in **alphabetical order** (see existing examples as a guide) and include links
 to relevant parts of the documentation. Thank you! Please write in N/A if there are no release notes.
 
+### <a name="platform_specific_features"></a>Platform Specific Features
+
+If this change involves any platform specific features (e.g. utilizing OS-specific socket options)
+or only implements new features for a limited set of platforms (e.g. Linux amd64 only), please
+include an explanation that addresses the reasoning behind this. Please also open a new tracking
+issue for each platform this change is not implemented on (and link them in the PR) to enable
+maintainers and contributors to triage. Reviewers will look for the change to avoid
+`#ifdef <OSNAME>` and rather prefer feature guards to not enable the change on a given platform
+using the build system.
+
 ### <a name="runtime_guard"></a>Runtime guard
 
 If this PR has a user-visible behavioral change, or otherwise falls under the

diff --git a/PULL_REQUEST_TEMPLATE.md b/PULL_REQUEST_TEMPLATE.md
@@ -18,6 +18,7 @@ Risk Level:
 Testing:
 Docs Changes:
 Release Notes:
+Platform Specific Features:
 [Optional Runtime guard:]
 [Optional Fixes #Issue]
 [Optional Deprecated:]
diff --git a/api/envoy/extensions/filters/http/ratelimit/v3/rate_limit.proto b/api/envoy/extensions/filters/http/ratelimit/v3/rate_limit.proto
@@ -19,7 +19,7 @@ option (udpa.annotations.file_status).package_version_status = ACTIVE;
 // Rate limit :ref:`configuration overview <config_http_filters_rate_limit>`.
 // [#extension: envoy.filters.http.ratelimit]
 
-// [#next-free-field: 9]
+// [#next-free-field: 10]
 message RateLimit {
   option (udpa.annotations.versioning).previous_message_type =
       "envoy.config.filter.http.rate_limit.v2.RateLimit";
@@ -60,7 +60,6 @@ message RateLimit {
   // The filter's behaviour in case the rate limiting service does
   // not respond back. When it is set to true, Envoy will not allow traffic in case of
   // communication failure between rate limiting service and the proxy.
-  // Defaults to false.
   bool failure_mode_deny = 5;
 
   // Specifies whether a `RESOURCE_EXHAUSTED` gRPC code must be returned instead
@@ -99,6 +98,11 @@ message RateLimit {
   // Disabled by default.
   XRateLimitHeadersRFCVersion enable_x_ratelimit_headers = 8
       [(validate.rules).enum = {defined_only: true}];
+
+  // Disables emitting the :ref:`x-envoy-ratelimited<config_http_filters_router_x-envoy-ratelimited>` header
+  // in case of rate limiting (i.e. 429 responses).
+  // Having this header not present potentially makes the request retriable.
+  bool disable_x_envoy_ratelimited_header = 9;
 }
 
 message RateLimitPerRoute {

diff --git a/api/envoy/extensions/transport_sockets/tls/v3/secret.proto b/api/envoy/extensions/transport_sockets/tls/v3/secret.proto
@@ -12,6 +12,7 @@ import "udpa/annotations/migrate.proto";
 import "udpa/annotations/sensitive.proto";
 import "udpa/annotations/status.proto";
 import "udpa/annotations/versioning.proto";
+import "validate/validate.proto";
 
 option java_package = "io.envoyproxy.envoy.extensions.transport_sockets.tls.v3";
 option java_outer_classname = "SecretProto";
@@ -33,7 +34,10 @@ message SdsSecretConfig {
   // Name (FQDN, UUID, SPKI, SHA256, etc.) by which the secret can be uniquely referred to.
   // When both name and config are specified, then secret can be fetched and/or reloaded via
   // SDS. When only name is specified, then secret will be loaded from static resources.
-  string name = 1 [(udpa.annotations.field_migrate).oneof_promotion = "name_specifier"];
+  string name = 1 [
+    (validate.rules).string = {min_len: 1},
+    (udpa.annotations.field_migrate).oneof_promotion = "name_specifier"
+  ];
 
   // Resource locator for SDS. This is mutually exclusive to *name*.
   // [#not-implemented-hide:]

diff --git a/api/envoy/extensions/transport_sockets/tls/v4alpha/secret.proto b/api/envoy/extensions/transport_sockets/tls/v4alpha/secret.proto
diff --git a/bazel/external/wee8.BUILD b/bazel/external/wee8.BUILD
@@ -13,6 +13,9 @@ cc_library(
         "wee8/include/v8-version.h",
         "wee8/third_party/wasm-api/wasm.hh",
     ],
+    copts = [
+        "-Wno-range-loop-analysis",
+    ],
     defines = ["ENVOY_WASM_V8"],
     includes = [
         "wee8/include",

diff --git a/bazel/external/wee8.genrule_cmd b/bazel/external/wee8.genrule_cmd
@@ -19,7 +19,7 @@ pushd $$ROOT/wee8
 rm -rf out/wee8
 
 # Export compiler configuration.
-export CXXFLAGS="$${CXXFLAGS-} -Wno-sign-compare -Wno-deprecated-copy -Wno-unknown-warning-option"
+export CXXFLAGS="$${CXXFLAGS-} -Wno-sign-compare -Wno-deprecated-copy -Wno-unknown-warning-option -Wno-range-loop-analysis"
 if [[ ( `uname` == "Darwin" && $${CXX-} == "" ) || $${CXX-} == *"clang"* ]]; then
   export IS_CLANG=true
   export CC=$${CC:-clang}

diff --git a/bazel/repositories.bzl b/bazel/repositories.bzl
@@ -351,31 +351,10 @@ def _com_github_zlib_ng_zlib_ng():
 def _com_google_cel_cpp():
     external_http_archive("com_google_cel_cpp")
     external_http_archive("rules_antlr")
-    external_http_archive(
-        name = "antlr4_runtimes",
-        build_file_content = """
-package(default_visibility = ["//visibility:public"])
-cc_library(
-    name = "cpp",
-    srcs = glob(["runtime/Cpp/runtime/src/**/*.cpp"]),
-    hdrs = glob(["runtime/Cpp/runtime/src/**/*.h"]),
-    includes = ["runtime/Cpp/runtime/src"],
-)
-""",
-        patch_args = ["-p1"],
-        # Patches ASAN violation of initialization fiasco
-        patches = ["@envoy//bazel:antlr.patch"],
-    )
 
     # Parser dependencies
     # TODO: upgrade this when cel is upgraded to use the latest version
-    external_http_archive(
-        name = "rules_antlr",
-        sha256 = "7249d1569293d9b239e23c65f6b4c81a07da921738bde0dfeb231ed98be40429",
-        strip_prefix = "rules_antlr-3cc2f9502a54ceb7b79b37383316b23c4da66f9a",
-        urls = ["https://github.com/marcohu/rules_antlr/archive/3cc2f9502a54ceb7b79b37383316b23c4da66f9a.tar.gz"],
-    )
-
+    external_http_archive(name = "rules_antlr")
     external_http_archive(
         name = "antlr4_runtimes",
         build_file_content = """
@@ -387,12 +366,9 @@ cc_library(
     includes = ["runtime/Cpp/runtime/src"],
 )
 """,
-        sha256 = "46f5e1af5f4bd28ade55cb632f9a069656b31fc8c2408f9aa045f9b5f5caad64",
         patch_args = ["-p1"],
         # Patches ASAN violation of initialization fiasco
         patches = ["@envoy//bazel:antlr.patch"],
-        strip_prefix = "antlr4-4.7.2",
-        urls = ["https://github.com/antlr/antlr4/archive/4.7.2.tar.gz"],
     )
 
 def _com_github_nghttp2_nghttp2():

diff --git a/bazel/repository_locations.bzl b/bazel/repository_locations.bzl
@@ -60,18 +60,18 @@ REPOSITORY_LOCATIONS_SPEC = dict(
         project_name = "BoringSSL",
         project_desc = "Minimal OpenSSL fork",
         project_url = "https://github.com/google/boringssl",
-        version = "597b810379e126ae05d32c1d94b1a9464385acd0",
-        sha256 = "1ea42456c020daf0a9b0f9e8d8bc3a403c9314f4f54230c617257af996cd5fa6",
+        version = "2192bbc878822cf6ab5977d4257a1339453d9d39",
+        sha256 = "bb55b0ed2f0cb548b5dce6a6b8307ce37f7f748eb9f1be6bfe2d266ff2b4d52b",
         strip_prefix = "boringssl-{version}",
         # To update BoringSSL, which tracks Chromium releases:
         # 1. Open https://omahaproxy.appspot.com/ and note <current_version> of linux/stable release.
         # 2. Open https://chromium.googlesource.com/chromium/src/+/refs/tags/<current_version>/DEPS and note <boringssl_revision>.
         # 3. Find a commit in BoringSSL's "master-with-bazel" branch that merges <boringssl_revision>.
         #
-        # chromium-85.0.4183.83
+        # chromium-86.0.4240.80
         urls = ["https://github.com/google/boringssl/archive/{version}.tar.gz"],
         use_category = ["controlplane", "dataplane_core"],
-        last_updated = "2020-06-23",
+        last_updated = "2020-07-30",
         cpe = "cpe:2.3:a:google:boringssl:*",
     ),
     boringssl_fips = dict(
@@ -611,13 +611,13 @@ REPOSITORY_LOCATIONS_SPEC = dict(
         project_name = "OpenCensus C++",
         project_desc = "OpenCensus tracing library",
         project_url = "https://github.com/census-instrumentation/opencensus-cpp",
-        version = "7877337633466358ed680f9b26967da5b310d7aa",
-        sha256 = "12ff300fa804f97bd07e2ff071d969e09d5f3d7bbffeac438c725fa52a51a212",
+        version = "ba631066779a534267fdb1321b19850eb2b0c000",
+        sha256 = "f239a40803f6e2e42b57c9e68771b0990c4ca8b2d76b440073cdf14f4211ad26",
         strip_prefix = "opencensus-cpp-{version}",
         urls = ["https://github.com/census-instrumentation/opencensus-cpp/archive/{version}.tar.gz"],
         use_category = ["observability_ext"],
         extensions = ["envoy.tracers.opencensus"],
-        last_updated = "2020-06-01",
+        last_updated = "2020-10-13",
         cpe = "N/A",
     ),
     # This should be removed, see https://github.com/envoyproxy/envoy/issues/11816.
@@ -859,7 +859,7 @@ REPOSITORY_LOCATIONS_SPEC = dict(
         # See: https://github.com/bazelbuild/rules_rust/issues/386
         strip_prefix = "rules_rust-{version}",
         urls = ["https://github.com/bazelbuild/rules_rust/archive/{version}.tar.gz"],
-        use_category = ["build"],
+        use_category = ["test_only"],
         last_updated = "2020-10-09",
     ),
     rules_antlr = dict(
@@ -886,8 +886,8 @@ REPOSITORY_LOCATIONS_SPEC = dict(
         project_name = "ANTLR v4",
         project_desc = "ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files",
         project_url = "https://github.com/antlr/antlr4",
-        version = "4.7.1",
-        sha256 = "4d0714f441333a63e50031c9e8e4890c78f3d21e053d46416949803e122a6574",
+        version = "4.7.2",
+        sha256 = "46f5e1af5f4bd28ade55cb632f9a069656b31fc8c2408f9aa045f9b5f5caad64",
         strip_prefix = "antlr4-{version}",
         urls = ["https://github.com/antlr/antlr4/archive/{version}.tar.gz"],
         use_category = ["dataplane_ext"],
@@ -898,7 +898,7 @@ REPOSITORY_LOCATIONS_SPEC = dict(
             "envoy.filters.network.wasm",
             "envoy.stat_sinks.wasm",
         ],
-        last_updated = "2020-07-29",
+        last_updated = "2020-10-09",
         cpe = "N/A",
     ),
 )
diff --git a/ci/api_mirror.sh b/ci/api_mirror.sh
@@ -3,16 +3,15 @@
 set -e
 
 CHECKOUT_DIR=../data-plane-api
+MAIN_BRANCH="refs/heads/master"
+API_MAIN_BRANCH="master"
 
-if [ -z "$CIRCLE_PULL_REQUEST" ] && [ "$CIRCLE_BRANCH" == "master" ]
-then
+if [[ "${AZP_BRANCH}" == "${MAIN_BRANCH}" ]]; then
   echo "Cloning..."
-  git clone git@github.com:envoyproxy/data-plane-api "$CHECKOUT_DIR"
+  git clone git@github.com:envoyproxy/data-plane-api "$CHECKOUT_DIR" -b "${API_MAIN_BRANCH}"
 
-  git -C "$CHECKOUT_DIR" config user.name "data-plane-api(CircleCI)"
+  git -C "$CHECKOUT_DIR" config user.name "data-plane-api(Azure Pipelines)"
   git -C "$CHECKOUT_DIR" config user.email data-plane-api@users.noreply.github.com
-  git -C "$CHECKOUT_DIR" fetch
-  git -C "$CHECKOUT_DIR" checkout -B master origin/master
 
   # Determine last envoyproxy/envoy SHA in envoyproxy/data-plane-api
   MIRROR_MSG="Mirrored from https://github.com/envoyproxy/envoy"
@@ -40,6 +39,6 @@ then
   done
 
   echo "Pushing..."
-  git -C "$CHECKOUT_DIR" push origin master
+  git -C "$CHECKOUT_DIR" push origin "${API_MAIN_BRANCH}"
   echo "Done"
 fi
diff --git a/ci/do_ci.sh b/ci/do_ci.sh
@@ -312,19 +312,21 @@ elif [[ "$CI_TARGET" == "bazel.compile_time_options" ]]; then
   collect_build_profile build
   exit 0
 elif [[ "$CI_TARGET" == "bazel.api" ]]; then
+  # Use libstdc++ because the API booster links to prebuilt libclang*/libLLVM* installed in /opt/llvm/lib,
+  # which is built with libstdc++. Using libstdc++ for whole of the API CI job to avoid unnecessary rebuild.
+  ENVOY_STDLIB="libstdc++"
   setup_clang_toolchain
+  export LLVM_CONFIG="${LLVM_ROOT}"/bin/llvm-config
   echo "Validating API structure..."
   ./tools/api/validate_structure.py
+  echo "Testing API and API Boosting..."
+  bazel_with_collection test "${BAZEL_BUILD_OPTIONS[@]}" -c fastbuild @envoy_api_canonical//test/... @envoy_api_canonical//tools/... \
+    @envoy_api_canonical//tools:tap2pcap_test @envoy_dev//clang_tools/api_booster/...
   echo "Building API..."
   bazel build "${BAZEL_BUILD_OPTIONS[@]}" -c fastbuild @envoy_api_canonical//envoy/...
-  echo "Testing API..."
-  bazel_with_collection test "${BAZEL_BUILD_OPTIONS[@]}" -c fastbuild @envoy_api_canonical//test/... @envoy_api_canonical//tools/... \
-    @envoy_api_canonical//tools:tap2pcap_test
-  echo "Testing API boosting (unit tests)..."
-  bazel_with_collection test "${BAZEL_BUILD_OPTIONS[@]}" -c fastbuild @envoy_dev//clang_tools/api_booster/...
   echo "Testing API boosting (golden C++ tests)..."
   # We use custom BAZEL_BUILD_OPTIONS here; the API booster isn't capable of working with libc++ yet.
-  LLVM_CONFIG="${LLVM_ROOT}"/bin/llvm-config BAZEL_BUILD_OPTIONS="--config=clang" python3.8 ./tools/api_boost/api_boost_test.py
+  BAZEL_BUILD_OPTIONS="${BAZEL_BUILD_OPTIONS[*]}" python3.8 ./tools/api_boost/api_boost_test.py
   exit 0
 elif [[ "$CI_TARGET" == "bazel.coverage" || "$CI_TARGET" == "bazel.fuzz_coverage" ]]; then
   setup_clang_toolchain

diff --git a/ci/go_mirror.sh b/ci/go_mirror.sh
@@ -2,7 +2,11 @@
 
 set -e
 
-if [ -z "$CIRCLE_PULL_REQUEST" ] && [ "$CIRCLE_BRANCH" == "master" ]
-then
-  tools/api/generate_go_protobuf.py
+MAIN_BRANCH="refs/heads/master"
+
+# shellcheck source=ci/setup_cache.sh
+. "$(dirname "$0")"/setup_cache.sh
+
+if [[ "${AZP_BRANCH}" == "${MAIN_BRANCH}" ]]; then
+  BAZEL_BUILD_OPTIONS="${BAZEL_BUILD_EXTRA_OPTIONS}" tools/api/generate_go_protobuf.py
 fi
diff --git a/ci/mac_ci_setup.sh b/ci/mac_ci_setup.sh
@@ -6,9 +6,12 @@
 # https://github.com/actions/virtual-environments/blob/master/images/macos/macos-10.15-Readme.md for
 # a list of pre-installed tools in the macOS image.
 
+# https://github.com/actions/virtual-environments/issues/1811
+brew uninstall openssl@1.0.2t
+
 export HOMEBREW_NO_AUTO_UPDATE=1
 HOMEBREW_RETRY_ATTEMPTS=10
-HOMEBREW_RETRY_INTERVAL=1
+HOMEBREW_RETRY_INTERVAL=3
 
 
 function is_installed {

diff --git a/ci/run_envoy_docker.sh b/ci/run_envoy_docker.sh
@@ -56,6 +56,7 @@ mkdir -p "${ENVOY_DOCKER_BUILD_DIR}"
 
 [[ -t 1 ]] && ENVOY_DOCKER_OPTIONS+=("-it")
 [[ -f .git ]] && [[ ! -d .git ]] && ENVOY_DOCKER_OPTIONS+=(-v "$(git rev-parse --git-common-dir):$(git rev-parse --git-common-dir)")
+[[ -n "${SSH_AUTH_SOCK}" ]] && ENVOY_DOCKER_OPTIONS+=(-v "${SSH_AUTH_SOCK}:${SSH_AUTH_SOCK}" -e SSH_AUTH_SOCK)
 
 export ENVOY_BUILD_IMAGE="${IMAGE_NAME}:${IMAGE_ID}"
 

diff --git a/docs/root/configuration/http/http_filters/rate_limit_filter.rst b/docs/root/configuration/http/http_filters/rate_limit_filter.rst
@@ -14,7 +14,9 @@ can optionally include the virtual host rate limit configurations. More than one
 apply to a request. Each configuration results in a descriptor being sent to the rate limit service.
 
 If the rate limit service is called, and the response for any of the descriptors is over limit, a
-429 response is returned. The rate limit filter also sets the :ref:`x-envoy-ratelimited<config_http_filters_router_x-envoy-ratelimited>` header.
+429 response is returned. The rate limit filter also sets the :ref:`x-envoy-ratelimited<config_http_filters_router_x-envoy-ratelimited>` header,
+unless :ref:`disable_x_envoy_ratelimited_header <envoy_v3_api_field_extensions.filters.http.ratelimit.v3.RateLimit.disable_x_envoy_ratelimited_header>` is
+set to true.
 
 If there is an error in calling rate limit service or rate limit service returns an error and :ref:`failure_mode_deny <envoy_v3_api_field_extensions.filters.http.ratelimit.v3.RateLimit.failure_mode_deny>` is 
 set to true, a 500 response is returned.

diff --git a/docs/root/configuration/other_features/rate_limit.rst b/docs/root/configuration/other_features/rate_limit.rst
@@ -14,5 +14,5 @@ gRPC service IDL
 
 Envoy expects the rate limit service to support the gRPC IDL specified in
 :ref:`rls.proto <envoy_v3_api_file_envoy/service/ratelimit/v3/rls.proto>`. See the IDL documentation
-for more information on how the API works. See Lyft's reference implementation
-`here <https://github.com/lyft/ratelimit>`_.
+for more information on how the API works. See Envoy's reference implementation
+`here <https://github.com/envoyproxy/ratelimit>`_.
diff --git a/docs/root/intro/arch_overview/other_features/global_rate_limiting.rst b/docs/root/intro/arch_overview/other_features/global_rate_limiting.rst
@@ -14,7 +14,7 @@ normally during typical request patterns but still prevent cascading failure whe
 to fail. Global rate limiting is a good solution for this case.
 
 Envoy integrates directly with a global gRPC rate limiting service. Although any service that
-implements the defined RPC/IDL protocol can be used, Lyft provides a `reference implementation <https://github.com/lyft/ratelimit>`_
+implements the defined RPC/IDL protocol can be used, Envoy provides a `reference implementation <https://github.com/envoyproxy/ratelimit>`_
 written in Go which uses a Redis backend. Envoy’s rate limit integration has the following features:
 
 * **Network level rate limit filter**: Envoy will call the rate limit service for every new