Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lua filter causes a crash when the script uses streamInfo():downstreamSslConnection() #14091

Closed
MarcinFalkowski opened this issue Nov 19, 2020 · 1 comment · Fixed by #14092
Closed

Lua filter causes a crash when the script uses streamInfo():downstreamSslConnection() #14091

MarcinFalkowski opened this issue Nov 19, 2020 · 1 comment · Fixed by #14092
Labels
area/lua bug help wanted Needs help!
Milestone
1.17.0

Comments

@MarcinFalkowski
Copy link
Contributor

MarcinFalkowski commented Nov 19, 2020 •
edited
Loading

Envoy crashes with segmentation fault, when Lua script contains streamInfo():downstreamSslConnection().
The crash happens after issuing many https requests.

The bug exists in v1.16.0 as well as in the latest master version.

I have a permission from envoy-security@googlegroups.com for fixing this publicly.

Repro steps:
Run envoy with attached config and make 1000 https requests.

Config:

Click to expand! 
static_resources:
  listeners:
    - name: listener_0
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 10000
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress_http
                http_filters:
                  - name: envoy.filters.http.lua
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
                      inline_code: |
                        function envoy_on_request(handle)
                          if handle:connection():ssl() ~= nil then
                            if handle:streamInfo() ~= nil then
                              if handle:streamInfo():downstreamSslConnection() ~= nil then
                                handle:logInfo("lua - envoy_on_request - SSL; streamInfo == present; downstreamSslConnection == present")
                              else
                                handle:logInfo("lua - envoy_on_request - SSL; streamInfo == present; downstreamSslConnection == nil")
                              end
                            else
                              handle:logInfo("lua - envoy_on_request - SSL; streamInfo == nil")
                            end
                          else
                            handle:logInfo("lua - envoy_on_request - RAW BUFFER")
                          end
                        end
                  - name: envoy.filters.http.router
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ["*"]
                      routes:
                        - match:
                            prefix: "/"
                          direct_response:
                            status: 200
                            body:
                              inline_string: "Envoy direct response"
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                  certificate_chain:
                    filename: ./certs/rsa-2048-signed-rsa-server/cert.crt
                  private_key:
                    filename: ./certs/rsa-2048-signed-rsa-server/key.prv

Call Stack:

Click to expand! 
Thread 11 "wrk:worker_2" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff280f700 (LWP 71)]
0x00005555567d8ef0 in lua_rawgeti ()
#0  0x00005555567d8ef0 in lua_rawgeti ()
#1  0x000055555682905c in luaL_unref ()
#2  0x00005555567b8790 in Envoy::Extensions::Filters::Common::Lua::LuaRef<Envoy::Extensions::Filters::Common::Lua::SslConnectionWrapper>::unref (
    this=<optimized out>)
    at bazel-out/k8-opt/bin/source/extensions/filters/common/lua/_virtual_includes/lua_lib/extensions/filters/common/lua/lua.h:277
#3  Envoy::Extensions::Filters::Common::Lua::LuaRef<Envoy::Extensions::Filters::Common::Lua::SslConnectionWrapper>::~LuaRef (this=<optimized out>)
    at bazel-out/k8-opt/bin/source/extensions/filters/common/lua/_virtual_includes/lua_lib/extensions/filters/common/lua/lua.h:239
#4  Envoy::Extensions::Filters::Common::Lua::LuaDeathRef<Envoy::Extensions::Filters::Common::Lua::SslConnectionWrapper>::~LuaDeathRef (this=<optimized out>)
    at bazel-out/k8-opt/bin/source/extensions/filters/common/lua/_virtual_includes/lua_lib/extensions/filters/common/lua/lua.h:294
#5  Envoy::Extensions::HttpFilters::Lua::StreamInfoWrapper::~StreamInfoWrapper
    (this=0x7ffff7e23c38)
    at bazel-out/k8-opt/bin/source/extensions/filters/http/lua/_virtual_includes/wrappers_lib/extensions/filters/http/lua/wrappers.h:181
#6  0x00005555567b4b27 in Envoy::Extensions::Filters::Common::Lua::BaseLuaObject<Envoy::Extensions::HttpFilters::Lua::StreamInfoWrapper>::registerType(lua_State*)::{lambda(lua_State*)#1}::operator()(lua_State*) const (
    this=<optimized out>, state=<optimized out>)
    at bazel-out/k8-opt/bin/source/extensions/filters/common/lua/_virtual_includes/lua_lib/extensions/filters/common/lua/lua.h:146
#7  0x00005555567c3b7a in lj_BC_FUNCC ()
#8  0x00005555567c62ec in gc_finalize ()
#9  0x00005555567c6a58 in gc_onestep ()
#10 0x00005555567c6628 in lj_gc_step ()
#11 0x00005555567d8929 in lua_newuserdata ()
#12 0x00005555567abd90 in Envoy::Extensions::Filters::Common::Lua::allocateLuaUserData<Envoy::Extensions::HttpFilters::Lua::StreamHandleWrapper> (state=
    0x7ffff7e23ce8)
    at bazel-out/k8-opt/bin/source/extensions/filters/common/lua/_virtual_includes/lua_lib/extensions/filters/common/lua/lua.h:82
#13 Envoy::Extensions::Filters::Common::Lua::BaseLuaObject<Envoy::Extensions::HttpFilters::Lua::StreamHandleWrapper>::create<Envoy::Extensions::Filters::Common::Lua::Coroutine&, Envoy::Http::HeaderMap&, bool&, Envoy::Extensions::HttpFilters::Lua::Filter&, Envoy::Extensions::HttpFilters::Lua::FilterCallbacks&> (
    state=0x7ffff7e23ce8, args=..., args=..., args=..., args=..., args=...)
    at bazel-out/k8-opt/bin/source/extensions/filters/common/lua/_virtual_includes/lua_lib/extensions/filters/common/lua/lua.h:121
#14 0x00005555567abc44 in Envoy::Extensions::HttpFilters::Lua::Filter::doHeaders (this=0x2d78bf1a3778, handle=..., coroutine=..., callbacks=...,
    function_ref=1, setup=<optimized out>, headers=..., end_stream=true)
    at source/extensions/filters/http/lua/lua_filter.cc:684
#15 0x00005555567ad6c5 in Envoy::Extensions::HttpFilters::Lua::Filter::decodeHeaders (this=0x2d78bf1a3778, headers=..., end_stream=true)
    at bazel-out/k8-opt/bin/source/extensions/filters/http/lua/_virtual_includes/lua_filter_lib/extensions/filters/http/lua/lua_filter.h:442
#16 0x000055555749a02f in Envoy::Http::ActiveStreamDecoderFilter::decodeHeaders (this=0x2d78bfa12900, headers=..., end_stream=true)
    at bazel-out/k8-opt/bin/source/common/http/_virtual_includes/filter_manager_lib/common/http/filter_manager.h:199
#17 Envoy::Http::FilterManager::decodeHeaders (this=<optimized out>,
    filter=<optimized out>, headers=..., end_stream=<optimized out>)
    at source/common/http/filter_manager.cc:452
#18 0x000055555748dc93 in Envoy::Http::FilterManager::decodeHeaders (
    this=0x2d78bf217278, headers=..., end_stream=<optimized out>)
    at bazel-out/k8-opt/bin/source/common/http/_virtual_includes/filter_manager_lib/common/http/filter_manager.h:582
#19 Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeHeaders (
    this=0x2d78bf217200, headers=..., end_stream=<optimized out>)
    at source/common/http/conn_manager_impl.cc:1024
#20 0x00005555574ac6f6 in Envoy::Http::Legacy::Http1::ServerConnectionImpl::onMessageComplete (this=0x2d78bf15e680)
    at source/common/http/http1/codec_impl_legacy.cc:970
#21 0x00005555574aa9e5 in Envoy::Http::Legacy::Http1::ConnectionImpl::onMessageCompleteBase (this=0x2d78bf15e690)
    at source/common/http/http1/codec_impl_legacy.cc:764
#22 0x00005555574a78dd in Envoy::Http::Legacy::Http1::ConnectionImpl::$_8::operator() (this=<optimized out>, parser=<optimized out>)
    at source/common/http/http1/codec_impl_legacy.cc:434
#23 Envoy::Http::Legacy::Http1::ConnectionImpl::$_8::__invoke (
    parser=<optimized out>)
    at source/common/http/http1/codec_impl_legacy.cc:433
#24 0x0000555557631beb in http_parser_execute (parser=0x2d78bf15e6d8,
    settings=<optimized out>, data=<optimized out>, len=<optimized out>)
    at external/com_github_nodejs_http_parser/http_parser.c:2167
#25 0x00005555574a91fa in Envoy::Http::Legacy::Http1::ConnectionImpl::dispatchSlice (this=0x2d78bf15e690, slice=0x0, len=140737261838648)
    at source/common/http/http1/codec_impl_legacy.cc:572
#26 0x00005555574a8d9f in Envoy::Http::Legacy::Http1::ConnectionImpl::innerDispatch (this=0x2d78bf15e690, data=...)
    at source/common/http/http1/codec_impl_legacy.cc:548
#27 0x00005555574b1552 in Envoy::Http::Legacy::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_14::operator()(Envoy::Buffer::Instance&) const (
    this=<optimized out>, data=...)
    at source/common/http/http1/codec_impl_legacy.cc:531
#28 _ZNSt3__18__invokeIRZN5Envoy4Http6Legacy5Http114ConnectionImpl8dispatchERNS1_6Buffer8InstanceEE4$_14JS8_EEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSB_DpOSC_ (__f=..., __args=...)
    at /opt/llvm/bin/../include/c++/v1/type_traits:3539
#29 std::__1::__invoke_void_return_wrapper<absl::Status>::__call<Envoy::Http::Legacy::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_14(Envoy::Buffer::Instance&)&>(Envoy::Http::Legacy::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_14(Envoy::Buffer::Instance&)&) (__args=..., __args=...)
    at /opt/llvm/bin/../include/c++/v1/__functional_base:317
#30 std::__1::__function::__alloc_func<Envoy::Http::Legacy::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_14(std::__1::allocator<std::__1::allocator>, absl::Status (Envoy::Buffer::Instance&))>::operator()(Envoy::Buffer::Instance&) (this=<optimized out>, __arg=...)
    at /opt/llvm/bin/../include/c++/v1/functional:1540
#31 std::__1::__function::__func<Envoy::Http::Legacy::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&)::$_14(std::__1::allocator<std::__1::allocator>, absl::Status (Envoy::Buffer::Instance&))>::operator()(Envoy::Buffer::Instance&) (this=<optimized out>, __arg=...)
    at /opt/llvm/bin/../include/c++/v1/functional:1714
#32 0x0000555557603847 in std::__1::__function::__value_func<absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) const (
    this=<optimized out>, __args=...)
    at /opt/llvm/bin/../include/c++/v1/functional:1867
#33 std::__1::function<absl::Status (Envoy::Buffer::Instance&)>::operator()(Envoy::Buffer::Instance&) const (this=<optimized out>, __arg=...)
    at /opt/llvm/bin/../include/c++/v1/functional:2473
#34 Envoy::Http::Utility::exceptionToStatus(std::__1::function<absl::Status (Envoy::Buffer::Instance&)>, Envoy::Buffer::Instance&) (dispatch=..., data=...)
    at source/common/http/utility.cc:43
#35 0x00005555574a8a51 in virtual thunk to Envoy::Http::Legacy::Http1::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) ()
    at source/common/http/http1/codec_impl_legacy.cc:530
#36 0x0000555557489e2c in Envoy::Http::ConnectionManagerImpl::onData (
    this=0x2d78bfb483c0, data=...)
    at source/common/http/conn_manager_impl.cc:282
#37 0x0000555557202873 in Envoy::Network::FilterManagerImpl::onContinueReading
    (this=0x2d78bf163bf8, filter=<optimized out>, buffer_source=...)
    at source/common/network/filter_manager_impl.cc:66
#38 0x00005555571fe845 in Envoy::Network::ConnectionImpl::onRead (
    this=0x2d78bf163b80, read_buffer_size=256)
    at source/common/network/connection_impl.cc:292
#39 Envoy::Network::ConnectionImpl::onReadReady (this=0x2d78bf163b80)
    at source/common/network/connection_impl.cc:574
#40 0x00005555571fc815 in Envoy::Network::ConnectionImpl::onFileEvent (
    this=0x2d78bf163b80, events=<optimized out>)
    at source/common/network/connection_impl.cc:534
#41 0x00005555571f2366 in std::__1::__function::__value_func<void (unsigned int)>::operator()(unsigned int&&) const (this=0x2d78bf127330,
    __args=@0x7ffff27fb0cc: 3)
    at /opt/llvm/bin/../include/c++/v1/functional:1867
#42 std::__1::function<void (unsigned int)>::operator()(unsigned int) const (
    this=0x2d78bf127330, __arg=3)
    at /opt/llvm/bin/../include/c++/v1/functional:2473
#43 Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb (
    this=0x2d78bf1272b0, events=3)
    at source/common/event/file_event_impl.cc:133
#44 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::operator()(int, short, void*) const (this=<optimized out>,
    what=<optimized out>, arg=0x2d78bf1272b0)
    at source/common/event/file_event_impl.cc:106
#45 Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_1::__invoke(int, short, void*) (what=<optimized out>, arg=0x2d78bf1272b0)
    at source/common/event/file_event_impl.cc:90
#46 0x000055555762afe8 in event_process_active_single_queue ()
#47 0x00005555576299be in event_base_loop ()
#48 0x00005555571e3838 in Envoy::Server::WorkerImpl::threadRoutine (
    this=0x2d78bfbcd1d0, guard_dog=...) at source/server/worker_impl.cc:132
#49 0x00005555577d88c3 in std::__1::__function::__value_func<void ()>::operator()() const (this=<optimized out>)
    at /opt/llvm/bin/../include/c++/v1/functional:1867
#50 std::__1::function<void ()>::operator()() const (this=<optimized out>)
    at /opt/llvm/bin/../include/c++/v1/functional:2473
#51 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, std::__1::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::operator()(void*) const (this=<optimized out>, arg=<optimized out>)
    at source/common/common/posix/thread_impl.cc:49
#52 Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__1::function<void ()>, std::__1::optional<Envoy::Thread::Options> const&)::{lambda(void*)#1}::__invoke(void*) (arg=<optimized out>)
    at source/common/common/posix/thread_impl.cc:48
#53 0x00007ffff74116db in start_thread (arg=0x7ffff280f700)
    at pthread_create.c:463
#54 0x00007ffff713a71f in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
A debugging session is active.
@MarcinFalkowski MarcinFalkowski added bug triage Issue requires triage labels Nov 19, 2020
@MarcinFalkowski MarcinFalkowski mentioned this issue Nov 19, 2020
Merged
@mattklein123 mattklein123 added area/lua help wanted Needs help! and removed triage Issue requires triage labels Nov 19, 2020
@mattklein123 mattklein123 added this to the 1.17.0 milestone Nov 22, 2020
@ccannell67
Copy link

Experiencing the same SegFault when using downstreamSslConnection:subjectPeerCertificate(), but at a lower request threshold. SegFaults at < 20 requests. Can provide config if needed.

lizan pushed a commit that referenced this issue Nov 23, 2020
@MarcinFalkowski
lua: reset downstream_ssl_connection in StreamInfoWrapper when object…
59e2968 
… is marked dead by Lua GC (#14092)

Fixes #14091

The problem and fix is similiar to #4312

Risk Level: Low
Testing: regression test, manual testing
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Marcin Falkowski <marcin.falkowski@allegro.pl>
andreyprezotto pushed a commit to andreyprezotto/envoy that referenced this issue Nov 24, 2020
@MarcinFalkowski @andreyprezotto
lua: reset downstream_ssl_connection in StreamInfoWrapper when object…
ce172b7 
… is marked dead by Lua GC (envoyproxy#14092)

Fixes envoyproxy#14091

The problem and fix is similiar to envoyproxy#4312

Risk Level: Low
Testing: regression test, manual testing
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Marcin Falkowski <marcin.falkowski@allegro.pl>
qqustc pushed a commit to qqustc/envoy that referenced this issue Nov 24, 2020
@MarcinFalkowski @qqustc
lua: reset downstream_ssl_connection in StreamInfoWrapper when object…
f46876c 
… is marked dead by Lua GC (envoyproxy#14092)

Fixes envoyproxy#14091

The problem and fix is similiar to envoyproxy#4312

Risk Level: Low
Testing: regression test, manual testing
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Marcin Falkowski <marcin.falkowski@allegro.pl>
Signed-off-by: Qin Qin <qqin@google.com>
cpakulski pushed a commit to cpakulski/envoy that referenced this issue Dec 16, 2020
@MarcinFalkowski @cpakulski
lua: reset downstream_ssl_connection in StreamInfoWrapper when object…
fff6c70 
… is marked dead by Lua GC (envoyproxy#14092)

Fixes envoyproxy#14091

The problem and fix is similiar to envoyproxy#4312

Risk Level: Low
Testing: regression test, manual testing
Docs Changes: N/A
Release Notes: N/A

Signed-off-by: Marcin Falkowski <marcin.falkowski@allegro.pl>
Signed-off-by: Christoph Pakulski <christoph@tetrate.io>
@cpakulski cpakulski mentioned this issue Dec 16, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/lua bug help wanted Needs help!
Projects
None yet
Milestone
1.17.0
3 participants
@mattklein123 @ccannell67 @MarcinFalkowski