Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automated cherry pick of #2429: Fix the deadlock between exporter and conntrack polling goroutines #2446

Conversation

srikartati
Copy link
Member

@srikartati srikartati commented Jul 21, 2021

Cherry pick of #2429 on release-1.2.

#2429: Fix the deadlock between exporter and conntrack polling goroutines

For details on the cherry pick process, see the cherry pick requests page.

Deadlock is due to the access of connection map from exporter goroutine waiting
to acquire connection map lock to update the "DoneExport" flag of the stored
connection. At the same time, the connection polling goroutine acquires the
connection map lock waiting to acquire flow record map, which is acquired by the
exporter goroutine.
This was caught in scale testing.
Resolved this through a temporary fix by adding the same flag in the flow record data struct.

The connection and record deletion logic will be re-evaluated through PR antrea-io#2360 as it refactors the related code quite a bit.

Signed-off-by: Srikar Tati <stati@vmware.com>
@srikartati srikartati added the kind/cherry-pick Categorizes issue or PR as related to the cherry-pick of a bug fix from the main branch to a release label Jul 21, 2021
@srikartati srikartati changed the title Automated cherry pick of #2429: Fix the deadlock between exporter and conntrack polling go Automated cherry pick of #2429: Fix the deadlock between exporter and conntrack polling goroutines Jul 21, 2021
@codecov-commenter
Copy link

codecov-commenter commented Jul 21, 2021

Codecov Report

Merging #2446 (61f3575) into release-1.2 (b1c7a5b) will decrease coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@               Coverage Diff               @@
##           release-1.2    #2446      +/-   ##
===============================================
- Coverage        59.87%   59.83%   -0.04%     
===============================================
  Files              284      284              
  Lines            22178    22170       -8     
===============================================
- Hits             13278    13265      -13     
- Misses            7474     7487      +13     
+ Partials          1426     1418       -8     
Flag Coverage Δ
e2e-tests ∅ <ø> (?)
kind-e2e-tests 47.04% <100.00%> (-0.05%) ⬇️
unit-tests 42.01% <66.66%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../flowexporter/connections/conntrack_connections.go 80.00% <100.00%> (+1.31%) ⬆️
pkg/agent/flowexporter/exporter/exporter.go 80.46% <100.00%> (+0.32%) ⬆️
pkg/agent/flowexporter/flowrecords/flow_records.go 83.01% <100.00%> (+4.58%) ⬆️
...agent/flowexporter/connections/deny_connections.go 82.92% <0.00%> (-9.76%) ⬇️
pkg/agent/flowexporter/connections/connections.go 75.00% <0.00%> (-6.82%) ⬇️
pkg/agent/flowexporter/utils.go 60.00% <0.00%> (-6.67%) ⬇️
pkg/controller/networkpolicy/status_controller.go 72.90% <0.00%> (-3.88%) ⬇️
pkg/controller/grouping/controller.go 67.24% <0.00%> (-1.73%) ⬇️
pkg/agent/memberlist/cluster.go 72.33% <0.00%> (-0.49%) ⬇️
... and 4 more

@srikartati
Copy link
Member Author

/test-all

@srikartati
Copy link
Member Author

/test-networkpolicy

@antoninbas antoninbas merged commit ea0f640 into antrea-io:release-1.2 Jul 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/cherry-pick Categorizes issue or PR as related to the cherry-pick of a bug fix from the main branch to a release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants