statistics: release session met error to avoid possible leak

[winoros]

What problem does this PR solve?

Issue Number: ref #54022 close #59524

Problem Summary:

We should release the session pool after abandoning it.

What changed and how does it work?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Fix the issue that the internal session used by statistics may not be released and cause possible memory leak.
修复统计信息是使用的内部会话在遇到错误时可能没有被释放的问题，该问题可能导致内存溢出。

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from winoros and additionally assign wjhuang2016 for approval(Please ensuring that each of them provides their approval before proceeding). For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS
• pkg/domain/OWNERS
• pkg/statistics/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[winoros]

/hold
feel free to unhold after 1 lgtm of sql-infra team

[winoros]

/unhold
oh, the approver from the domain package is enough.
no need to add the manually checking

[codecov]

Codecov Report

Attention: Patch coverage is 35.29412% with 11 lines in your changes missing coverage. Please review.

Project coverage is 74.8468%. Comparing base (27f8ff3) to head (410d3ef).
Report is 16 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #59522        +/-   ##
================================================
+ Coverage   73.0108%   74.8468%   +1.8359%     
================================================
  Files          1693       1739        +46     
  Lines        468044     476199      +8155     
================================================
+ Hits         341723     356420     +14697     
+ Misses       105314      97299      -8015     
- Partials      21007      22480      +1473     

Flag	Coverage Δ
integration	48.8469% <11.7647%> (?)
unit	72.1958% <35.2941%> (-0.0175%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.6910% <ø> (ø)
parser	∅ <ø> (∅)
br	61.6709% <ø> (+16.1884%)	⬆️

[tangenta]

I think it is better to put the session in the pool after cleaning up the dirty state..

[winoros]

but how to clean it(

[lance6716]

I see that if we return the session back to pool, the pool should be able to call DeleteInternalSession

tidb/pkg/domain/domain.go

Lines 1332 to 1346 in f6bf8e8

	sysSessionPool: util.NewSessionPool(
	capacity, factory,
	func(r pools.Resource) {
	_, ok := r.(sessionctx.Context)
	intest.Assert(ok)
	infosync.StoreInternalSession(r)
	},
	func(r pools.Resource) {
	sctx, ok := r.(sessionctx.Context)
	intest.Assert(ok)
	intest.AssertFunc(func() bool {
	txn, _ := sctx.Txn(false)
	return txn == nil || !txn.Valid()
	})
	infosync.DeleteInternalSession(r)

Can you reproduce this problem in our environment first? So we can use goref or other ways to know why the reference is kept

[Rustin170506]

I see that if we return the session back to pool, the pool should be able to call DeleteInternalSession

It is challenging to determine whether an error stems from the business logic or from the internal session. This complicates error handling when issues arise. As a result, we decided not to return the session to the pool, but this decision has led to memory leak problems. If we continue failing and create numerous sessions without properly releasing them, it exacerbates the issue.

I propose that we introduce a new API for the session pool that allows the caller to destroy a session instead of directly returning it.

[lance6716]

I see that if we return the session back to pool, the pool should be able to call DeleteInternalSession

It is challenging to determine whether an error stems from the business logic or from the internal session. This complicates error handling when issues arise. As a result, we decided not to return the session to the pool, but this decision has led to memory leak problems. If we continue failing and create numerous sessions without properly releasing them, it exacerbates the issue.

I propose that we introduce a new API for the session pool that allows the caller to destroy a session instead of directly returning it.

This is not related to check the source of error? Line 88 of pkg/statistics/handle/util/util.go should also call DeleteInternalSession

[Rustin170506]

This is not related to check the source of error? Line 88 of pkg/statistics/handle/util/util.go should also call DeleteInternalSession

If we do not check the source of the error, we cannot put it back in the pool directly. This is also why we call DeleteInternalSession from the caller side.

[winoros]

closed since a better one is implemented