Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(batch): let QueryManager hold FrontendEnv #3347

Closed
wants to merge 2 commits into from
Closed

Conversation

lmatz
Copy link
Contributor

@lmatz lmatz commented Jun 20, 2022

What's changed and what's your intention?

QueryManager is lightweight as it is just holding several Arc referring to fields contained in FrontendEnv. And it is very likely that QueryManager will have more and more fields referring to fields in FrontendEnv. Let's just replace them with a single FrontendEnv.

context in functions, i.e. schedule and schedule_single, is not being used now and it will not be needed to pass into these two functions anymore after this change. Previously, QueryManager is held by the FrontendEnv which in turn is held by context. We now make sure a QueryManager will not be mismatched with a context to which it does not belong.

Note that this change effectively makes QueryManager only responsible for the execution of a single query. This is fine for now as currently, no role requires a view of all the queries.

By implementing lazy initialization, the QueryManager is still held by FrontendEnv and manages all the batch queries.

Checklist

- [ ] I have written necessary docs and comments
- [ ] I have added necessary unit tests and integration tests

  • All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

@codecov
Copy link

codecov bot commented Jun 20, 2022

Codecov Report

Merging #3347 (ed0d540) into main (dc8631e) will increase coverage by 0.00%.
The diff coverage is 4.65%.

@@           Coverage Diff           @@
##             main    #3347   +/-   ##
=======================================
  Coverage   73.67%   73.68%           
=======================================
  Files         760      760           
  Lines      104133   104111   -22     
=======================================
- Hits        76723    76709   -14     
+ Misses      27410    27402    -8     
Flag Coverage Δ
rust 73.68% <4.65%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/frontend/src/handler/dml.rs 0.00% <0.00%> (ø)
src/frontend/src/handler/query.rs 0.00% <0.00%> (ø)
src/frontend/src/lib.rs 50.00% <ø> (ø)
...rontend/src/scheduler/distributed/query_manager.rs 0.00% <0.00%> (-11.33%) ⬇️
src/frontend/src/session.rs 40.28% <10.52%> (-1.26%) ⬇️
src/frontend/src/expr/utils.rs 98.74% <0.00%> (-0.51%) ⬇️
src/storage/src/hummock/local_version_manager.rs 84.53% <0.00%> (+0.15%) ⬆️
... and 3 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@lmatz lmatz marked this pull request as ready for review June 21, 2022 03:05
@lmatz lmatz requested a review from liurenjie1024 June 21, 2022 03:12
@liurenjie1024
Copy link
Contributor

The reason we put QueryManager in FrontendEnv is that it's supposed to manage all mpp queries, and treated as a component of FrontendEnv. As you said, if we move it out, it can only manage one query, which conflicts with our design.

@lmatz
Copy link
Contributor Author

lmatz commented Jun 21, 2022

I see, let me consider another approach.

The possible inconsistency of the env variables XXXRef held by QueryManager and the context passed into the schedule functions of QueryManager could be a concern.

@lmatz lmatz marked this pull request as draft June 21, 2022 06:49
@lmatz lmatz changed the title refactor(batch): move QueryManager out of FrontendEnv refactor(batch): let QueryManager hold FrontendEnv Jun 21, 2022
@lmatz
Copy link
Contributor Author

lmatz commented Jun 21, 2022

Now this version uses Arc<tokio::sync::OnceCell<>>

@lmatz lmatz marked this pull request as ready for review June 21, 2022 08:38
@chinawch007
Copy link

chinawch007 commented Jun 21, 2022

Can this be done in the opposite way by putting Arc<FrontendEnv> in QueryManager?

}

pub fn server_address(&self) -> &HostAddr {
&self.server_addr
}

pub async fn query_manager(&self) -> &QueryManager {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's really wierd to me to introduce circular dependency here 😂.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the goal of this pr:

Simplifying fields of QueryManager to avoid misuse in different contexts.

is not correct enginnering practice. When we design a component, we should only consider its dependencies, and avoid introducing extra unnecessary dependencies and unnessary assumptions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QueryManager have Ref to fields in FrontendEnv, and also has context, i.e. a FrontendEnv, as the function parameters when someone calls schedule and schedule_single.

My intention is to remove one of them so they won't mismatch with each other, I am totally fine with other approaches

Calling with context seems a bit error-prone unless we make QueryManager stateless,

open to discussion

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to remove the context in QueryManager's schedule methods.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I don't want store whole FrontedEnv in QueryManager since this introduces two problems:

  1. Add unnecessary depencies to QueryManager
  2. Introduce circular dependencies.

@lmatz
Copy link
Contributor Author

lmatz commented Jun 27, 2022

Let me create an issue for this and discuss it later, close this for now.

@lmatz lmatz closed this Jun 27, 2022
@lmatz lmatz deleted the lz/schedule branch September 16, 2022 07:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants