Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mongodb setup bug + tests #2257

Merged
merged 16 commits into from
Apr 29, 2024

Conversation

pranshugupta54
Copy link
Member

@pranshugupta54 pranshugupta54 commented Apr 23, 2024

What kind of change does this PR introduce?

Bug fix + added tests

Issue Number:

Fixes #2255

Did you add tests for your changes?

yes

Snapshots/Videos:

image

If relevant, did you update the documentation?

NA

Summary

  • Currently it was checking localhost connection even if user had not set the env causing no value in the final env.

Does this PR introduce a breaking change?

Other information

Have you read the contributing guide?

Copy link

Our Pull Request Approval Process

We have these basic policies to make the approval process smoother for our volunteer team.

Testing Your Code

Please make sure your code passes all tests. Our test code coverage system will fail if these conditions occur:

  1. The overall code coverage drops below the target threshold of the repository
  2. Any file in the pull request has code coverage levels below the repository threshold
  3. Merge conflicts

The process helps maintain the overall reliability of the code base and is a prerequisite for getting your PR approved. Assigned reviewers regularly review the PR queue and tend to focus on PRs that are passing.

Reviewers

Do not assign reviewers. Our Queue Monitors will review your PR and assign them.
When your PR has been assigned reviewers contact them to get your code reviewed and approved via:

  1. comments in this PR or
  2. our slack channel

Reviewing Your Code

Your reviewer(s) will have the following roles:

  1. arbitrators of future discussions with other contributors about the validity of your changes
  2. point of contact for evaluating the validity of your work
  3. person who verifies matching issues by others that should be closed.
  4. person who gives general guidance in fixing your tests

CONTRIBUTING.md

Read our CONTRIBUTING.md file. Most importantly:

  1. PRs with issues not assigned to you will be closed by the reviewer
  2. Fix the first comment in the PR so that each issue listed automatically closes

Other

  1. 🎯 Please be considerate of our volunteers' time. Contacting the person who assigned the reviewers is not advised unless they ask for your input. Do not @ the person who did the assignment otherwise.
  2. Read the CONTRIBUTING.md file make

@pranshugupta54 pranshugupta54 changed the base branch from main to develop April 23, 2024 10:55
@pranshugupta54
Copy link
Member Author

Oops, made the PR on main branch and it automatically added Peter for review 😅

@pranshugupta54
Copy link
Member Author

pranshugupta54 commented Apr 23, 2024

Added localhost as default instead of testing it ourselves

image
image

@Cioppolo14
Copy link
Contributor

@pranshugupta54 Please fix the failed tests.

Copy link

codecov bot commented Apr 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.52%. Comparing base (6e196fa) to head (cd9fe28).
Report is 1 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2257      +/-   ##
===========================================
+ Coverage    98.45%   98.52%   +0.06%     
===========================================
  Files          349      349              
  Lines        25168    25173       +5     
  Branches      2347     2351       +4     
===========================================
+ Hits         24779    24801      +22     
+ Misses         386      369      -17     
  Partials         3        3              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@palisadoes palisadoes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments

@@ -71,7 +64,7 @@ export async function askForMongoDBUrl(): Promise<string> {
type: "input",
name: "url",
message: "Enter your MongoDB URL:",
default: process.env.MONGO_DB_URL,
default: "mongodb://localhost:27017",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. What happens if the parameter is already defined and it's not localhost or port 27017? Will the application break?
  2. Can a ternary be used?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have already defined then it's going to show us
image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but if another url is set in our env, it won't show that, it'll always show "mongodb://localhost:27017"
the line should be this default: process.env.MONGO_DB_URL ?? "mongodb://localhost:27017"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check the screenshot in previous comment, it's showing different. The port is different from default.

default means - when we have Yes/No type, do you see one of them marked automatically and u press enter?

Copy link
Contributor

@meetulr meetulr Apr 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the screenshot is for a log, this one. The check that you modified is different, this one

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You said it doesn't show from env. That url is from env itself, check the port. It's different port from default

Copy link
Contributor

@meetulr meetulr Apr 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what? we're talking about the line that Peter commented on, you've hard coded mongodb://localhost:27017 there, so when that function runs, it'll always show mongodb://localhost:27017 for that prompt.

You said it doesn't show from env. That url is from env itself, check the port. It's different port from default

the screenshot you've provided is just for a log link, it will show what's in the .env. The prompt you've modified is different, with its default value hard-coded, that's what we mean here.

@palisadoes
Copy link
Contributor

Thanks,

  1. I really don't want setup to break and though this is a test PR, there are changes to the code.
  2. @Anubhav-2003, @meetulr, @adi790uu and @Doraemon012 have all worked on this. I'd appreciate your comments.

Setup is the first DevOps and developer experience with the code. It needs to be perfect.

@pranshugupta54
Copy link
Member Author

pranshugupta54 commented Apr 24, 2024

@palisadoes
The current implementation is wrong. It's currently testing [env MONGO url and localhost url]. If any of them is found to work properly, it moves to next step. But whats wrong in this? If user did not add the url and left the env like MONGO_DB_URL= then it's going to test local url and move forward, without even fixing the env.
Basically it's checking localhost as a fallback when no env key found but it does NOT add that key to env.

Steps:

  • Make MONGO_DB_URL empty in your env like, MONGO_DB_URL=
  • Run setup with npm run setup
  • It's going to show that localhost is detected
  • Continue with it
  • Now in the last step for import data, it will throw error that can not connect to DB.
  • It did not save anything in env for MONGO_DB_URL

What I've done now,

  • If there is empty string MONGO_DB_URL, it will ask you to enter a new url or use the default localhost url

  • It's going to check if it can connect to that url and then adds it to the env

  • Continues to work

  • If MONGO_DB_URL exists, it will simply ask if you want to change or not (same like other fields)

  • If no change then continues

  • If change then checks and adds

We also kinda discussed a bit about this on Slack before this PR: https://thepalisadoes-dyb6419.slack.com/archives/CSWH4PN0M/p1713863659923539

@meetulr
Copy link
Contributor

meetulr commented Apr 24, 2024

This is bad, things are messy here, the way the setup for mongoDB is written is very ambiguous and confusing.

The purpose of setup.ts is basically to setup our .env file (after checking for things like valid connections, etc.) so that the app would run properly when we start it, right?
So, the way to go is to check the url where a mongoDB instance is running and add it as MONGO_DB_URL in our .env.

These are the things that need attention in the current implementation:

  1. Ambiguous variable naming and comments: The way the functions and variables are named, comments are written, are the direct cause of confusion.

    • The comments for this function link, say in one line that it checks if an existing mongoDB instance is running, but in the next line it says its checking for an existing mongoDB connection (which it's not).
    • It is checking if a mongoDB instance is running on a given mongo URL, if so, it sets that as the MONGO_DB_URL in our .env so that the app can connect to it when we run it, as I mentioned before.
    • Checking for an existing connection implies whether the connection to that mongoDB instance exists or not, which is wrong. The function link is not checking whether a connection exists, it's checking whether a connection can be established (which is the thing to do here).
  2. Incomplete checks: Here, after finding a mongo url in .env:

    • if we select to change the url, it checks whether an instance is running on that/other url or not (the variable name isConnected is again wrong here, i.e. it implies that there's a mongo connection established, but what it actually represents is whether a url exists where a mongoDB instance is running).
    • On the other hand, if we select not to change the url, it doesn't check whether an instance is running on that url or not, just moves on to the next step in the setup.
    • Plus, what Pranshu mentioned above.

So, the comments and function/variable names need to be modified, and more/proper checks need to be added.

@pranshugupta54
Copy link
Member Author

I agree with the bad naming of variables and comments. I'll probably fix the naming and comments in this PR itself today.

On the other hand, if we select not to change the url, it doesn't check whether an instance is running on that url or not, just moves on to the next step in the setup.

@meetulr, we are actually not checking this for any step. If variable is found then we skip for every step like Redis etc.
All your points are valid except the above, if we add checks for found variables then the setup will take too long.

@pranshugupta54
Copy link
Member Author

@meetulr, other than naming and comments - do you find any issue with the fixed implementation? Or PR looks better than current situation

@meetulr
Copy link
Contributor

meetulr commented Apr 24, 2024

@meetulr, we are actually not checking this for any step. If variable is found then we skip for every step like Redis etc.
All your points are valid except the above, if we add checks for found variables then the setup will take too long.

The purpose is to setup up a running url right, I'm saying that if a url is found in .env and the user opts not to change it, it's not checking whether an instance is running on that url or not, which would cause that "connection refused" error when we run the app after "successfully setting up the app". If this is done in other places, they need to be fixed too I guess. It won't take too long, the procedure should be the same right? I think things can be refactored a little here.

other than naming and comments - do you find any issue with the fixed implementation? Or PR looks better than current situation

  1. this comment
  2. this line, that adds talawa-api after the url, this check is not done properly. If I have the url as mongodb://localhost:27017/talawa, it will change it to mongodb://localhost:27017/talawa/talawa-api. The check should rather just replace the talawa part with talawa-api, i.e. just the database name.

@pranshugupta54
Copy link
Member Author

We're not checking if it's already in env, it will directly be seen when we start the server. There's no point checking for previous values in setup. We don't run setup everytime we start the server so it's just for setting up mostly on the first time.

Will fix the 2nd point, Yes!

@meetulr
Copy link
Contributor

meetulr commented Apr 24, 2024

We're not checking if it's already in env, it will directly be seen when we start the server.

This is confusing:

  1. So, if a url is in the .env and the developer doesn't change it, we're assuming that the developer knows that it's an active url (having an active mongoDB instance).
  2. If the developer changes it, we're assuming that the developer doesn't know whether it's an active url or not, and we need to check for that?

There's no point checking for previous values in setup.

Previous values? setup is meant to run first, and check if things are ready to run. It can be run again, and do the same thing.

We don't run setup every time we start the server so it's just for setting up mostly on the first time.

exactly, setting up is meant to set things up so that the app can run properly.

@Anubhav-2003
Copy link
Contributor

The purpose is to setup up a running url right, I'm saying that if a url is found in .env and the user opts not to change it, it's not checking whether an instance is running on that url or not, which would cause that "connection refused" error when we run the app after "successfully setting up the app". If this is done in other places, they need to be fixed too I guess. It won't take too long, the procedure should be the same right? I think things can be refactored a little here.

look what you are saying, makes complete sense that we should check for whether a connection can be established, even if the user opts not to change the URL. It definitely won’t take a long time, so this is a valid point.

But you people must also realise that this setup was written. I’m not sure maybe more than a year ago. In fact when I worked on the set up a few months ago, my job was to refactor it and write tests. And yes, comments and the variable names are definitely ambiguous sometimes. But again, these were written by someone maybe more than a year ago, so we don’t know what was going on in his mind and maybe these were written by multiple people. The repository was quite different back then. So all we can do is if we find any errors, then we should discuss like @pranshugupta54 took initiative, and fix them.

@Anubhav-2003
Copy link
Contributor

We're not checking if it's already in env, it will directly be seen when we start the server.

This is confusing:

  1. So, if a url is in the .env and the developer doesn't change it, we're assuming that the developer knows that it's an active url (having an active mongoDB instance).
  2. If the developer changes it, we're assuming that the developer doesn't know whether it's an active url or not, and we need to check for that?

I don’t think this is confusing, look the purpose of the set up was to ensure that while you are initially setting up the application, all the variables and parameters provided are working properly in order for the application to start up.
So if we know that a variable is correct and the developer chooses not to change it, then it is not completely unreasonable to not check it again, assuming that it will be correct. Now again, you can obviously add an extra lay of check. As I wrote above, but the present implementation is not wrong and definitely not confusing.

@Anubhav-2003
Copy link
Contributor

@Anubhav-2003, the default value is only used when there is no value given and user presses enter to select default.

Just like when you init a new npm folder then it asks for so many of the things, you can either choose the default or add your value.

Like, Q) What Port do you want? (4000)

Now, if you press "Enter/Return" - it will use the default 4000 value. If you start typing something and enter 4321 then it will use 4321 and save it.

Ok understood. Thanks👍

@meetulr
Copy link
Contributor

meetulr commented Apr 24, 2024

The purpose is to setup up a running url right, I'm saying that if a url is found in .env and the user opts not to change it, it's not checking whether an instance is running on that url or not, which would cause that "connection refused" error when we run the app after "successfully setting up the app". If this is done in other places, they need to be fixed too I guess. It won't take too long, the procedure should be the same right? I think things can be refactored a little here.

look what you are saying, makes complete sense that we should check for whether a connection can be established, even if the user opts not to change the URL. It definitely won’t take a long time, so this is a valid point.

But you people must also realise that this setup was written. I’m not sure maybe more than a year ago. In fact when I worked on the set up a few months ago, my job was to refactor it and write tests. And yes, comments and the variable names are definitely ambiguous sometimes. But again, these were written by someone maybe more than a year ago, so we don’t know what was going on in his mind and maybe these were written by multiple people. The repository was quite different back then. So all we can do is if we find any errors, then we should discuss like @pranshugupta54 took initiative, and fix them.

Brother, the purpose of my comments is not to blame anyone😂, but to point out the things that can be / need to be improved😅.

look the purpose of the set up was to ensure that while you are initially setting up the application, all the variables and parameters provided are working properly in order for the application to start up.

Exactly, thus my insistence on comprehensive checks.

So if we know that a variable is correct and the developer chooses not to change it, then it is not completely unreasonable to not check it again, assuming that it will be correct.

I'm saying that if there exist a url and user doesn't change it we're not checking if there's running instance there. That contradicts your first point which is what I was saying.

Assuming part is my concern, when user changes the url, we "assume" that we need to check for a running instance, when they don't, we "assume" that it running. "again"? we're checking for a url for a running instance right?

Now again, you can obviously add an extra lay of check. As I wrote above, but the present implementation is not wrong and definitely not confusing.

Again, you're looking at it from the perspective of defending the code😂 which no one is blaming you for. It is definitely not confusing is definitely a strong comment, because it is if you look at it from my perspective (and because we're having this discussion😂).
Your line of reasoning... obviously, I thought of that too, I wanted to be comprehensive in testing that's why I said we could simply add a check for that too.

Ofcourse, It is not a completely ureasonable assumption, but it is that, an "assumption". We could go with the current implementation assuming that. I Just thought it wouldn't be much work to just check it always.

The main reason why I got confused was the variable/function naming and comments. This assumption on checks was another thing that came to attention, so I wanted to discuss it here.😅

See this

@meetulr
Copy link
Contributor

meetulr commented Apr 24, 2024

@pranshugupta54 The default is not just used for when there's no url, it would be promped even when there's no instance running on a provided url, for cases where a user forgot to run the instance before starting the setup script, then, the user would be promped with their provided url (if not hard-coded like it is here), they can simply start the instance and continue the setup process. This could be a very common situation right? This is why I'm saying it should be this default: process.env.MONGO_DB_URL ?? "mongodb://localhost:27017", serves both purposes, and I see nothing wrong with it.

@pranshugupta54
Copy link
Member Author

pranshugupta54 commented Apr 24, 2024

It's not really important to check if instance is running or not, this is just setup!!

Instance may or may not be running when user runs the actual server 😂

So it's like we're checking running instance today but user starts development mode tomorrow, so no point checking running instances while setup

@meetulr
Copy link
Contributor

meetulr commented Apr 24, 2024

It's not really important to check if instance is running or not, this is just setup!!

Now this is just flipping the whole thing😂. It is for checking a running instance, try running it without a running instance (when you opt to change the url), you would be stuck in a loop.

Instance may or may not be running when user runs the actual server 😂

We're checking for running instances, if not, we wouldn't be able to proceed. this is what the check is doing link

So it's like we're checking running instance today but user starts development mode tomorrow, so no point checking running instances while setup

That's what I assumed until today, before I looked at the code, it is checking for running a instance.
If setting up today and running tomorrow is the case, then I don't know why we're checking for connection at all😅. We should just check if the provided url is a valid mongoDB url or not. The implementation implies that after setting up, the user would be able to run the app right away.

See this

@meetulr
Copy link
Contributor

meetulr commented Apr 24, 2024

These were the confusions😂 maybe a clear description is need as to what it is meant to do.

Is it this:

That we're just checking whether the user is able to establish a connection with the mongo url they provide, regardless of when they actually mean to run the app, right after, or tomorrow.
If that's the case, then checking only when the user changes the url makes sense, and the only things to consider/change here are the name/comment fixes, and that hard-coded url.

@pranshugupta54
Copy link
Member Author

pranshugupta54 commented Apr 24, 2024

I've made a flow on how everything will work after this PR is merged.

We do NOT need default: process.env.MONGO_DB_URL ?? "mongodb://localhost:27017".
Assuming that we are having beginner developers,
If the user added some random value in their env, suppose MONGO_DB_URL=localhost:4000 (VERY WRONG). So our setup will first try to connect -> then fail -> prompt to ask new value (but default will show localhost:4000 - very wrong). Now if we only keep default: mongodb://localhost:27017, then whatever messy value is being added by the user - they'll be suggested to use mongodb://localhost:27017

The following is how setup will work:
image

Apart from this, we need to add some more comments for each function.

Everything in this PR looks fine except some function comments.
(Already discussed with @meetulr on Slack) - if someone else has anything else to say then please let me know

@palisadoes / @Anubhav-2003

@Anubhav-2003
Copy link
Contributor

We do NOT need default: process.env.MONGO_DB_URL ?? "mongodb://localhost:27017".
Assuming that we are having beginner developers,
If the user added some random value in their env, suppose MONGO_DB_URL=localhost:4000 (VERY WRONG). So our setup will first try to connect -> then fail -> prompt to ask new value (but default will show localhost:4000 - very wrong). Now if we only keep default: mongodb://localhost:27017, then whatever messy value is being added by the user - they'll be suggested to use mongodb://localhost:27017

This seems to be a valid point. Suggesting new developers and first time comers the exact value of the URL, which is probably used 90% of times in localhost, will aid them in the setup as many people use mongoDB for the first time.

The flowchart is also correct. The PR looks good. But one question, why is the event field changed in the schema in this pull request? Is that relevant to this issue?

@varshith257
Copy link
Member

But one question, why is the event field changed in the schema in this pull request? Is that relevant to this issue?

It is the recent work of @meetulr to remove event status. I think it added into this PR of pulling with latest upstream in Pranshu local branch.

@pranshugupta54
Copy link
Member Author

It got automatically added by husky - pre-commit -> gen:schema

@meetulr
Copy link
Contributor

meetulr commented Apr 24, 2024

It is the recent work of @meetulr to remove event status. I think it added into this PR of pulling with latest upstream in Pranshu local branch.

Yeah sorry about that😅, schema file didn't get updated in my pr because i was having weird local issues with lint-staged, so i made a commit with --no-verify, and forgot to add schema file later😅

@varshith257
Copy link
Member

varshith257 commented Apr 24, 2024

And one thing I like to add, A long back we had a PR merged to check the replicaSet or not and we have standardised it for a reason and me, @meetulr and @palisadoes had reviewed them. But it didn't introduced a way to use it, except checking for replicaSet or not. Why don't we standardise this check strictly in setup to connect for a replicaSet?

@palisadoes
Copy link
Contributor

What's the status on this? It's not clear whether there approval for this PR or not

@pranshugupta54
Copy link
Member Author

@palisadoes, the flow is fine. I've added comments for functions which looked unclear.

Check #2257 (comment)

@palisadoes
Copy link
Contributor

@varshith257 @meetulr @Anubhav-2003 PTAL once more

@varshith257
Copy link
Member

varshith257 commented Apr 28, 2024

And one thing I like to add, A long back we had a PR merged to check the replicaSet or not and we have standardised it for a reason and me, @meetulr and @palisadoes had reviewed them. But it didn't introduced a way to use it, except checking for replicaSet or not. Why don't we standardise this check strictly in setup to connect for a replicaSet?

@pranshugupta54 Everything seems fine but what about this?

@meetulr
Copy link
Contributor

meetulr commented Apr 28, 2024

@varshith257 @meetulr @Anubhav-2003 PTAL once more

I still have my doubts about variable naming😅. But others agree that these changes are good. We can see and change them later if people get confused.

@meetulr
Copy link
Contributor

meetulr commented Apr 28, 2024

Everything seems fine but what about this?

@varshith257 do you mean to make replica set mandatory? Is that an appealing solution🤔

But it didn't introduced a way to use it, except checking for replicaSet or not.

The way that check worked is that if we connect to a replica set, session would be initiated and we'll have transactions support, if not, session would not be initiated and the functionalities would still work.

If making replica set mandatory is what's required, it should be its own separate issue?

@Anubhav-2003
Copy link
Contributor

If making replica set mandatory is what's required, it should be its own separate issue?

I agree on this point. This could be handled in a separate issue which I think @varshith257 you can create. As for this PR. Everything seems fine including the flowchart.

@Anubhav-2003
Copy link
Contributor

@varshith257 @meetulr @Anubhav-2003 PTAL once more

I still have my doubts about variable naming😅. But others agree that these changes are good. We can see and change them later if people get confused.

If you have some doubts then you can suggest them in the review here. Then we can workout something all together.

@pranshugupta54
Copy link
Member Author

Most of the setup file is using similar variable name, so changing one of mongodb variable would require to change redis and more too. So I've simply added comments in a better way to make them easy.

We can take replicaSet in a separate issue, I'm not very sure about how it works.

Copy link
Member

@varshith257 varshith257 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@palisadoes
Copy link
Contributor

Before merging. Will this allow setup to work with a local Mongo installation either:

  1. Using docker?
  2. Not using docker?

We had problems with this before. We need functionality in all scenarios.

@varshith257
Copy link
Member

It's functionality is without docker

@palisadoes palisadoes merged commit 85c2684 into PalisadoesFoundation:develop Apr 29, 2024
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tests: src/setup/MongoDB.ts
6 participants