Discuss: workflow for managing Dist files (i.e. on a 'stable' branch)

[dojutsu-user]

A new demo folder should be created, containg the deployment code, so that each time dist folder does not raises Merge Conflict errors.

• A demo folder or a separate branch can be created from where the code can be deployed. This will ensure that if any wrong code commited to main branch does not get deployed immediately.
• Also this will solve the problem of having Merge Conflicts with the dist files.
• dist folder should be added to .gitignore.

What's your thought on this? @jywarren @tech4GT

[welcome]

Thanks for opening your first issue here! Please follow the issue template to help us help you 👍🎉😄
If you have screenshots to share demonstrating the issue, that's really helpful! 📸 You can make a gif too!

[Divy123]

@dojutsu-user great thoughts.
Can you give some example where such pattern has been followed?
I would be really helpful.
Thanks for your help. 😄

[harshkhandeparkar]

@dojutsu-user there has been a lot of discussion about managing dist files. I think the decision was to create a pre-commit hook to ensure that nothing wrong is committed.

#611

[dojutsu-user]

@Divy123
Yes.

1. You can see a similar pattern in this repo: https://github.com/mozilla/kuma. If you see the list of branches there (https://github.com/mozilla/kuma/branches), a separate branch prod-push is used for the deployment purpose.
2. Also, you can see a similar pattern in Read the Docs as mentioned in their docs: https://docs.readthedocs.io/en/stable/faq.html#what-commit-of-read-the-docs-is-in-production

Many github repositories (like elasticsearch: https://github.com/elastic/elasticsearch/) also use separate branches for maintaining their different releases.

[harshkhandeparkar]

@dojutsu-user your idea of separate branches is pretty cool. I had suggested doing that before but @jywarren said that cloning the repo and having the code ready is much better than building it manually or pulling from somewhere else. What do you think?

[dojutsu-user]

@harshkhandeparkar
Contributors need not to clone another branch except the main for the development purpose. Only the deployment will be done through another branch.

deployment branch can be merged with the main branch later.

[harshkhandeparkar]

@dojutsu-user that is not what @jywarren meant(i think). What he meant was that adding dist to gitignore would create problem. Since the code version is not bumped very often, the code is not linked on github releases. jywarren wants the users yo be able to clone/zip the repo and have the latest dist readily available.

[dojutsu-user]

@harshkhandeparkar Let's see his thoughts on this.

[harshkhandeparkar]

Sure 😊

[Divy123]

Cool examples dojutsu-user.
Harsh one thing I recommend that tagging someone one time will send an email to the concerned people.
And thanks for your reviews, they mean a lot.

[harshkhandeparkar]

Divy123 sometimes people prioritize their emails by whether they are tagged or not. If they are tagged earlier, they will receive the email even if they wre not concerned so it is a good way to sort/prioritize. Also I don't like referencing people by name, rather by username. Thanks.

[Divy123]

Sorry closed by mistake.

[jywarren]

Hi, all, there are a few different threads here, and I do want to bring the discussion back to #601 where we talk about how to manage dist files. One suggestion there was to not ask for dist files on each commit, and to just generate them periodically. If we want to /also/ have a discussion about an alternative branch, the plots2 project (https://github.com/publiclab/plots2/) uses a stable branch to deploy from, so there is one extra layer between merged code and publication. But while this is cool, we mainly make use of it by publishing to a stable build server so that people can test things out one last time before they go live. We'd want to do the same here or it might not really add much to just have a separate branch (unless it's part of the dist files discussion).

We could consider the stable branch idea here, and break off the dist files discussion back to #601 maybe?

[jywarren]

And -- thank you!!!!!

[Divy123]

That would be great to have a stable branch to deploy and I think that could make the maintenance work divided and cleaner.
Thanks.
I think we can implement it right away.
I would like to help in this.

[dojutsu-user]

@jywarren
Aren't the problem of dist files and stable branch related?
Making a stable branch will resolve the problem of dist folder and then dist folder can be added to .gitignore.

[harshkhandeparkar]

We can discuss on gitignoring dist files. Moved the pre-commit hook discussion to this issue.

[jywarren]

OK, I've opened a documentation issue to clarify the policy on submitting dist files to PRs (i.e. you don't have to anymore, and I'll compile them in periodically). I think we can do this without stable for starters, but a bigger issue for me is that I'd like some help regularly (every week? 2 weeks?) building everything and publishing a new release. This could be done via a stable branch, maybe.

But is it something we could share responsibility for, and is it something we could manage via PRs?

What if anyone in @publiclab/is-reviewers could open a PR with built files from main to stable branch once per week, and we had a chance to look it over, and if it worked, we'd try to automate the process of:

1. merging to stable
2. publishing a new release on npm
3. updating gh-pages demo

Anything else?

Thank you all!

[jywarren]

OK, I've opened a documentation issue to clarify the policy on submitting dist files to PRs (i.e. you don't have to anymore, and I'll compile them in periodically). I think we can do this without stable for starters, but a bigger issue for me is that I'd like some help regularly (every week? 2 weeks?) building everything and publishing a new release. This could be done via a stable branch, maybe.

But is it something we could share responsibility for, and is it something we could manage via PRs?

What if anyone in @publiclab/is-reviewers could open a PR with built files from main to stable branch once per week, and we had a chance to look it over, and if it worked, we'd try to automate the process of:

1. merging to stable
2. publishing a new release on npm
3. updating gh-pages demo

Anything else?

Thank you all!

[harshkhandeparkar]

If the tasks listed above are done by running commands, I can create scripts for those like the one created in #857

[jywarren]

So summary, i think our options are:

1. branch off a release branch like v3.2.0 and publish to NPM from there each time after merging to stable
2. track dist files in main but only build and track built versions just before merging to stable; stable and NPM versions will be same and include dist files

[harshkhandeparkar]

@harshkhandeparkar you mean stable and NPM wouldn't have dist files? I think we can't do that, as people really rely on dist files for downstream production work. Or do you mean we'd branch off a release branch like v3.2.0 and publish to NPM from there each time we did that?

Oops not at all. I mean that we will always push dist files to npm and branch tags. What will be the latest on npm will be on stable. What will be on main will be on a new unstable branch but will also have dist files. This will be difficult but can be managed(maybe it can be automated?)

[jywarren]

Oh ok thanks, so that is pretty close to (2) above, right?

[harshkhandeparkar]

Like right now, because we don't track dist files at all, i would need to do a merge commit (or worse, rebase the entire main branch), in order to merge to stable and rebuild the dist files. So there's a pretty messy tangle in our git history each time that happens. I think tracking dist files but also .gitignoring them makes sense as a result. I'm not sure how to publish to NPM right now, is the issue.

I don't think clean git history is a requirement (in my opinion it isn't) because people(users) will only pull the stable branch, they won't care about the history, for a history they can have a look at the main branch commit history.

For managing files and npm publishes, I can create a script for you so that you can open your terminal, tyoe one command, type your username and password. Done. What OS do you use?

[harshkhandeparkar]

Oh ok thanks, so that is pretty close to (2) above, right?

Not really, it is a mix of both. So I mean that we never push dist files to main.

[jywarren]

Oh, so where do we build them? Maybe I don't exactly follow What will be on main will be on a new unstable branch but will also have dist files.. Thank you for explaining!

[jywarren]

Whoa... just noting: Downloading Chromium r641577 - 109.8 Mb [================= ] 85% 31.4s

since we started including puppeteer... but yeah, cool! @tech4GT so did you say this actually uses the dist files locally in main? I think that may be a good reason to have them in main as well.

[harshkhandeparkar]

We have to install pupeteer and other deps anyways. For that we use the setup script. The setup script already builds dist files.

[jywarren]

Well, some people may use npm install but in any case, can you describe the disadvantage to having dist files in main, so we can weigh the pros and cons? Thanks!

[harshkhandeparkar]

Maybe I don't exactly follow What will be on main will be on a new unstable branch but will also have dist files.

Unstable branch(which we need to create) can hold the latest beta. We will build dist files here. After each PR, we can push to this branch. Or at least once every 2 to 3 PRs. I can semi-automate this using a script. Only barrier between fully automated and semi-automated(you will have to run the command) is authentication.

Ok. I can create a highly specialised script with different options for npm pushes, unstable pushing, etc. Each time you push to npm, it will also push to stable. This will only be done when bumping npm version. I love writing scripts(I had mentioned harshith's update-is before).

So, what OS do you use?

[jywarren]

I use Linux but sorry @harshkhandeparkar i just don't want to make work for you on script writing before we've come to a consensus on this. Thanks!

[jywarren]

Also, shouldn't we try to write an NPM script that would be cross-platform? But i think we may be getting ahead of the goal here, which is to decide a workflow. Can you tell a bit about the disadvantages you see to including dist files in main? Thank you!!!

[harshkhandeparkar]

Well, some people may use npm install but in any case, can you describe the disadvantage to having dist files in main, so we can weigh the pros and cons? Thanks!

Ok. Pros and cons of not having dist files:(in my opinion)

Pros

• No conflicts
• There will be no version conflicts(difference in browser and node as mentioned above

Cons

• end users need to take the extra step of cloning or pulling from stable
• devs and users have to be extra careful

We can actually manage the dist files building on main(when it is freshly pulled) by documenting it properly. We can directly link to stable branch in the docs wherever it links to dist files.

[harshkhandeparkar]

Also, shouldn't we try to write an NPM script that would be cross-platform?

Yes that can be done.

[harshkhandeparkar]

Pros and cons of having dist files:

Pros

• Clone it, use it
• No need to manage extra branches

Cons

• There can be differences in browser and node
• Dist files can confuse newcomers

[jywarren]

This is really helpful. Let's hear from some others now. Thanks, Harsh!

[harshkhandeparkar]

Sure.

[harshkhandeparkar]

Majority opinion should be full and final.

[jywarren]

Just found this guide to making gh-pages build from Travis!

“Automate deployment of projects to GitHub-pages using Travis CI” by Dhruv Barochiya https://link.medium.com/9wOQZZr4JX

[tech4GT]

@jywarren I would then suggest that, we keep a beta branch for all the dev work and only push to master on releases. We don't want buggy code to get deployed.

[harshkhandeparkar]

@tech4GT what do you mean by dev-work?

[tech4GT]

I mean the any new code should be pushed to the beta branch first! It should actually get deployed with the next release.
This will make sure that even if a PR with buggy code get's merged, it's not immediately deployed.

[harshkhandeparkar]

We have a beta branch on Harshith's repo. The code is pushed to it either by me or him.
https://beta.sequencer.publiclab.org. I had also suggested the creation of unstable branch.

[harshkhandeparkar]

@publiclab/is-maintainers Should we close this now?

[jywarren]

sounds good!