Make released engine stability a priority of the release cycle

[lekoder]

Context:

• I released an Early Access title ΔV: Rings of Saturn using Godot 3.1.1-stable.
• I am getting frequent crash reports, which I traced back to Revert "Fix AudioStreams::stop possibly causing a small noise" #28469 (or other audio-related issues). The fix is not in current published stable build of the engine.

TL;DR: I think you should make it a priority to publish every bug that causes a engine crash into a patch version of the stable engine and release it as soon as possible.

Look at it from my perspective. I have released a title. There is engine-related bug, that's fixed in current master, that causes my game to crash every couple of hours. I did some workarounds, but there are some areas that I just can't reach with game code. I'm releasing cross-platform, so compiling my own version of the engine is not really feasible.

It's bad with Early Access release and would be a catastrophe in a full one. You just can't have a (stable marked!) engine that crashes a game.

Ideally, the bugfix should get into 3.1.2 and be published days after it was fixed.

I took a leap of faith by developing my game using Godot 3 - when I started it was in early beta stage. Now I feel like your focus is on Godot 4, but the 3.1.1 you are publishing currently on your site is not stable as advertised. I would implore you to revise the release cycle to favor stability of published project over development of next feature or major version. Having an "unstable stable" engine is not doing anyone any favors.

[KoBeWi]

I'm releasing cross-platform, so compiling my own version of the engine is not really feasible.

There are also nightly-builds, which are compiled for 3 major platforms and 6 exports. They are mostly ok, from my experience, although using them in production might be a bit risky (you could test it in "beta" branch of your game, if you have one).

I think the problem here is that minor stable releases are too far apart. Last one was 5 months ago. The engine got so many improvements since then that I don't use it anymore, unless I really have to.

[lekoder]

They are mostly ok, from my experience, although using them in production might be a bit risky

Yes, I know them and they are my fallback. But you should not, really, resolve to using a nightly build to have a stable version of a software that is marked as stable.

I think the problem here is that minor stable releases are too far apart.

Agreed, this is exactly what I'm trying to address here.

[mwerezak]

I'm spitballing here, and this would require someone to maintain it, but perhaps there could be a branch off of the latest stable release for backporting important fixes? I'm sure everyone tries their best to ensure stable builds are bug-free but it's inevitable that something will slip through the cracks sometimes.

[clayjohn]

@mwerezak We actually already do that. Which is why you can see the releases do not stop at e.g. 3.1, it goes up to 3.1.1 currently and we have a 3.1.2 in the works. See the 3.1 branch on github which recieves backported fixed tagged "cherrypick 3.1".

@lekoder I understand the frustration you are feeling right now. It has been a long time since we have had a minor release. This is not due to the core devs ignoring the importance of fixing bugs in the latest stable, but rather it is the result of a manpower issue. Our release maintainer has been very busy with work/personal life and has been unable to put together a stable release for some months now. I hope you can understand that putting out a release is an immense amount of work and due to the fact that the people who do so, do so in their free time, deadlines often get pushed back to the detriment of users.

There is progress on 3.1.2, but it is later than what would be ideal see this PR

[lekoder]

I completely understand the problem we are facing, and indeed I talked it over with @akien-mga a month ago. Let's make it clear - I'm not holding grudges against anyone here, I'm just calling for action.

Since it's a open-source, community-driven project I felt the best way to address this was though an issue. This is my call to focus on deploying bug-fixes and patch releases, since I feel it's important for well-being of the Godot Engine project. After all - it needs games released, and games in turn need stable engine above all else.

[Xrayez]

it is the result of a manpower issue

I think this sums it up really. I learned with Godot (or open-source projects really), that one can't possibly rely on stability, so if you do get bugs be prepared to fix them yourself.

I believe @xsellier has run into similar issues with stability, and he had to stay on 2.1 for his game and would confirm that he'd rather stay on master branch to keep up with all bug fixes and features, that's what I've been doing so far. 😛

Actually, I caught some handful of regression bugs while staying on master branch for some period of time: #23437, #24283, #25284, #27489, #30269, #30795, #31604.

[xsellier]

Im still on version 2.1 because I started working on my game back in 2017. Version 3 was not out at the moment, and I was told that it was not production ready. So I started using the version 2.1.3.

When I first released my game (2019) I had the same feeling (ie, the stable version is not stable). So what I learnt from Godot Engine is:

• Official releases are great for Jams or prototype
• If you want a stable version you need to build your own version of Godot by forking the repo from the branch you are using (3.1 in your case). Then once a week, you merge the 3.1 branch to your fork, recompile and test your game. You let your game running for several hours with random inputs. And if it works, then it is stable enough to be published. However sometime, you are going to encounter a critical bug that will never get fixed (Distorted sound on Windows with Corsair or SteelSeries Wireless Headset #25875 for example), and it is up to you to know if you can live with it or not. And if you cant live with it, then you shall port your game to the master branch of Godot Engine (and repeat).

I made a suggestion few weeks ago about that:
godotengine/godot-proposals#10 (comment)

It would be awesome if they could name someone who is responsible to maintain a specific older version. I mean, you don't have to make any announcement when you release a patch version of Godot Engine 2.1 or 3.0, because we know that it won't include any new feature but bugfixes. And it would be even cooler if they can release a patch version per month.

[Calinou]

@xsellier @akien-mga is usually the one who performs releases. He also cherry-picks commits to the 3.1 branch (@hpvb sometimes does this too). I don't know how much bureaucracy is involved in making new releases – we do have automated builds, but I believe there's still a fair bit of manual work to do for every point release.

If we ever get an official nightly builds system, it'd make sense to have builds of the 3.1 branch available in addition to master builds. This way, people who need fixes quickly while staying on 3.1 can easily download builds.

[akien-mga]

Haven't had time to read this so I'll comment more next week, but to clarify: the lack of 3.1.x release since 3.1.1 is not a maintenance decision, but a consequence of @hpvb (the release maintainer for 3.1) being unavailable the past few months, and thus an unplanned bottleneck.

I recently got access to the official buildsystem that he set up for 3.0.x and 3.1.x (which I'm working on to match the needs of 3.2.x), so work on releasing 3.1.2 will resume shortly (ideally together with @hpvb if he's more available, or I'll handle it alone if not).

[Feniks-Gaming]

@akien-mga any update on potential progress regarding this issue with potential time frame?

[girng]

Wow, if I never made that issue in January reporting the static, all of this would not have happened. @lekoder I am so sorry. I feel terrible. You should have not experienced that for a game release, especially regarding your beautiful game. (players would have just reported static sounds instead of crashing).

I hope this never, EVER happens again. I hope reduz/lead devs see this issue

[Calinou]

My Godot builds system is now configured to build and upload a new build of the 3.1 branch every week (in addition to daily master builds): https://archive.hugo.pro/builds/godot/3.1/

I'll update the web page to link to stable builds soon.

[Feniks-Gaming]

Will that result in Steam version being updated as well more often?

[akien-mga]

Steam version is updating when there are new stable builds, this won't change. If you want to use nightly builds, Steam is not the proper platform.

[starry-abyss]

Look at it from my perspective. I have released a title. There is engine-related bug, that's fixed in current master, that causes my game to crash every couple of hours. I did some workarounds, but there are some areas that I just can't reach with game code. I'm releasing cross-platform, so compiling my own version of the engine is not really feasible.

I agree with the points about the Godot release delay.
The apparent option I see here is to cherry-pick the needed patch from master to 3.1.x, then ask a friend who has missing (I assume that is why it's not feasible) platforms to recompile. Sad, but welcome to software dev, one should gain access to platforms before promising support for them... :/

[aaronfranke]

In my opinion, a great way to increase the stability of engine builds would be to make it easier for testers to test out new engine builds by providing automatically-updated daily builds. I opened this proposal to describe the idea further.

[lekoder]

FWIW current 3.2.3-rc* is perfectly stable (my builds are currently using rc5) and I see vast improvement in general stability of the builds. It was months since last time a game crash was traced back to an engine bug, so I'm really satisfied.

[Calinou]

Closing in favor of godotengine/godot-proposals#126, as feature proposals are now tracked in the Godot proposals repository.