Suggestion to stop hosting server-api on k8s

[derberg]

Imho we should stop using K8s on Digital Ocean. The future is not how we envisioned it 2 years ago when we decided that the best will be to have all our services on K8s

more details: asyncapi-archived-repos/event-gateway#161

thoughts?

[smoya]

I'm up for migrating into Digital Ocean App instead, which should reduce the cost considerably.

[derberg]

@smoya thanks

other codeowners? @BOLT04 @magicmatatjahu ?
it will be also much easier to plug in some official DO github action to deploy new server-api directly after merge (cause I don't think you folks have automation for that in place)

@smoya we would be archiving infra repo I guess?

[smoya]

@smoya we would be archiving infra repo I guess?

No, we still can automate the creation of the infrastructure in Digital Ocean trough Terraform

[derberg]

@smoya but we simplify things here. With DO and droplets the infra becomes much simpler, no terraform is needed. We basically have individual apps in place, easily to manage by DO GitHub Actions

[smoya]

@smoya but we simplify things here. With DO and droplets the infra becomes much simpler, no terraform is needed. We basically have individual apps in place, easily to manage by DO GitHub Actions

Terraform automates the creation of those applications in Digital Ocean so no manual process should take in place. You just define your resources, run Terraform and that's it. We don't need you or anyone else to manually create anything, which is basically error-prone and not reproducible at all in case you want to destroy and recreate, etc.

We already use that for creating the K8s cluster in DO, so nothing new is added here. See the Digital Ocean Terraform provider https://registry.terraform.io/providers/digitalocean/digitalocean/latest/docs

[derberg]

but even if we go forward and still use terraform, I don't think we need separate repo for it right?

[smoya]

but even if we go forward and still use terraform, I don't think we need separate repo for it right?

Yeah, we could move it (in fact, create the Terraform files) here that's a good point!

[magicmatatjahu]

Whatever you decide I am with you :) Just makes me wonder why we can't use Github action with image bump (on each merge to master) on k8s? But from previous comments it seems too expensive.

[derberg]

yeah, k8s is basically not needed, we do not take any benefit from it, so no point of using and wasting money

[smoya]

We should really move into this asap, remove the K8s cluster and move the app to another place. As a side note, we are never deploying to the K8s cluster whenever a new version of server-api is out because the process is manual and, afaik, no one is doing it 😅

Unfortunately I don't have the bandwidth atm, but will suggest this issue to be picked up in a near ShapeUp cycle

cc @BOLT04 @magicmatatjahu

[aeworxet]

Bounty Issue's service comment

Text labels: bounty, bounty/2024-Q1, level/medium, bounty/coding
First assignment to third-party contributors: 2023-12-23 00:00:00 UTC+12:00
End Of Life: 2024-05-31 23:59:59 UTC-12:00

[derberg]

To complete this issue you will need to :

• create GitHub workflow that will make sure that on merge to master, server-api application is deployed to Digital Ocean droplet solution
• to create such workflow and make sure it works you will have to for sure create some trial account in DO, deploy server-api there to understand how it works on DO side

In the past when I checked there were already official GitHub Actions for DO that worked well, so best would be to use them imho. From discussion in issue you also see suggestion and link to docs how this could be done with terraform

[AayushSaini101]

@derberg I can take this issue, I have worked on the GitHub workflows. I can learn more during working on this issue.

[sambhavgupta0705]

@derberg I would like to work on this issue.
I have past experience of working with GH actions and K8s

[Shurtu-gal]

@derberg @smoya Would be glad to work on this, if possible.

[aeworxet]

@asyncapi/bounty_team

[derberg]

@Shurtu-gal is most experienced with GH Actions here and already knows how our setup works so let me allow Ashish work on it

[Shurtu-gal]

Just a question, is this deployed somewhere currently so I see look at what I am supposed to be replicating.

Update: https://api.asyncapi.com/v1/help/validate Got it

[smoya]

Do you need something from me @Shurtu-gal ? Happy to help

[Shurtu-gal]

Do you need something from me @Shurtu-gal ? Happy to help

Currently, I am trying to replicate the current deployment on droplets and checking various. Will absolutely let you know as soon as I face any problems 🍻

[aeworxet]

Bounty Issue's Timeline

Complexity Level	Assignment date (by GitHub)	Start date (by BP rules)	End date (by BP rules)	Draft PR submission	Final PR submission	Final PR merge
Medium	2024-01-08	2024-01-15	2024-02-23	2024-01-26	2024-02-09	2024-02-23

Please note that the dates given represent deadlines, not specific dates, so if the goal is reached sooner, it's better.

[smoya]

I noticed we are following an approach of using Digital Ocean droplets directly.
Even though it might seem simple and minimalistic enough, it can be problematic in the long run. Droplets are just VM in the cloud, they not provide any service control over what is running inside, no metrics, no alarms, etc. Less important at this moment but worth to mention the fact they neither provide scale mechanisms. More here.
For example, managing SSL certificates is a manual thing. In our particular case, we would need to issue a new certificate (e.g. Let's Encrypt one), install it in the machine, taking care of renovation of it, etc. Something we also need to care is about application lifecycle; when do we stop a container for deploying the new one? how to rollback in case of failure? Whats the current status after an error?

I know we are migrating to anything else rather than K8s on Digital Ocean because of pricing. I completely agree that, atm, using K8s is completely far from our needs being an overengineered solution.
Luckily, Digital Ocean has the App Platform service. In short, lets you run your containers in the cloud under a managed environment, providing application life-cycle control, monitoring (including alarms on things are wrong, such as bad deployment), domain and SSL certificate handling including automatic renewal of Let's Encrypt certificates, etc etc etc.

The pricing (in our case) should not be way more expensive than just running droplets isolated and, definitely, way more cheaper than K8s solution. The benefits are really worth.

Regarding the deployment, you just deploy to Digital Ocean telling "hey, just deploy latest version" and the system will pick up latest container image tag and deploy it, taking care of the lifecycle from that moment on. Btw, we are already publish the image to Docker Registry as asyncapi/server-api (See the GH Action).

There is an existing Digital Ocean App Platform GH action that lets you trigger the deploy easily.
Additionally, we can manage the App Platform via Terraform (as you are doing with the Droplet).

cc @derberg

[derberg]

In short, lets you run your containers in the cloud under a managed environment, providing application life-cycle control, monitoring (including alarms on things are wrong, such as bad deployment), domain and SSL certificate handling including automatic renewal of Let's Encrypt certificates, etc etc etc.

we were on k8s that was giving us full freedom and all this features. With k8s enabled a solution for future, which costed us a lot of credits from DO, and we did not used that opportunity. I want to decrease initiative costs to bare minimum. If in few months more people will use server-api and maintain and traffic will increase to the size that will cause us think about monitoring - then switch will be easy, there will be people to do it.

Now we use bounty budget to quickly fix stuff that nobody did since July last year.

So if App Platform is more expensive than simple droplet, we should not use it

[derberg]

btw, I hope it doesn't sound like I'm blaming anyone - I do remember I was main advocate for k8s, like pushing a lot...and I regret

[derberg]

K8s costed us $118.08 monthly last year, but after removing event-gateway deployment it went down to $84.00.

Current DO budget we have is $1033.51.

Let's assume need for memory 1GB

• Droplet will cost us 6$
• App Platform 10$

Both for me are good.

Problem is that scope of the task was droplet and it is almost done, so up to @Shurtu-gal if he agrees to do modifications and @smoya to help out with his DO App Platform experience

[Shurtu-gal]

Yeah, I am happy to do them. @smoya I guess we can schedule a meeting and talk a little more regarding it.

[Shurtu-gal]

create GitHub workflow that will make sure that on merge to master, server-api application is deployed to Digital Ocean droplet solution

@derberg just need a clarification should it be after release or on push ?

Because if its after release maybe we can combine both.

[smoya]

@derberg just need a clarification should it be after release or on push ?

I would say we are fine to make it after release. We can add it to https://github.com/asyncapi/server-api/blob/master/.github/workflows/release-docker.yml.

[Shurtu-gal]

@smoya @derberg Shifted from apps to droplets.

Also there is no need of a Let's Encrypt cert as SSL is managed by DO on its own. For further info :- https://www.digitalocean.com/community/questions/ssl-for-digital-ocean-apps

[aeworxet]

@Shurtu-gal, please provide an update to the PR.

[Shurtu-gal]

@aeworxet PR is already ready, just waiting for @smoya to come back from holidays 😁

[smoya]

I couldnt test yet the GH action for deploying, because there is a bug in the deploy command (unrelated with this work) I’m fixing now
I will drop a line here once I test it 👍

[smoya]

I confirm all is working like a charm. Deployment went successful 🎉 Also confirmed in Digital Ocean's control panel.
🥇 goes to @Shurtu-gal.

@aeworxet work related to this bounty issue can be now considered done.

[aeworxet]

Bounty Issue Completed 🎉

@Shurtu-gal, please go to the AsyncAPI page on Open Collective and submit an invoice for USD 200.00 with the expense title Bounty server-api#317, tag bounty, and full URL of this Bounty Issue in the description.

[Shurtu-gal]

Done @aeworxet https://opencollective.com/asyncapi/expenses/184952