Deployment fails and Functions return "The service is unavailable"

[maersk96]

Describe the bug
Starting today all my production Azure Static Web App services have been returning "The service is unavailable" responses. I am also experiencing issues with deployment to these SWA services. This is the DevOps deployment response:

2023-03-08T09:46:48.8086600Z ##[section]Finished Upload. Polling on deployment.
2023-03-08T09:46:48.8556874Z Status: InProgress. Time: 0.0467032(s)
2023-03-08T09:47:03.9068232Z Status: InProgress. Time: 15.0980397(s)
2023-03-08T09:47:18.9516267Z Status: InProgress. Time: 30.1429502(s)
2023-03-08T09:47:34.0105878Z Status: Failed. Time: 45.2015801(s)
2023-03-08T09:47:34.0106351Z ##[error]Deployment Failed :(
2023-03-08T09:47:34.0106747Z ##[error]Deployment Failure Reason: Failed to deploy the Azure Functions.

I cannot find any official status message stating that the SWA service is down, so I am suspecting that the issue lies within our setup/subscription. Is anyone experiencing the same issue? Did any breaking changes happen in regards to SWA services that i am unaware of?

To Reproduce
Steps to reproduce the behavior:

1. Create and deploy an SWA with Functions
2. Check if SWA deployment fails
3. If not, check if Functions return "Service unavailable" response

[ksdaniel]

Also facing the same issue - doubled by the fact that for some existing SWA's the functions are not running.

[maersk96]

Also facing the same issue - doubled by the fact that for some existing SWA's the functions are not running.

Exactly my issue. My Static Web App frontends are still reachable to some extend, but Functions are unavailable.

I am wondering why nothing shows up on the Azure status page

[KongMats]

Also having this issue at the moment. Getting the error "Deployment Failure Reason: Failed to deploy the Azure Functions."

[polaroi8d]

Also having this issue at the moment. Getting the error "Deployment Failure Reason: Failed to deploy the Azure Functions."

Same issue.

[valstu]

Facing the same issue

[myticketgit]

We have the same issue :-(

[erikknorren]

We have the same issue. Multiple tenants, free plan/standard plan/enterprise grade edge. Sent support ticket via Azure.

[ottosulin]

We have been experiencing a total outage of Static Web Apps at least since 08:15 UTC this morning. I'm honestly very surprised to see that Azure has not yet communicated about this to customers with service being totally out for at least 2h30min now and counting.

[matthiasbruch]

Deployment Failed :(
Deployment Failure Reason: Failed to deploy the Azure Functions.

For further information, please visit the Azure Static Web Apps documentation at https://docs.microsoft.com/en-us/azure/static-web-apps/
If you believe this behavior is unexpected, please raise a GitHub issue at https://github.com/azure/static-web-apps/issues/
Exiting
##[debug]Exit code 1 received from tool '/usr/bin/bash'
##[debug]STDIO streams have closed for tool '/usr/bin/bash'
##[debug]task result: Failed
##[error]Error: The process '/usr/bin/bash' failed with exit code 1
##[debug]Processed: ##vso[task.issue type=error;]Error: The process '/usr/bin/bash' failed with exit code 1
##[debug]Processed: ##vso[task.complete result=Failed;]Error: The process '/usr/bin/bash' failed with exit code 1
Finishing: AzureStaticWebApp

[ChristianFahrner]

We are facing the same issue. Sadly this is not the first time that the deployment service is flaky, but until now it at least did not cause an outage for the static web app itself.

[erikknorren]

@ChristianFahrner we have had a similar issue on 14 december 2022, with the same consequences.

[erijss]

@erikknorren: Did you found a solution or workaround that you can share?

[eodeg]

Experiencing the sam issue

[hohenp]

Same issue here. The site stopped working today. Only message "The service is unavailable." is shown.
Redeployment fails with message "Deployement failed :("
Anything we can do here?

[andrem0]

Same here. Running Next.js.

[erikknorren]

@erijss No it seemed back then as though it was downtime.

[pakeecbala]

same issue here with Nextjs 13.

[matthiasbruch]

Are you guys with nextjs also using this preview feature?

[Even-MW]

Same issue here, Nextjs 13 with Static Web App (preview)

[ottosulin]

Got hold of Azure Support on Twitter, they said they're looking into this.

[tlaundal]

Same issue here. Static assets work, but any function invocation yields a 503 response with the text "The service is unavailable.".

[hohenp]

Our page is back running since some minutes. Anyone knows what happened?

[vd-aloker]

Same issue here. Deployment fails. Existing deployments that use functions see outages (503 The service is unavailable).

[cfillies]

I am having the same issue. no API and no deployment

[sshehab1]

same here

[filipkowal]

Same issue. Next.js 13 with appDir

[erikknorren]

Hello Erik,

From case verbatim we understand that you are facing issue while accessing Azure App Services.

I would like to inform you that seems like an issue going on in West Europe region and our product group team is actively working towards the fix and we are expecting an update soon.

Impact Statement: Starting at 07:18 UTC on 08 Mar 2023, you have been identified as a customer using App Service in West Europe who may receive intermittent HTTP 500-level response codes, experience timeouts or high latency when creating or deleting containers for resources hosted in this region. Existing containers would not be impacted by this issue.

Current Status: Our investigation indicates that some container instances which App Service utilizes became unresponsive due to an internal backend service becoming unhealthy. Impact is limited to resources hosted in the West Europe region. This caused operational functions to stop processing and resulted in App Service users experiencing the errors mentioned above.

We have performed recovery actions to help the internal backend service return to a healthy state. We are currenting monitoring the error rates to validate its efficiency. In parallel, we're exploring alternative options to clean up the backlog of stuck requests.

We are constantly following up with them and shall get back to you as soon as we have further update to share with you.

For latest official updates on this outage please visit: https://azure.status.microsoft/th-th/status/

Apologies for the inconvenience caused.

[HaikoF]

Thanks Erik
I got no answer from support yet....
and the status page is showing everything green! the whole time! Same with the azure service health page in the portal.
Like nobody ever noticed anything about this outage :(

[costinBol]

Yeah, that's really frustrating for me, the Azure status page doesn't seem to detect any incidents. Not sure how they managed to not detect this, since it looks like the complete Static Web App service is down, you can just ping any website and get the error "The service is unavailable."

[niicojs]

Same here. Not nextjs but a Javascript function so it may be related.

[ottosulin]

Here we go again, down here too. Azure should now clearly communicate that they have identified the root causes and are doing something about them, if they want customers to retain confidence in the service.

[espenja]

We've had downtime again today as well. Same issue as before - "Azure Functions runtime is unreachable"

[maersk96]

Aaaand, we're down again 🎉🎉🎉

[hohenp]

We too.. WTF!!

[tlaundal]

Same here.

Get 503 response with the text "The service is unavailable." on our server-rendered routes. App is deployed in West Europe. Haven't tried deploy

[ksdaniel]

+1

[erikknorren]

Down again as well.

[erikknorren]

https://twitter.com/AzureSupport/status/1653696849583169540 Azure support replied to me

[tlaundal]

I just got off the phone with a support representative who said the outtake was confirmed and that the product team was working on a fix which would be live shortly.

[thomas-bingel]

+1

[olarsson]

this is not good, this is a major outage and its lasting a very long time. this effectively kills any production-environment. why does this keep occuring? i was considering moving our future web infrastructure to static web apps, but im very hesitant now.

[nanonandi]

We are just about to ship to production and this happens... idk what to think :S

[jikuja]

The answer I got from support in twitter:

In this case we would ask that you open a support case to have this directly investigated by an engineer as we no longer see any active outages at the moment which would explain why you're still having these issues. Please let us know if we can be of help with that. ^CR

Got 503 just three minutes before that answer.

[ottosulin]

Our SWAs are down once again, like they were also yesterday.

Microsoft should begin to at least acknowledge in the Service Health when the service is not working. That would be a start and the bare minimum Microsoft should be doing. If these severe outages (without any communications!) continue we will have to consider alternatives.

[ksdaniel]

This feels more and more like a social experiment: "how low can you push service levels before customers start leaving".

We feel very betrayed...

[hohenp]

yes.. down again... Hello Microsoft... Is there anyone not working on the CoPilot stuff and cares about the other services around? A info about the reasons and status would be the least to inform about...

[fwmoor]

We've moved away from SWA now. It just not stable enough

[costinBol]

Just out of curiosity, which other providers have you all considered? I was looking at Hostinger and Vercel since they work well with SvelteKit but I'd like to keep my current Azure DevOps CI/CD pipelines,

[DavidAtImpactCubed]

We have redeployed our Linux function apps to North Europe (from West Europe) and touch wood, so far no issues. It does seem like this is a West Europe problem.

[tlaundal]

Down again now.

Had some downtime this morning, then fine for a few hours and down again now.

[projectsbydan]

down in germany as well. paid plan. next.js swa.
a redeploy seems to "wake it up".

[niels-ws]

Down for us. Deployment fails with "Deployment Failure Reason: Failed to deploy the Azure Functions."

[TheSirLan]

Re-running the deployment job worked for me just then - I am up again.

This has been intermittent for a couple of days now though, I will open my app and see "The service is unavailable."

Re-deploying sometimes brings it back up right away or it just fails to deploy.

While it's in this state I notice that the APIs tab in the SWA resource shows no backend function assigned at all.

Hopefully these stability issues can be fixed soon.

[TheSirLan]

Re-running the deployment job worked for me just then - I am up again.

This has been intermittent for a couple of days now though, I will open my app and see "The service is unavailable."

Re-deploying sometimes brings it back up right away or it just fails to deploy.

While it's in this state I notice that the APIs tab in the SWA resource shows no backend function assigned at all.

Hopefully these stability issues can be fixed soon.

5 minutes later, down again

[costinBol]

Has this started again? We've been getting "The service is unavailable." for at least 3h now. Wondering if it's only UK South/West or other regions as well.

[cfillies]

Same 503 in "West Europe" as soon as it tries to access app

[ottosulin]

It's ridiculous how Microsoft engineers can ignore these here.

I suggest everyone keeps making support tickets and remember to ask for refund for breaking the SLA. Because I guess that is the only way of getting attention of some director in Redmond guarding his/her bonus who thinks fixing these issues is not important as integrating Copilot to yet another service.

[jikuja]

At least this time they created service health alerts without contacting support.

[peterboba]

We're also in contact with MS support since we're experiencing HTTP 503/502 responses when trying to access
https://{func}.scm.azurewebsites.net/. Mostly WestEurope, but seen also in other regions.

[Chizaruu]

Aye, same issue again.

Update 2 (6 hrs):

Service is still down.

I can confirm it's a region issue; my East US 2 static web apps are working as intended.

Managed Azure Functions for East Asia is not working.

Update 3 (8hrs):

One of the East Asia sites is now fully operational, waiting on the rest.

All sites are have started to come back up. I would love it if a Microsoft representative could explain why this happened.