fallback: true serves app shell to crawlers

[leo]

What version of Next.js are you using?

10.1.3

What version of Node.js are you using?

14.16.0

What browser are you using?

All browsers

What operating system are you using?

All operating systems

How are you deploying your application?

Vercel, next start and next dev

Describe the Bug

If the fallback property within getStaticPaths is set to true, an incoming request for a parameter that doesn't yet match a generated path will cause the app shell to be delivered and the contents of the page to be fetched on the client side afterwards. Subsequent requests will then receive the generated page instead of the app shell.

This behavior is expected and matches the documentation. However, in the case that a crawler (like Google) sends such a request and there isn't yet a generated version of the path available, it will also be served the app shell, which makes it impossible for certain crawlers to parse the content of the page, because the content is only available on the client side, which doesn't even exist in the case of many crawlers.

Expected Behavior

Whenever a crawler (like Google) visits a page that has fallback set to true, the response should be blocked on generating the page — as if fallback was temporarily set to blocking.

To Reproduce

The description above already outlines how to reproduce the bug.

[ijjk]

Closing as this is now resolved by #29121

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.