[Misc] Validate grammar and fail early

[comaniac]

If the grammar is invalid, we now let xGrammar throw RuntimeError when compiling it. However, this happens in logits processor, so the exception is raised from model executor. Since we don't expect model executor to throw any exception now, the exception will crash the engine and kill the worker process.

This PR adds a validation to make sure the grammar is valid when constructing the GrammarConfig to solve this issue.

Note that there is another issue with the xgrammar backend that isn't addressed by this PR #11118

cc @mgoin

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[kouroshHakha]

Hey @comaniac,

1/ Let's add unittests for this. Let's make sure it doesn't diverge from this behavior later on.
2/ We should do a similar thing for json schema as well. Use the following extra_body and it will still kill the engine:
extra_body={"guided_json": {"type": "str"}}

[comaniac]

I don't know where to add unit tests for xgrammar backend. It seems not much unit tests have been added for this component. @mgoin do you have any pointer or should we merge this PR first and make up unit tests later?

Per offline discussion with @mgoin, this PR also fixes the Lark-like issue. I also found another issue that crashes the engine with my fix:

1. Send an invalid grammar. It failed with bad request. However, at this moment the tokenizer data is cached.
2. Send a valid grammar. Since the tokenizer data is cached, we have encoded_vocab = None, which results in crash in get_compiler. This is because here is a hidden assumption that when encoded_vocab = None, the compiler must be initialized already, but this is no longer guaranteed.

The fix in this PR is to make sure encoded_vocab would never be None. This shouldn't hurt performance because the tokenizer data is cached anyways.

[comaniac]

encoded_vocab cannot be None anymore because the compiler may not be initialized even the tokenizer data is cached if the grammar is invalid. This change shouldn't hurt performance because the tokenizer data is cached anyways.

[AlbertoCastelo]

This would fix my issue!

[mgoin]

Thanks!