Add syntax highlight to datafusion-cli

[trungda]

Which issue does this PR close?

#8701

Rationale for this change

More user-friendly and readability when using datafusion-cli.

What changes are included in this PR?

Add SQL syntax highlighting capability for datafusion-cli: SyntaxHighlighter.

Are these changes tested?

1. Added unit tests to validate the ANSI string generated;
2. Manually tested.
   

Are there any user-facing changes?

Users of datafusion-cli.

[trungda]

The current implementation is not the best. If the input string is invalid, e.g., unterminated: select * from tab_1 where col_1 = ', the whole string will be _un_highlighted. I think a better approach is to make change to sqlparser crate to expose a token iterator just so we can keep highlighting as we scan through the original string, and only stop when we encounter a failure. Something along this line: https://github.com/sqlparser-rs/sqlparser-rs/blob/main/src/tokenizer.rs#L555-L558.

I am happy to go down that path too. Let me know :)

On that note, can we add datafusion-cli to members attribute in [workspace] just so it will be recognized by the rust-analyzer plugin in IDEs?

[comphead]

I really love this one, thanks @trungda
I checked it in the codespace, it has some bugs mostly on when you type a key word and then press left arrow.

[comphead]

[comphead]

[trungda]

Thanks for the feedback @comphead . What is your terminal setup? I am not able to repro yet. The current approach is to take the whole input string and tokenize the input as a whole, the arrow keys shouldn't matter.

[Weijun-H]

Awesome feature! Thanks, @trungda 👍 . I am wondering if could make the default color lighter because the current version is hard to track. Maybe you could consider the DuckDB setting.

[alamb]

I am happy to go down that path too. Let me know :)

That would be great -- thank you @trungda (this is somewhat related); apache/datafusion-sqlparser-rs#1094

On that note, can we add datafusion-cli to members attribute in [workspace] just so it will be recognized by the rust-analyzer plugin in IDEs?

The reason it is not part of the workspace is that as datafusion-cli we want to release a binary and thus have the versions locked (via a checked in Cargo.lock) However, for the datafusion crates we release them as a library where having Cargo.lock is not the recommended.

This is not clear I will add it to the README

BTW what I do is to "open" the datafusion/datafusion-cli project as its own top level project in my IDE (rather than open datafusion)

Update: #8938

[alamb]

Thank you @trungda -- I tried this out and it was great.

Once CI passes I think we could merge it.

In terms of changing the default colors / improving the highlighting, it would be great to improve this PR but also I think we can iterate in the future as well

Another feature that would be neat would be a CLI flag to enable/disable the highlighting so people who find the color scheme not to their liking could turn it off

[trungda]

Thanks @alamb ! I've updated the license + changed the color to be lighter as suggested.

That would be great -- thank you @trungda (this is somewhat related); apache/datafusion-sqlparser-rs#1094

This is great! I'll look into that. It'd be easier if somehow we know the position in the original string where the tokernization fails just so we can just copy from that position to the highlighted string.

Another feature that would be neat would be a CLI flag to enable/disable the highlighting so people who find the color scheme not to their liking could turn it off

Sounds great! Let me look into that after this PR.

[Weijun-H]

LGTM! Thanks @trungda

[comphead]

LGTM thanks @trungda We can fix highlighting in following PRs