Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to skip utf-8 check #1924

Closed
1 task done
Mesteery opened this issue Aug 1, 2021 · 14 comments · Fixed by #1928
Closed
1 task done

Option to skip utf-8 check #1924

Mesteery opened this issue Aug 1, 2021 · 14 comments · Fixed by #1928

Comments

@Mesteery
Copy link
Contributor

Mesteery commented Aug 1, 2021

  • I've searched for any related issues and avoided creating a duplicate
    issue.

Description

I was wondering if it would be interesting to have an option (disabled by default) to skip the UTF-8 check. This would give a slight performance gain in my opinion. (Only for the client)

@lpinca
Copy link
Member

lpinca commented Aug 1, 2021

I'm -1 because it is mandatory per specification and skipping it might lead to unexpected behavior. See #1868.

@Mesteery
Copy link
Contributor Author

Mesteery commented Aug 1, 2021

Even if the utf8 check is voluntarily disabled because the developer is sure of the origin and the encoding?

@lpinca
Copy link
Member

lpinca commented Aug 1, 2021

The spec does not say anything about that. You might trust a sever but I do not see a real need for the option. utf-8-validate is already pretty fast and I'll make it use https://github.com/simdutf/simdutf soon so the overhead will be insignificant.

@lpinca
Copy link
Member

lpinca commented Aug 1, 2021

Usually an application does not open thousands of WebSocket connections to the same trusted server or set of servers, which is the only scenario I can think of where the option would be useful.

@Mesteery
Copy link
Contributor Author

Mesteery commented Aug 1, 2021

Indeed, but for example, a bot, which obviously trusts the server, can receive a lot of messages, and can also have many connections.

@lpinca
Copy link
Member

lpinca commented Aug 1, 2021

I'm not sure I understand, what would that bot do?

@Mesteery
Copy link
Contributor Author

Mesteery commented Aug 1, 2021

A Discord bot for example.

@lpinca
Copy link
Member

lpinca commented Aug 1, 2021

I don't know how a discord bot works but wouldn't it open a single (client) connection to a server? Why would it need more?

@Mesteery
Copy link
Contributor Author

Mesteery commented Aug 1, 2021

When a Discord bot is on a large number of guilds (> 1000), we use the "sharding", this system allows to "divide" the bot in several parts (called shard), each one takes care of a chunk of guilds/events. Each shard is independent and has its own websocket connection.

https://discord.com/developers/docs/topics/gateway#sharding

@lpinca
Copy link
Member

lpinca commented Aug 2, 2021

Ok but it seems an edge case and even if there is no limit I think the number of shards is usually in the order or tens?

@Mesteery
Copy link
Contributor Author

Mesteery commented Aug 2, 2021

It can start from a few shards up to several thousands for big bots.

@lpinca
Copy link
Member

lpinca commented Aug 2, 2021

Adding the option is easy but I want it to meet a real need/demand. The same argument made here for the client can also be done for the server if clients are trusted. If the option is added for the client, it should also be added for the server.

I am personally not interested but let's keep this open for a few days to see if other people are interested in having this feature.

@lpinca
Copy link
Member

lpinca commented Aug 2, 2021

Anyway does a shard run in its own process or do shards share the same process?

@Mesteery
Copy link
Contributor Author

Mesteery commented Aug 2, 2021

In its own process.

lpinca added a commit that referenced this issue Aug 11, 2021
Add the `skipUTF8Validation` option to skip UTF-8 validation for text
and close messages.

Refs: #1878
Closes #1924
lpinca added a commit that referenced this issue Aug 11, 2021
Add the `skipUTF8Validation` option to skip UTF-8 validation for text
and close messages.

Refs: #1878
Closes #1924
lpinca added a commit that referenced this issue Aug 11, 2021
Add the `skipUTF8Validation` option to skip UTF-8 validation for text
and close messages.

Refs: #1878
Closes #1924
lpinca added a commit that referenced this issue Aug 11, 2021
Add the `skipUTF8Validation` option to skip UTF-8 validation for text
and close messages.

Refs: #1878
Closes #1924
lpinca added a commit that referenced this issue Aug 11, 2021
Add the `skipUTF8Validation` option to skip UTF-8 validation for text
and close messages.

Refs: #1878
Closes #1924
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants