Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Willing to PR) Make tokenizer.padding_side an argument instead of only being a field #30447

Open
fzyzcjy opened this issue Apr 24, 2024 · 2 comments
Labels
Core: Tokenization Internals of the library; Tokenization. Feature request Request for a new feature

Comments

@fzyzcjy
Copy link
Contributor

fzyzcjy commented Apr 24, 2024

Feature request

Hi thanks for the library! When using tokenizer, for example, for batch-generation with GPT2 (in https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517), it seems that currently I have to do something like:

tokenizer.padding_side = 'left'
data = tokenizer(['sentence one', 'another'])
tokenizer.padding_side = 'right'

Therefore, it would be great to have:

data = tokenizer(['sentence one', 'another'], padding_side = 'left')

just like what we do today for many options like padding_strategy etc.

Motivation

(see above)

Your contribution

Yes, I am willing to PR

@amyeroberts amyeroberts added Core: Tokenization Internals of the library; Tokenization. Feature request Request for a new feature labels Apr 24, 2024
@amyeroberts
Copy link
Collaborator

cc @ArthurZucker

@ArthurZucker
Copy link
Collaborator

Sure, feel free to open a PR and ping @itazap 🤗

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Core: Tokenization Internals of the library; Tokenization. Feature request Request for a new feature
Projects
None yet
Development

No branches or pull requests

3 participants