Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] add input formatter decorator #2158

Merged
merged 1 commit into from
Jul 12, 2024

Conversation

sindhuvahinis
Copy link
Contributor

@sindhuvahinis sindhuvahinis commented Jul 10, 2024

Description

This PR introduces a new decorator input_formatter. Users are expected to annotate this in their input_formatter function in model.py.

This PR also expects the user to define their input_formatter with signature

def my_custom_input_formatter(input_item: Input, **kwargs) -> RequestInput:

Example of UX for users writing their own input formatter:

from djl_python import Input
from djl_python.input_parser import input_formatter
from djl_python.request_io import TextInput
from djl_python.encode_decode import decode

## **kwargs -> has most of the class attributes of TRTLLMService/HuggingFaceService
## example, for TRTLLMService, it has configs (serving.properties), tokenizer, rolling_batch

@input_formatter
def custom_input_formatter(input_item: Input, **kwargs):
    request_input = TextInput()
    content_type = input_item.get_property("Content-Type")
    input_map = decode(input_item, content_type)

    inputs = input_map.pop("inputs", input_map)
    params = input_map.pop("parameters", {})
    
    request_input.input_text = inputs
    request_input.parameters = params
    
    return request_input

@sindhuvahinis sindhuvahinis requested review from zachgk, frankfliu and a team as code owners July 10, 2024 20:18
@tosterberg tosterberg merged commit 63cca6a into deepjavalibrary:master Jul 12, 2024
6 of 9 checks passed
@sindhuvahinis sindhuvahinis deleted the sales branch July 31, 2024 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants