Skip to content

Commit

Permalink
Simplified model loading (#32)
Browse files Browse the repository at this point in the history
This PR simplifies the model loading taking advantage of the new
functionality of `get_model()` from `fms.models`.
The current implementation automatically infers `architecture` and
`variant` from a given `model_path` pointing to directory with weights
in **hf** (hugging face) format.

### Changes:

- replacing as_fms_model() by get_model() for **hf** models. 
- removing if condition for **meta** weights

Note: make sure to use the **hf** format of the weights for model
**7B-F** (checkpoint trained by meta) from now on...
  • Loading branch information
yannicks1 authored and GitHub Enterprise committed Sep 12, 2024
1 parent f9961c5 commit bd8457a
Showing 1 changed file with 3 additions and 28 deletions.
31 changes: 3 additions & 28 deletions vllm/model_executor/model_loader/sendnn.py
Original file line number Diff line number Diff line change
Expand Up @@ -182,34 +182,9 @@ def sample(
def load_weights(self, model_name_or_path: str, device_type,
max_prompt_length, max_decode_length, **kwargs):

# check model source: hf or meta
files = os.listdir(model_name_or_path)
model_source = 'hf'
# default huggingface, but if .pth file in model directory, then it is meta weights
for f in files:
if f.endswith('.pth'):
model_source = 'meta'
break

if model_source == 'hf': # hugging face
# load hugging face model
self.model = as_fms_model(model_name_or_path)
else: # meta
variant = "7b"
architecture = "llama"
distr_param = None

# Load the weights from the cached or downloaded files.
self.model = get_model(
architecture=architecture,
variant=variant,
model_path=model_name_or_path,
source=model_source,
device_type=device_type,
distributed_strategy=distr_param,
group=dist.group.WORLD,
)

# function will infer architecture and variant for hf models based on model_name_or_path
self.model = get_model("hf_pretrained", model_name_or_path)

compile_mode = "default"
dynamo_backend = DYN_BACKEND

Expand Down

0 comments on commit bd8457a

Please sign in to comment.