Skip to content

Commit

Permalink
Set use_hf attribute in LlamaConfig of export_paged_llm_v1 to fix e…
Browse files Browse the repository at this point in the history
…xport error (#190)

We need to set the `use_hf` attribute for LlamaModelConfig when
exporting the IR's because the params from gguf do not include the
`use_hf` attribute. Fixes the export error below:
```
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/avsharma/sharktank/sharktank/sharktank/examples/export_paged_llm_v1.py", line 226, in <module>
    main()
  File "/home/avsharma/sharktank/sharktank/sharktank/examples/export_paged_llm_v1.py", line 64, in main
    model = PagedLlamaModelV1(dataset.root_theta, llama_config)
  File "/home/avsharma/sharktank/sharktank/sharktank/models/llama/llama.py", line 130, in __init__
    self.use_hf = config.use_hf
AttributeError: 'LlamaModelConfig' object has no attribute 'use_hf'
```

Signed-off-by: aviator19941 <avinash.sharma@amd.com>
  • Loading branch information
aviator19941 authored Sep 12, 2024
1 parent f088189 commit 3911b3e
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions sharktank/sharktank/examples/export_paged_llm_v1.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ def main():

hp = configs.LlamaHParams.from_gguf_props(dataset.properties)
llama_config = LlamaModelConfig(hp)
llama_config.use_hf = False
llama_config.static_tables = False # Rely on the compiler for hoisting tables.
llama_config.kv_cache_type = "direct" if args.bs == [1] else "paged"
if llama_config.hp.expert_count:
Expand Down

0 comments on commit 3911b3e

Please sign in to comment.