Add an example #357
Add an example #357
Conversation
Code Metrics Report=============================================================================== Language Files Lines Code Comments Blanks =============================================================================== Dockerfile 1 34 25 0 9 Happy 1 442 369 0 73 JSON 5 9 9 0 0 Python 21 741 622 21 98 TOML 15 389 352 1 36 ------------------------------------------------------------------------------- Jupyter Notebooks 1 0 0 0 0 |- Markdown 1 60 30 22 8 |- Python 1 96 87 1 8 (Total) 156 117 23 16 ------------------------------------------------------------------------------- Markdown 15 1050 0 777 273 |- BASH 6 203 190 0 13 |- Python 6 121 110 0 11 |- Rust 3 185 172 9 4 (Total) 1559 472 786 301 ------------------------------------------------------------------------------- Rust 85 28309 25914 370 2025 |- Markdown 42 439 0 427 12 (Total) 28748 25914 797 2037 =============================================================================== Total 145 30974 27291 1169 2514 =============================================================================== |
| let ConversionResult { | ||
| tokenizer, | ||
| bos, | ||
| eos, | ||
| unk, | ||
| } = if paths.get_tokenizer_filename().to_string_lossy().is_empty() { | ||
| convert_ggml_to_hf_tokenizer(&model)? | ||
| } else { | ||
| ConversionResult { | ||
| tokenizer: get_tokenizer(paths.get_tokenizer_filename())?, | ||
| bos: None, | ||
| eos: None, | ||
| unk: None, | ||
| } | ||
| }; |
There was a problem hiding this comment.
I really don't want to deal with resolving conflicts again, this will be annoying for me to rebase against like the debug addition was.
There was a problem hiding this comment.
and... you just merged it 😑
There was a problem hiding this comment.
Ah, sorry! I didn't see the comment in time. Is there anything I can do to help it?
There was a problem hiding this comment.
Not really, but due to the number of commits and my change being early at the point straight after this addition, I get conflicts rippled through the history.
I'll probably be lazy and just use a merge commit to resolve instead of a proper rebase. I raised the PR before going to bed since there is a lot of activity recently on the files I was refactoring and it's starting to get tedious, so best to get what I've done so far in when you have time to review 👍
There was a problem hiding this comment.
Ok, sounds good, a merge commit is fine. I'll make a review now and then maybe you can do the rebase/merge?
Also some fixes