Basic local LLM support #216
Conversation
| except ImportError: | ||
| LocalCUDACluster = None | ||
|
|
||
| config.set({ |
There was a problem hiding this comment.
Okay I follow and yeah we can punt. I guess I was thinking this may represent a bug for ML Ops teams using LDP, because they'll likely want to configure this themselves
| backward_prefetch: BackwardPrefetch | None = Field( | ||
| BackwardPrefetch.BACKWARD_PRE, |
There was a problem hiding this comment.
Do we need to allow | None here if we have a default? And if we want | None, can you document what that means in the description
There was a problem hiding this comment.
Yeah None is a valid option to pass to FSDP (i.e. no prefetch). The link I put in the description describes this.
| ) | ||
| self._initialize_workers(config, parallel_mode_config) | ||
|
|
||
| def _initialize_workers(self, config, parallel_mode_config): |
There was a problem hiding this comment.
| def _initialize_workers(self, config, parallel_mode_config): | |
| def _initialize_workers(self, dask_client_config, parallel_mode_config): |
Having this take a config and a parallel_mode_config makes one wonder, what is this first config? Let's be more verbose on the name for the first config
There was a problem hiding this comment.
It's not a dask_client_config though. I have added typing to avoid ambiguity.
| bf16 = "bfloat16" | ||
| fp16 = "float16" | ||
| fp32 = "float32" | ||
| auto = "auto" |
There was a problem hiding this comment.
| auto = "auto" | |
| unspecified = "unspecified" |
It seems like auto is not really the right word here, it's more like unspecified.
And if I am wrong, can you add an inline code comment stating what auto means here?
There was a problem hiding this comment.
It's already documented here: https://huggingface.co/docs/transformers/v4.48.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained.torch_dtype . Per above, I don't want to re-document parameters we are forwarding
LMConfigcaptures LLM configuration optionsTransformerHandleradds support for buffered LLM execution & multi-device/node distribution