Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hf llm additional values #1480

Merged
merged 3 commits into from
Jan 30, 2025
Merged

hf llm additional values #1480

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
89300ef
hf llm additional values
stefanmcshane Jan 30, 2025
a75650a
removing file
stefanmcshane Jan 30, 2025
bbf1f29
Merge branch 'main' into sms/hf-speedrun
stefanmcshane Jan 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions addons/hf-llm-models/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,19 @@ spec:
- --tensor-parallel-size={{ .Values.tensorParallelSize }}
{{- if .Values.maxModelLen }}
- --max-model-len={{ .Values.maxModelLen }}
{{- end }}
{{- if .Values.extraArgs }}
{{- range .Values.extraArgs }}
- {{ . }}
{{- end }}
{{- end }}
image: {{ .Values.vllmImage }}
imagePullPolicy: IfNotPresent
env:
- name: HF_TOKEN
value: {{ .Values.huggingFaceToken }}
- name: NCCL_DEBUG
value: INFO
ports:
- containerPort: 8000
protocol: TCP
Expand Down Expand Up @@ -105,13 +112,19 @@ spec:
nvidia.com/gpu: {{ .Values.resources.limits.nvidiaGpu }}
{{- end }}
volumeMounts:
- name: dshm
mountPath: /dev/shm
- name: model-volume
mountPath: {{ .Values.modelDir }}
name: vllm
securityContext:
allowPrivilegeEscalation: false
terminationGracePeriodSeconds: 10
volumes:
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 4Gi
- name: model-volume
persistentVolumeClaim:
claimName: "{{ .Release.Name }}-hf-llm"