SageMaker LLM Falcon 40b fails to start

[pdifranc]

Describe the bug
When enabling the SageMaker endpoint with Falcon 40b, the endpoint fails to start

> #033[2m2023-10-01T14:42:56.994807Z#033[0m #033[31mERROR#033[0m #033[1mshard-manager#033[0m: #033[2mtext_generation_launcher#033[0m#033[2m:#033[0m Error when initializing model
> File "/opt/conda/lib/python3.9/site-packages/text_generation_server/server.py", line 124, in serve_inner
    model = get_model(model_id, revision, sharded, quantize, trust_remote_code)
  File "/opt/conda/lib/python3.9/site-packages/text_generation_server/models/__init__.py", line 286, in get_model
    raise ValueError("sharded is not supported for AutoModel")

To Reproduce
just deploy the stack and enable LLM on a ml.g5.12xlarge instance

Expected behavior
The deployment is successful

Please complete the following information about the solution:

• Version: [v5.4.2]

To get the version of the solution, you can look at the description of the created CloudFormation stack. For example, "(SO0189) QnABot [...] [v5.4.2]".

• Region: eu-west-1
• Was the solution modified from the version published on this repository? no
• If the answer to the previous question was yes, are the changes available on GitHub?
• Have you checked your service quotas for the services this solution uses? yes
• Were there any errors in the CloudWatch Logs? yes

Screenshots
logs specified above

Additional context
Add any other context about the problem here.

[kroeter]

Hey pdifranc - thank you for the note. We have an open ticket with the SageMaker team to resolve. Will keep you updated on progress.

[rstrahan]

Thanks! Meanwhile, @pdifranc, now that Amazon Bedrock is generally available, you might want to try using QnABot with the Bedrock plugin. See https://github.com/aws-samples/qnabot-on-aws-plugin-samples Sent from Workspace ONE Boxer On Oct 1, 2023 11:16 AM, kroeter ***@***.***> wrote: Hey pdifranc - we noticed this issue on Friday and have an open ticket with the SageMaker team to resolve. Will keep you updated on progress. — Reply to this email directly, view it on GitHub<#635 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACTSFHW6ZY35YXSRELPM5NTX5GCM5ANCNFSM6AAAAAA5OMA4B4>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[marcburnie]

Thanks @pdifranc for reporting the issue. This problem affects v5.4.0, v5.4.1, and the version of v5.4.2 that you are using.

To summarize the issue, a recent commit on the model repo (https://huggingface.co/tiiuae/falcon-40b-instruct/commit/ecb78d97ac356d098e79f0db222c9ce7c5d9ee5f) has broken compatibility with the model image used in QnABot. This problem has been resolved in the v5.4.2 release by specifying an earlier commit from the model repo (see https://github.com/aws-solutions/qnabot-on-aws/blob/main/templates/sagemaker-qa-summarize-llm/index.js#L53).

If you are deploying from this repo, please pull and deploy the v5.4.2 tag/branch: https://github.com/aws-solutions/qnabot-on-aws/releases/tag/v5.4.2. If you deployed from the Implementation Guide, please pull and deploy the latest template.