Add performance benchmark config: MPS 8da4w

[manuelcandales]

Adds a new performance benchmark config to keep track of performance on MPS backend when running Llama 3.2 1B inference with 8da4w quantization

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8429

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures, 1 Cancelled Job

As of commit 7d3ca20 with merge base 931bb8b ():

NEW FAILURES - The following jobs have failed:

• apple-perf / benchmark-on-device (llama, coreml_fp16, apple_iphone_15, arn:aws:devicefarm:us-west-2:3085353851... / mobile-job (ios) (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/executorch/executorch/test-infra/.github/actions/get-workflow-job-id'. Did you forget to run actions/checkout before running your local action?
• apple-perf / benchmark-on-device (llama, xnnpack_q8, apple_iphone_15, arn:aws:devicefarm:us-west-2:30853538511... / mobile-job (ios) (gh)
  Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under '/home/ec2-user/actions-runner/_work/executorch/executorch/test-infra/.github/actions/get-workflow-job-id'. Did you forget to run actions/checkout before running your local action?
• apple-perf / build-benchmark-app / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• apple-perf / export-models (llama, coreml_fp16, apple_iphone_15, arn:aws:devicefarm:us-west-2:308535385114:dev... / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• apple-perf / export-models (llama, xnnpack_q8, apple_iphone_15, arn:aws:devicefarm:us-west-2:308535385114:devi... / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 2
• apple-perf / upload-benchmark-results (gh)
  Could not assume role with user credentials: User: arn:aws:sts::308535385114:assumed-role/gh-ci-github-action-runners-runner-role/i-065a8293b6aede394 is not authorized to perform: sts:TagSession on resource: arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results
• Lint / lintrunner / linux-job (gh)
  >>> Lint for install_executorch.py:
• pull / unittest / linux / linux-job (gh)
  .ci/scripts/tests/test_gather_benchmark_configs.py::TestGatehrBenchmarkConfigs::test_generate_compatible_configs_llama_model

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-llama-runner-mac (fp32, xnnpack+kv+custom) / macos-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[guangy10]

@manuelcandales Here is the instructions of how to trigger an on-demand benchmark job on your PR using the newly added benchmark config:

1. On GitHub, got to "Actions" tab, and select "apple-perf" workflow
2. Click on "Run workflow", then select your branch, then enter the HF model_id (meta-llama/Llama-3.2-1B for example) and config name llama3_mps_8da4w, and click the green "Run workflow" button (as shown in the screenshot)

[guangy10]

@manuelcandales Please note that the all infra backed by pytorch dev infra including the benchmarking infra can only run on a non-forked PR. If your PR is created on your own fork (seems like yes), you have to recreate a non-fork PR.

[manuelcandales]

@guangy10 Thank you for the detailed instructions. I created a non-fork PR #8461. However, when I select my branch under Run Workflow, it doesn't show the Backend delegates field. Notice that when I select the bench-debug branch from your example, I can see it. Do you know why are these templates different from branch to branch? Am I missing something on my branch?