Trying to access private repo that I own inside of a github action that I also own in a different repository

[tiffanyseale]

Context: using a github action to deploy to the azure container registry using a private repo, then cloning another private repository I own inside of the docker build. (trying to build a POC of it to then share because none of the guides or message boards I've found are actually working for me). I DO NOT want to use a PAT if I can avoid it, although that seems to be giving me the same issue.

I have successfully authenticated with github using ssh -t git@github.com inside of the dockerfile so I know that the ssh-agent is being passed into the container successfully and reading the right public key (from deploy keys), and have tried with public keys signed as git@github.com, git@github.com/username/repo and git@github.com/username/repo.git. I keep getting this error and it seems it is because I am trying to access a repository with the same owner (me):
#7 ERROR: process "/bin/sh -c git clone git@github.com:tiffanyseale/automatic-system.git" did not complete successfully: exit code: 128

[stage-0 4/4] RUN --mount=type=ssh git clone git@github.com:tiffanyseale/automatic-system.git:
0.256 Cloning into 'automatic-system'...
0.476 ERROR: Repository not found.
0.476 fatal: Could not read from remote repository.
0.476
0.476 Please make sure you have the correct access rights
0.476 and the repository exists.

This is what the Dockerfile looks like:

Get alpine image

FROM alpine:3.14

install ssh client and git

RUN apk add --no-cache openssh-client git

download public key for github.com

RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

clone our private repository

RUN --mount=type=ssh git clone git@github.com:USERNAME/REPONAME.git

This is what the github action looks like:
name: Deploy Docker Image With SSH-Forwarding
permissions:
contents: read
packages: write
on:
push:
branches: [ main ]

jobs:

build:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- name: Build and push Docker image
  env:
    SSH_AUTH_SOCK: /tmp/ssh_agent.sock
    SSH_PRIVATE_KEY: ${{ secrets.ACR_DEPLOY_SECRET_0 }}
  run: |
    # Start SSH agent and add SSH key
    ssh-agent -a $SSH_AUTH_SOCK > /dev/null
    ssh-add <(echo "$SSH_PRIVATE_KEY")
    
- name: Azure Container Registry Login 
  uses: Azure/docker-login@v1 
  with:
    # Container registry server url
    login-server: ${{ secrets.ACR_URL}}
    # Container registry username
    username: ${{ secrets.ACR_USERNAME }}
    # Container registry password
    password: ${{ secrets.ACR_PASSWORD }} 

- name: Build and Push the Docker image
  env:
    SSH_AUTH_SOCK: /tmp/ssh_agent.sock
  run:  | 
    ls
    docker build --build-arg $SSH_AUTH_SOCK --ssh default -f Dockerfile . -t ${{secrets.ACR_URL}}/${{ secrets.ACR_REPO }}:0.0.01
    docker push ${{secrets.ACR_URL}}/${{ secrets.ACR_REPO }}:0.0.01