[BUG] lots of jpeg lost after update

[xv44586]

The bug

after update follow #5723 and lots of img lost link

lots of "no such file or directory" error

immich_server            | [Nest] 7  - 01/14/2024, 5:33:07 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
immich_server            | [Nest] 7  - 01/14/2024, 5:33:07 AM   ERROR [ExceptionsHandler] ENOENT: no such file or directory, access 'upload/thumbs/c23eeb9f-5501-4536-9796-e34e0844116b/18/74/1874d962-6586-4eb0-9d5c-884e5bebd2ad.webp'
immich_server            | Error: ENOENT: no such file or directory, access 'upload/thumbs/c23eeb9f-5501-4536-9796-e34e0844116b/18/74/1874d962-6586-4eb0-9d5c-884e5bebd2ad.webp'
immich_server            | [Nest] 7  - 01/14/2024, 5:33:07 AM   ERROR [Error: ENOENT: no such file or directory, access 'upload/c23eeb9f-5501-4536-9796-e34e0844116b/thumb/dc36f6a40b36b5ae4232070e43798dc86c2bfbd3ec5423772d7b595989d5fec5/281f44ae-d475-43e2-b76f-b759e48bcd7a.jpeg'] Unable to send file: Error
immich_server            | [Nest] 7  - 01/14/2024, 5:33:07 AM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
immich_server            | [Nest] 7  - 01/14/2024, 5:33:07 AM   ERROR [ExceptionsHandler] ENOENT: no such file or directory, access 'upload/c23eeb9f-5501-4536-9796-e34e0844116b/thumb/dc36f6a40b36b5ae4232070e43798dc86c2bfbd3ec5423772d7b595989d5fec5/281f44ae-d475-43e2-b76f-b759e48bcd7a.jpeg'

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

release

Version of Immich Mobile App

1.92.0 build.133

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.m.daocloud.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.m.daocloud.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.m.daocloud.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/data/imgs


###################################################################################
# Log message level - [simple|verbose]
###################################################################################

LOG_LEVEL=simple

###################################################################################
# Typesense
###################################################################################
# TYPESENSE_ENABLED=false
TYPESENSE_API_KEY=some-random-text
# TYPESENSE_HOST: typesense
# TYPESENSE_PORT: 8108
# TYPESENSE_PROTOCOL: http

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE="My Family Photos and Videos Backup Server"

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

###################################################################################
# Immich Version - Optional
#
# This allows all immich docker images to be pinned to a specific version. By default,
# the version is "release" but could be a specific version, like "v1.59.0".
###################################################################################

#IMMICH_VERSION=

Reproduction steps

1. del docker-compose.yml and .env
2. re-download docker-compose.yml and .env from release
3. docker compose pull && docker compose up -d
4. open from web or app,you will see lots of imgs cannot open
...

Additional information

No response

[alextran1502]

this means something happen to your mount point

[xv44586]

no, mount point is the same as before, and the jpeg acturely in the path.

[xv44586]

no, mount point is the same as before, and the jpeg acturely in the path.

[bo0tzz]

Are the files also where they should be when you look from inside the containers?

[micahflack]

I am experiencing the same issues. The logs report the error...

ERROR [Error: ENOENT: no such file or directory, access '/photos/thumbs/cdfbb8ed-7785-4b41-a1ea-72508c4aa28c/2a/ee/2aee37cd-e47a-42c4-a9dd-1cbe8338396c.webp'] Unable to send file: Error

And my mountpoint has never changed. All I have done is update the docker container and successfully migrated my postgresql db for the 1.90 update.

Under the repair page for administration settings, it reports that almost all of my photos are...

These files are the results of manually deletion of the default upload library

And if I check the mountpoint for those files/directories, they don't exist anymore.

[micahflack]

Nevermind. In my case it looks like another service that was using a mountpoint further up must have deleted a portion of the directory used.

Luckily, I had some backups ;)

[douo]

I have experienced the same problem and spent the whole night without finding the reason. Finally, I found inspiration through this post and realized that during the downtime of my disk, immich uploaded photos and videos to the SSD.

A solution to avoid the same problem in the future. Use systemctl list-units --type=mount to find the unit that mounts the disk array and append it to the After= field of the immich service unit. Alternatively, directly use local-fs.target as an After dependency.