Immich-Go Upload Fails for Large Photo Import (120K+ Files)

[Didilusse]

Hello,

One of my 120,000 photos import keeps erroring out ,a nd not sure how to fix it:

Command:
./immich-go upload from-folder --into-album "Backup" --api-trace --log-file "debug.log" --server=http://192.168.1.90:8080/ --api-key=*********************** "/mnt/photos"

First log lines:

2025-03-23 11:32:17 INF immich-go version:0.25.0, commit:561f56503aa27b5d5870315cfa1a787e3b745dd4, date:2025-03-22T15:56:48Z
2025-03-23 11:32:17 INF Running environment: architecture=amd64 os=linux
2025-03-23 11:32:17 INF Command: immich-go upload from-folder
2025-03-23 11:32:17 INF Flags:
2025-03-23 11:32:17 INF --album-path-joiner= /
2025-03-23 11:32:17 INF --album-picasa=false
2025-03-23 11:32:17 INF --api-key=**********************************girY
2025-03-23 11:32:17 INF --api-trace=true
2025-03-23 11:32:17 INF --ban-file='@eaDir/', '@_thumb/', 'SYNOFILE_THUMB.', 'Lightroom Catalog/', 'thumbnails/', '.DS_Store/', '/._', '.photostructure/'
2025-03-23 11:32:17 INF --client-timeout=5m0s
2025-03-23 11:32:17 INF --date-from-name=true
2025-03-23 11:32:17 INF --date-range=unset
2025-03-23 11:32:17 INF --device-uuid=HomeLab
2025-03-23 11:32:17 INF --dry-run=false
2025-03-23 11:32:17 INF --exclude-extensions=
2025-03-23 11:32:17 INF --folder-as-album=NONE
2025-03-23 11:32:17 INF --folder-as-tags=false
2025-03-23 11:32:17 INF --help=false
2025-03-23 11:32:17 INF --icloud-takeout=false
2025-03-23 11:32:17 INF --ignore-sidecar-files=false
2025-03-23 11:32:17 INF --include-extensions=
2025-03-23 11:32:17 INF --include-type=
2025-03-23 11:32:17 INF --into-album=Backup
2025-03-23 11:32:17 INF --log-file=debug.log
2025-03-23 11:32:17 INF --log-level=INFO
2025-03-23 11:32:17 INF --log-type=text
2025-03-23 11:32:17 INF --manage-burst=NoStack
2025-03-23 11:32:17 INF --manage-epson-fastfoto=false
2025-03-23 11:32:17 INF --manage-heic-jpeg=NoStack
2025-03-23 11:32:17 INF --manage-raw-jpeg=NoStack
2025-03-23 11:32:17 INF --no-ui=false
2025-03-23 11:32:17 INF --on-server-errors=stop
2025-03-23 11:32:17 INF --recursive=true
2025-03-23 11:32:17 INF --server=http://192.168.1.90:8080/
2025-03-23 11:32:17 INF --session-tag=false
2025-03-23 11:32:17 INF --skip-verify-ssl=false
2025-03-23 11:32:17 INF --tag=[]
2025-03-23 11:32:17 INF --time-zone=
2025-03-23 11:32:17 INF Arguments:
2025-03-23 11:32:17 INF "/mnt/photos"
2025-03-23 11:32:17 INF Connection to the server http://192.168.1.90:8080
2025-03-23 11:32:17 INF Server status: OK
2025-03-23 11:32:18 INF Server information: version=v1.129.0
2025-03-23 11:32:18 INF Connected, user: XXXX, ID: XXXX
2025-03-23 11:32:18 INF Check the API-TRACE file: debug.trace.log

Last lines:

2025-03-23 13:39:48 INF server has same asset file=photos:backup/Recovered data 03-04 11_09_15/Deep Scan result/Special Lost Files(Lab)/Camera/FUJIFILM/FinePix265021927.JPG reason=An asset with the same name:"FinePix265021925.JPG", date:"2002-06-24 00:39:32" and size:308.8 KB exists on the server. No need to upload

2025-03-23 13:40:55 INF
2025-03-23 13:40:55 INF Input analysis:
2025-03-23 13:40:55 INF ---------------
2025-03-23 13:40:55 INF scanned image file : 63547
2025-03-23 13:40:55 INF scanned video file : 354
2025-03-23 13:40:55 INF scanned sidecar file : 3
2025-03-23 13:40:55 INF discarded file : 1
2025-03-23 13:40:55 INF unsupported file : 0
2025-03-23 13:40:55 INF file duplicated in the input : 9
2025-03-23 13:40:55 INF associated metadata file : 0
2025-03-23 13:40:55 INF missing associated metadata file : 0
2025-03-23 13:40:55 INF
2025-03-23 13:40:55 INF Uploading:
2025-03-23 13:40:55 INF ----------
2025-03-23 13:40:55 INF uploaded : 0
2025-03-23 13:40:55 INF upload error : 0
2025-03-23 13:40:55 INF file not selected : 0
2025-03-23 13:40:55 INF server's asset upgraded with the input : 0
2025-03-23 13:40:55 INF server has same asset : 55291
2025-03-23 13:40:55 INF server has a better asset : 0
2025-03-23 13:40:55 INF
2025-03-23 13:40:55 ERR Some errors have occurred. Look at the log file for details

Here's the API Trace log:

2025-03-23T13:39:21-04:00 RESPONSE 64 createStack POST http://192.168.1.90:8080/api/stacks
Header:
Keep-Alive : timeout=5
X-Powered-By : Express
X-Immich-Cid : 9rq5ly0p
Content-Type : application/json; charset=utf-8
Content-Length : 2561
Etag : "a01-leg8Fe0TyFIkUteLYI86TGfIE+8"
Date : Sun, 23 Mar 2025 17:39:21 GMT
Connection : keep-alive
Status: 201 Created
-- response body start --
{"id":"xxx","primaryAssetId":"a651521e-c09d-4356-bb9d-2d272b03227d","assets":[{"id":"a651521e-c09d-4356-bb9d-2d272b03227d","deviceAssetId":"WpdThumbnail.png-8831","ownerId":"535cff05-2f35-4543-99ee-1ef9c46f33af","deviceId":"xxx","libraryId":null,"type":"IMAGE","originalPath":"/photos/library/shared/2012/2012-03-07/WpdThumbnail.png","originalFileName":"WpdThumbnail.png","originalMimeType":"image/png","thumbhash":"8+cJFwLJt5h5h3dvhVaH12iHhArnyXUP","fileCreatedAt":"2012-03-07T09:21:00+00:00","fileModifiedAt":"2012-03-07T09:21:00+00:00","localDateTime":"2012-03-07T09:21:00+00:00","updatedAt":"2025-03-23T17:39:21.736117+00:00","isFavorite":false,"isArchived":false,"isTrashed":false,"duration":"0:00:00.00000","exifInfo":{"make":null,"model":null,"exifImageWidth":96,"exifImageHeight":96,"fileSizeInByte":8831,"orientation":null,"dateTimeOriginal":"2012-03-07T09:21:00+00:00","modifyDate":"2012-03-07T09:21:00+00:00","timeZone":null,"lensModel":null,"fNumber":null,"focalLength":null,"iso":null,"exposureTime":null,"latitude":null,"longitude":null,"city":null,"state":null,"country":null,"description":"","projectionType":null,"rating":null},"livePhotoVideoId":null,"people":[],"checksum":"JzSca0aWkxzSkfFi81fRlTSs52M=","isOffline":false,"hasMetadata":true,"duplicateId":null,"resized":true},{"id":"7644c76b-43cf-485f-a57e-31ced3859f21","deviceAssetId":"WpdThumbnail.tiff-56860","ownerId":"xxx","deviceId":"xxx","libraryId":null,"type":"IMAGE","originalPath":"/photos/library/shared/2007/2007-07-25/WpdThumbnail.tiff","originalFileName":"WpdThumbnail.tiff","originalMimeType":"image/tiff","thumbhash":"8+cJFwLJtph5h3dvhVaH12iHhArnyXUP","fileCreatedAt":"2007-07-25T15:22:50+00:00","fileModifiedAt":"2012-03-07T09:21:00+00:00","localDateTime":"2007-07-25T11:22:50+00:00","updatedAt":"2025-03-23T17:39:21.73538+00:00","isFavorite":false,"isArchived":false,"isTrashed":false,"duration":"0:00:00.00000","exifInfo":{"make":null,"model":null,"exifImageWidth":96,"exifImageHeight":96,"fileSizeInByte":56860,"orientation":"1","dateTimeOriginal":"2007-07-25T15:22:50+00:00","modifyDate":"2007-07-25T15:22:50+00:00","timeZone":"UTC-4","lensModel":null,"fNumber":null,"focalLength":null,"iso":null,"exposureTime":null,"latitude":null,"longitude":null,"city":null,"state":null,"country":null,"description":"","projectionType":null,"rating":null},"livePhotoVideoId":null,"people":[],"checksum":"3B9jqNF6qZzpQfq9d4/ZqE2MXPg=","isOffline":false,"hasMetadata":true,"duplicateId":null,"resized":true}]}

I've looked through the log but I don't see anything that could be wrong, I tried yesterday and it gave me a 500 internal error. I have also ran one with --dry-run and it completed just fine.

Thank you

[simulot]

The dry-run doesn't write anything on the server. As no action is posted to the server, errors are less prone to come.

This log part doesn't show any errors.
Could you search for ERR in the immich-go log file?

The summary reports shows 63547 photos + 354 videos, but only 55291 processed files (already present on the server), were are the others? I'd need the full log for investigation.

The api trace part shows a success. Was it the last call in the log?
The call was for stacking JPG and a TIFF, despite the --manage-raw-jpeg=NoStack option, and the time difference between the two files.

[Didilusse]

The summary reports shows 63547 photos + 354 videos, but only 55291 processed files (already present on the server), were are the others? I'd need the full log for investigation.

After getting an error the program asks me to quit. This was the all the files it scanned before asking me to exit.

The api trace part shows a success. Was it the last call in the log?

Yes that was the last call in the log despite it showing success. That's why I am having trouble troubleshooting the issue since there doesn't appear to be anything wrong with it.

Btw that log was not from the run I did use dry-run so all actions were posted. I'm gonna try to delete all the files and try again. I will post the new log to see if it improved

[Didilusse]

Here is the debug.log I had to stop it in the middle but restarted it back up an hour later. I have looked through the file and the only Error i see is "ERR ReadDir not a directory".

Here's the API trace.log. I don't see anything wrong with it but I could've missed something.

Please let me know what the next step should be.

[simulot]

ReadDir $RECYCLE.BIN/S-1-5-21-1796673143-239304482-1539857752-4545/$RTDX8NL.artist_states....

I think you are mounting a windows share, and some windows' hidden directory at share root are not accessible...
Could you move your photo into a sub directory and upload from /mnt/photos/subdir ?

---

ERR stat backup/Recovered data 03-04 11_09_15/Deep Scan result/Special Lost Files(Lab)/Camera/APPLE/iPhone 41970.JPG: input/output error

I can't tell more than the error message: IO error. Is you disk reliable?

---

You have to know that uploading from a remote folder can be very slow and error prone depending on you configuration.
The best option is to run immich-go on the machine that owns the takeout while accessing to the remote immich. The reason is that SMB share is inefficient and the http connection to immich is designed for remote access.

[Didilusse]

So I have all my photos on an external SSD and they are scattered throughout the SSD into various directories, so putting all the photos into the same sub directory is not really an option.

The disks is reliable after running some tests.

The best option is to run immich-go on the machine that owns the takeout while accessing to the remote immich. The reason is that SMB share is inefficient and the http connection to immich is designed for remote access.

What do you mean by this? How would I set this up?