Trying to get advice for photo/files storing/syncing
Trying to get advice for photo/files storing/syncing
Hi. Sorry for the vague title. Nowadays I'm using multiple computers and get to need files and such from other machines pretty often. My music and photos library has also increased and it's getting much harder to maintain with it being scattered across many machines. Basically I'm trying to have a photo library and plain files(documents, music, etc) shared across computers.
For plain files I'm thinking nfs+samba would be the best approach, but there are problems. They have speed issues, but as I can't afford large space for all my machines I can't keep a full rsync'd local copies everywhere too.
The photos are my bigger concern, as I'm looking for a tagging feature. A plain directory structure would be easy to sync but those tags would differ by programs.. desktop programs like digikam or xnview(sadly proprietary) would work well if I didn't need syncing, but I'm not sure if they'd work reliably with all their configs/files stored over nfs. Plus, these programs would have incompatibilities by platform and not work at all on android.
Web based solutions like Immich or NextCloud Photos appear to be pretty famous nowadays, but I'm not sure about them as well. They seem to be overkill for my purpose, and those mostly tend to be very new & i'm not too sure about their future, as they store tags and such on their own formats.
Edit: Oops, forgot to say. I have multiple servers right now, one offsite running FreeBSD, another running Devuan, and one at home running FreeBSD.
I'd love to hear how others are maintaining their system. Thanks for reading.
I would honestly consolidate your media and start storing it on a NAS, then have everything sync to the main shares on the NAS.
I wish lol. I don't have a dedicated(?) hardware, just multiple laptops in various locations. They've ran pretty stable for the last few years of 24/7 use though. I'm thinking having some files synced to local would be enough backup, compared to none i have now (aside from few manual work).
How much data in total do you have? Perhaps 2 or 3 external drives (each large enough to hold all your data) on those laptops could bridge the gap for now. Externals are relatively inexpensive, and using something like Syncthing, they could stay in sync.
I took this approach to consolidate my data to free up drives from machines so I could build a NAS running Proxmox. Then copied that data to the NAS, which is the authoritative data store, the other drives now act as local duplicates.
Alternatively, upgrade the drives in the laptops (depending on how much data you have).
Also, keep in mind growth - once you have your data sorted, watch it grow and use that to predict your need for new storage.
As far as photo librabry, you can compare the different solutions at https://github.com/meichthys/foss_photo_libraries
Thanks. I've actually already seen it, but it wasn't enough as it only listed web hosted solutions and I found most of them lacking for my needs. Thankfully others' comments below are pretty inspiring :)
I don't see why you think those are web based, usually people run this at home but glad you find what you are looking for
I use Syncthing and a bunch of rsync scripts to keep my machines in sync. The stuff I want synced continuously is handled by Syncthing. Other stuff is synced on a daily basis using the rsync scripts and anachron. For Photos, I use PhotoPrism. I simply sync the Photos from my smartphone to a folder and make PhotoPrism scan it on a regular basis using ofelia. For cameras, I need to copy the photos manually, but I don't think there's a way around that.
Seems like I'll be going on a setup pretty similar to this. Thanks!
I use Nextcloud and Immich and would recommend both. Immich might be a bit overkill, but it's also well maintained, feature-rich and has a large community. It's super easy to set up and works great.
Thanks for replying. Is there a reason you use Immich over NextCloud's NextCloud Photos? Also I've occasionally heard NextCloud is pretty slow, is it okay for you? You're using their official client program to sync files?
Immich has image and facial recognition by default and a very neat Android app. Also it's running in my home server, which has more power if Immich needs it. In that case I'd say software should serve one purpose and serve that good. Immich is just for picture management and does that very good. Nextcloud is a cloud and the Photos app is just a small extra that can't compete with a full-fledged software. Nextcloud runs fine on my Raspberry Pi 4, but it's only used by me and three friends. It's mainly limited by your network speed and disk speed I'd say. And I'm using an external hard drive without issues.
What is the cost associated with Immich? I keep hearing about it, but I still don't know how that really works I guess
What's costs do you mean? It's free and open source.
I use nextcloud for syncing between different computers, because I tend to have different machines that are far separated geographically, and it works well. I put all my home folders on each computer into the nextcloud directory so I have all the same files everywhere I go and if I don't have one of my computers I can still log in and access those files.
I used to use nextcloud as my solution for everything, but a big problem with photos is it isn't really very navigable, and a problem with nextcloud as a general platform is everything is a plugin so if the plugin doesn't get updated you can be stuck on an older version of the software which carries its own risks. As well, given the interface, You have your media but you can't really go back and look at it. What I did instead is I set up a library in jellyfin with all my photos sorted into directories, and you can scroll and navigate through them fairly intuitively. I pulled my data out of google and facebook before deleting the accounts and so had many many photos but no way to really enjoy them, but that solution worked really well for me and I've been able to look at my old photos easily.
Nice. So you have home folders of each machines in NextCloud? Are you using their official client to sync automatically or on a certain schedule? I never knew Jellyfin can be used as a photo library. Thanks :)
The way I've got it set up is I have a Nextcloud\Desktop, Nextcloud\Downloads, Nextcloud\Documents, Nextcloud\Pictures, and Nextcloud\Videos folder, and on each machine I use I point each of those points in windows to use the folder in the nextcloud folder instead of my users folder, then I run the official client to sync the entire nextcloud folder. By doing that, whichever computer I'm on I've got the same stuff in my main folders and anything else I have I can keep in the nextcloud folder. I've also got it on my mobile device just to automatically upload new pictures to the InstantUploads folder, but the app is a bit limited.
I live equally on the road working as at home, and I've got completely different computers for home and travel, so in this way I've always got all my files available since once I start up the computer it automatically starts pulling the local files. If you don't want a full copy of everything on both machines, I think you can tell it to just create links of the files and the client will download the files from the server as they're required, but I prefer having a local copy of the files myself.
Maybe I have just had back luck but syncing my files across all of my devices has always burned me at some point. No matter what software I have used I have overwritten something by accident or I try and delete something and it doesnt get deleted on all device. Or get a bunch of conflicting files and now i need to figure out what file I want etc...
I do use some syncing but it is mostly between only two devices. Often times it is only a 1 way sync. For example photos on my phone get synced to my NAS automatically.
For me keeping all of my files in one place is the way to go. I just have everything on a NAS. (TrueNAS) All of my devices connect to that and i just edit them directly over the network.
I have a VPN for remote access.
I use digikam with sidecar files on my main photo editing PC.
We sync this directory with a media server on the local network that enables all of the local devices to access the photos and tags. In theory, it means we could run digikam on another device as well, and sync data between them via updates to the sidecar files, but in practice, we don't do this and the media server is effectively read only.
Then, we sync the media server images with a photoprism instance that we have running on external hosting. Photoprism recognises the keywords and sidecar data from digikam, which lets me search and access the images from anywhere.
That's great, I never knew Photoprism can detect and work with digikam's sidecar files. Thanks for sharing.
Lots of good advice here.
The best place to start would be to ignore what you currently have, for the moment, and think about your requirements, at a high level.
In the corporate world, we start with Business/User Requirements - think "what does a user need to be able to do", these are pretty abstract things like:
Have all photos accessible on mobile
Have all photos accesible by App A
Have all photos accesible by App B
Etc
Then take all those User requirements as a guide to the Functional/System/Technical requirements (what solution meets which requirement?)
I kind of just focus on data stability myself (3 local copies, one cloud backup, with local copies being sync'd manually, to act as a sort of buffer from my own fuckups), and implement different solutions for each requirement/system.
Like Syncthing on Windows/Linux/Android, because it just works for regular sync, Resilio on my Media server and Mobile devices, because it has Selective Sync, Tailscale on mobile devices and a single server at home for remote access and remote control.
Thanks. I should try making my requirements clearer. I'm curious, is there a reason you use Resilio despite it being proprietary over other solutions?
Selective Sync is the one feature that Resilio provides that I use.
It enables me to grab any file, using any device, at any time, from anywhere, over any network, simply and quickly. I really wish Syncthing had this capability. Oh well.
So if I'm traveling, I can download a movie from my library with my phone or iPad while connected to hotel wifi. The Resilio UI is simpler than turning on Tailscale, launching a file explorer connecting to my server, then copying. Plus it's a robust sync job - I don't have to think about it, if the network goes away, Resilio will pick up the sync again when it can. On my mobile devices, Resilio is only run if started by the user, but Syncthing runs all the time to ensure stuff like photos, downloaded files, Backups, etc, are sync'd to my server.
I switched from Resilio to Syncthing for everything else (mobile devices mostly, since I can use other tools on laptops), because it's much lighter to run. Resilio is hell on mobile devices if you have a large library, as it keeps the index in memory, while Syncthing uses a file-based approach for indexes. Resilio is also resource intensive on my server - again because of the large media library.