Merge Two Photo Libraries Without Duplicates
How to combine photo folders from a phone, a Takeout export and old backups into one library without copying the same photo twice, using content hashes.
You have the photos from your old laptop, a Google Takeout export, a backup drive from 2018 and a phone. They overlap. A lot. You want one library and you do not want every photo three times.
Why libraries overlap
Every backup you ever made is a copy of what you had then. The Takeout export contains everything you ever uploaded, which includes what you copied from the laptop. The phone has what the laptop had plus new photos. Nobody meant to make duplicates. They accumulate.
Filenames do not help. IMG_0042.JPG exists in every set and means different photos in each. File dates do not help either, since each copy reset them.
What makes two files the same
The bytes. Two files with identical contents are the same photo, regardless of name, folder or date. The quick way to compare bytes is a content hash: run the file through SHA-256 or similar and get a fixed-length fingerprint. Same bytes, same hash. One byte different, completely different hash.
Comparing hashes is how every serious dedupe tool works. It is fast, it never gives a false match in practice, and it does not care about names.
What it does not catch is near-duplicates: the same photo saved at a different quality, resized, or with the EXIF changed. Those are different bytes. A Takeout edited version and its original are different files. A WhatsApp copy of a photo you also have in full quality is a different file. Content hashing treats them as two photos, because they are.
Edited copies and Takeout
Google Takeout ships both the original and an -edited version when you edited a photo in Google Photos. They have one sidecar between them and differ in pixels. Whether you want both is up to you. Most people keep the original and let the edit go, or keep both and accept the pair. /blog/takeout-duplicate-photos-edited-versions goes through the choices.
The other Takeout duplicate is the album copy. A photo in three albums appears three times in the export, in three album folders, byte for byte identical. Content hashing collapses those to one.
A safe order of operations
Fix dates first. Merging a folder where half the photos say today with a folder where the dates are right produces a library where half the photos say today. Sort out EXIF in each source before combining. For Takeout that means reading the .json sidecars; see /blog/merge-json-metadata-into-photos.
Pick a base. The library with the best dates and names becomes the destination. Everything else merges into it.
Merge one source at a time. Copy into the base, skipping anything whose hash already exists there. Check the result. Then the next source.
Never delete a source until the merged library has been checked and backed up. Disk is cheap, photos are not.
Doing it by hand
fdupes on Linux and macOS, and its faster cousin jdupes on all three platforms, find identical files across folders by hash and can delete or hard-link the extras. Run them after copying everything into one tree:
jdupes -r -d -N ./Library-d -N deletes duplicates without asking, keeping the first of each set. Run without -N first to see what it would do.
rmlint does the same on Linux with more options. dupeGuru is a free graphical tool for all three platforms and has a picture mode that finds near-duplicates by look, which is useful for the WhatsApp-versus-original case, with the caveat that it needs a human to check each match.
digiKam finds exact and similar duplicates inside its own library and lets you review them side by side.
Doing it with the app
Takeout JSON Metadata Fixer merges into an existing library and skips duplicates by content hash as it goes, so the duplicates never get written in the first place.
Set the destination to your existing library folder. Before copying each photo, the app hashes it and checks against what is already in the destination. Identical files are skipped and counted, not copied. Different files are copied, dated, renamed with the template you chose and filed into the folder layout. Album copies from Takeout collapse to one. Edited versions are kept as their own files, since they are different pictures.
The dates and GPS are fixed on the way in, from sidecars or EXIF, so the merged library is consistent. A dry run lists what would be copied and what would be skipped before anything is written. If you organize the destination in place at the same time, the original files can be kept in an "Originals (before organizing)" folder.
Offline on macOS, Windows and Linux. Free for 500 photos, $9.99 one-time for unlimited.
Frequently asked questions
Will it find the same photo at different sizes?
No. Different pixels mean different bytes and different hashes. That is a near-duplicate, and it needs a visual tool like dupeGuru or digiKam and a person to decide which to keep.
What if the same photo has different EXIF in two places?
They are different files and both are kept. This happens when one copy had its date fixed and the other did not. Fix dates in every source with the same tool before merging and the copies become identical again.
Is the hash comparison slow on a big library?
It reads every file once. On a spinning disk with a hundred thousand photos, expect an hour or two. On an SSD, minutes.
Which copy does it keep?
The one already in the destination. Sources are merged into the base, and anything already present wins. Choose the base with the best metadata for that reason.