bash-scripts

A collection of Bash utilities. Works on Linux, macOS, WSL, and Git Bash (Windows).

Scripts

Script	Purpose
find_duplicate_pictures.sh	Find duplicate pictures across one or more filesystems/directories using name and content comparison

---

find_duplicate_pictures.sh

Finds all picture files and reports duplicates. Because more than one picture can legitimately share the same file name (e.g. IMG_0001.jpg from two different cameras), the script does not trust names alone — it combines three independent checks:

1. Same name — groups files sharing the same file name (case-insensitive). Fast, but only a hint.
2. Same content (authoritative) — file size is used as a cheap pre-filter, then SHA-256 hashes are computed only for size-matched candidates. Identical hashes = true duplicates, regardless of name. Also reports approximate disk space wasted by redundant copies.
3. "False friends" — files with the same name but different content (the exact case where names are misleading).

Usage

# Scan specific roots
./find_duplicate_pictures.sh /home/user/Pictures /mnt/backup

# Scan a collection while skipping cache and export directories
./find_duplicate_pictures.sh --exclude ~/Pictures/Cache --exclude ~/Pictures/Exports ~/Pictures

# Scan the whole filesystem (default when no arguments given)
./find_duplicate_pictures.sh

# On Windows (Git Bash) — drives are /c, /d, ...
"/c/Program Files/Git/bin/bash.exe" find_duplicate_pictures.sh /c /d

# Make it executable first if needed
chmod +x find_duplicate_pictures.sh

Options (environment variables)

Variable	Default	Description
HASH_CMD	auto-detected	Hash tool to use: sha256sum, shasum -a 256, or md5sum (last resort)
FOLLOW_LINKS	0 (off)	Set to 1 to follow symlinks while scanning
REPORT	(none)	Path to a file where the full report is also saved

Directory exclusions

Use -x DIR or --exclude DIR to skip a directory and all of its contents. The option may be repeated and accepts paths containing spaces. Relative paths are supported for both scan roots and exclusions.

./find_duplicate_pictures.sh -x ~/Pictures/Cache -x ~/Pictures/Exports ~/Pictures

Examples

# Save the report to a file
REPORT=dupes.txt ./find_duplicate_pictures.sh ~/Pictures

# Follow symlinks, scan two drives
FOLLOW_LINKS=1 ./find_duplicate_pictures.sh /mnt/c /mnt/d

# Force a specific hash tool
HASH_CMD="shasum -a 256" ./find_duplicate_pictures.sh ~/Pictures

Sample output

=== Step 1: scanning for picture files ===
  -> /tmp/picdup_test
  found 3 picture file(s)

=== Step 2: duplicates by file name ===
--- name: photo1.jpg ---
         100 bytes  /tmp/picdup_test/a/photo1.jpg
         120 bytes  /tmp/picdup_test/b/photo1.jpg
  1 name group(s) share the same file name

=== Step 3: duplicates by content (SHA-256) ===
  hashing 2 candidate file(s) with matching sizes...
--- identical content (sha256: 57e8310931615cb7...) ---
         100 bytes  /tmp/picdup_test/a/photo1.jpg
         100 bytes  /tmp/picdup_test/b/copy_of_photo1.jpg
  1 content group(s) are true duplicates
  approx. space wasted by extra copies: 100 bytes (100)

=== Step 4: same name, different content (false friends) ===
--- 'photo1.jpg' is used by 2 DIFFERENT pictures: ---
  [57e831093161] /tmp/picdup_test/a/photo1.jpg
  [unique-size] /tmp/picdup_test/b/photo1.jpg

Supported image formats

jpg, jpeg, png, gif, bmp, tiff, tif, webp, heic, heif, svg, ico, raw, cr2, nef, arw, dng, psd, avif, jfif

Matching is case-insensitive (.JPG, .Jpg, … all match). To change the list, edit the EXTENSIONS variable at the top of the script.

Notes & limitations

• Safety: the script is read-only — it never deletes or modifies files.
• Skipped paths: /proc, /sys, /dev, /run, /snap, /tmp are pruned during full-system scans (edit PRUNE_DIRS to change).
• Performance: only files whose size appears more than once are hashed, which keeps large collections fast. Whole-filesystem scans still take time.
• Filenames: names with spaces, tabs, or other special characters are handled correctly (NUL-delimited scanning).
• Permissions: unreadable files/directories are skipped silently.
• Exit codes: 0 on success; 1 if no hashing tool is available or the temp directory cannot be created.

Requirements

• Bash 4+ (for mapfile)
• One of: sha256sum, shasum, or md5sum
• Standard tools: find, awk, sort, stat, cut, grep, wc (all present on Linux/macOS/Git Bash/WSL)