120 lines
4.2 KiB
Markdown
120 lines
4.2 KiB
Markdown
# bash-scripts
|
|
|
|
A collection of Bash utilities. Works on Linux, macOS, WSL, and Git Bash (Windows).
|
|
|
|
## Scripts
|
|
|
|
| Script | Purpose |
|
|
|---|---|
|
|
| [find_duplicate_pictures.sh](find_duplicate_pictures.sh) | Find duplicate pictures across one or more filesystems/directories using name **and** content comparison |
|
|
|
|
---
|
|
|
|
## find_duplicate_pictures.sh
|
|
|
|
Finds all picture files and reports duplicates. Because more than one picture
|
|
can legitimately share the same file name (e.g. `IMG_0001.jpg` from two
|
|
different cameras), the script does **not** trust names alone — it combines
|
|
three independent checks:
|
|
|
|
1. **Same name** — groups files sharing the same file name (case-insensitive).
|
|
Fast, but only a hint.
|
|
2. **Same content** *(authoritative)* — file size is used as a cheap
|
|
pre-filter, then SHA-256 hashes are computed **only for size-matched
|
|
candidates**. Identical hashes = true duplicates, regardless of name.
|
|
Also reports approximate disk space wasted by redundant copies.
|
|
3. **"False friends"** — files with the *same name but different content*
|
|
(the exact case where names are misleading).
|
|
|
|
### Usage
|
|
|
|
```bash
|
|
# Scan specific roots
|
|
./find_duplicate_pictures.sh /home/user/Pictures /mnt/backup
|
|
|
|
# Scan the whole filesystem (default when no arguments given)
|
|
./find_duplicate_pictures.sh
|
|
|
|
# On Windows (Git Bash) — drives are /c, /d, ...
|
|
"/c/Program Files/Git/bin/bash.exe" find_duplicate_pictures.sh /c /d
|
|
|
|
# Make it executable first if needed
|
|
chmod +x find_duplicate_pictures.sh
|
|
```
|
|
|
|
### Options (environment variables)
|
|
|
|
| Variable | Default | Description |
|
|
|---|---|---|
|
|
| `HASH_CMD` | auto-detected | Hash tool to use: `sha256sum`, `shasum -a 256`, or `md5sum` (last resort) |
|
|
| `FOLLOW_LINKS` | `0` (off) | Set to `1` to follow symlinks while scanning |
|
|
| `REPORT` | *(none)* | Path to a file where the full report is also saved |
|
|
|
|
### Examples
|
|
|
|
```bash
|
|
# Save the report to a file
|
|
REPORT=dupes.txt ./find_duplicate_pictures.sh ~/Pictures
|
|
|
|
# Follow symlinks, scan two drives
|
|
FOLLOW_LINKS=1 ./find_duplicate_pictures.sh /mnt/c /mnt/d
|
|
|
|
# Force a specific hash tool
|
|
HASH_CMD="shasum -a 256" ./find_duplicate_pictures.sh ~/Pictures
|
|
```
|
|
|
|
### Sample output
|
|
|
|
```
|
|
=== Step 1: scanning for picture files ===
|
|
-> /tmp/picdup_test
|
|
found 3 picture file(s)
|
|
|
|
=== Step 2: duplicates by file name ===
|
|
--- name: photo1.jpg ---
|
|
100 bytes /tmp/picdup_test/a/photo1.jpg
|
|
120 bytes /tmp/picdup_test/b/photo1.jpg
|
|
1 name group(s) share the same file name
|
|
|
|
=== Step 3: duplicates by content (SHA-256) ===
|
|
hashing 2 candidate file(s) with matching sizes...
|
|
--- identical content (sha256: 57e8310931615cb7...) ---
|
|
100 bytes /tmp/picdup_test/a/photo1.jpg
|
|
100 bytes /tmp/picdup_test/b/copy_of_photo1.jpg
|
|
1 content group(s) are true duplicates
|
|
approx. space wasted by extra copies: 100 bytes (100)
|
|
|
|
=== Step 4: same name, different content (false friends) ===
|
|
--- 'photo1.jpg' is used by 2 DIFFERENT pictures: ---
|
|
[57e831093161] /tmp/picdup_test/a/photo1.jpg
|
|
[unique-size] /tmp/picdup_test/b/photo1.jpg
|
|
```
|
|
|
|
### Supported image formats
|
|
|
|
`jpg`, `jpeg`, `png`, `gif`, `bmp`, `tiff`, `tif`, `webp`, `heic`, `heif`,
|
|
`svg`, `ico`, `raw`, `cr2`, `nef`, `arw`, `dng`, `psd`, `avif`, `jfif`
|
|
|
|
Matching is case-insensitive (`.JPG`, `.Jpg`, … all match). To change the
|
|
list, edit the `EXTENSIONS` variable at the top of the script.
|
|
|
|
### Notes & limitations
|
|
|
|
- **Safety:** the script is read-only — it never deletes or modifies files.
|
|
- **Skipped paths:** `/proc`, `/sys`, `/dev`, `/run`, `/snap`, `/tmp` are
|
|
pruned during full-system scans (edit `PRUNE_DIRS` to change).
|
|
- **Performance:** only files whose size appears more than once are hashed,
|
|
which keeps large collections fast. Whole-filesystem scans still take time.
|
|
- **Filenames:** names with spaces, tabs, or other special characters are
|
|
handled correctly (NUL-delimited scanning).
|
|
- **Permissions:** unreadable files/directories are skipped silently.
|
|
- **Exit codes:** `0` on success; `1` if no hashing tool is available or the
|
|
temp directory cannot be created.
|
|
|
|
### Requirements
|
|
|
|
- Bash 4+ (for `mapfile`)
|
|
- One of: `sha256sum`, `shasum`, or `md5sum`
|
|
- Standard tools: `find`, `awk`, `sort`, `stat`, `cut`, `grep`, `wc`
|
|
(all present on Linux/macOS/Git Bash/WSL)
|