- Jul 19, 2020
- 18
- 13
- 669
Disclaimer:
- This script is being made completely with AI.
- I do not know how to code.
- I put all the features I wanted in, tested it and completely happy with how it works.
- Please don't make a comment about AI etc.
- If you want to try, you are welcome to do so.
- If you like it, please leave a comment.
You don't need to save submissions or search wayback one by one anymore. It will download archived submissions, hidden profiles, redgifs profiles. Everything is automated.
Usage:
Change the file name in the command according to script's version number.
To use it under Windows, you need python3 and powershell (comes with the system). It is not hard. Search for tutorials.
For a single user
python arctic_reddit_downloader REDDITUSERNAME
For a batch download you need to create a usernames text file next to the script with 1 username per line:
python arctic_reddit_downloader usernames.txt
The script works over Arctic. Scans the profile 100 submissions per page, collects media links and downloads. If the media cannot be downloaded, tries archive.org. For older submissions, I also added Imgur support.
The script supports redgifs profile downloading. It only works with single downloads since batch download complicate things. If script detects a redgifs link it downloads the media in both modes. In single mode however, it asks if you want to download the whole redgifs profile and keeps the media in seperate Redgifs folder.
I have been working on it for a while. Fixing errors I've encountered.
Can update with fixes and more features. Check the link once in a while.
The complete feature list is below.
[Login or Register to view]
- This script is being made completely with AI.
- I do not know how to code.
- I put all the features I wanted in, tested it and completely happy with how it works.
- Please don't make a comment about AI etc.
- If you want to try, you are welcome to do so.
- If you like it, please leave a comment.
You don't need to save submissions or search wayback one by one anymore. It will download archived submissions, hidden profiles, redgifs profiles. Everything is automated.
Usage:
Change the file name in the command according to script's version number.
To use it under Windows, you need python3 and powershell (comes with the system). It is not hard. Search for tutorials.
For a single user
python arctic_reddit_downloader REDDITUSERNAME
For a batch download you need to create a usernames text file next to the script with 1 username per line:
python arctic_reddit_downloader usernames.txt
The script works over Arctic. Scans the profile 100 submissions per page, collects media links and downloads. If the media cannot be downloaded, tries archive.org. For older submissions, I also added Imgur support.
The script supports redgifs profile downloading. It only works with single downloads since batch download complicate things. If script detects a redgifs link it downloads the media in both modes. In single mode however, it asks if you want to download the whole redgifs profile and keeps the media in seperate Redgifs folder.
I have been working on it for a while. Fixing errors I've encountered.
Can update with fixes and more features. Check the link once in a while.
The complete feature list is below.
Code:
ARCTIC REDDIT DOWNLOADER — COMPLETE FEATURE LIST
1. PLATFORM SUPPORT
- Works on Linux, Windows 10 and Windows 11.
- Works with Python3.
- Automatically detects the operating system.
- Uses pathlib to create platform-compatible file paths.
- Requires no external Python packages.
- Uses only Python’s standard library.
2. RUNNING THE SCRIPT
Change the file name in the command according to script's version number.
Download one Reddit user:
python arctic_reddit_downloader.py USERNAME
Process a list of users:
python arctic_reddit_downloader.py usernames.txt
Apply a date limit:
python arctic_reddit_downloader.py USERNAME --before YYYY-MM-DD
Retry only temporary errors:
python arctic_reddit_downloader.py USERNAME --retry-errors
3. ARCTIC SHIFT SEARCH
- Does not search for posts directly through Reddit.
- Retrieves the user’s posts through the Arctic Shift API.
- Reads results page by page, with up to 100 records per request.
- Checks up to 20,000 posts for each user.
- Prevents the same post from being processed twice.
- Supports an optional --before date limit.
- Extracts original media addresses from post records.
- Checks the url, url_overridden_by_dest and link_url fields.
- Checks media_metadata from Reddit galleries.
- Checks fallback_url for Reddit videos.
4. DIRECTORY STRUCTURE
- Saves downloads next to the script.
- Uses the following main directory structure: Reddit/USERNAME/
- Creates a separate directory for each Reddit user.
- Automatically removes unsafe characters from directory names.
- Never mixes content belonging to different Reddit users.
- Uses the post date, post ID and sequence number in filenames.
Example: Reddit/$username/2024-10-15_abc123_1.jpg
5. REDDIT MEDIA DOWNLOADS
- Downloads i.redd.it images.
- Processes v.redd.it video links.
- Downloads original files from Reddit galleries.
- Converts preview.redd.it URLs to original i.redd.it URLs when possible.
- Does not download external-preview.redd.it thumbnails as real media.
- Accepts only image/* and video/* content types.
- Does not save HTML, error, blocking or login pages as media files.
- Determines the appropriate file extension from the URL or content type.
6. REDDIT THUMBNAIL AND DELETED-MEDIA DETECTION
- Detects Reddit’s 130×60 deleted-media placeholder image.
- Does not save the placeholder as a real image.
- Sends deleted or unavailable media to the Archive.org queue.
- Rejects small external-preview.redd.it thumbnails.
7. IMGUR SUPPORT
- Processes direct i.imgur.com media URLs.
- Recognizes extensionless imgur.com post pages.
- Searches Archive.org for the original Imgur page if the current link fails.
- Extracts media URLs from archived Imgur HTML pages.
- Checks HTML metadata tags.
- Searches embedded JSON and JavaScript for older Imgur media URLs.
- Can process multiple media files found on one Imgur page.
- Detects Imgur’s 161×81 “The image you are requesting does not exist” placeholder.
- Does not save the Imgur placeholder as a real image.
- Removes unnecessary query-string duplicates of the same Imgur file.
8. REDGIFS POST SUPPORT
- Recognizes redgifs.com/watch/... URLs.
- Recognizes redgifs.com/ifr/... URLs.
- Automatically requests a temporary Redgifs access token.
- Reuses the same access token while it remains valid.
- Automatically refreshes an invalid or expired token.
- Selects the HD media version first.
- Falls back to the SD version when HD is unavailable.
- Supports both video and image content.
- Respects Retry-After instructions when Redgifs returns HTTP 429.
- Retries temporary Redgifs API failures.
9. REDGIFS PROFILE SELECTION FOR SINGLE-USER RUNS
- This feature is enabled only when a single Reddit username is provided: python arctic_reddit_downloader.py USERNAME
- Detects Redgifs links in the Arctic Shift results.
- Uses the Redgifs API to identify the owner of each detected post.
- Displays the detected Redgifs profile URL in the terminal: https://www.redgifs.com/users/PROFILE_NAME
- Offers three options:
[1] Download only Redgifs content shared on Reddit
[2] Download all images and videos from the Redgifs profile
[3] Skip this Redgifs profile
- If several Redgifs profiles are detected, each profile is presented
separately.
- If the profile owner cannot be identified, the profile selection is not shown.
- The profile question is not displayed during usernames.txt batch processing.
- Batch processing downloads only the Redgifs content shared on Reddit.
10. FULL REDGIFS PROFILE DOWNLOADS
- Downloads the complete profile when the user selects “Full profile.”
- Retrieves profile content page by page.
- Checks up to 100 API pages.
- Requests up to 40 profile items per page.
- Saves profile content in: Reddit/REDDIT_USERNAME/Redgifs/REDGIFS_PROFILE/
- Keeps each Redgifs profile in a separate directory.
- Does not mix profile content with the Reddit user’s main downloads.
- Selects HD media first and falls back to SD.
- Creates a separate sources file for every Redgifs profile: sources.txt
- Creates a separate progress state file for every Redgifs profile: download_state.json
- Does not download completed profile items again.
11. ARCHIVE.ORG SUPPORT
- Searches Archive.org when a direct media download fails.
- Uses the CDX API to find exact URL matches first.
- Considers only archived records with HTTP status 200.
- Collapses duplicate captures of the same content.
- Tries the oldest archived capture first.
- Tries the newest distinct capture if the oldest one fails.
- Uses raw id_ capture URLs for downloads.
- Tries the 1id_ and 2id_ shortcuts only if the CDX request itself fails.
- Does not repeat the same lookup through 1id_/2id_ when CDX confirms that no capture exists.
- Applies controlled intervals between Archive.org requests.
- Retries temporary HTTP 429, 500, 502, 503 and 504 errors.
- Respects the Retry-After response header.
- Does not save archived HTML pages as media files.
12. ARCHIVE.ORG CACHE
- Stores CDX results in archive_cdx_cache.json.
- Prevents unnecessary repeated Archive.org queries for the same URL.
- Caches both successful results and confirmed “no capture” results.
- Keeps cached records for seven days.
- Uses cache version control.
- Continues downloading even if the cache is damaged or cannot be written.
13. RESUME SUPPORT
- Creates download_state.json inside each user directory.
- Stores the processing status of every original media URL.
- Immediately skips completed URLs on later runs.
- Does not redownload a completed URL just to compare the resulting file.
- Distinguishes between temporary and permanent errors.
- Automatically skips permanently failed URLs during normal reruns.
- Allows temporary errors to be retried later.
- Preserves completed download records when the script is interrupted.
- Never considers an unfinished .part file complete.
14. FILE INTEGRITY AND DUPLICATE PREVENTION
- Calculates a SHA-256 hash for every downloaded file.
- Prevents the same content from being saved again under another URL or name.
- Scans existing files and calculates their hashes during the first run.
- Uses stored hashes on later runs for faster continuation.
- Checks an existing file with the same filename.
- Replaces the file atomically if its existing contents are incomplete or different.
- Downloads files using a temporary .part filename.
- Renames the .part file only after the download completes.
- Never overwrites a valid file with an incomplete download.
15. SOURCES FILE
- Creates sources.txt inside every Reddit user directory.
- Records the direct URL used for each successful download.
- The recorded source may be Reddit, Imgur, Redgifs or Archive.org.
- Does not write the same successful URL twice.
- Places unsuccessful URLs in a separate section at the bottom: --- FAILED DOWNLOADS ---
- Records the reason next to every unsuccessful URL.
- Removes a URL from the error section if it is downloaded successfully later.
- Can read the JSON-line sources format used by older script versions.
- Updates the sources file atomically through a temporary file.
16. USERNAMES.TXT BATCH PROCESSING
- Checks for usernames.txt next to the script.
- Treats each line as a separate Reddit username.
- Ignores empty lines.
- Ignores comment lines beginning with #.
- Removes duplicate usernames.
- Normalizes usernames written as u/USERNAME.
- Processes Reddit users sequentially.
- Processes only one to three Reddit user at a time depending on speed profile.
- Automatically starts the next user after the current user finishes.
17. BATCH QUEUE CHECKPOINT
- Stores batch progress in queue_control_point.json.
- Calculates a SHA-256 signature for the usernames.txt list.
- Detects an unfinished run using the same user list.
- Asks the user whether to resume or restart:
[K] Resume from the checkpoint
Start from the beginning
- Automatically resumes in a non-interactive terminal.
- Does not add already completed users back to the queue.
- Marks the queue as complete after all users finish.
18. LIVE USERNAMES.TXT RELOADING
- Reloads usernames.txt when R or r is pressed.
- Adds newly listed users to the pending queue.
- Removes deleted users from the pending queue.
- Does not interrupt or replace the user currently being processed.
- Does not re-add users already completed during the current run.
- Displays the number of added, removed and remaining users.
19. LIVE KEYBOARD CONTROLS ON LINUX
- P or p: Pause or resume downloads.
- R or r: Reload usernames.txt.
- S or s: Skip the current Reddit user.
- H or h: Display the keyboard help again.
- First Ctrl+C: Stop creating new jobs and allow the current file to finish.
- Second Ctrl+C: Force the program to exit immediately.
- Keyboard commands are case-insensitive.
20. LIVE KEYBOARD CONTROLS ON WINDOWS
- P or p: Pause or resume downloads.
- R or r: Reload usernames.txt.
- S or s: Skip the current Reddit user.
- H or h: Display the keyboard help again.
- Q or q: Start a safe shutdown.
- Safe shutdown prevents new jobs from starting.
- The active file is allowed to finish.
- Keyboard commands are case-insensitive.
21. PAUSE AND RESUME BEHAVIOR
- Pressing P pauses new network operations.
- Pressing P again resumes processing.
- Every consecutive P press toggles the current state.
- Pausing does not close the script.
- Queue and progress information remain available while paused.
22. DISK-SPACE PROTECTION
- Checks the free space on the output disk.
- Automatically pauses when free space drops below 5 GB.
- Displays the disk warning only once.
- Allows the user to continue manually by pressing P.
- Does not repeatedly display the same warning after the user chooses
to continue.
23. SPEED PROFILES
The script offers three speed profiles at startup:
[1] Slow
[2] Normal
[3] Fast
Slow:
- Uses longer intervals between Archive.org requests.
- Uses longer network timeouts.
- Runs one simultaneous direct Reddit download.
- May make additional Archive.org attempts after temporary failures.
Normal:
- Default profile.
- Runs up to three simultaneous direct Reddit downloads.
- Uses balanced Archive.org intervals and timeout values.
Fast:
- Uses shorter timeout values.
- Runs up to five simultaneous direct Reddit downloads.
- Sends Archive.org requests at shorter intervals.
- May produce more timeout errors on slower connections.
24. STARTUP CONNECTION TEST
The following services are checked before downloading:
- Arctic Shift
- Reddit
- Archive.org
- Displays “Available” or “Unavailable” for every tested service.
- Asks whether processing should continue when a service is unavailable.
- Does not start automatically in a non-interactive terminal when a required
service is unavailable.
- Redgifs is contacted only when Redgifs content is detected.
25. ERROR MANAGEMENT
Examples of permanent errors:
- No Archive.org CDX capture found.
- Content was deleted.
- Imgur content is no longer available.
- Confirmed HTTP 404 response.
- No Redgifs HD or SD media URL found.
Examples of temporary errors:
- Connection failure.
- Timeout.
- HTTP 429.
- HTTP 500.
- HTTP 502.
- HTTP 503.
- HTTP 504.
- Retries temporary network errors with increasing delays.
- Skips permanent errors during later normal runs.
- Records error reasons in sources.txt and download_state.json.
- Never reports an unsuccessful download as successful.
26. RETRYING ERRORS AFTER BATCH COMPLETION
- Runs only after every user in usernames.txt has been processed.
- Does not ask after every individual user.
- If temporary errors remain, asks: Retry temporary errors now? [Y/N]
- Selecting Y retries only temporary errors.
- Permanent errors are not retried.
- The --retry-errors argument starts the script directly in retry mode.
27. LIVE PROGRESS DISPLAY
- Displays the active Reddit username.
- Displays the number of Arctic Shift records found.
- Displays the total number of detected media items.
- Displays the number of processed media items.
- Displays the number of direct downloads.
- Displays the number of Archive.org downloads.
- Displays the number of skipped items.
- Displays the number of errors.
- Displays Archive.org queue progress.
- Displays Redgifs profile download progress.
- Updates the same status line in an interactive terminal.
- When output is redirected to a log file, reports progress periodically instead of producing one line for every file.
28. PER-USER SUMMARY
After each Reddit user finishes, the script displays a one-line summary with:
- Number of Arctic Shift records
- Number of media items
- Total downloads
- Direct downloads
- Archive.org downloads
- Skipped items
- Failed items
Possible states:
[COMPLETED] Completed
[SKIPPED] Skipped
[ERROR] Error
29. FINAL SUMMARY
After the complete run finishes, the script displays:
- Final state: completed or stopped
- Total number of users
- Total Arctic Shift records
- Total media items
- Total downloaded files
- Total direct downloads
- Total Archive.org downloads
- Total skipped items
- Total errors
- Permanent error count
- Temporary error count
- Total running time
30. CURRENT LIMITATIONS
- Processes Reddit posts, not Reddit comments.
- Can only process posts returned by Arctic Shift.
- Does not currently search for Redgifs URLs written only inside the post’s selftext/body.
- Redgifs profile searches are limited to 100 API pages.
- Deleted, private or restricted Redgifs content may be unavailable.
- An archived web page does not guarantee that its media file was archived.
- Some Reddit videos store audio separately; the script does not currently combine separate video and audio streams.
- Does not create new Archive.org captures.[Login or Register to view]