View difference between Paste ID: wXyiB4pY and znXyf01z

 Data Hoarding General /dhg/ (sauce - https://github.com/simon987/awesome-datahoarding)
 
 ### Web Archiving
 * Collect - https://github.com/xarantolus/Collect: A server to collect & archive websites that also supports video downloads
 * grab-site - https://github.com/ludios/grab-site: The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
 * Heritrix - https://github.com/internetarchive/heritrix3: Extensible, web-scale, archival-quality web crawler
 * HTTrack - https://www.httrack.com/: Download a website from the Internet to a local directory
 * wail - https://github.com/machawk1/wail: Web Archiving Integration Layer: One-Click User Instigated Preservation
 * wikiteam - https://github.com/WikiTeam/wikiteam: set of tools for archiving wikis
 
 ### General
 * annie - https://github.com/iawia002/annie: Youtube-DL alternative writtent in Golang
 * aria2 - https://aria2.github.io/: A lightweight multi-protocol & multi-source command-line download utility
 * CrowLeer - https://github.com/ERap320/CrowLeer: Powerful C++ web crawler based on libcurl
 * curl - https://github.com/curl/curl: Tool and library for transferring data with URL syntax, supporting many protocols
 * Plowshare - https://github.com/mcrapet/plowshare: Command-line tool to manage file-sharing site
 * Rclone - https://github.com/ncw/rclone: A command line program to sync files and directories to and from various cloud storage providers
 * wget - https://savannah.gnu.org/git/?group=wget: Utility for non-interactive download of files from
 * you-get - https://github.com/soimort/you-get: Dumb downloader that scrapes the web
 * Youtube-DL - https://github.com/rg3/youtube-dl: A command-line program to download videos from YouTube and a few hundred more sites
 
 ### Application-specific
 * ChanThreadWatch - https://github.com/SuperGouge/ChanThreadWatch: Saves threads from \*chan-style boards and checks for updates until the thread dies
 * floatplane_ripper - https://gist.github.com/simon987/0756c378ca2dfb0003931e26ff7fe270: Script to rip all videos from https://floatplane.rip/
 * gallery-dl - https://github.com/mikf/gallery-dl: Fownload image galleries and collections from pixiv, exhentai, danbooru and more
 * dzi-dl - https://github.com/ryanfb/dzi-dl: Deep Zoom Image Downloader
 * FanFicFare - https://github.com/JimmXinu/FanFicFare: Tool for making eBooks from stories on fanfiction and other web sites
 * FicSave - https://github.com/waylaidwanderer/FicSave: Online fanfiction downloader
 * Google Images Download - https://github.com/hardikvasa/google-images-download: Python script for downloading images
 * iiif-dl - https://github.com/ryanfb/iiif-dl: Command-line tile downloader/assembler for IIIF endpoints/manifests
 * Instagram Scraper - https://github.com/dankmemes/instagram-scraper: Instagram-scraper is a command-line application written in Python that scrapes and downloads an instagram user's photos and videos. Use responsibly.
 * PyInstaLive - https://github.com/notcammy/PyInstaLive: Instagram live stream downloader.
 * RedditDownloader - https://github.com/shadowmoose/RedditDownloader: Scrapes Reddit to download media of your choice
 * Scribd-Downloader - https://github.com/ritiek/scribd-downloader: Allows downloading of Scribd documents
 * RipMe - https://github.com/RipMeApp/ripme: RipMe is an album ripper for various websites. Runs on your computer. Requires Java 8.
 * yt-mango - https://github.com/terorie/yt-mango: Youtube metadata archiver
 the Web (HTTP & FTP)
 * Youtube-MA - https://github.com/CorentinB/YouTube-MA: Youtube metadata archiver
 
 ### Download automation
 * bazarr - https://github.com/morpheus65535/bazarr: Companion application to Sonarr and Radarr for downloading subtitles
 * FlexGet - https://github.com/Flexget/Flexget: Multipurpose automation tool for content like torrents, nzbs, podcasts, comics, series, movies, etc
 * Jackett - https://github.com/Jackett/Jackett: API support for torrent trackers (works with Sonarr, Radarr and others)
 * Lidarr - https://github.com/lidarr/Lidarr: Music collection manager for Usenet and BitTorrent users
 * Mylar - https://github.com/evilhero/mylar: An automated Comic Book downloader (cbr/cbz) for use with SABnzbd, NZBGet and torrents
 * Sick-Beard - https://github.com/midgetspy/Sick-Beard: PVR for newsgroup users (with limited torrent support)
 * Radarr - https://github.com/Radarr/Radarr: A fork of Sonarr to work with movies à la Couchpotato
 * Sonarr - https://github.com/Sonarr/Sonarr: PVR for Usenet and BitTorrent users
 
 ## Handling Data Rot and it's Corruption
 * m5 deep - http://md5deep.sourceforge.net/: md5deep is a set of programs to compute MD5, SHA-1, SHA-256, Tiger, or Whirlpool message digests on an arbitrary number of files.
 
 ## Compression
 * KGB Archiver - https://github.com/RandallFlagg/kgbarchiver: compression tool with unbelievable high compression rate
 * peazip - http://www.peazip.org/: File archiver utility
 * PIGZ - https://zlib.net/pigz/: Multi-threaded gzip
 * WinRAR - https://www.rarlab.com/download.htm: Can decompress RAR and zip files.
 
 ## Network
 * NetLimiter - https://www.netlimiter.com/: Internet traffic control and monitoring tool for Windows
 
 ## File systems
 * httpdirfs - https://github.com/fangfufu/httpdirfs/:  A filesystem which allows you to mount HTTP directory listings
 * mergerfs - https://github.com/trapexit/mergerfs: a featureful union filesystem
 * NTFS drivers for MacOS - https://www.seagate.com/ca/en/support/downloads/item/ntfs-driver-for-mac-os-master-dl/
 
 ## File conversion
 * AAXtoMP3 - https://github.com/KrumpetPirate/AAXtoMP3: convert AAX files to common MP3, M4A, M4B, flac and ogg formats through a basic bash script frontend to FFMPEG
 * html2warc - https://github.com/steffenfritz/html2warc: Convert web resources to a single warc file
 
 
 ## Utility Scripts
 * Backblaze B2 sync backup script - https://gist.github.com/AlexanderProd/cb645cf858fd5c89780e7df267226b80: Script to sync mutliple directories with Backblaze B2
 * Misc download scripts - https://github.com/simon987/Misc-Download-Scripts: Scripts for downloading content from various websites
 * rclone_dirsize - https://gist.github.com/simon987/7aff5ca3e9ae6c755055ca7b350ef9f8: Get size of http directory listing with rclone
 * rm_empty_subdir - https://gist.github.com/simon987/f5c2cd7602898615ac9bc8c762d9fe1d: Remove empty sub-directories on Windows
 * void-cat-uploader - https://github.com/takky1154/void-cat-uploader: This script automatically uploads all files inside a directory to https://void.cat.
 * youtube-dl_soundcloud - https://gist.github.com/simon987/2dd7c57d65a741c93f5791bc984b97d1: snippet for using youtube-dl to download soundcloud playlists
 
 ## Content sharing
 * h5ai - https://github.com/lrsjng/h5ai: HTTP web server index for Apache httpd, lighttpd, nginx and Cherokee
 * ipfs - https://ipfs.io/: Protocol and network designed to create a content-addressable, peer-to-peer method of storing and sharing hypermedia in a distributed file system
 * opds - https://opds.io/: Easy to use, Open & Decentralized Content Distribution
 
 ## Data curation
 * baobab - https://github.com/GNOME/baobab: Graphical disk usage analyzer
 * beets - https://github.com/beetbox/beets: Music library manager and MusicBrainz tagger
 * Calibre - https://github.com/kovidgoyal/calibre: Ebook manager
 * DeepSort - https://github.com/CorentinB/DeepSort/: AI powered image tagger backed by DeepDetect
 * diskover - https://github.com/shirosaidev/diskover: File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch
 * Everything - https://www.voidtools.com/: Locate files and folders by name instantly (Windows)
 * FileBot - https://www.filebot.net/: FileBot is the ultimate tool for organizing and renaming your Movies, TV Shows and Anime
 * fucking-weeb - https://github.com/cosarara/fucking-weeb: A library manager for animu (and TV shows, and whatever).
 * grepWin - https://github.com/stefankueng/grepWin: A powerful and fast search tool using regular expressions (Windows)
 * jdupes - https://github.com/jbruchon/jdupes: Powerful duplicate file finder
 * MediaElch - https://github.com/komet/mediaelch: Media manager for Kodi
 * MediaInfo - https://github.com/MediaArea/MediaInfo: Convenient unified display of the most relevant technical and tag data for video and audio files
 * Mp3tag - https://www.mp3tag.de: Powerful and easy-to-use tool to edit metadata of audio files (Windows/Mac)
 * phockup - https://github.com/ivandokov/phockup: Media sorting tool to organize photos and videos from your camera
 * picard - https://github.com/metabrainz/picard: MusicBrainz tagger
 * TeraCopy - https://www.codesector.com/downloads: Copy your files faster and more securely
 * tree - http://mama.indstate.edu/users/ice/tree/: 'tree' command for linux
 * WinDirStat - https://windirstat.net/: Disk usage statistics viewer and cleanup tool for Windows
 * SyncToy - https://www.microsoft.com/en-us/download/details.aspx?id=15155: Microsoft windows file parity across locations tool
 * DupeGuru - https://dupeguru.voltaicideas.net/: finds duplicate files
 
 ## File Utilities
 * __Batch Renamer__: GPRename - http://gprename.sourceforge.net/ -> qmv (renameutils) - http://www.nongnu.org/renameutils/
 * __File Archiver__: PeaZip -> Xarchiver -> Atool - http://www.nongnu.org/atool/
 * __File Search__: DocFetcher - http://docfetcher.sourceforge.net/en/index.html -> ANGRYsearch - https://github.com/DoTheEvo/ANGRYsearch -> Puggle - http://puggle.sourceforge.net/ -> regain - http://regain.sourceforge.net/index.php -> find
 * __File Synchronization__: Unison - https://github.com/bcpierce00/unison -> git-annex - https://git-annex.branchable.com/ -> Rsync
 * __Image Organizer__: hydrus network -> Shotwell -> GTKRawGallery -> digiKam -> gThumb (+ gphoto) -> Mapivi - http://mapivi.sourceforge.net/mapivi.shtml -> BASH-Booru - https://github.com/ChristianSilvermoon/BASH-Booru
 * __RegEx Builder__: regexxer - https://directory.fsf.org/wiki/Regexxer -> Visual REGEXP - http://laurent.riesterer.free.fr/regexp/ -> txt2regex - https://github.com/aureliojargas/txt2regex
 
 ## Filesharing
 * __Direct Connect__: LinuxDC++ -> ncdc - https://github.com/srijan/ncdc -> microdc2  -http://corsair626.no-ip.org/microdc/
 * __Download Manager__: giFT - https://sourceforge.net/projects/gift/ + giFTcurs - http://www.nongnu.org/giftcurs/ -> aria2 - https://aria2.github.io/ -> cURL -> Wget
 * __File Scraper__: megatools -> JDownloader - https://github.com/Bobmk/JDownloader) -> Plowshare - https://github.com/mcrapet/plowshare
 * __FTP Client__: FileZilla -> lftp - https://github.com/lavv17/lftp
 * __LAN Sharing__: NitroShare -> Dukto
 * __Media Center__: Plex -> Emby -> Popcorn Time -> Kodi ("XBMC", + Sonarr)
 * __Media Miner__: FlexGet -> Sonarr - https://github.com/Sonarr/Sonarr
 * __Offline Reader__: Kiwix - http://www.kiwix.org/ -> Darcy Ripper -> HTTrack -> Wget
 * __Soulseek__: Nicotine Plus -> Museek (mucous) - https://museek-plus.org/
 * __Stream Catcher__: Streamripper -> youtube-dl -> cclive - https://github.com/legatvs/cclive -> youtube-pl - http://ronja.twibright.com/youtube-pl.php -> quvi - https://github.com/mogaal/quvi, RTMPDump - https://github.com/mstorsjo/rtmpdump
 * __Torrent Client__: qBittorrent -> RTorrent -> transmission-daemon (comes with a web interface - https://github.com/transmission/transmission/wiki/Web-Interface by default but other frontends - https://github.com/fagga/transmission-remote-cli exist.
 * __Torrent Tracker Scraper__: Torrtux - https://github.com/l333k0/torrtux -> Torrench - https://github.com/kryptxy/torrench -> Jackett - https://github.com/Jackett/Jackett
 * __Usenet (File Grabber)__: LottaNZB -> SABnzbd -> NZBGet - https://github.com/nzbget/nzbget -> nzb - https://directory.fsf.org/wiki/Nzb -> nzbperl - https://github.com/eghm/nzbperl
 
 ## Command Line Tools
 * __Command Line Cheatsheet__: CLI Companion - https://launchpad.net/clicompanion -> xman -> cheat / howdoi / clf / fu / bro -> cheat.sh - https://github.com/chubin/cheat.sh
 * __Directory Browsing__: fasd - https://github.com/clvv/fasd, xd - https://github.com/fbb-git/xd, fzy - https://github.com/jhawthorn/fzy
 * __Framebuffer Environment__: Fbterm - https://code.google.com/archive/p/fbterm/ -> yaft (because sixel) - https://github.com/uobikiemukot/yaft -> hterm (because regis) - https://github.com/new299/hackterm
 * __Hacker Culture__: ddate, fortune, The Hacker Test, The Jargon File
 * __Multiplexer__: Tmux  -> Byobu -> GNU Screen (+sixel patch - https://gist.github.com/saitoha/7546579
 * __Progress Viewers__: progress - https://github.com/Xfennec/progress) -> pv - Pipe Viewer - https://github.com/icetee/pv -> Advanced Copy - https://github.com/atdt/advcpmv
 
 ## Disk Tools
 * __CD-DVD Burn and Copy (Backends)__: cdrtools -> cdrkit -> cdrskin - https://dev.lovelyhq.com/libburnia/web/wikis/cdrskin
 * __CD-DVD Burn and Copy (Frontends)__: K3b -> Brasero -> cdw - http://cdw.sourceforge.net/
 * __CD-DVD Ripping__: Sound Juicer -> fre ac -> cdparanoia - https://www.xiph.org/paranoia/ (+ ABCDE - http://lly.org/~rcw/abcde/page/
 * __Custom Install CD__: Respin -> Remastersys -> Distroshare -> PinguyBuilder -> Customizer -> Ubuntu Customization Kit -> Mklivecd
 * __Device Management__: Udisks (+ udevil) -> pmount -> bashmount  -https://github.com/jamielinux/bashmount/blob/master/INSTALL
 * __Disk Cloning and Writing__: dd -> dcfldd -> dc3dd - https://sourceforge.net/projects/dc3dd/
 * __Live USB__: UNetbootin -> MultiCD - https://multicd.us/
 * __Partitioning__: Gparted -> cfdisk -> GNU Parted -> fdisk / sfdisk
 * __System Backup__: Systemback - https://sourceforge.net/projects/systemback/ -> Bacula - https://blog.bacula.org/ -> FSArchiver - http://www.fsarchiver.org/ -> CYA - https://www.cyberws.com/bash/cya/
 
 ## APIs & Online tools
 * iqdb - https://iqdb.org/: Multi-service reverse image search
 * thetvdb - https://www.thetvdb.com/: TV shows metadata (used by plex)
 
 ## Hardware / Monitoring
 * CrystalDiskInfo - https://crystalmark.info/en/software/crystaldiskinfo/: A HDD/SSD utility software which supports a part of USB, Intel RAID and NVMe.
 * Hard Drive Sentinel - https://www.hdsentinel.com/: Multi-OS SSD and HDD monitoring and analysis software
 * smartmontools - https://www.smartmontools.org/: Control and monitor storage systems using the (SMART) built into most modern ATA/SATA, SCSI/SAS and NVMe disks
 
 ## Data recovery
 * PhotoRec - https://www.cgsecurity.org/wiki/PhotoRec: FOSS powerful gui data recovery tool.
 * TestDisk - https://www.cgsecurity.org/wiki/TestDisk_Download: Another FOSS tool by the author of PhotoRec, but this one is for cli