Guest User

data hoarding general - 4ch

a guest
Jun 23rd, 2020
251
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!

Data Hoarding General /dhg/ (sauce - https://github.com/simon987/awesome-datahoarding)

Web Archiving

  • Collect - https://github.com/xarantolus/Collect: A server to collect & archive websites that also supports video downloads
  • grab-site - https://github.com/ludios/grab-site: The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
  • Heritrix - https://github.com/internetarchive/heritrix3: Extensible, web-scale, archival-quality web crawler
  • HTTrack - https://www.httrack.com/: Download a website from the Internet to a local directory
  • wail - https://github.com/machawk1/wail: Web Archiving Integration Layer: One-Click User Instigated Preservation
  • wikiteam - https://github.com/WikiTeam/wikiteam: set of tools for archiving wikis

General

  • annie - https://github.com/iawia002/annie: Youtube-DL alternative writtent in Golang
  • aria2 - https://aria2.github.io/: A lightweight multi-protocol & multi-source command-line download utility
  • CrowLeer - https://github.com/ERap320/CrowLeer: Powerful C++ web crawler based on libcurl
  • curl - https://github.com/curl/curl: Tool and library for transferring data with URL syntax, supporting many protocols
  • Plowshare - https://github.com/mcrapet/plowshare: Command-line tool to manage file-sharing site
  • Rclone - https://github.com/ncw/rclone: A command line program to sync files and directories to and from various cloud storage providers
  • wget - https://savannah.gnu.org/git/?group=wget: Utility for non-interactive download of files from
  • you-get - https://github.com/soimort/you-get: Dumb downloader that scrapes the web
  • Youtube-DL - https://github.com/rg3/youtube-dl: A command-line program to download videos from YouTube and a few hundred more sites

Application-specific

  • ChanThreadWatch - https://github.com/SuperGouge/ChanThreadWatch: Saves threads from *chan-style boards and checks for updates until the thread dies
  • floatplane_ripper - https://gist.github.com/simon987/0756c378ca2dfb0003931e26ff7fe270: Script to rip all videos from https://floatplane.rip/
  • gallery-dl - https://github.com/mikf/gallery-dl: Fownload image galleries and collections from pixiv, exhentai, danbooru and more
  • dzi-dl - https://github.com/ryanfb/dzi-dl: Deep Zoom Image Downloader
  • FanFicFare - https://github.com/JimmXinu/FanFicFare: Tool for making eBooks from stories on fanfiction and other web sites
  • FicSave - https://github.com/waylaidwanderer/FicSave: Online fanfiction downloader
  • Google Images Download - https://github.com/hardikvasa/google-images-download: Python script for downloading images
  • iiif-dl - https://github.com/ryanfb/iiif-dl: Command-line tile downloader/assembler for IIIF endpoints/manifests
  • Instagram Scraper - https://github.com/dankmemes/instagram-scraper: Instagram-scraper is a command-line application written in Python that scrapes and downloads an instagram user's photos and videos. Use responsibly.
  • PyInstaLive - https://github.com/notcammy/PyInstaLive: Instagram live stream downloader.
  • RedditDownloader - https://github.com/shadowmoose/RedditDownloader: Scrapes Reddit to download media of your choice
  • Scribd-Downloader - https://github.com/ritiek/scribd-downloader: Allows downloading of Scribd documents
  • RipMe - https://github.com/RipMeApp/ripme: RipMe is an album ripper for various websites. Runs on your computer. Requires Java 8.
  • yt-mango - https://github.com/terorie/yt-mango: Youtube metadata archiver
    the Web (HTTP & FTP)
  • Youtube-MA - https://github.com/CorentinB/YouTube-MA: Youtube metadata archiver

Download automation

  • bazarr - https://github.com/morpheus65535/bazarr: Companion application to Sonarr and Radarr for downloading subtitles
  • FlexGet - https://github.com/Flexget/Flexget: Multipurpose automation tool for content like torrents, nzbs, podcasts, comics, series, movies, etc
  • Jackett - https://github.com/Jackett/Jackett: API support for torrent trackers (works with Sonarr, Radarr and others)
  • Lidarr - https://github.com/lidarr/Lidarr: Music collection manager for Usenet and BitTorrent users
  • Mylar - https://github.com/evilhero/mylar: An automated Comic Book downloader (cbr/cbz) for use with SABnzbd, NZBGet and torrents
  • Sick-Beard - https://github.com/midgetspy/Sick-Beard: PVR for newsgroup users (with limited torrent support)
  • Radarr - https://github.com/Radarr/Radarr: A fork of Sonarr to work with movies à la Couchpotato
  • Sonarr - https://github.com/Sonarr/Sonarr: PVR for Usenet and BitTorrent users

Handling Data Rot and it's Corruption

  • m5 deep - http://md5deep.sourceforge.net/: md5deep is a set of programs to compute MD5, SHA-1, SHA-256, Tiger, or Whirlpool message digests on an arbitrary number of files.

Compression

  • KGB Archiver - https://github.com/RandallFlagg/kgbarchiver: compression tool with unbelievable high compression rate
  • peazip - http://www.peazip.org/: File archiver utility
  • PIGZ - https://zlib.net/pigz/: Multi-threaded gzip
  • WinRAR - https://www.rarlab.com/download.htm: Can decompress RAR and zip files.

Network

  • NetLimiter - https://www.netlimiter.com/: Internet traffic control and monitoring tool for Windows

File systems

  • httpdirfs - https://github.com/fangfufu/httpdirfs/: A filesystem which allows you to mount HTTP directory listings
  • mergerfs - https://github.com/trapexit/mergerfs: a featureful union filesystem
  • NTFS drivers for MacOS - https://www.seagate.com/ca/en/support/downloads/item/ntfs-driver-for-mac-os-master-dl/

File conversion

  • AAXtoMP3 - https://github.com/KrumpetPirate/AAXtoMP3: convert AAX files to common MP3, M4A, M4B, flac and ogg formats through a basic bash script frontend to FFMPEG
  • html2warc - https://github.com/steffenfritz/html2warc: Convert web resources to a single warc file

Utility Scripts

  • Backblaze B2 sync backup script - https://gist.github.com/AlexanderProd/cb645cf858fd5c89780e7df267226b80: Script to sync mutliple directories with Backblaze B2
  • Misc download scripts - https://github.com/simon987/Misc-Download-Scripts: Scripts for downloading content from various websites
  • rclone_dirsize - https://gist.github.com/simon987/7aff5ca3e9ae6c755055ca7b350ef9f8: Get size of http directory listing with rclone
  • rm_empty_subdir - https://gist.github.com/simon987/f5c2cd7602898615ac9bc8c762d9fe1d: Remove empty sub-directories on Windows
  • void-cat-uploader - https://github.com/takky1154/void-cat-uploader: This script automatically uploads all files inside a directory to https://void.cat.
  • youtube-dl_soundcloud - https://gist.github.com/simon987/2dd7c57d65a741c93f5791bc984b97d1: snippet for using youtube-dl to download soundcloud playlists

Content sharing

  • h5ai - https://github.com/lrsjng/h5ai: HTTP web server index for Apache httpd, lighttpd, nginx and Cherokee
  • ipfs - https://ipfs.io/: Protocol and network designed to create a content-addressable, peer-to-peer method of storing and sharing hypermedia in a distributed file system
  • opds - https://opds.io/: Easy to use, Open & Decentralized Content Distribution

Data curation

  • baobab - https://github.com/GNOME/baobab: Graphical disk usage analyzer
  • beets - https://github.com/beetbox/beets: Music library manager and MusicBrainz tagger
  • Calibre - https://github.com/kovidgoyal/calibre: Ebook manager
  • DeepSort - https://github.com/CorentinB/DeepSort/: AI powered image tagger backed by DeepDetect
  • diskover - https://github.com/shirosaidev/diskover: File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch
  • Everything - https://www.voidtools.com/: Locate files and folders by name instantly (Windows)
  • FileBot - https://www.filebot.net/: FileBot is the ultimate tool for organizing and renaming your Movies, TV Shows and Anime
  • fucking-weeb - https://github.com/cosarara/fucking-weeb: A library manager for animu (and TV shows, and whatever).
  • grepWin - https://github.com/stefankueng/grepWin: A powerful and fast search tool using regular expressions (Windows)
  • jdupes - https://github.com/jbruchon/jdupes: Powerful duplicate file finder
  • MediaElch - https://github.com/komet/mediaelch: Media manager for Kodi
  • MediaInfo - https://github.com/MediaArea/MediaInfo: Convenient unified display of the most relevant technical and tag data for video and audio files
  • Mp3tag - https://www.mp3tag.de: Powerful and easy-to-use tool to edit metadata of audio files (Windows/Mac)
  • phockup - https://github.com/ivandokov/phockup: Media sorting tool to organize photos and videos from your camera
  • picard - https://github.com/metabrainz/picard: MusicBrainz tagger
  • TeraCopy - https://www.codesector.com/downloads: Copy your files faster and more securely
  • tree - http://mama.indstate.edu/users/ice/tree/: 'tree' command for linux
  • WinDirStat - https://windirstat.net/: Disk usage statistics viewer and cleanup tool for Windows
  • SyncToy - https://www.microsoft.com/en-us/download/details.aspx?id=15155: Microsoft windows file parity across locations tool
  • DupeGuru - https://dupeguru.voltaicideas.net/: finds duplicate files

File Utilities

  • Batch Renamer: GPRename - http://gprename.sourceforge.net/ -> qmv (renameutils) - http://www.nongnu.org/renameutils/
  • File Archiver: PeaZip -> Xarchiver -> Atool - http://www.nongnu.org/atool/
  • File Search: DocFetcher - http://docfetcher.sourceforge.net/en/index.html -> ANGRYsearch - https://github.com/DoTheEvo/ANGRYsearch -> Puggle - http://puggle.sourceforge.net/ -> regain - http://regain.sourceforge.net/index.php -> find
  • File Synchronization: Unison - https://github.com/bcpierce00/unison -> git-annex - https://git-annex.branchable.com/ -> Rsync
  • Image Organizer: hydrus network -> Shotwell -> GTKRawGallery -> digiKam -> gThumb (+ gphoto) -> Mapivi - http://mapivi.sourceforge.net/mapivi.shtml -> BASH-Booru - https://github.com/ChristianSilvermoon/BASH-Booru
  • RegEx Builder: regexxer - https://directory.fsf.org/wiki/Regexxer -> Visual REGEXP - http://laurent.riesterer.free.fr/regexp/ -> txt2regex - https://github.com/aureliojargas/txt2regex

Filesharing

  • Direct Connect: LinuxDC++ -> ncdc - https://github.com/srijan/ncdc -> microdc2 -http://corsair626.no-ip.org/microdc/
  • Download Manager: giFT - https://sourceforge.net/projects/gift/ + giFTcurs - http://www.nongnu.org/giftcurs/ -> aria2 - https://aria2.github.io/ -> cURL -> Wget
  • File Scraper: megatools -> JDownloader - https://github.com/Bobmk/JDownloader) -> Plowshare - https://github.com/mcrapet/plowshare
  • FTP Client: FileZilla -> lftp - https://github.com/lavv17/lftp
  • LAN Sharing: NitroShare -> Dukto
  • Media Center: Plex -> Emby -> Popcorn Time -> Kodi ("XBMC", + Sonarr)
  • Media Miner: FlexGet -> Sonarr - https://github.com/Sonarr/Sonarr
  • Offline Reader: Kiwix - http://www.kiwix.org/ -> Darcy Ripper -> HTTrack -> Wget
  • Soulseek: Nicotine Plus -> Museek (mucous) - https://museek-plus.org/
  • Stream Catcher: Streamripper -> youtube-dl -> cclive - https://github.com/legatvs/cclive -> youtube-pl - http://ronja.twibright.com/youtube-pl.php -> quvi - https://github.com/mogaal/quvi, RTMPDump - https://github.com/mstorsjo/rtmpdump
  • Torrent Client: qBittorrent -> RTorrent -> transmission-daemon (comes with a web interface - https://github.com/transmission/transmission/wiki/Web-Interface by default but other frontends - https://github.com/fagga/transmission-remote-cli exist.
  • Torrent Tracker Scraper: Torrtux - https://github.com/l333k0/torrtux -> Torrench - https://github.com/kryptxy/torrench -> Jackett - https://github.com/Jackett/Jackett
  • Usenet (File Grabber): LottaNZB -> SABnzbd -> NZBGet - https://github.com/nzbget/nzbget -> nzb - https://directory.fsf.org/wiki/Nzb -> nzbperl - https://github.com/eghm/nzbperl

Command Line Tools

  • Command Line Cheatsheet: CLI Companion - https://launchpad.net/clicompanion -> xman -> cheat / howdoi / clf / fu / bro -> cheat.sh - https://github.com/chubin/cheat.sh
  • Directory Browsing: fasd - https://github.com/clvv/fasd, xd - https://github.com/fbb-git/xd, fzy - https://github.com/jhawthorn/fzy
  • Framebuffer Environment: Fbterm - https://code.google.com/archive/p/fbterm/ -> yaft (because sixel) - https://github.com/uobikiemukot/yaft -> hterm (because regis) - https://github.com/new299/hackterm
  • Hacker Culture: ddate, fortune, The Hacker Test, The Jargon File
  • Multiplexer: Tmux -> Byobu -> GNU Screen (+sixel patch - https://gist.github.com/saitoha/7546579
  • Progress Viewers: progress - https://github.com/Xfennec/progress) -> pv - Pipe Viewer - https://github.com/icetee/pv -> Advanced Copy - https://github.com/atdt/advcpmv

Disk Tools

  • CD-DVD Burn and Copy (Backends): cdrtools -> cdrkit -> cdrskin - https://dev.lovelyhq.com/libburnia/web/wikis/cdrskin
  • CD-DVD Burn and Copy (Frontends): K3b -> Brasero -> cdw - http://cdw.sourceforge.net/
  • CD-DVD Ripping: Sound Juicer -> fre ac -> cdparanoia - https://www.xiph.org/paranoia/ (+ ABCDE - http://lly.org/~rcw/abcde/page/
  • Custom Install CD: Respin -> Remastersys -> Distroshare -> PinguyBuilder -> Customizer -> Ubuntu Customization Kit -> Mklivecd
  • Device Management: Udisks (+ udevil) -> pmount -> bashmount -https://github.com/jamielinux/bashmount/blob/master/INSTALL
  • Disk Cloning and Writing: dd -> dcfldd -> dc3dd - https://sourceforge.net/projects/dc3dd/
  • Live USB: UNetbootin -> MultiCD - https://multicd.us/
  • Partitioning: Gparted -> cfdisk -> GNU Parted -> fdisk / sfdisk
  • System Backup: Systemback - https://sourceforge.net/projects/systemback/ -> Bacula - https://blog.bacula.org/ -> FSArchiver - http://www.fsarchiver.org/ -> CYA - https://www.cyberws.com/bash/cya/

APIs & Online tools

  • iqdb - https://iqdb.org/: Multi-service reverse image search
  • thetvdb - https://www.thetvdb.com/: TV shows metadata (used by plex)

Hardware / Monitoring

  • CrystalDiskInfo - https://crystalmark.info/en/software/crystaldiskinfo/: A HDD/SSD utility software which supports a part of USB, Intel RAID and NVMe.
  • Hard Drive Sentinel - https://www.hdsentinel.com/: Multi-OS SSD and HDD monitoring and analysis software
  • smartmontools - https://www.smartmontools.org/: Control and monitor storage systems using the (SMART) built into most modern ATA/SATA, SCSI/SAS and NVMe disks

Data recovery

  • PhotoRec - https://www.cgsecurity.org/wiki/PhotoRec: FOSS powerful gui data recovery tool.
  • TestDisk - https://www.cgsecurity.org/wiki/TestDisk_Download: Another FOSS tool by the author of PhotoRec, but this one is for cli
RAW Paste Data