/hsg/ + /dhg/ links

***************************************************************************************************************************************
Due to pastebin's new "Policy" of deleting random shit for no reason, the information in this paste has been migrated to the /g/ wiki.
If you would like to help contribute and make our page better please do so here.

> https://wiki.installgentoo.com/wiki/Home_server

Feel free to Keep this paste bookmarked, just know I will no longer be maintaining it as of now.
***************************************************************************************************************************************


Included here are some helpful links for those who are looking to start hoarding but don't really have a grasp of where to get started.
This list is a work in progress. I will add to it and change it over time.

----------Hardware-------------------------------------------------------------------------------------------------------------------

// Simple shucking guide.
https://www.ifixit.com/Guide/How+to+Shuck+a+WD+Elements+External+Hard+Drive/137646

// Price tracker for external drives.
https://shucks.top/

// WD Red guide.
https://www.westerndigital.com/products/internal-drives/wd-red-hdd

// List of SMR drives.
https://www.ixsystems.com/community/resources/list-of-known-smr-drives.141/

// SSD buying guide.
https://ssd.borecraft.com/SSD_Buying_Guide_List.pdf

// List of good NAS cases (WIP).
https://docs.google.com/document/d/13Pw67e5dP2rt0OzIrLXLoa3-mgYlf2IDtNhhASO2Y4Q/edit?usp=sharing\

// Low-power x86 systems.
https://docs.google.com/spreadsheets/d/1yl414kIy9MhaM0-VrpCqjcsnfofo95M1smRTuKN6e-E

// List of ARM-based SBCs.
https://docs.google.com/spreadsheets/d/1PGaVu0sPBEy5GgLM8N-CvHB2FESdlfBOdQKqLziJLhQ

// HP T620 plus. Decent cheap computer. You can use it as a VPN, pfSense firewall, and more.
https://www.servethehome.com/hp-t620-plus-thin-client-and-firewall-vpn-appliance/
https://youtu.be/pV1iPJ6vmhE  // Upgrade Ram.
https://youtu.be/cRSZ_pDO1SY // Buyers guide.
https://youtu.be/VCiIsDHIoU8 // Overview.

// Ondroid HC4 storage server. Cheap two drive system.
https://forum.odroid.com/viewtopic.php?f=29&t=40609

// Good place to fine old server hardware.
https://www.labgopher.com/  // Website which gathers ebay listings for server hardware.
https://www.bargainhardware.co.uk/  // Decent place for UK

// How to spot counterfeit LSI cards
https://forums.servethehome.com/index.php?threads/how-to-buy-non-counterfeit-lsi-9207-8i.26304/
https://www.youtube.com/watch?v=QQUWwhDU_KQ // Good video on how to spot counterfeits.

// Some videos on SAS controllers and cables which I found very helpful. If you are new to using SAS you should watch these.
https://youtu.be/hTbKzQZk21w  // Overview of typical SAS controllers
https://youtu.be/OW419HwU7sg  // Overview of SAS cables.
https://youtu.be/qccpopxc_Uo  // Using a SAS expander to connect 24 HDDs using only one controller.

// Lack rack: Meme-y but practical and cheap solution for rack-mount equipment. Be wary of putting too much weight on them though.
https://wiki.eth0.nl/index.php/LackRack

// Quiet SC846 with Noctua mod.
https://youtu.be/0UjyL6ZiMkI



----------Software-------------------------------------------------------------------------------------------------------------------

This section is for software that is useful for troubleshooting and other related things. For software for streaming etc check below for the Awesome Selfhosted link in the next section.


-----Windows-----
// Crystal Disk info: Graphical utility for checking SMART data on drives. Works on Externals and SSDs.
// Crystal Disk mark: Graphical disk benchmark software to test HDD/SSD performance.
https://crystalmark.info/en/software/crystaldiskinfo/

// Graphical utility for viewing disk usage statistics.
https://windirstat.net/

// YUMI: Graphical tool for creating a multiboot USB with multiple Bootable ISO files.
// Can be used to install a lot of commonly used Linux OS's, as well as live bootable troubleshooting tools and more.
// No official support For BSD or specialized OS's like proxmox/FreeNas/TrueNas
https://www.pendrivelinux.com/yumi-multiboot-usb-creator/

// Graphical bootable USB creator.
// Can do some BSD distros.
// Only does one bootable image per USB.
https://rufus.ie/

// Graphical duplicate file finder.
// Runs well in WINE
http://www.joerg-rosenthal.com/en/antitwin/

-----Linux-----
// Gparted: Graphical frontend of Parted. Use to crate, delete, alter partitions on drives.
https://gparted.org/

// Qdirstat: Graphical utility for viewing disk usage statistics.
https://github.com/shundhammer/qdirstat

// Badblocks: Linux CLI utility to check for bad sectors on HDDs
https://wiki.archlinux.org/index.php/badblocks

// Smartmontools: Linux CLI utilities package used to analyze and monitor SMART information on storage devices.
https://wiki.archlinux.org/index.php/S.M.A.R.T.

// Memtest86+: Memory testing tool available in most linux distros as well as the BIOS of motherboards.
// Also available as a bootable ISO that can be installed onto a USB drive.
https://www.memtest.org/


-----Multiplatform-----
// iperf: CLI Network performance tester.
https://iperf.fr/

// Wireshark: Graphical network protocol analyzer. Can be used for troubleshooting.
https://www.wireshark.org/

// PuTTy: Graphical Terminal emulator. also supports SCP, SSH.
https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

// Etcher: Graphical bootable USB creator. Simple and easy to use.
// Only does one bootable image per USB.
https://www.balena.io/etcher/

// Graphical multiboot USB creation tool.
// supports 560+ ISOs, including Proxmox, vmware ESXi, Freenas and more.
https://www.ventoy.net/en/index.html

// FOSS CLI Partition recovery tool
// Finds lost or deleted partitions from a wide variety of filesystem types
// Also available as a live bootable ISO.
https://www.cgsecurity.org/wiki/TestDisk

// CLI data recovery tool
// Recovers deleted files regardless of filesystem type
// GUI frontend Qphotorec also available
https://www.cgsecurity.org/wiki/PhotoRec

// Graphical duplicate file finder
https://dupeguru.voltaicideas.net/


----------/hsg/ + /dhg/ Links -------------------------------------------------------------------------------------------------------

// Misc links.
https://haydenjames.io/home-lab-beginners-guide-hardware/
https://wiki.debian.org/FreedomBox/Features

// STH forums. Good general resource for server questions.
https://forums.servethehome.com/index.php

// List of software for self hosting and administration.
https://github.com/awesome-selfhosted/awesome-selfhosted
https://github.com/n1trux/awesome-sysadmin

// Script to secure Debian and Debian based Linux installs.
https://github.com/x08d/lockdown.sh

// Server tips.
https://pastebin.com/SXuHp12J

// Can use as a reference, I wouldn't post there though.
https://old.reddit.com/r/datahoarder
https://www.reddit.com/r/homelab/wiki/index

// Setting up pfSense.
https://gist.github.com/deergod1/818ec78ab70947a2f89df2bb5bb28896

// Setting up a simple Samba share in ubuntu.
https://ubuntu.com/tutorials/install-and-configure-samba#1-overview

// Raspberry Pi KVM for managing servers remotely.
https://github.com/pikvm/pikvm

----------Linux/UNIX links-----------------------------------------------------------------------------------------------------------

// Linux Wikis. Most stuff on here applies to all distros.
https://wiki.archlinux.org
https://wiki.gentoo.org

// /g/ wiki. Good general resource.
https://wiki.installgentoo.com/index.php/Category:GNU/Linux

// Learn Command line. Will always be useful even if you mostly use a GUI.
http://mywiki.wooledge.org/BashGuide
http://wiki.bash-hackers.org/
http://www.grymoire.com/Unix
http://overthewire.org/wargames/bandit



----------Containers-----------------------------------------------------------------------------------------------------------------

// Pre-configured container images.
https://docs.linuxserver.io/


-----LXC-----
// Standard Linux containers. Available on most distros.
// You will likely be using these if you are running a Proxmox server.
// Since Linux containers are essentially just separate instances of Linux, you can't run Windows programs in them without using WINE
// LXD is a newer, more user friendly version of LXC. Has better management options for containers.
https://linuxcontainers.org/
https://wiki.debian.org/LXC
https://www.linuxjournal.com/content/everything-you-need-know-about-linux-containers-part-i-linux-control-groups-and-process
https://www.linuxjournal.com/content/everything-you-need-know-about-linux-containers-part-ii-working-linux-containers-lxc


-----Docker-----
// Instead of running as though it was an entire OS like LXC, Docker only virtualizes a single application.
// Can run on Windows as well as Linux. You will still need WINE to run Windows apps on Linux.
// Freemium software. Base software is free for individuals (you).
https://www.docker.com/
https://docs.docker.com/
https://forums.docker.com/
https://docs.docker.com/storage/bind-mounts/

-----Podman-----
// An alternative to Docker.
// Those using Docker can easily switch without issues.
// Unlike Docker, it does not use a single large server daemon.
// Uses "pods" which can contain more than one container.
https://podman.io/
http://docs.podman.io/en/latest/


-----Jails-----
// Jails are BSD's version of containers.
// Since TrueNas Core/FreeNas is freeBSD based you will be using these instead of LXC/LXD.
https://www.truenas.com/docs/hub/tasks/advanced/jails/
https://www.ixsystems.com/documentation/freenas/11.3-U5/jails.html#additional-storage
https://www.freebsd.org/doc/handbook/jails-build.html

----------Operating systems----------------------------------------------------------------------------------------------------------

-----Linux-----
// It's just Linux. Nothing too fancy. Just use whatever distro you like.
// Debian based distros like Ubuntu are easy to work with and have plenty of documentation.
// CentOS is a free alternative to RedHat and is also fairly popular.
// Use Mdadm + LVM, ZFS on Linux (ZoL), or Snapraid + mergerFS for Raid.


-----TrueNAS Core-----
// The successor and replacement for FreeNas. Still free to use.
https://www.truenas.com/download-truenas-core/
https://www.truenas.com/docs/hub/


-----FreeNAS----- OFFICIALLY DEPRECATED USE TRUENAS CORE.
// A FreeBSD based OS that utilizes ZFS. Has many available plugins for things like PLEX, bitorrent, and more.
// Keep in mind it will install to the ENTIRE DRIVE and you won't be able to use the install drive for anything else.
// Server grade hardware is recommended.
https://www.ixsystems.com/community/
https://www.freenas.org/hardware-requirements/
https://www.ixsystems.com/documentation/freenas/11.3-U5/freenas.html


-----Open Media Vault-----
// Good for storing infrequently changed files like media files.
// Website doesn't have much to show for. Check the forums for guides.
// Supports Snapraid as a plugin.
// Can be used with mergerFS to pool drives together.
https://www.openmediavault.org/
https://forum.openmediavault.org/
https://omv-extras.org/ // needed for mergerFS
https://forum.openmediavault.org/index.php?thread/23005-installation-and-setup-videos-beginning-intermediate-and-advanced/


-----Xpenology-----
// A free version of the OS Synology uses for their NAS products.
// Obviously you won't get any support from Synology if you use this.
// Supports differently sized physical disks and adding hard drives to expand as needed.
https://xpenology.org/
https://xpenology.com/forum/

-----unRAID-----
// Not free. You need to fork over some money to buy it.
// Comes with it's own RAID solution that technically isn't real RAID because all parity is stored on one or two disks.
// Supports differently sized physical disks and adding hard drives to expand as needed.
https://unraid.net/
https://wiki.unraid.net/Official_Documentation


-----Proxmox-----
// A Linux based Virtualization Environment that has built in ZFS support, Linux container support, and more.
// Your CPU must support Virtualization in order to run Proxmox. (VT-x on Intel, AMD-V for AMD)
// ECC Ram is recommended as per usual with ZFS.
https://www.proxmox.com/en/
https://pve.proxmox.com/wiki/Main_Page
https://pve.proxmox.com/wiki/ZFS_on_Linux



----------File Systems & RAID--------------------------------------------------------------------------------------------------------

-----LVM-----
// Required learning for management of drives on Linux.
// Can be used with Mdadm to pool multiple arrays into one logical volume.
// A bit more confusing than partitioning drives normally but is very flexible.
https://wiki.archlinux.org/index.php/LVM
https://wiki.gentoo.org/wiki/LVM#Thin_provisioning
https://wiki.archlinux.org/index.php/LVM_on_software_RAID


-----MDADM-----
// A tool for creating and managing Linux software RAID arrays.
// Can be used with LVM to pool multiple arrays into one logical volume.
// You can also create file systems directly on the raid arrays, and then use mergerFS to pool the file systems together.
// *Technically* supports disks of different sizes, but it requires multiple partitions on the drives and is not recommended.
// No built in checksums, Can use dm-integrity to detect errors, but has no way of dealing with them.
https://www.digitalocean.com/community/tutorial_series/how-to-configure-raid-arrays-on-ubuntu-16-04
https://raid.wiki.kernel.org/index.php/RAID_setup



-----ZFS-----
// A long standing, reliable file system and software raid solution that works on BSD and Linux.
// Supports up to 3 disk redundancy (RaidZ3) and has checksums and self healing to prevent data corruption.
// Has some limitations, one major one being expansion is cumbersome.
// If you plan on using RaidZ make sure you know what your array is going to be beforehand, you won't be able to add to it later.
// Server hardware and at least 8gb ECC memory recommended. Using old hardware or consumer hardware is not recommended.
// See the ZFS section of the FAQs pastebin for a more in-depth look at the limitations of ZFS.
https://openzfs.github.io/openzfs-docs/index.html


-----XFS-----
// Another reliable file system. Unlike ZFS it doesn't have built in software raid features.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-xfs


-----ext4-----
// Default file system for most Linux distros. Does everything a good file system should do and more.
https://ext4.wiki.kernel.org/index.php/Ext4_Howto


-----Btrfs-----
// It's "B-Tree", not "Butter"
// Has many of the same features as ZFS, including checksums and self healing.
// STILL UNDER DEVELOPMENT USE AT OWN RISK. Potential for data loss.
// Raid 1 features are stable on the most recent Linux kernels.
https://btrfs.wiki.kernel.org/index.php/Main_Page
https://wiki.archlinux.org/index.php/btrfs


-----mergerFS-----
// A Union file system that pools multiple file systems together under one mount point, allowing them to appear as one.
// Has some advantages over LVM, you can use multiple disks with data already on them instead of having to create LVM volumes/groups.
// If a disk fails, since the data is not striped across multiple disks (like with LVM), data loss can be less drastic.
// Works with multiple different file systems at the same time, including Windows's NTFS.
// Use with Snapraid or mdadm for disk redundancy.
// Available as a plugin for OMV.
https://www.teknophiles.com/2018/02/19/disk-pooling-in-linux-with-mergerfs/
https://github.com/trapexit/mergerfs


-----Snapraid-----
// Has an impressive list of features including up to 6 disk redundancy and the ability to add hard drives to expand as needed.
// Supports differently sized disks, allowing for more flexibility with expansion (your data disks must be equal to or smaller than your parity disks! If you try to add a new data disk that is larger than your parity drives you will run into issues).
// Technically not "real" raid and has some limitations. Read the manual.
// Available as a plugin for OMV.
// Can be used with mergerFS to pool drives together while retaining a level of redundancy.
https://www.snapraid.it/
https://sourceforge.net/p/snapraid/discussion/1677233/
https://www.snapraid.it/manual
https://www.snapraid.it/faq
https://selfhostedhome.com/combining-different-sized-drives-with-mergerfs-and-snapraid/