SCAN documentation

# SCAN
atS Cache ANalyzer

## User Guide
SCAN's primary use is as a library for inspecting Apache Traffic Server<sup>TM</sup> caches. For that usage, see the [API Reference Guide](./docs/documentation.pdf). SCAN also provides a command-line utility (`scan`), which is described here.

### Installation

#### Prerequisites
SCAN is a pretty large python project, and I used the `typing` package to help keep track of what types of values are being passed around. If you have Python version 3.5 or greater, you already have this. SCAN also requires `setuptools` to find its scripts. If you have `pip3` (for any Python version > 3.4.0), you already have this. To install `typing` on Python versions < 3.5, simply run `sudo -H pip3 install typing`. If you don't have `pip3`, then to install either dependency on CentOS/Fedora/RHEL distros, do `sudo yum install -y python34-<dependency>`, on Ubuntu/Mint/Debian distros do `sudo apt-get install python3-<dependency>`, and on Arch/Gentoo(?) do `sudo pacman -S python3-<dependency>`. If you need the dependencies and you're on MacOS/BSD/Windows, then gods help you - I can't.

#### Installing 'scan'
For right now, if you're reading this, you're a Comcast employee or contractor. So that means to install, you'll want to download the source from [the Comcast Github](https://github.comcast.com/URL_REDACTED). Once you've done that, go to the downloaded folder and run

```bash
sudo -H pip3 install .
```

... or, if you don't have `pip3`:

```bash
sudo python3 setup.py install
```

Note that SCAN is only guaranteed to work for Python versions 3.4.1 and greater.
If you want to run the tests, make sure you're in the project's root directory (the downloaded folder), and run `tests/test.py`.

#### Building the Documentation
I highly recommend that anyone seriously considering building the docs themselves see a licensed physician. The LaTeX source files can be found under "docs", and technically to compile them it's only necessary to run them through an `xelatex` engine. However, you're almost certainly missing the document class file, as well any included images or api reference material. The api reference material is generated using a *highly experimental* documentation generator that can be found [here](https://github.com/Sensibility/dochelper) (which also has the class file). Note that the generator itself is - ironically - undocumented and the installer doesn't work, even if you could guess how to use it. Building the images requires that you have [inkscape](https://inkscape.org) and [imagemagick](https://www.imagemagick.org) installed, and they have to be in your `$PATH` - which they won't be if you're using MacOS/Windows, even if they are installed. Plus you need GNU `make`, which I know for a fact is impossible on Windows (though `msys` and Cygwin should work, and Windows now comes with a Linux subsystem that should do the trick).

### Usage
The basic usage of `scan` is pretty simple at the moment; to start the utility simply run:

```bash
scan [ -f --fips ]
```

where the options have the following meanings:

* `-f` or `--fips`
	You **must** use this option if the ATS running on your system was compiled with `ENABLE_FIPS` enabled. If you don't, everything will be messed up. Actually, some things will still be messed up even if you do.


Once the utility is started, you'll be faced with a pretty basic prompt. At first, your only option will be `[1] Read Storage Config`. After you select this option, you'll be prompted to enter the location of your ATS configuration files. SCAN will expect all of them to be in the same directory, and will guess that they are in `/opt/trafficserver/etc/trafficserver/` by default. **Note that the use of FIPS at compilation time cannot be determined from the config files, and MUST be given on the command line.** Once the configuration has been read, all menu options will be unlocked. They are as follows:

#### `[1] Show Cache Setup`
This option will print out the spans and volumes declared in the configuration. Output will look like:

```
Cache files:
/path/to/a/span	Span of <n> stripes	XXX.XB

Volumes:
#1	<type>	XXX.XB

```
where `<n>` is the number of stripes in the span on that line, and XXX.XB is the size of a span/volume (but it will be displayed in human-readable approximations in units of B, kB, MB, or GB as appropriate). Volumes defined as a percent of total storage will have their size calculated at runtime, and displayed in absolute terms. `<type>` will be the type of volume declared. In nearly all cases, this will be `http`, but certain plugins could define other volume types. Finally, it should be noted that while this example shows one volume on one span, this menu option will display _all_ volumes and _all_ spans, in no particular order and with no distinction between cache spans on files, block devices, or ram devices.

#### `[2] List Settings`
This option will list the settings declared in `records.config`, in proper ATS syntax. An example:
```
proxy.config.log.collation_host STRING NULL
proxy.config.ssl.compression INT 1
```
Only one or two of these settings actually has any impact on the function of `scan`, but all values are read in to facilitate future extension.

#### `[3] Search for Setting`
This option will bring up a prompt to type a search string for a specific setting from `records.config`. Python-syntax regex is supported and enabled by default (meaning searching for 'proxy.config' will match 'proxyZconfig' as well as the exact string typed).

#### `[4] List Stripes in a Span`
This option will prompt you to enter a span (which is the **full** path to the span file) and then list all stripes within it. The output is in the format:
```
XXX.XB stripe, created Www Mmm D hh:mm:ss (version XX.X)
```

where XXX.XB is the size of a stripe (but it will be displayed in human-readable approximations in units of B, kB, MB, or GB as appropriate), `Www Mmm D hh:mm:ss` is the date of the stripe's creation (in the system's `ctime(3)` format) and XX.X is the decimal-separated major and minor version numbers of the cache system that created it. Note that this version is **not** the same as the version of ATS using the cache. Also note that as of this time **only version 24.0+ is supported by** `scan`, and using lower versions with `scan` **will cause to crash and/or give incorrect output**.

#### `[5] View URLs of objects in a Span`
When selected, this option will first prompt you for a span. It will then search all of the stripes on that span for stored objects, and catalog their URLs, printing them to the screen as they are found. Each URL is printed in the format:
```
protocol://[[user]:password@]host/path/to/content	 - XXX.XB - 	x<Y>
```
where `protocol` is the protocol used to retrieve the content (nearly always `http` or `https`), `[[user]:password@]` is the username (if used, usually not) 'colon' password (if used, usually not) used to access the content 'at' the `host` - which is the fully-qualified domain name of the content host, and `path/to/content` is the location on that host of the content stored in the cache. A typical example of a path is `images/test/testquest.png`. XXX.XB is the size of this content (but it will be displayed in human-readable approximations in units of B, kB, MB, or GB as appropriate). Finally, `<Y>` will be the number of times this same URL is stored in the cache (typically in 'alternate' forms). For example, if a given item is stored only once in the cache span, its line will end in `x1`, and if it is encountered 42 times, then it will end in `x42`. Note that the size of a given object is reported as the size of _one_ instance of this item, regardless of the number actually stored.

**Warning:** When tested on a span of a single, roughly 830GB stripe, this operation took between 39 and 44 seconds to complete. Be aware that the time this takes is directly proportional to the size of the spans, and the number of spans that it is searching. However, results are cached so that subsequent searches (or uses of menu option 6) on the same span should be significantly quicker. To help recognize that the program has not frozen, findings are printed to the screen as they are found, and the main menu will display upon completion.

#### `[6] View Usage of a Span broken down by host`
This option will first prompt for a span, then it will list the hosts that have content stored in that span, as well as the total storage size used, the storage size as a percent of the total available storage, and the storage size as a percent of the storage currently in use. The output format for each host is as follows:
```
<host>	 - XXX.XB - 	YY.YY% of available space - 	ZZ.ZZ% of used space
```
where `<host>` is the fully-qualified domain name of the host, XXX.XB is the total size of that host's content on disk (but it will be displayed in human-readable approximations in units of B, kB, MB, or GB as appropriate), YY.YY is the percent of available space taken up by this host's content, and ZZ.ZZ is the percent of space currently being used to store objects that is taken up by this host's content.

**Warning:** When tested on a span of a single, roughly 830GB stripe, this operation took between 39 and 44 seconds to complete. Be aware that the time this takes is directly proportional to the size of the spans, and the number of spans that it is searching. However, results are cached so that subsequent searches (or uses of menu option 5) on the same span should be significantly quicker. To help recognize that the program has not frozen, findings are printed to the screen as they are found, and the main menu will display upon completion.

#### `[7] Dump cache usage stats to file (Tabular YAML format)}}`
This option will ask you to first name a file for output (relative or absolute paths - doesn't matter which), then it will dump the output of a call to the 'View URLs of objects in a Span' for **ALL** spans in the cache system to the named file in Tabular YAML (TYAML) format (which is just YAML but indented with tabs instead of spaces and accepts `None` as a null value.)