brute - ibb

/*

# brute

BRUTE stands for: Bibliotik Retail Uploader Torrent Epubs.
It does not mean anything, but BRUTEs are not known to be particularly
articulate.

In short, BRUTE allows uploading a retail epub to Bib from the comfort of the
command line.

It's not the prettiest thing, being hacked together and mashed up in a single
file for ease of sharing, but BRUTE will try its best to help make quality
uploads.
This means it will automate the boring stuff, and let the user focus on making
sure the torrent metadata is accurate and complete.

BRUTE retrieves metadata from the epub and Goodreads automatically, and forces
the user to merge and edit the results as needed.
Everything is saved to a yaml file, for (optional) manual editing before the
actual upload.

## Building

You need a working Go installation: see https://golang.org/doc/install.
Put this file in a 'brute' directory in your $GO_PATH, cd to it and just run:

    $ go get .
    $ go install ...brute

or if you don't want to install it:

    $ go get .
    $ go run brute.go

To update its dependancies, rebuild from time to time:

    $ go get -u .
    $ go install ...brute

BRUTE was tested on Linux, but it should probably work wherever you can install
Go.

## Requirements

- a WhatIMG account
- a GoodReads account for which you've requested an API Key: see
  https://www.goodreads.com/api/keys

## Usage

BRUTE only requires one existing epub file as argument.

    $ brute /path/to/book.epub

It will:

 - read epub metadata
 - retrieve information from Goodreads if ISBN is found
 - allow (alright, force) the user to edit the merged information to prepare
   the bibliotik upload form
 - save that information to /path/to/book.yaml for manual editing if deemed
   necessary
 - upload the cover to whatimg and retrieve its url

At this point BRUTE will ask if you want to upload directly or not.
If you say no, you will be able to manually edit the yaml file as you wish.
Run BRUTE again with the same argument, and it will load the yaml file with your
changes, and proceed to:

 - generate the .torrent from the epub with the user's passkey
 - upload the torrent with the cover and information collected in previous steps
   or loaded from a previously saved yaml file
 - copy the .torrent and .epub where you want to begin seeding
 - archive everything for later reference: epub, cover, torrent, and yaml in a
   tarball.
 - (optionally) remove the original files

Conventions:
- the script assumes /path/to/book.jpg exists and is the cover for this epub.
- it will create /path/to/book.torrent and /path/to/book.yaml

## Configuration

BRUTE needs a configuration file ("config.yaml") with the following information:

    whatimg_login: xx
    whatimg_password: xx
    bibliotik_login: xx
    bibliotik_password: xx
    bibliotik_passkey: https://bibliotik.me/announce.php?passkey=xx
    torrent_watch_directory: /home/user/torrents
    torrent_seed_directory: /home/user/downloads
    archive_directory: /home/user/uploads
    goodreads_api_key: xx
    tag_aliases:
        science-fiction:
        - sf
        - sci-fi

tag_aliases allows automatic removal of duplicate tags (from duplicate shelves
on Goodreads).
This configuration file must be in the working directory.
Lots of information in a clear text file, I know.
Keep it close to your heart.

## Things BRUTE isn't good at

BRUTE is not good at:
- retrieving information when it cannot find an ISBN number or if Goodreads
does not know about an ISBN
- retrieving editors, contributors, translators, languages (defaults to english)
- it will try to extract tags from Goodreads shelves, and clean out the most
blatantly awful ones. ALWAYS EDIT TAGS MANUALLY.

Fortunately, you can overcome these shortcomings by manually editing
/path/to/book.yaml!
No excuses!

## Things BRUTE is useless at

BRUTE can't help you if you're dealing with anything other than epubs.

You could use it to upload non-retail epubs, but then you MUST manually set the
RetailField to "0" in the yaml file.

## Changelog

v1.0.4: Updated to work with latest version of third party packages.
v1.0.3: Updated to work with latest version of third party packages.
v1.0.2: In description, title is in italics and author name in bold.
v1.0.1: BRUTE should now correctly retrieve edition year instead of original
publication year from GR.
v1.0: original release.

*/

package main

import (
    "bytes"
    "errors"
    "fmt"
    "io"
    "io/ioutil"
    "log"
    "mime/multipart"
    "net/http"
    "net/http/cookiejar"
    "net/url"
    "os"
    "path/filepath"
    "regexp"
    "strings"
    "time"

    "github.com/anacrolix/torrent/metainfo"
    b "github.com/barsanuphe/endive/book"
    e "github.com/barsanuphe/endive/endive"
    u "github.com/barsanuphe/endive/ui"
    "github.com/jhoonb/archivex"
    "github.com/skratchdot/open-golang/open"
    "github.com/spf13/viper"
    "golang.org/x/net/publicsuffix"
    "gopkg.in/yaml.v2"
)

func main() {
    var ui e.UserInterface
    ui = u.UI{}

    // 1. load configuration
    cfg := Config{}
    if err := cfg.load("config.yaml"); err != nil {
        fmt.Println("Error loading config.")
        return
    }
    if err := cfg.check(); err != nil {
        fmt.Println("Error checking config: " + err.Error())
        return
    }
    // 2. check argument
    if len(os.Args) != 2 {
        fmt.Println("Please provide an epub filename.")
        return
    }
    c := UploadCandidate{
        epubFile:    os.Args[1],
        torrentFile: strings.TrimSuffix(os.Args[1], ".epub") + ".torrent",
        imageFile:   strings.TrimSuffix(os.Args[1], ".epub") + ".jpg",
        infoFile:    strings.TrimSuffix(os.Args[1], ".epub") + ".yaml",
    }
    if err := c.check(); err != nil {
        fmt.Println(err.Error())
        return
    }
    // 3. get info from file + GR
    if err := c.getInfo(cfg, ui); err != nil {
        fmt.Println(err.Error())
        return
    }
    // 4. upload cover if necessary
    if c.info.ImageField == "" || ui.YesOrNo("Image URL found, upload again") {
        if err := c.uploadImage(cfg); err != nil {
            fmt.Println(err.Error())
            return
        }
    }
    // 5. save info
    err := c.saveInfo()
    if err != nil {
        fmt.Println(err.Error())
    }
    ui.Title("You can now either manually edit " + filepath.Base(c.infoFile) + " and run this script again, or directly upload to bibliotik if you are satisfied the quality of the metadata is excellent.")
    if ui.YesOrNo("Upload to bibliotik") {
        // 6. generate torrent
        if err := c.generateTorrent(cfg); err != nil {
            fmt.Println(err.Error())
            return
        }
        // 7. upload to bibliotik
        if err := c.uploadTorrent(cfg); err != nil {
            fmt.Println(err.Error())
            return
        }
        // 8. copy torrent to watch dir && epub to incoming
        if err := c.seed(cfg); err != nil {
            fmt.Println(err.Error())
            return
        }
        // 9. backup all relevant files to zip with date
        if err := c.archive(cfg); err != nil {
            fmt.Println(err.Error())
            return
        }
        // 10. clean up
        if ui.YesOrNo("Remove original files") {
            if err := c.cleanUp(); err != nil {
                fmt.Println(err.Error())
                return
            }
        }
    } else if ui.YesOrNo("Launch this book's Goodreads page in your browser for reference?") {
        err = open.Start("https://www.goodreads.com/search?q=" + c.info.IsbnField)
        if err != nil {
            fmt.Println("Error: could not open goodreads page!")
        }
    }
}

//------------------------------------------------------------------------------

type Config struct {
    bibUser         string
    bibPassword     string
    bibPasskey      string
    whatimgUser     string
    whatimgPassword string
    torrentWatchDir string
    torrentSeedDir  string
    archiveDir      string
    grApiKey        string
    TagAliases      map[string][]string
}

func (c *Config) load(path string) (err error) {
    conf := viper.New()
    conf.SetConfigType("yaml")
    conf.SetConfigFile(path)

    err = conf.ReadInConfig()
    if err != nil {
        return
    }
    c.bibUser = conf.GetString("bibliotik_login")
    c.bibPassword = conf.GetString("bibliotik_password")
    c.bibPasskey = conf.GetString("bibliotik_passkey")
    c.whatimgUser = conf.GetString("whatimg_login")
    c.whatimgPassword = conf.GetString("whatimg_password")
    c.torrentSeedDir = conf.GetString("torrent_seed_directory")
    c.torrentWatchDir = conf.GetString("torrent_watch_directory")
    c.archiveDir = conf.GetString("archive_directory")
    c.grApiKey = conf.GetString("goodreads_api_key")
    c.TagAliases = conf.GetStringMapStringSlice("tag_aliases")
    return
}

func (c *Config) check() (err error) {
    if !e.DirectoryExists(c.torrentWatchDir) {
        return errors.New("Torrent Watch directory does not exist")
    }
    if !e.DirectoryExists(c.torrentSeedDir) {
        return errors.New("Torrent download directory does not exist")
    }
    if !e.DirectoryExists(c.archiveDir) {
        return errors.New("Archive directory does not exist")
    }
    return
}

//------------------------------------------------------------------------------

// newTrue allows setting the value of a *bool in a struct.
// it is arguably awful.
func newTrue() *bool {
    b := true
    return &b
}

func checkErrors(errs ...error) error {
    for _, err := range errs {
        if err != nil {
            return err
        }
    }
    return nil
}

func login(siteUrl, username, password string) (hc *http.Client, returnData string, err error) {
    form := url.Values{}
    form.Add("username", username)
    form.Add("password", password)
    req, err := http.NewRequest("POST", siteUrl, strings.NewReader(form.Encode()))
    if err != nil {
        fmt.Println(err.Error())
    }
    req.Header.Add("Content-Type", "application/x-www-form-urlencoded")

    options := cookiejar.Options{
        PublicSuffixList: publicsuffix.List,
    }
    jar, err := cookiejar.New(&options)
    if err != nil {
        log.Fatal(err)
    }
    hc = &http.Client{Jar: jar}

    resp, err := hc.Do(req)
    if err != nil {
        fmt.Println(err.Error())
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        err = errors.New("Returned status: " + resp.Status)
    }

    data, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        log.Fatal(err)
    }
    returnData = string(data)
    return
}

func uploadImage(client *http.Client, uploadUrl, image string) (responseData string, err error) {
    // preparing a form
    b := new(bytes.Buffer)
    w := multipart.NewWriter(b)
    // adding image to form
    f, err := os.Open(image)
    if err != nil {
        return "", err
    }
    defer f.Close()
    fw, err := w.CreateFormFile("userfile[]", filepath.Base(image))
    if err != nil {
        return "", err
    }
    if _, err = io.Copy(fw, f); err != nil {
        return
    }
    if err = w.WriteField("upload_to", "0"); err != nil {
        return
    }
    if err = w.WriteField("private_upload", "1"); err != nil {
        return
    }
    if err = w.WriteField("upload_type", "standard"); err != nil {
        return
    }
    w.Close()

    req, err := http.NewRequest("POST", uploadUrl, b)
    if err != nil {
        return "", err
    }
    req.Header.Set("Content-Type", w.FormDataContentType())

    resp, err := client.Do(req)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        err = errors.New("Returned status: " + resp.Status)
    }

    data, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return "", err
    }
    responseData = string(data)
    return
}

//------------------------------------------------------------------------------

// BUInfo tracks all of the upload form fields.
type BUInfo struct {
    TorrentFileField  string
    authkey           string
    upload            string // default: empty
    TitleField        string
    EditorsField      string
    ContributorsField string
    TranslatorsField  string
    PublishersField   string
    PagesField        string
    AuthorsField      string
    FormatField       string // EPUB == "15"
    IsbnField         string
    TagsField         string
    DescriptionField  string
    RetailField       string // retail == "1"
    NotifyField       string // default "1"
    LanguageField     string // english == "1"
    YearField         string
    ImageField        string
}

// ShowInfo returns a table with relevant information about a book.
func (i *BUInfo) ShowInfo() (desc string) {
    var rows [][]string
    rows = append(rows, []string{"TorrentFile", i.TorrentFileField})
    rows = append(rows, []string{"Title", i.TitleField})
    rows = append(rows, []string{"Authors", i.AuthorsField})
    rows = append(rows, []string{"Editors", i.EditorsField})
    rows = append(rows, []string{"Contributors", i.ContributorsField})
    rows = append(rows, []string{"Translators", i.TranslatorsField})
    rows = append(rows, []string{"Publishers", i.PublishersField})
    rows = append(rows, []string{"Isbn", i.IsbnField})
    rows = append(rows, []string{"Pages", i.PagesField})
    rows = append(rows, []string{"Year", i.YearField})
    rows = append(rows, []string{"Format", i.FormatField})
    rows = append(rows, []string{"Language", i.LanguageField})
    rows = append(rows, []string{"Retail", i.RetailField})
    rows = append(rows, []string{"Tags", i.TagsField})
    rows = append(rows, []string{"Image", i.ImageField})
    rows = append(rows, []string{"Description", i.DescriptionField})
    rows = append(rows, []string{"Notify", i.NotifyField})
    return e.TabulateRows(rows, "Info", "Book")
}

func (i *BUInfo) load(path string) (err error) {
    data, err := ioutil.ReadFile(path)
    if err != nil {
        return
    }
    return yaml.Unmarshal(data, i)
}

func (i *BUInfo) save(path string) (err error) {
    d, err := yaml.Marshal(i)
    if err != nil {
        return err
    }
    return ioutil.WriteFile(path, d, 0777)
}

func (i *BUInfo) fill(m b.Metadata, torrentPath string) {
    i.TorrentFileField = torrentPath
    i.authkey = ""
    i.upload = ""
    if m.HasAny() {
        seriesInfo := ""
        if m.Series.HasAny() {
            for i := range m.Series {
                seriesInfo += fmt.Sprintf(" (%s, Book %s)", m.Series[i].Name, m.Series[i].Position)
            }
        }
        i.TitleField = m.Title() + seriesInfo
        i.EditorsField = ""      // TODO
        i.ContributorsField = "" // TODO
        i.TranslatorsField = ""  // TODO
        i.PublishersField = m.Publisher
        i.PagesField = m.NumPages
        i.AuthorsField = strings.Join(m.Authors, ", ")
        i.FormatField = "15" // EPUB
        i.IsbnField = m.ISBN
        i.TagsField = m.Category + ", " + m.MainGenre + ", " + m.Tags.String()

        r := strings.NewReplacer(
            i.AuthorsField, "[b]"+i.AuthorsField+"[/b]",
            m.Title(), "[i]"+m.Title()+"[/i]",
        )
        i.DescriptionField = r.Replace(m.Description)
        i.RetailField = "1"
        i.NotifyField = "1"
        i.LanguageField = "1" // English
        i.YearField = m.EditionYear
    }
}

func (i *BUInfo) generateUploadRequest(uploadURL string) (req *http.Request, err error) {
    // setting up the form
    b := new(bytes.Buffer)
    w := multipart.NewWriter(b)
    // adding the torrent file
    f, err := os.Open(i.TorrentFileField)
    if err != nil {
        return nil, err
    }
    defer f.Close()
    fw, err := w.CreateFormFile("TorrentFileField", filepath.Base(i.TorrentFileField))
    if err != nil {
        return nil, err
    }
    if _, err = io.Copy(fw, f); err != nil {
        return
    }
    errors := []error{}
    errors = append(errors, w.WriteField("authkey", i.authkey))
    errors = append(errors, w.WriteField("upload", i.upload))
    errors = append(errors, w.WriteField("TitleField", i.TitleField))
    errors = append(errors, w.WriteField("EditorsField", i.EditorsField))
    errors = append(errors, w.WriteField("ContributorsField", i.ContributorsField))
    errors = append(errors, w.WriteField("TranslatorsField", i.TranslatorsField))
    errors = append(errors, w.WriteField("PublishersField", i.PublishersField))
    errors = append(errors, w.WriteField("PagesField", i.PagesField))
    errors = append(errors, w.WriteField("AuthorsField", i.AuthorsField))
    errors = append(errors, w.WriteField("FormatField", i.FormatField))
    errors = append(errors, w.WriteField("IsbnField", i.IsbnField))
    errors = append(errors, w.WriteField("TagsField", i.TagsField))
    errors = append(errors, w.WriteField("DescriptionField", i.DescriptionField))
    errors = append(errors, w.WriteField("RetailField", i.RetailField))
    errors = append(errors, w.WriteField("NotifyField", i.NotifyField))
    errors = append(errors, w.WriteField("LanguageField", i.LanguageField))
    errors = append(errors, w.WriteField("YearField", i.YearField))
    errors = append(errors, w.WriteField("ImageField", i.ImageField))
    if err := checkErrors(errors...); err != nil {
        return nil, err
    }
    w.Close()

    req, err = http.NewRequest("POST", uploadURL, b)
    if err != nil {
        return nil, err
    }
    req.Header.Set("Content-Type", w.FormDataContentType())
    return
}

//------------------------------------------------------------------------------

type UploadCandidate struct {
    epubFile    string
    torrentFile string
    imageFile   string
    infoFile    string
    info        BUInfo
}

func (t *UploadCandidate) loadInfo() (err error) {
    return t.info.load(t.infoFile)
}

func (t *UploadCandidate) saveInfo() (err error) {
    fmt.Println("Saving all gathered information to " + t.infoFile)
    return t.info.save(t.infoFile)
}

func (t *UploadCandidate) check() (err error) {
    //  assert epub exists
    if _, err := e.FileExists(t.epubFile); err != nil {
        return errors.New("Epub does not exist!")
    }
    // assert it's an epub
    if !strings.HasSuffix(strings.ToLower(t.epubFile), ".epub") {
        return errors.New(t.epubFile + " is not an epub")
    }
    // assert jpg exists
    if _, err := e.FileExists(t.imageFile); err != nil {
        return errors.New("Cover " + t.imageFile + " does not exist!")
    }
    //  assert t does not exist yet
    if _, err := e.FileExists(t.torrentFile); err == nil {
        return errors.New("Torrent " + t.torrentFile + " should not exist!")
    }
    return
}

func (t *UploadCandidate) getInfo(cfg Config, ui e.UserInterface) (err error) {
    loadedFromFile := false
    if _, err := e.FileExists(t.infoFile); err == nil {
        if ui.YesOrNo("Info file already exists! Load") {
            if err := t.loadInfo(); err != nil {
                return err
            }
            fmt.Println(t.info.ShowInfo())
            if !ui.YesOrNo("Confirm") {
                return errors.New("Incorrect book information rejected.")
            } else {
                loadedFromFile = true
            }
        }
    }
    if !loadedFromFile {
        bk := b.NewBook(ui, 0, t.epubFile, e.Config{GoodReadsAPIKey: cfg.grApiKey, TagAliases: cfg.TagAliases}, true)
        // read epub metadata
        bk.Metadata, err = bk.RetailEpub.ReadMetadata()
        if err != nil {
            return err
        }
        // find GoodReads metadata and merge
        err = bk.SearchOnline()
        if err != nil {
            return err
        }
        t.info.fill(bk.Metadata, t.torrentFile)
        fmt.Println(t.info.ShowInfo())
        // review/edit field by field, confirm
        for !ui.YesOrNo("Confirm") {
            relevantFields := []string{"series", "title", "author", "year", "publisher", "description", "isbn", "category", "maingenre", "tags"}
            for _, f := range relevantFields {
                if err := bk.EditField(f); err != nil {
                    fmt.Println("Could not assign new value to field " + f + ", continuing.")
                }
            }
            t.info.fill(bk.Metadata, t.torrentFile)
            fmt.Println(t.info.ShowInfo())
        }
    }
    return
}

func (t *UploadCandidate) uploadImage(cfg Config) error {
    // login
    hc, data, err := login("https://whatimg.com/users.php?act=login-d", cfg.whatimgUser, cfg.whatimgPassword)
    if err != nil {
        return err
    }
    if !strings.Contains(data, "You have been successfully logged in.") {
        return errors.New("Failed to log in")
    }
    // upload image and get link
    fmt.Printf("Uploading " + filepath.Base(t.imageFile) + " to WhatIMG... ")
    data, err = uploadImage(hc, "https://whatimg.com/upload.php", t.imageFile)
    r := regexp.MustCompile(".*value=\"(https://whatimg.com/i/[[:alnum:]]+?.jpg)\".*")
    if r.MatchString(data) {
        t.info.ImageField = r.FindStringSubmatch(data)[1]
        fmt.Println("OK: " + t.info.ImageField)
    } else {
        fmt.Println("KO")
        err = errors.New("Could not find image URL.")
    }
    return nil
}

func (t *UploadCandidate) generateTorrent(cfg Config) error {
    fmt.Println("Building torrent file " + filepath.Base(t.torrentFile))
    mi := &metainfo.MetaInfo{Announce: cfg.bibPasskey}
    mi.Comment = "for bibliotik"
    mi.CreatedBy = "BRUTE"
    mi.CreationDate = time.Now().Unix()
    mi.Info.PieceLength = 256 * 1024
    mi.Info.BuildFromFilePath(t.epubFile)
    mi.Info.Private = newTrue()

    f, err := os.Create(t.torrentFile)
    if err != nil {
        return err
    }
    defer f.Close()
    return mi.Write(f)
}

func (t *UploadCandidate) uploadTorrent(cfg Config) (err error) {
    // login
    client, data, err := login("https://bibliotik.me", cfg.bibUser, cfg.bibPassword)
    if err != nil {
        fmt.Println(err.Error())
        return err
    }
    // getting authkey
    r := regexp.MustCompile(".*authkey=([[:alnum:]]{40}).*")
    if r.MatchString(data) {
        t.info.authkey = r.FindStringSubmatch(data)[1]
    } else {
        return errors.New("Could not log in.")
    }
    // prepare request
    req, err := t.info.generateUploadRequest("https://bibliotik.me/upload/ebooks")
    if err != nil {
        return err
    }
    // submit the request
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    // check what URL the response came from
    finalURL := resp.Request.URL.String()
    if finalURL == "https://bibliotik.me/upload/ebooks" {
        return errors.New("Upload probably failed, response was from upload form.")
    }
    return
}

func (t *UploadCandidate) seed(cfg Config) (err error) {
    fmt.Println("Copying files to begin seeding.")
    err = e.CopyFile(t.epubFile, filepath.Join(cfg.torrentSeedDir, filepath.Base(t.epubFile)))
    if err != nil {
        return
    }
    err = e.CopyFile(t.torrentFile, filepath.Join(cfg.torrentWatchDir, filepath.Base(t.torrentFile)))
    if err != nil {
        return
    }
    return
}

func (t *UploadCandidate) archive(cfg Config) (err error) {
    // generate archive name
    currentTime := time.Now().Local()
    currentDay := currentTime.Format("2006-01-02")
    archiveName := filepath.Join(cfg.archiveDir, fmt.Sprintf("%s - %s - %s (%s) %s.tar.gz", currentDay, t.info.IsbnField, t.info.AuthorsField, t.info.YearField, t.info.TitleField))
    if _, err := e.FileExists(archiveName); err == nil {
        return errors.New("Archive " + archiveName + " already exists")
    }
    fmt.Println("Backing up files to " + filepath.Base(archiveName))

    // generate archive
    tar := new(archivex.TarFile)
    if err := tar.Create(archiveName); err != nil {
        return errors.New("Error creating tar.")
    }
    if err := tar.AddFile(t.epubFile); err != nil {
        return errors.New("Error adding epub " + t.epubFile)
    }
    if err := tar.AddFile(t.imageFile); err != nil {
        return errors.New("Error adding image " + t.imageFile)
    }
    if err := tar.AddFile(t.torrentFile); err != nil {
        return errors.New("Error adding torrent " + t.torrentFile)
    }
    if err := tar.AddFile(t.infoFile); err != nil {
        return errors.New("Error adding yaml " + t.infoFile)
    }
    if err := tar.Close(); err != nil {
        return errors.New("Error writing tar.")
    }
    return
}

func (t *UploadCandidate) cleanUp() (err error) {
    if err := os.Remove(t.imageFile); err != nil {
        return errors.New("Could not remove " + t.imageFile)
    }
    if err := os.Remove(t.torrentFile); err != nil {
        return errors.New("Could not remove " + t.torrentFile)
    }
    if err := os.Remove(t.epubFile); err != nil {
        return errors.New("Could not remove " + t.epubFile)
    }
    if err := os.Remove(t.infoFile); err != nil {
        return errors.New("Could not remove " + t.infoFile)
    }
    return
}