{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo for cohort's mini-gsutil library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import cohorts\n",
    "from cohorts.io import gcloud_storage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a barebones files to be used for testing the new functionality. We will populate it manually for now and then will try to fetch it, modify it, and put it back onto the GS:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file:///tmp/gsio.txt [Content-Type=text/plain]...\n",
      "/ [1 files][   13.0 B/   13.0 B]                                                \n",
      "Operation completed over 1 objects/13.0 B.                                       \n",
      "Cohorts test\n"
     ]
    }
   ],
   "source": [
    "# file URI on the GS\n",
    "gsuri = \"gs://arman-hammerlab/test/gsio.txt\"\n",
    "org_tmp = \"/tmp/gsio.txt\"\n",
    "\n",
    "!echo 'Cohorts test' > {org_tmp}\n",
    "!gsutil cp {org_tmp} {gsuri}\n",
    "!gsutil cat {gsuri}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see if we can pull it without breaking anything:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the class that will help us with those:\n",
    "gcio = gcloud_storage.GoogleStorageIO()  # Look ma, no passwords or anything!\n",
    "# and it handles the auth part very smoothly, which is great.\n",
    "\n",
    "gcio.download_to_path(gsuri=gsuri, localpath=\"{}.dl\".format(org_tmp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohorts test\r\n"
     ]
    }
   ],
   "source": [
    "!cat {org_tmp}.dl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Weee! So we can pull files to our own local, that is useful; but even more useful would be to open them as if they are local files, change the content, and seemlessly putting it back. We will do this with the Python's `ContextManager`-style of doing things:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohorts test\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# This is our context class that hands you a file handler\n",
    "# once you are in and takes of the rest of the legwork (download/upload)\n",
    "# for you:\n",
    "from cohorts.io.gcloud_storage import GoogleStorageFile\n",
    "\n",
    "# The following is similar to what we do with `with open(file, mode)`:\n",
    "with GoogleStorageFile(gcio=gcio, gsuri=gsuri, mode='r+') as fgcs:\n",
    "    print(fgcs.read()) # See the contents first\n",
    "    # And since the mode is `r+`, this addition should go to the end\n",
    "    # of the file (append mode):\n",
    "    fgcs.write(\"Additional text\")   \n",
    "# And we are out of the context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All right! No errors, but just the content we printed out is here; but now let's see things are OK on the GS side:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohorts test\r\n",
      "Additional text"
     ]
    }
   ],
   "source": [
    "!gsutil cat {gsuri}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we have a few other utils that might come handy, too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Blob: arman-hammerlab, CIBERSORT.R>\n",
      "<Blob: arman-hammerlab, gs://arman-hammerlab/test/gsio.txt>\n",
      "<Blob: arman-hammerlab, lowresp/SI9946.bam>\n",
      "<Blob: arman-hammerlab, lowresp/SI9946.bam.bai>\n",
      "<Blob: arman-hammerlab, rcc/reDEFAULT.bam>\n",
      "<Blob: arman-hammerlab, rcc/reDEFAULT.bam.bai>\n",
      "<Blob: arman-hammerlab, rcc/reL0-sorted.bam>\n",
      "<Blob: arman-hammerlab, rcc/reL0-sorted.bam.bai>\n",
      "<Blob: arman-hammerlab, rcc/reL3-sorted.bam>\n",
      "<Blob: arman-hammerlab, rcc/reL3-sorted.bam.bai>\n",
      "<Blob: arman-hammerlab, rcc/tumor.bam>\n",
      "<Blob: arman-hammerlab, rcc/tumor.bam.bai>\n",
      "<Blob: arman-hammerlab, test-htslib/sample-readwrite.bam>\n",
      "<Blob: arman-hammerlab, test-htslib/sample.bam>\n",
      "<Blob: arman-hammerlab, test/C509.TCGA-78-7536-10A-01D-2063-08.1_gdc_realn.bam>\n",
      "<Blob: arman-hammerlab, test/a9fa477c-8299-41e4-aa5d-e0ae6801a201_gdc_realn_rehead.bam>\n",
      "<Blob: arman-hammerlab, test/b37decoy.fasta>\n",
      "<Blob: arman-hammerlab, test/bla>\n",
      "<Blob: arman-hammerlab, test/cnv>\n",
      "<Blob: arman-hammerlab, test/d295cec33ca4e862fc06cb13aa9afa3d26ed6aa7cfc46893691aafbc404b250en_PGDX2811T_Excleaned-b2fq-PE_R1.fastq>\n",
      "<Blob: arman-hammerlab, test/gsio.txt>\n",
      "<Blob: arman-hammerlab, test/gsio.txt.upl>\n",
      "<Blob: arman-hammerlab, test/tumor>\n"
     ]
    }
   ],
   "source": [
    "# Listing files, maybe?\n",
    "for afile in gcio.list_files(\"gs://arman-hammerlab/test\"):\n",
    "    print(afile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have another one: not as powerful as the gsutil is, but still a small/frequent task that keep doing over and over again: upload file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cohorts test\r\n",
      "Yet another addition\r\n"
     ]
    }
   ],
   "source": [
    "!echo \"Yet another addition\" >> {org_tmp}\n",
    "gcio.upload_file(localpath=org_tmp, gsuri=\"{}.upl\".format(gsuri))\n",
    "!gsutil cat {gsuri}.upl"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}