Kobold Notebook

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_Khe2ahSRqw"
      },
      "source": [
        "This notebook is loosely based on the original 4bit Kobold notebook by Peepy, OshieteKudasai, and Otis and subsequently modified by Anon. This should work with both Colab and Runpod.\n",
        "\n",
        "First, we'll install Kobold (may take a few minutes, particularly the last\n",
        "command)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oSoUZXZ4QO2H"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Package / Python updates.\n",
        "\n",
        "!apt-get -y update\n",
        "!apt-get -y install git-lfs\n",
        "!python -m pip install --upgrade pip\n",
        "\n",
        "!git-lfs install\n",
        "!git clone https://github.com/0cc4m/KoboldAI -b latestgptq --recurse-submodules\n",
        "\n",
        "# Install KoboldAI.\n",
        "\n",
        "!cd KoboldAI && ./install_requirements.sh cuda > /dev/null"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ey9AO8UrTb9S"
      },
      "source": [
        "Next, provide the URL for a 4bit model from Huggingface, along with the group size.  Your URL should look something like https://huggingface.co/groupname/modelname/ . The linked repo should contain exactly one safetensors or pt, otherwise weird shit will happen.\n",
        "\n",
        "If \"version\" is specified, you can load a particular commit of the Huggingface repository."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eS_7mSYuTccr"
      },
      "outputs": [],
      "source": [
        "num_params = \"13b\" #@param [\"7b\", \"13b\", \"30b\", \"65b\"]\n",
        "\n",
        "group_size = \"128\" #@param [\"None\", \"32\", \"128\"]\n",
        "\n",
        "huggingface_url = '' #@param {type:\"string\"}\n",
        "\n",
        "version = '' #@param {type:\"string\"}\n",
        "\n",
        "group_and_model = huggingface_url[len('https://huggingface.co/'):].split('/')[:2]\n",
        "group = group_and_model[0]\n",
        "model = group_and_model[1]\n",
        "\n",
        "if not version:\n",
        "    !cd KoboldAI/models && git clone https://huggingface.co/{group}/{model}/\n",
        "else:\n",
        "    !cd KoboldAI/models && git clone https://huggingface.co/{group}/{model}/ && cd {model} && git checkout {version}\n",
        "\n",
        "if group_size == 'None':\n",
        "    !cd KoboldAI/models/{model} ; mv *.pt 4bit.pt ; mv *.safetensors 4bit.safetensors\n",
        "else:\n",
        "    !cd KoboldAI/models/{model} ; mv *.pt 4bit-{group_size}g.pt ; mv *.safetensors 4bit-{group_size}g.safetensors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JWFvZKfJa0c2"
      },
      "source": [
        "Next, perform a quick hotfix to fix bug with the Flask Session package (make sure to only ever run this cell once). Fixed now, no longer necessary."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ib-L2kajbkFp"
      },
      "outputs": [],
      "source": [
        "'''\n",
        "def insert_lines(fpath, positions, lines):\n",
        "    with open(fpath, 'r') as f:\n",
        "        contents = f.readlines()\n",
        "\n",
        "    for pos, line in zip(positions, lines):\n",
        "        contents.insert(pos, line)\n",
        "\n",
        "    with open(fpath, 'w') as f:\n",
        "        contents = ''.join(contents)\n",
        "        f.write(contents)\n",
        "\n",
        "positions = [328, 329]\n",
        "lines = ['        if not hasattr(app, \\'session_cookie_name\\'):\\n',\n",
        "         '            app.session_cookie_name = \\'session\\'\\n']\n",
        "\n",
        "if os.path.exists('KoboldAI/runtime/envs/koboldai/lib/python3.8/site-packages/flask_session/sessions.py'):\n",
        "    insert_lines('KoboldAI/runtime/envs/koboldai/lib/python3.8/site-packages/flask_session/sessions.py', positions, lines)\n",
        "'''"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6uipr3t5en52"
      },
      "source": [
        "Run Kobold (use UI2, then when you select the model it should give you the option to load it in 4bit)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UidO9J3xe9PM"
      },
      "outputs": [],
      "source": [
        "!cd KoboldAI && ./play.sh --remote --quiet"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}