{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Progress tracking\n",
    "\n",
    "AutoTransformers supports [ClearML](https://clear.ml/clearml-experiment/) to log and visualize a model's metrics during training. This example shows how you can track your model's training progress using the ClearML Web Interface.\n",
    "\n",
    "Before starting, create a ClearML account and set up the credentials on your machine. Refer to the [ClearML tutorial](https://clear.ml/docs/latest/docs/getting_started/ds/ds_first_steps) for further instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure ClearML is installed\n",
    "%pip install ClearML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from autotransformers import AutoTransformer, DatasetLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, we use the googleplay dataset to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The text snippets in this dataset are from \"googleplay\", a public dataset of app reviews on Google's Play Store.\n",
    "dataset = {\n",
    "    \"meta\": {\n",
    "        \"name\": \"example_singlelabel\",\n",
    "        \"version\": \"1.0.0\",\n",
    "        \"created_with\": \"wizard\"\n",
    "    },\n",
    "    \"config\": [\n",
    "        {\n",
    "            \"domain\": \"text\",\n",
    "            \"type\": \"IText\"\n",
    "        },\n",
    "        {\n",
    "            \"task_id\": \"task1\",\n",
    "            \"classes\": [\"positive\", \"neutral\", \"negative\"],\n",
    "            \"type\": \"TSingleClassification\"\n",
    "        }\n",
    "    ],\n",
    "    \"train\": [\n",
    "        [\n",
    "            {\"value\": \"None of the notifications work. other people in the forums teport similar problems bht no fix. the app is nice but it isnt nearly as functional without notifications\"},\n",
    "            {\"value\": \"negative\"},\n",
    "        ],\n",
    "        [\n",
    "            {\"value\": \"It's great\"},\n",
    "            {\"value\": \"positive\"},\n",
    "        ],\n",
    "        [\n",
    "            {\"value\": \"Not allowing me to delete my account\"},\n",
    "            {\"value\": \"negative\"},\n",
    "        ],\n",
    "        [\n",
    "            {\"value\": \"So impressed that I bought premium on very first day\"},\n",
    "            {\"value\": \"positive\"},\n",
    "        ],\n",
    "    ],\n",
    "    \"test\": [\n",
    "        [\n",
    "            {\"value\": \"Can't set more than 7 tasks without paying an absurdly expensive weekly subscription\"},\n",
    "            {\"value\": \"negative\"},\n",
    "        ]\n",
    "    ],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the configuration, we enable ClearML tracking and disable tracking on the console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl = DatasetLoader(dataset)\n",
    "\n",
    "config = [\n",
    "    (\"engine/stop_condition/type\", \"MaxEpochs\"),\n",
    "    (\"engine/stop_condition/value\", 6),\n",
    "    (\"engine/modules/tracking.ClearML/enabled\", True),\n",
    "    (\"engine/modules/tracking.ClearML/task\", \"MyExperiment\"),\n",
    "    (\"engine/modules/tracking.Console/enabled\", False),\n",
    "]\n",
    "at = AutoTransformer(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When starting a training run, a new ClearML page is automatically created, and logs are uploaded to it during training.   \n",
    "Once you start executing the cell below, a link to the tracking page will be displayed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "at.init(dataset_loader=dl, path=\".models/example08\")\n",
    "at.train(dl)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "83ee97f1e4ad98a710574577955c6720418d3d8f987616cd4f238f891737d017"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}