Lab05_classifying_movie_reviews.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04-classifying-movie-reviews.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jeonggunlee/DeepLearningBasics/blob/master/Lab05_classifying_movie_reviews.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fxPd_GitEP9E",
        "colab_type": "text"
      },
      "source": [
        "###한림대학교 소프트웨어 융합 대학 \n",
        "##Keras를 활용한 딥러닝 기초 (2020년 8월 5일)\n",
        "\n",
        "이정근 교수\n",
        "\n",
        "   - Homepage: http://www.onchip.net\n",
        "   - Email: jeonggun.lee@hallym.ac.kr / jeonggun.lee@gmail.com \n",
        "\n",
        "\n",
        "---\n",
        "## IMDB 리뷰 감성 분류하기\n",
        "\n",
        "imdb의 데이터 로딩 화일이 이전 버전의 numpy를 활용하기 때문에, 이전 numpy 버전(1.16.2)을 새롭게 설치해야 합니다.\n",
        "\n",
        "이를 위하여 다음 명령어를 수행\n",
        "\n",
        "**!pip install numpy==1.16.2**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0h4XpDobECSZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "558b0375-00e5-4e01-f6f7-e268da481e64"
      },
      "source": [
        "import keras\n",
        "keras.__version__"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'2.3.1'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 1
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "THe9NBvLEhIk",
        "colab_type": "text"
      },
      "source": [
        "참고 사이트\n",
        "\n",
        "   - https://github.com/rickiepark/deep-learning-with-python-notebooks\n",
        "   - 원작자: François Chollet, https://github.com/fchollet\n",
        "   - 한글화: Haesun Park (rickiepark) https://github.com/rickiepark\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9BFFhbm_ECSh",
        "colab_type": "text"
      },
      "source": [
        "# 영화 리뷰 분류: 이진 분류 예제\n",
        "\n",
        "이진 분류 (binary classification)는 아마도 가장 널리 적용된 머신 러닝 문제일 것입니다. 이 예제에서 리뷰 텍스트를 기반으로 영화 리뷰를 긍정 (positive)과 부정(negative)로 분류하는 법을 배우겠습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xcXwwGXnECSi",
        "colab_type": "text"
      },
      "source": [
        "## IMDB 데이터셋\n",
        "\n",
        "- 인터넷 영화 데이터베이스 (Internet Movie Database, IMDB)\n",
        "- 양극단의 리뷰 50,000개로 이루어진 IMDB 데이터셋을 사용\n",
        "- 데이터셋은 훈련 데이터 25,000개와 테스트 데이터 25,000개로 구성\n",
        "   - 각각 50%는 부정, 50%는 긍정 리뷰로 구성\n",
        "\n",
        "- IMDB 데이터셋도 케라스에 포함\n",
        " ```python\n",
        "  from keras.datasets import imdb\n",
        "  ```\n",
        "- 데이터는 전처리되어 있어 각 리뷰(단어 시퀀스)가 숫자 시퀀스로 변환되어 있음\n",
        "   - 각 숫자는 사전에 있는 고유한 단어를 나타냅니다."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1QrzWZ2_ECSj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "2eab0140-7951-4a7b-d7cf-4db4a24c3190"
      },
      "source": [
        "from keras.datasets import imdb\n",
        "\n",
        "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eesTEuVZECSl",
        "colab_type": "text"
      },
      "source": [
        "- `num_words=10000`은 데이터에서 가장 자주 나타나는 단어 10,000개만 사용함을 의미\n",
        "- 변수 `train_data`와 `test_data`는 리뷰의 목록\n",
        "- 각 리뷰는 단어 인덱스의 리스트(단어 시퀀스가 인코딩된 것)\n",
        "-  `train_labels`와 `test_labels`는 부정을 나타내는 0과 긍정을 나타내는 1의 리스트"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OGHQjs11ECSn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "7cbf6139-95a0-4388-ceb0-221c88166897"
      },
      "source": [
        "train_data[0]"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[1,\n",
              " 14,\n",
              " 22,\n",
              " 16,\n",
              " 43,\n",
              " 530,\n",
              " 973,\n",
              " 1622,\n",
              " 1385,\n",
              " 65,\n",
              " 458,\n",
              " 4468,\n",
              " 66,\n",
              " 3941,\n",
              " 4,\n",
              " 173,\n",
              " 36,\n",
              " 256,\n",
              " 5,\n",
              " 25,\n",
              " 100,\n",
              " 43,\n",
              " 838,\n",
              " 112,\n",
              " 50,\n",
              " 670,\n",
              " 2,\n",
              " 9,\n",
              " 35,\n",
              " 480,\n",
              " 284,\n",
              " 5,\n",
              " 150,\n",
              " 4,\n",
              " 172,\n",
              " 112,\n",
              " 167,\n",
              " 2,\n",
              " 336,\n",
              " 385,\n",
              " 39,\n",
              " 4,\n",
              " 172,\n",
              " 4536,\n",
              " 1111,\n",
              " 17,\n",
              " 546,\n",
              " 38,\n",
              " 13,\n",
              " 447,\n",
              " 4,\n",
              " 192,\n",
              " 50,\n",
              " 16,\n",
              " 6,\n",
              " 147,\n",
              " 2025,\n",
              " 19,\n",
              " 14,\n",
              " 22,\n",
              " 4,\n",
              " 1920,\n",
              " 4613,\n",
              " 469,\n",
              " 4,\n",
              " 22,\n",
              " 71,\n",
              " 87,\n",
              " 12,\n",
              " 16,\n",
              " 43,\n",
              " 530,\n",
              " 38,\n",
              " 76,\n",
              " 15,\n",
              " 13,\n",
              " 1247,\n",
              " 4,\n",
              " 22,\n",
              " 17,\n",
              " 515,\n",
              " 17,\n",
              " 12,\n",
              " 16,\n",
              " 626,\n",
              " 18,\n",
              " 2,\n",
              " 5,\n",
              " 62,\n",
              " 386,\n",
              " 12,\n",
              " 8,\n",
              " 316,\n",
              " 8,\n",
              " 106,\n",
              " 5,\n",
              " 4,\n",
              " 2223,\n",
              " 5244,\n",
              " 16,\n",
              " 480,\n",
              " 66,\n",
              " 3785,\n",
              " 33,\n",
              " 4,\n",
              " 130,\n",
              " 12,\n",
              " 16,\n",
              " 38,\n",
              " 619,\n",
              " 5,\n",
              " 25,\n",
              " 124,\n",
              " 51,\n",
              " 36,\n",
              " 135,\n",
              " 48,\n",
              " 25,\n",
              " 1415,\n",
              " 33,\n",
              " 6,\n",
              " 22,\n",
              " 12,\n",
              " 215,\n",
              " 28,\n",
              " 77,\n",
              " 52,\n",
              " 5,\n",
              " 14,\n",
              " 407,\n",
              " 16,\n",
              " 82,\n",
              " 2,\n",
              " 8,\n",
              " 4,\n",
              " 107,\n",
              " 117,\n",
              " 5952,\n",
              " 15,\n",
              " 256,\n",
              " 4,\n",
              " 2,\n",
              " 7,\n",
              " 3766,\n",
              " 5,\n",
              " 723,\n",
              " 36,\n",
              " 71,\n",
              " 43,\n",
              " 530,\n",
              " 476,\n",
              " 26,\n",
              " 400,\n",
              " 317,\n",
              " 46,\n",
              " 7,\n",
              " 4,\n",
              " 2,\n",
              " 1029,\n",
              " 13,\n",
              " 104,\n",
              " 88,\n",
              " 4,\n",
              " 381,\n",
              " 15,\n",
              " 297,\n",
              " 98,\n",
              " 32,\n",
              " 2071,\n",
              " 56,\n",
              " 26,\n",
              " 141,\n",
              " 6,\n",
              " 194,\n",
              " 7486,\n",
              " 18,\n",
              " 4,\n",
              " 226,\n",
              " 22,\n",
              " 21,\n",
              " 134,\n",
              " 476,\n",
              " 26,\n",
              " 480,\n",
              " 5,\n",
              " 144,\n",
              " 30,\n",
              " 5535,\n",
              " 18,\n",
              " 51,\n",
              " 36,\n",
              " 28,\n",
              " 224,\n",
              " 92,\n",
              " 25,\n",
              " 104,\n",
              " 4,\n",
              " 226,\n",
              " 65,\n",
              " 16,\n",
              " 38,\n",
              " 1334,\n",
              " 88,\n",
              " 12,\n",
              " 16,\n",
              " 283,\n",
              " 5,\n",
              " 16,\n",
              " 4472,\n",
              " 113,\n",
              " 103,\n",
              " 32,\n",
              " 15,\n",
              " 16,\n",
              " 5345,\n",
              " 19,\n",
              " 178,\n",
              " 32]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PXy3U4DjECSr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "92c95154-3671-4379-e3dc-061d17f639c2"
      },
      "source": [
        "train_labels[0]"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "escBRmzXECSv",
        "colab_type": "text"
      },
      "source": [
        "- 자주 등장하는 단어 10,000개로 제한했기 때문에 단어 인덱스는 10,000을 넘지 않음"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wZpZZUGiECSv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7c62bee0-a6b6-4c72-ebd6-fdd672ffc881"
      },
      "source": [
        "max([max(sequence) for sequence in train_data])"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "9999"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f-lPLyx4ECSz",
        "colab_type": "text"
      },
      "source": [
        "- 리뷰 데이터 하나를 원래 영어 단어로 어떻게 바꾸는 과정"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-icY3WyDECS0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "eddae4e4-5e61-4726-d72f-1a8d1e881b6c"
      },
      "source": [
        "# word_index는 단어와 정수 인덱스를 매핑한 딕셔너리입니다\n",
        "word_index = imdb.get_word_index()\n",
        "# 정수 인덱스와 단어를 매핑하도록 뒤집습니다\n",
        "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
        "# 리뷰를 디코딩합니다. \n",
        "# 0, 1, 2는 '패딩', '문서 시작', '사전에 없음'을 위한 인덱스이므로 3을 뺍니다\n",
        "decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://s3.amazonaws.com/text-datasets/imdb_word_index.json\n",
            "1646592/1641221 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FeRrS0gOECS2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 178
        },
        "outputId": "c46c8db4-138f-4e8a-9d40-dc3dfa619788"
      },
      "source": [
        "decoded_review"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hOu9z0UtECS6",
        "colab_type": "text"
      },
      "source": [
        "## 데이터 준비\n",
        "\n",
        "신경망에 숫자 리스트를 주입할 수는 없습니다. 리스트를 텐서로 바꾸는 방법:\n",
        "\n",
        "- 같은 길이가 되도록 리스트에 패딩을 추가하고 `(samples, sequence_length)` 크기의 정수 텐서로 변환\n",
        "- 리스트를 원-핫 인코딩하여 0과 1의 벡터로 변환\n",
        "- 예를 들면 시퀀스 `[3, 5]`를 인덱스 3과 5의 위치는 1이고 그 외는 모두 0인 10,000차원의 벡터로 각각 변환\n",
        "- 그다음 부동 소수 벡터 데이터를 다룰 수 있는 `Dense` 층을 신경망의 첫 번째 층으로 사용\n",
        "\n",
        "이해를 돕기 위해 직접 데이터를 원-핫 벡터로 만들겠습니다:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1uX4svmhECS7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "def vectorize_sequences(sequences, dimension=10000):\n",
        "    # 크기가 (len(sequences), dimension))이고 모든 원소가 0인 행렬을 만듭니다\n",
        "    results = np.zeros((len(sequences), dimension))\n",
        "    for i, sequence in enumerate(sequences):\n",
        "        results[i, sequence] = 1.  # results[i]에서 특정 인덱스의 위치를 1로 만듭니다\n",
        "    return results\n",
        "\n",
        "# 훈련 데이터를 벡터로 변환합니다\n",
        "x_train = vectorize_sequences(train_data)\n",
        "# 테스트 데이터를 벡터로 변환합니다\n",
        "x_test = vectorize_sequences(test_data)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RRpscTxhECS9",
        "colab_type": "text"
      },
      "source": [
        "샘플은 다음과 같이 나타남:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NqQBT6A3ECS-",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6784653b-6e5e-4bed-f276-0a8342ac2b41"
      },
      "source": [
        "x_train[0]"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0., 1., 1., ..., 0., 0., 0.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sLG3Y7rxECTB",
        "colab_type": "text"
      },
      "source": [
        "레이블은 쉽게 벡터로 바꿀 수 있음:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8O_JlZGPECTC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 레이블을 벡터로 바꿉니다\n",
        "y_train = np.asarray(train_labels).astype('float32')\n",
        "y_test = np.asarray(test_labels).astype('float32')"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "43kK_YsPECTF",
        "colab_type": "text"
      },
      "source": [
        "이제 신경망에 주입할 데이터가 준비"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wp7KsWgnECTG",
        "colab_type": "text"
      },
      "source": [
        "## 신경망 모델 만들기\n",
        "\n",
        "* 16개의 은닉 유닛을 가진 두 개의 은닉층\n",
        "* 현재 리뷰의 감정을 스칼라 값의 예측으로 출력하는 세 번째 층\n",
        "\n",
        "- 중간에 있는 은닉층은 활성화 함수로 `relu`를 사용\n",
        "- 마지막 층은 확률(0과 1 사이의 점수로, 어떤 샘플이 타깃 '1'일 가능성이 높다는 것은 그 리뷰가 긍정일 가능성이 높다는 것을 의미합니다)을 출력하기 위해 시그모이드 활성화 함수를 사용\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZfyOYeGaECTH",
        "colab_type": "text"
      },
      "source": [
        "![3-layer network](https://s3.amazonaws.com/book.keras.io/img/ch3/3_layer_network.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PYdwr5R6ECTJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras import models\n",
        "from keras import layers\n",
        "\n",
        "model = models.Sequential()\n",
        "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
        "model.add(layers.Dense(16, activation='relu'))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eqXMbF7nECTL",
        "colab_type": "text"
      },
      "source": [
        "손실 함수와 옵티마이저를 선택\n",
        "   - 이진 분류 문제이고 신경망의 출력이 확률이기 때문에(네트워크의 끝에 시그모이드 활성화 함수를 사용한 하나의 유닛으로 된 층을 놓았습니다), `binary_crossentropy` 손실이 적합\n",
        "\n",
        "다음은 `rmsprop` 옵티마이저와 `binary_crossentropy` 손실 함수로 모델을 설정\n",
        "   - 훈련하는 동안 정확도를 사용해 모니터링"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ejzUuLe5ECTN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(optimizer='rmsprop',\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5XZKz8zaECTQ",
        "colab_type": "text"
      },
      "source": [
        "- 케라스에 `rmsprop`, `binary_crossentropy`, `accuracy`가 포함"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RXTPm7WAECTR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras import optimizers\n",
        "\n",
        "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zc3Fm6giECTX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras import losses\n",
        "from keras import metrics\n",
        "\n",
        "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
        "              loss=losses.binary_crossentropy,\n",
        "              metrics=[metrics.binary_accuracy])"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BIB5VW9DECTa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(optimizer='rmsprop',\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OODVEBG2ECTe",
        "colab_type": "text"
      },
      "source": [
        "## 훈련 검증\n",
        "\n",
        "훈련하는 동안 처음 본 데이터에 대한 모델의 정확도를 측정하기 위해서는 원본 훈련 데이터에서 10,000의 샘플을 떼어서 검증 세트를 만듦:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4aA5gRkpECTf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_val = x_train[:10000]\n",
        "partial_x_train = x_train[10000:]\n",
        "\n",
        "y_val = y_train[:10000]\n",
        "partial_y_train = y_train[10000:]"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "Cvh88iY3ECTh",
        "colab_type": "text"
      },
      "source": [
        "- 모델을 512개 샘플씩 미니 배치를 만들어 20번의 에포크 동안 훈련\n",
        "   - `x_train`과 `y_train` 텐서에 있는 모든 샘플에 대해 20번 반복\n",
        "- 동시에 따로 떼어 놓은 10,000개의 샘플에서 손실과 정확도를 측정\n",
        "- `validation_data` 매개변수에 검증 데이터를 전달:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MYV7YTZPECTi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 758
        },
        "outputId": "a9332d92-3540-4e5e-c5c4-fea64f63dc85"
      },
      "source": [
        "history = model.fit(partial_x_train,\n",
        "                    partial_y_train,\n",
        "                    epochs=20,\n",
        "                    batch_size=512,\n",
        "                    validation_data=(x_val, y_val))"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 15000 samples, validate on 10000 samples\n",
            "Epoch 1/20\n",
            "15000/15000 [==============================] - 3s 193us/step - loss: 0.5292 - accuracy: 0.7935 - val_loss: 0.4098 - val_accuracy: 0.8577\n",
            "Epoch 2/20\n",
            "15000/15000 [==============================] - 1s 73us/step - loss: 0.3261 - accuracy: 0.8975 - val_loss: 0.3142 - val_accuracy: 0.8859\n",
            "Epoch 3/20\n",
            "15000/15000 [==============================] - 1s 74us/step - loss: 0.2393 - accuracy: 0.9235 - val_loss: 0.2860 - val_accuracy: 0.8877\n",
            "Epoch 4/20\n",
            "15000/15000 [==============================] - 1s 73us/step - loss: 0.1895 - accuracy: 0.9396 - val_loss: 0.2968 - val_accuracy: 0.8800\n",
            "Epoch 5/20\n",
            "15000/15000 [==============================] - 1s 73us/step - loss: 0.1554 - accuracy: 0.9509 - val_loss: 0.2907 - val_accuracy: 0.8842\n",
            "Epoch 6/20\n",
            "15000/15000 [==============================] - 1s 71us/step - loss: 0.1293 - accuracy: 0.9588 - val_loss: 0.2826 - val_accuracy: 0.8893\n",
            "Epoch 7/20\n",
            "15000/15000 [==============================] - 1s 72us/step - loss: 0.1070 - accuracy: 0.9681 - val_loss: 0.3067 - val_accuracy: 0.8849\n",
            "Epoch 8/20\n",
            "15000/15000 [==============================] - 1s 73us/step - loss: 0.0897 - accuracy: 0.9741 - val_loss: 0.3440 - val_accuracy: 0.8777\n",
            "Epoch 9/20\n",
            "15000/15000 [==============================] - 1s 74us/step - loss: 0.0760 - accuracy: 0.9790 - val_loss: 0.3374 - val_accuracy: 0.8815\n",
            "Epoch 10/20\n",
            "15000/15000 [==============================] - 1s 75us/step - loss: 0.0619 - accuracy: 0.9847 - val_loss: 0.3779 - val_accuracy: 0.8745\n",
            "Epoch 11/20\n",
            "15000/15000 [==============================] - 1s 75us/step - loss: 0.0512 - accuracy: 0.9874 - val_loss: 0.3884 - val_accuracy: 0.8789\n",
            "Epoch 12/20\n",
            "15000/15000 [==============================] - 1s 74us/step - loss: 0.0421 - accuracy: 0.9898 - val_loss: 0.4091 - val_accuracy: 0.8765\n",
            "Epoch 13/20\n",
            "15000/15000 [==============================] - 1s 75us/step - loss: 0.0347 - accuracy: 0.9923 - val_loss: 0.4407 - val_accuracy: 0.8712\n",
            "Epoch 14/20\n",
            "15000/15000 [==============================] - 1s 75us/step - loss: 0.0274 - accuracy: 0.9948 - val_loss: 0.4968 - val_accuracy: 0.8642\n",
            "Epoch 15/20\n",
            "15000/15000 [==============================] - 1s 75us/step - loss: 0.0220 - accuracy: 0.9965 - val_loss: 0.5090 - val_accuracy: 0.8718\n",
            "Epoch 16/20\n",
            "15000/15000 [==============================] - 1s 74us/step - loss: 0.0197 - accuracy: 0.9958 - val_loss: 0.5489 - val_accuracy: 0.8674\n",
            "Epoch 17/20\n",
            "15000/15000 [==============================] - 1s 73us/step - loss: 0.0113 - accuracy: 0.9991 - val_loss: 0.5698 - val_accuracy: 0.8681\n",
            "Epoch 18/20\n",
            "15000/15000 [==============================] - 1s 72us/step - loss: 0.0114 - accuracy: 0.9986 - val_loss: 0.6083 - val_accuracy: 0.8661\n",
            "Epoch 19/20\n",
            "15000/15000 [==============================] - 1s 74us/step - loss: 0.0095 - accuracy: 0.9987 - val_loss: 0.6544 - val_accuracy: 0.8674\n",
            "Epoch 20/20\n",
            "15000/15000 [==============================] - 1s 74us/step - loss: 0.0052 - accuracy: 0.9999 - val_loss: 0.7062 - val_accuracy: 0.8645\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hQLfpUezECTl",
        "colab_type": "text"
      },
      "source": [
        "- `model.fit()` 메서드는 `History` 객체를 반환\n",
        "- 이 객체는 훈련하는 동안 발생한 모든 정보를 담고 있는 딕셔너리인 `history` 속성을 가지고 있음:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nyYGnGpUECTm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "04e1e9f5-3a7d-412c-9c58-970de6f0c06c"
      },
      "source": [
        "history_dict = history.history\n",
        "history_dict.keys()"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_keys(['val_loss', 'val_accuracy', 'loss', 'accuracy'])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4qc1dSLGECTp",
        "colab_type": "text"
      },
      "source": [
        "- 이 딕셔너리는 훈련과 검증하는 동안 모니터링할 측정 지표당 하나씩 모두 네 개의 항목을 포함\n",
        "- 맷플롯립을 사용해 훈련과 검증 데이터에 대한 손실과 정확도 display:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WaTZZqWpECTp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OT-hbiMZECTs",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "308498c3-508c-42f2-c266-3b8867a57c5f"
      },
      "source": [
        "acc = history.history['accuracy']\n",
        "val_acc = history.history['val_accuracy']\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "epochs = range(1, len(acc) + 1)\n",
        "\n",
        "# ‘bo’는 파란색 점을 의미합니다\n",
        "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
        "# ‘b’는 파란색 실선을 의미합니다\n",
        "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
        "plt.title('Training and validation loss')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5xU5dn/8c9FEaRYKDa6CUg0SFtARQl2RAQLRhALQUVI7LFgiMKD8jOJ5gkhohE1tqDYHgkqKEFF7LLiiqAQEUGXWBARUIqU6/fHfRaGdRvMnpnZme/79ZrXzDlz5pxrz86ea+9y7tvcHRERyV3V0h2AiIiklxKBiEiOUyIQEclxSgQiIjlOiUBEJMcpEYiI5DglAqlUZjbdzM6v7G3TycyWmtlxMezXzeyn0eu/m9kNFdl2F44zyMxm7GqcZey3p5kVVvZ+JfVqpDsAST8z+y5hsQ6wEdgSLV/s7pMqui93PymObbOduw+rjP2YWUvgE6Cmu2+O9j0JqPDvUHKPEoHg7vWKXpvZUuBCd59ZfDszq1F0cRGR7KGqISlVUdHfzK4zsy+A+8xsbzN7xsxWmNmq6HXThM/MMrMLo9eDzexVM7st2vYTMztpF7dtZWazzWytmc00swlm9s9S4q5IjDeZ2WvR/maYWaOE9881s2VmttLMRpZxfrqZ2RdmVj1h3WlmNi963dXM3jCzb83sczO73cx2K2Vf95vZzQnL10Sf+a+ZDSm27clm9q6ZrTGzz8xsdMLbs6Pnb83sOzM7vOjcJnz+CDObY2aro+cjKnpuymJmP4s+/62ZLTCzvgnv9TazD6J9Ljezq6P1jaLfz7dm9o2ZvWJmui6lmE64lGc/oAHQAhhK+M7cFy03B9YDt5fx+W7AIqAR8CfgXjOzXdj2YeBtoCEwGji3jGNWJMazgV8B+wC7AUUXpoOBO6P9HxAdryklcPe3gO+BY4rt9+Ho9RbgyujnORw4Fvh1GXETxdAriud4oDVQvH3ie+A8YC/gZGC4mZ0avdcjet7L3eu5+xvF9t0AeBYYH/1s/ws8a2YNi/0MPzo35cRcE3gamBF97lJgkpkdFG1yL6GasT7wc+DFaP1vgUKgMbAv8DtA496kmBKBlGcrMMrdN7r7endf6e5Puvs6d18LjAV+Ucbnl7n73e6+BXgA2J/wB1/hbc2sOdAFuNHdf3D3V4GppR2wgjHe5+7/cff1wGNAh2h9f+AZd5/t7huBG6JzUJpHgIEAZlYf6B2tw93fcfc33X2zuy8F7iohjpL8Mopvvrt/T0h8iT/fLHd/3923uvu86HgV2S+ExPGRuz8UxfUIsBA4JWGb0s5NWQ4D6gF/iH5HLwLPEJ0bYBNwsJnt4e6r3H1uwvr9gRbuvsndX3ENgJZySgRSnhXuvqFowczqmNldUdXJGkJVxF6J1SPFfFH0wt3XRS/r7eS2BwDfJKwD+Ky0gCsY4xcJr9clxHRA4r6jC/HK0o5F+O//dDOrBZwOzHX3ZVEcbaJqjy+iOP4foXRQnh1iAJYV+/m6mdlLUdXXamBYBfdbtO9lxdYtA5okLJd2bsqN2d0Tk2bifs8gJMllZvaymR0erb8VWAzMMLMlZjaiYj+GVCYlAilP8f/OfgscBHRz9z3YXhVRWnVPZfgcaGBmdRLWNStj+2Ri/Dxx39ExG5a2sbt/QLjgncSO1UIQqpgWAq2jOH63KzEQqrcSPUwoETVz9z2Bvyfst7z/pv9LqDJL1BxYXoG4yttvs2L1+9v26+5z3L0fodpoCqGkgbuvdfffuvuBQF/gKjM7NslYZCcpEcjOqk+oc/82qm8eFfcBo/+w84HRZrZb9N/kKWV8JJkYnwD6mNmRUcPuGMr/O3kYuJyQcB4vFsca4DszawsMr2AMjwGDzezgKBEVj78+oYS0wcy6EhJQkRWEqqwDS9n3NKCNmZ1tZjXM7CzgYEI1TjLeIpQerjWzmmbWk/A7mhz9zgaZ2Z7uvolwTrYCmFkfM/tp1Ba0mtCuUlZVnMRAiUB21jhgd+Br4E3guRQddxChwXUlcDPwKOF+h5LscozuvgD4DeHi/jmwitCYWZaiOvoX3f3rhPVXEy7Sa4G7o5grEsP06Gd4kVBt8mKxTX4NjDGztcCNRP9dR59dR2gTeS3qiXNYsX2vBPoQSk0rgWuBPsXi3mnu/gPhwn8S4bzfAZzn7gujTc4FlkZVZMMIv08IjeEzge+AN4A73P2lZGKRnWdql5GqyMweBRa6e+wlEpFspxKBVAlm1sXMfmJm1aLulf0Idc0ikiTdWSxVxX7A/xEabguB4e7+bnpDEskOqhoSEclxqhoSEclxVa5qqFGjRt6yZct0hyEiUqW88847X7t745Leq3KJoGXLluTn56c7DBGRKsXMit9Rvo2qhkREcpwSgYhIjlMiEBHJcVWujaAkmzZtorCwkA0bNpS/saRV7dq1adq0KTVr1kx3KCISyYpEUFhYSP369WnZsiWlz3ki6eburFy5ksLCQlq1apXucEQkkhVVQxs2bKBhw4ZKAhnOzGjYsKFKbiIZJisSAaAkUEXo9ySSeWJNBGbWy8wWmdnikmYeMrO/mFlB9PiPmX0bZzwiIlXRpk1w9dXwWanz8iUntkQQTQs4gTA++cHAwGhi8G3c/Up37+DuHYC/EQYVq3JWrlxJhw4d6NChA/vttx9NmjTZtvzDDz+U+dn8/Hwuu+yyco9xxBFHVEqss2bNok+fPpWyLxGJ37p1cNpp8Oc/w7Rp8RwjzhJBV2Cxuy+JJq2YTBg6uDQDiSb9jtukSdCyJVSrFp4nTUpufw0bNqSgoICCggKGDRvGlVdeuW15t912Y/PmzaV+Ni8vj/Hjx5d7jNdffz25IEWkyvnmGzj+eJg+Hf7+d7j44niOE2ciaMKOE3AXsuME2duYWQugFT+eiano/aFmlm9m+StWrEgqqEmTYOhQWLYM3MPz0KHJJ4PiBg8ezLBhw+jWrRvXXnstb7/9NocffjgdO3bkiCOOYNGiRcCO/6GPHj2aIUOG0LNnTw488MAdEkS9evW2bd+zZ0/69+9P27ZtGTRoEEUjyE6bNo22bdvSuXNnLrvssnL/8//mm2849dRTOfTQQznssMOYN28eAC+//PK2Ek3Hjh1Zu3Ytn3/+OT169KBDhw78/Oc/55VXXqncEyYiOygshB49ID8fHnssviQAmdN9dADwhLtvKelNd58ITATIy8tLatzskSNDUSvRunVh/aBBJX9mVxUWFvL6669TvXp11qxZwyuvvEKNGjWYOXMmv/vd73jyySd/9JmFCxfy0ksvsXbtWg466CCGDx/+oz737777LgsWLOCAAw6ge/fuvPbaa+Tl5XHxxRcze/ZsWrVqxcCBA8uNb9SoUXTs2JEpU6bw4osvct5551FQUMBtt93GhAkT6N69O9999x21a9dm4sSJnHjiiYwcOZItW7awrvhJFJFKs3AhnHgirFoFzz0HRx8d7/HiTATLgWYJy02jdSUZQJgnNnaffrpz65Nx5plnUr16dQBWr17N+eefz0cffYSZsWnTphI/c/LJJ1OrVi1q1arFPvvsw5dffknTpk132KZr167b1nXo0IGlS5dSr149DjzwwG398wcOHMjEiRPLjO/VV1/dloyOOeYYVq5cyZo1a+jevTtXXXUVgwYN4vTTT6dp06Z06dKFIUOGsGnTJk499VQ6dOiQ1LkRkZK9/Tb07g3Vq8PLL0PHjvEfM86qoTlAazNrZWa7ES72U4tvZGZtgb0JE1fHrnnznVufjLp16257fcMNN3D00Uczf/58nn766VL70teqVWvb6+rVq5fYvlCRbZIxYsQI7rnnHtavX0/37t1ZuHAhPXr0YPbs2TRp0oTBgwfz4IMPVuoxRQRmzIBjjoE994TXXktNEoAYE4G7bwYuAZ4HPgQec/cFZjbGzPombDoAmOwpmipt7FioU2fHdXXqhPVxWr16NU2ahCaS+++/v9L3f9BBB7FkyRKWLl0KwKOPPlruZ4466igmRY0js2bNolGjRuyxxx58/PHHtGvXjuuuu44uXbqwcOFCli1bxr777stFF13EhRdeyNy5cyv9ZxDJZY88An36wE9/Cq++Gp5TJdY2AnefBkwrtu7GYsuj44yhuKJ2gJEjQ3VQ8+YhCVR2+0Bx1157Leeffz4333wzJ598cqXvf/fdd+eOO+6gV69e1K1bly5dupT7maLG6UMPPZQ6derwwAMPADBu3DheeuklqlWrxiGHHMJJJ53E5MmTufXWW6lZsyb16tVTiUCkEo0fD5dfHhqHp04NJYJUqnJzFufl5XnxiWk+/PBDfvazn6Uposzx3XffUa9ePdyd3/zmN7Ru3Zorr7wy3WH9iH5fIoE73HBD+Gf01FNDqaB27XiOZWbvuHteSe9lzRATAnfffTcdOnTgkEMOYfXq1VwcZ38zEUnKli2hS+jYsXDhhfD44/ElgfJkSvdRqQRXXnllRpYARGRHGzbA2WfDU0/B734HN98M6RyGS4lARCSFVq+Gfv1C19C//hUqMMJM7JQIRERS5IsvoFcvWLAgjGZw9tnpjihQIhARSYGPP4YTTgjJ4Jlnwp3DmUKJQEQkZq+/HkYQ3bIFXnwRunVLd0Q7Uq+hSnD00Ufz/PPP77Bu3LhxDB8+vNTP9OzZk6JusL179+bbb388FcPo0aO57bbbyjz2lClT+OCDD7Yt33jjjcycOXNnwi+RhqsWqRz33gs9e0L9+uFGsUxLAqBEUCkGDhzI5MmTd1g3efLkCg38BmHU0L322muXjl08EYwZM4bjjjtul/YlIpVn0ya49NLQNbRnzzCGUNu26Y6qZEoElaB///48++yz2yahWbp0Kf/973856qijGD58OHl5eRxyyCGMGjWqxM+3bNmSr7/+GoCxY8fSpk0bjjzyyG1DVUO4R6BLly60b9+eM844g3Xr1vH6668zdepUrrnmGjp06MDHH3/M4MGDeeKJJwB44YUX6NixI+3atWPIkCFs3Lhx2/FGjRpFp06daNeuHQsXLizz59Nw1SI7Z8WK0B5w++3w29+GCWUaNEh3VKXLujaCK66AgoLK3WeHDjBuXOnvN2jQgK5duzJ9+nT69evH5MmT+eUvf4mZMXbsWBo0aMCWLVs49thjmTdvHoceemiJ+3nnnXeYPHkyBQUFbN68mU6dOtG5c2cATj/9dC666CIAfv/733Pvvfdy6aWX0rdvX/r06UP//v132NeGDRsYPHgwL7zwAm3atOG8887jzjvv5IorrgCgUaNGzJ07lzvuuIPbbruNe+65p9SfT8NVi1RcQUG4S/iLL+Chh+Ccc9IdUflUIqgkidVDidVCjz32GJ06daJjx44sWLBgh2qc4l555RVOO+006tSpwx577EHfvtvH5ps/fz5HHXUU7dq1Y9KkSSxYsKDMeBYtWkSrVq1o06YNAOeffz6zZ8/e9v7pp58OQOfOnbcNVFeaV199lXPPPRcoebjq8ePH8+2331KjRg26dOnCfffdx+jRo3n//fepX79+mfsWySaPPgpHHBEahV99tWokAcjCEkFZ/7nHqV+/flx55ZXMnTuXdevW0blzZz755BNuu+025syZw957783gwYNLHX66PIMHD2bKlCm0b9+e+++/n1mzZiUVb9FQ1skMYz1ixAhOPvlkpk2bRvfu3Xn++ee3DVf97LPPMnjwYK666irOO++8pGIVyXRbtoQxg265JSSCJ5+E/fZLd1QVpxJBJalXrx5HH300Q4YM2VYaWLNmDXXr1mXPPffkyy+/ZPr06WXuo0ePHkyZMoX169ezdu1ann766W3vrV27lv33359NmzZtGzoaoH79+qxdu/ZH+zrooINYunQpixcvBuChhx7iF7/4xS79bBquWqR0q1dD374hCVx0UegeWpWSAGRhiSCdBg4cyGmnnbatiqh9+/Z07NiRtm3b0qxZM7p3717m5zt16sRZZ51F+/bt2WeffXYYSvqmm26iW7duNG7cmG7dum27+A8YMICLLrqI8ePHb2skBqhduzb33XcfZ555Jps3b6ZLly4MGzZsl34uDVctUrJFi8JwER9/DHfcAcOGpXfMoF2lYagl5fT7kmwwbRoMHAi1asETT4S5BDKZhqEWEakk7qEaqE8f+MlPID8/85NAeVQ1JCJSQd9/DxdcEHoHDRgQ7houPvVtVZQ1JYKqVsWVq/R7kqpq6VLo3h0eewz++Ed4+OHsSAIQcyIws15mtsjMFpvZiFK2+aWZfWBmC8zs4V05Tu3atVm5cqUuMhnO3Vm5ciW10zUNk8gu+r//g7y8kAyeeQauvbZqNgqXJraqITOrDkwAjgcKgTlmNtXdP0jYpjVwPdDd3VeZ2T67cqymTZtSWFjIihUrKiN0iVHt2rVp2rRpusMQqZBVq8J4QZMmQceOMHkyRPdoZpU42wi6AovdfQmAmU0G+gGJt9ZeBExw91UA7v7VrhyoZs2atGrVKslwRUS2mz49DBj31VcwenSYUrJmzXRHFY84q4aaAJ8lLBdG6xK1AdqY2Wtm9qaZ9SppR2Y21MzyzSxf//WLSJzWroWhQ6F3b9h7b3jzTRg1KnuTAKS/sbgG0BroCQwE7jazH43H7O4T3T3P3fMaN26c4hBFJFfMmgWHHgr33BPaAfLzIRr3MavFmQiWA80SlptG6xIVAlPdfZO7fwL8h5AYRERSZt06uPxyOPpoqFEjDBj3xz9CrvRriDMRzAFam1krM9sNGABMLbbNFEJpADNrRKgqWhJjTCIiO3jjjTDU/PjxoWG4oCAMHJdLYksE7r4ZuAR4HvgQeMzdF5jZGDMrGl/5eWClmX0AvARc4+4r44pJRKTIxo0wYgQceST88AO88EJIBnXrpjuy1MuKsYZERHbG3Llw/vkwf37oGfTnP8Mee6Q7qnhprCEREcI8wmPGhAnkV66EZ5+Fu+/O/iRQHo01JCI5YcGCUAp45x0YNChUA2XyPMKppEQgIllr48YwJMSDD4Zho/faKwwZfcYZ6Y4ssygRiEhWcQ83gT34YBgldNWqMGPYFVfANdfAPrs0kE12UyIQkaywdCk89FBIAIsXw+67w2mnwXnnwbHHhvsDpGQ6NSJSZa1eHap6HnwQZs8O63r2DOMCnXGGGoErSolARKqUzZvh3/8OF/8pU2DDhjAi6M03wznnQIsW6Y6w6lEiEJEq4f334YEHwpDQX3wRevwMGRKqfrp2za75AVJNiUBEMtr69XDddfC3v4URQE8+OVz8e/cOE8dL8pQIRCRjvftu6PP/4Ydw2WVwww3QqFG6o8o+urNYRDLOli1h9M9u3UKD8IwZ8Ne/KgnEJScSwaRJ0LIlVKsWnidNSndEIlKaZcvgmGPCgHB9+8K8eXD88emOKrtlfdXQpElhtqF168LysmVhGUKRU0Qygzs8/DD8+tewdSvcf39oC1AjcPyyvkQwcuT2JFBk3bqwXkQyw6pVcPbZoftnu3bw3nthXCAlgdTI+kTw6ac7t15EUuull8L0kE88Ee4FmDULDjww3VHllqxPBM2b79x6EUmNjRvh6qvD8A916sDrr4eSuoaCSL2sTwRjx4YvWaI6dcJ6EUmP+fPDTWB//jNcfHGYKKZLl3RHlbuyPhEMGgQTJ4bbzs3C88SJaigWSYetW2HcOMjLC3cHP/MM3Hlnbk4PmUlyohA2aJAu/CLptnw5DB4MM2fCKafAPfdoSOhMEWsiMLNewF+B6sA97v6HYu8PBm4Flkerbnf3e+KMSURSY/VqePtteOut8Hj55XCj2F13wUUXqUdQJoktEZhZdWACcDxQCMwxs6nu/kGxTR9190viikNE4rdpUxgUruii/9ZbsHDh9vfbtoX+/cNNYm3apC9OKVmcJYKuwGJ3XwJgZpOBfkDxRCAiVYh76H6deNGfOzcMDgfQuHEYGmLQoPDcpUuYIlIyV5yJoAnwWcJyIdCthO3OMLMewH+AK939s+IbmNlQYChAc/X7FEm5DRtgwgR45ZUwDeSXX4b1tWpBp06h50+3buHRsqWqfaqadDcWPw084u4bzexi4AHgmOIbuftEYCJAXl6epzZEkdy2YUOY8vG556B16zDuT7ducNhh4Uaw3XZLd4SSrDgTwXKgWcJyU7Y3CgPg7isTFu8B/hRjPCKyk9avh379Qk+fu++GCy9Md0QShzjvI5gDtDazVma2GzAAmJq4gZntn7DYF/gwxnhEZCesWxe6ec6cCffeqySQzWIrEbj7ZjO7BHie0H30H+6+wMzGAPnuPhW4zMz6ApuBb4DBccUjIhX3/fchCcyatX0UUMle5l61qtzz8vI8Pz8/3WGIZK3vvgvTQb76apggXjdjZgcze8fd80p6L92NxSKSQdauDXMBv/FGmMtjwIB0RySpoEQgIgCsWQO9eoW7gR95BM48M90RSaooEYgIq1eHJJCfD48+Cmecke6IJJWUCERy3LffwgknQEEBPP44nHpquiOSVFMiEMlh33wTksD778OTT4aeQpJ7lAhEctTKleEu4QUL4KmnQiOx5CYlApEc9PXXcNxxYYTQf/0rtA9I7lIiEMkxK1aEeYI/+giefjqUCiS3KRGI5JAvvwxJYMmSME3kscemOyLJBEoEIjniiy/gmGNg2TJ49lk4+uh0RySZIusnrxcReOcd6NkzTCgzfbqSgOxIiUAki336KZx7LuTlha6izz0HPXqkOyrJNEoEIllo9Wq4/vowP/ATT8DvfgeLF8ORR6Y7MslEaiMQySKbNoUJZEaNCl1Ezz0Xbr4ZNMOrlEUlApEs4A5Tp0K7dvCb38DPfx7GDXrwQSUBKZ8SgUgV9847oTdQv35heepUePFF6Nw5vXFJ1aFEIFJFJTYEL1gAd9wRxgw65RQwS3d0UpWojUCkilmzBv7wB/jLX8Ly9dfDddfBnnumNy6punIqEaxZA3vske4oRHZNUUPw6NFhmAg1BEtlibVqyMx6mdkiM1tsZiPK2O4MM3MzK3E+zcrw17+GrnSrVsV1BJF4fPVV+P4WNQQffLAagqVyxZYIzKw6MAE4CTgYGGhmB5ewXX3gcuCtuGIB+MUvwn9Ro0bFeRSRyrF+fZgprE8fOOAAuOIKqFcvjBT60ktqCJbKFWeJoCuw2N2XuPsPwGSgXwnb3QT8EdgQYyx06ADDh8OECfDee3EeSWTXbN0Ks2fDhRfCfvuFieMLCuDqq2H+/FAK6NtXDcFS+eJMBE2AzxKWC6N125hZJ6CZuz8bYxzbjBkDDRrAJZeEftcimeA//4EbboADDwwl18mT4bTTYObMMEDcH/4AhxyS7iglm6Wt+6iZVQP+F/htBbYdamb5Zpa/YsWKXT5mgwZwyy3w6qvw8MO7vBuRpH39dSiddusGBx0E/+//hed//jMMFX3//WGI6OrV0x2p5ALzmP41NrPDgdHufmK0fD2Au98SLe8JfAx8F31kP+AboK+755e237y8PM/PL/Xtcm3dCocdBp99BosWqReRpM7GjWEOgIceCsNAb94Mhx4aev+cfXZoCxCJi5m94+4ldsiJs/voHKC1mbUClgMDgLOL3nT31UCjhCBnAVeXlQQqQ7VqcPvtIRncdBPcemucR5NctWlTmAayoCC0SRUUhDr+1atD/f/ll4cE0L59uiMViTERuPtmM7sEeB6oDvzD3ReY2Rgg392nxnXs8nTtChdcAOPGwZAh8LOfpSsSyQarVm2/2L/3XngsWAA//BDer107jP3zy1/CGWeEKp8aOXUHj2S6ClUNmVldYL27bzWzNkBbYLq7b4o7wOKSrRoqsmJFuK8gLw9mzFBPDCnf1q3wySc7/pf/3nthqIci++4beqi1b7/9uU0bXfgl/Sqjamg2cJSZ7Q3MIFT7nAUMqpwQU69x43BX5iWXwJNPQv/+6Y5IMpV7GMfn97+Hb78N66pVg7ZtoXv3cJNX+/bhsd9+6Y1VZFdUtEQw1907mdmlwO7u/iczK3D3DvGHuKPKKhFAaKwrmrnpww+hbt1K2a1kka++gl/9CqZNg+OOg7POCv/pH3II7L57uqMTqbiySgQV7T5qUS+gQUBRn/8q37GtRo3QcPzZZ6FbqUii6dPDsA4vvADjx4cqxAsvDP88KAlINqloIrgCuB54KmrwPRB4Kb6wUufII+Gcc0LvocWL0x2NZIL16+Gyy6B3b9hnn9Db59JL1Y4k2Wun7yOIbgSr5+5r4gmpbJVZNVTk88/DzTw9eoR+3pK73n8/9OmfPz908fzDH0KvH5GqLumqITN72Mz2iHoPzQc+MLNrKjPIdNp//zC077PPKhHkqq1bwwifXbqEHmXTp4fuxUoCkgsqWjV0cFQCOBWYDrQCzo0tqjS49NJwP8Hll8OGWIe/k0zzxRehGuiKK+D442HePOjVK91RiaRORRNBTTOrSUgEU6P7B7Jq2LaaNeFvf4MlS+C229IdjaTK00+HBuGXXw5dRKdODe0CIrmkoongLmApUBeYbWYtgLS0EcTp2GPhzDPDAGDLlqU7GonTunXw61+HYZ2bNAkTwA8frgZhyU0VSgTuPt7dm7h7bw+WAUfHHFta3HZbuBhcdVW6I5FE330HTzwBL74IH32UXPVdQUHoAnrnnfDb38Jbb4VZv0RyVYXuLI5GCh0F9IhWvQyMAVbHFFfaNG8OI0eGx4wZcMIJ6Y5IZs4M/feLl9IaNw6/r+bNoVmzH7/eb79wB3CRrVvDhO/XXw+NGoXf7/HHp/ZnEclEFb2z+ElCb6EHolXnAu3d/fQYYytRHN1Hi9u4MQwSVr16aDjcbbdYDyelWL06zM51zz2he29RL55PPw03AX766fbXy5aFUkOiGjWgadPtyeGzz8IMYP36hX02alTycUWyUWWMNfQTdz8jYfl/zKwg+dAyU61aoSvhySeH5wMOCCWETz8NF5WxY2FQlR1lqWp45hm4+OLQo2fEiDDXdFldOd1D4iieIIpev/ZaaBe46y646CK1BYgkqmgiWG9mR7r7qwBm1h1YH19Y6de7d2hIvPHGsFxUJ71sGQwdGl4rGVS+lStDF95Jk0Jvnn/9K9Tnl8cM9torPNq1iz9OkWxS0Uv7Ls4AABHxSURBVF5Dw4AJZrbUzJYCtwMXxxZVhvjLX0ICKN4wuW5dKCFI5XriidBo++ijoQSQn1+xJCAiyalQicDd3wPam9ke0fIaM7sCmBdncOl24IGlv5c4Br0k58svw1DOTz4JnTrBv/8dpnAUkdTYqcnr3X1NwhhDOdHBslmzktc3b57aOLKRe5is/eCDQ5vALbeErpxKAiKptVOJoJicaG675ZYf9xqqUyc0GMuuKyyEU04J8/YedFDo2z9ihGbyEkmHZBJBVg0xUZpBg+Dee7f3WNlrr9CNUQ3Fu8Y9dN085JBwc9i4cfDKK2G2LxFJjzITgZmtNbM1JTzWAgekKMa0O+eccDdrnz5hqsLrroObbgrdFaXiPvkk3MB10UWhLeD990MPoepVfoojkaqtzETg7vXdfY8SHvXdvdxCvJn1MrNFZrbYzEaU8P4wM3vfzArM7FUzy9gb/Zs2DQOUzZkTJrO58UZo2RLGjFFCKM/HH4chO9q1g7ffhr//Pcz69ZOfpDsyEYHkqobKZGbVgQnAScDBwMASLvQPu3u7aO7jPwH/G1c8lSUvL4xQmZ8fJrIZNSokhP/5n+0Tm0sYzuH550MpqnXrMLLrKaeECV8uvnjHoR9EJL3i/HPsCix29yXu/gMwGeiXuEGxWc7qUoXaHTp3Djc7zZ0LPXuGiW1atgyJYdWqyj+eO6xdG54z2Zo14aL/s5+FMf3nzIHf/z7ciPfII+ptJZKJ4uyj0QT4LGG5EOhWfCMz+w2hK+puwDEl7cjMhgJDAZpn2JWkY0d46qnQ6+Wmm0JV0bhxoe77iiugQYOd3+fmzbBoEbz7btjvu++Gx6pVocdSkybh0bTpj183bQr77pv6evdFi+D22+H++8OYP926ha6h/fuHITtEJHPt9JzFFd6xWX+gl7tfGC2fC3Rz90tK2f5s4ER3P7+s/aZi0LlkzJsXksGTT0L9+mES9KuuKj0hrF8fGk2LLvbvvhv2UXQ3c61aoW69Y8dwg9uKFaHr5fLl4fm//4VNm3bcZ/XqYeTN4omiWbNQL9+6Ney9d/I/65YtMG1aSAAzZoRutmedFWZ769Il+f2LSOUpa9C5OBPB4cBodz8xWr4ewN1vKWX7asAqd9+zrP1meiIo8v77oYTw+OMhIVx6KVxwASxduuNFf+HCUJ8OoWtqhw7hol/0OOigMHtaabZuha+/3p4cihJE4vPy5aHKJlHDhiEhtG4NP/3p9tetW8OeZf4GQsnkH/8IM3otWRIG5Rs+PIzBpNm9RDJTuhJBDeA/wLHAcmAOcLa7L0jYprW7fxS9PgUYVVqgRapKIigyf/72hJB4qps02fGC36FDaGOIa1TMtWvDsBiLF4eusImPwsIdt23cuOQEAWH0zn/+M4y3dNRRIcGdemrZyUpE0i8tiSA6cG9gHFAd+Ie7jzWzMUC+u081s78CxwGbgFXAJYmJoiRVLREUWbAgdJls2zZc+Bs3TndE261fH7p4Fk8QH30Uqp4S1a4dbqa75JKQvESkakhbIohDVU0EVdX3329PEqtXh0ldGjZMd1QisrMqY2IayVF164ZB4DQQnEj20m09IiI5TolARCTHKRGIiOQ4JQIRkRynRCAikuOUCEREcpwSgYhIjlMiSIFJk8LwEdWqhedJk9IdkYjIdrqhLGaTJoXB2NatC8vLloVl0LzHIpIZVCKI2ciR25NAkXXrwnoRkUygRBCzTz/dufUiIqmmRBCz0iZUy7CJ1kQkhykRxGzs2DC9ZKI6dcJ6EZFMoEQQs0GDYOJEaNEiTDrTokVYVkOxiGQK9RpKgUGDdOEXkcylEoGISI5TIhARyXFKBCIiOS7WRGBmvcxskZktNrMRJbx/lZl9YGbzzOwFM2sRZzwiIvJjsSUCM6sOTABOAg4GBprZwcU2exfIc/dDgSeAP8UVj4iIlCzOEkFXYLG7L3H3H4DJQL/EDdz9JXcvGoDhTaBpjPGIiEgJ4kwETYDPEpYLo3WluQCYXtIbZjbUzPLNLH/FihWVGGLVoNFLRSROGXEfgZmdA+QBvyjpfXefCEwEyMvL8xSGlnYavVRE4hZniWA50CxhuWm0bgdmdhwwEujr7htjjKdK0uilIhK3OBPBHKC1mbUys92AAcDUxA3MrCNwFyEJfBVjLFWWRi8VkbjFlgjcfTNwCfA88CHwmLsvMLMxZtY32uxWoB7wuJkVmNnUUnaXszR6qYjELdY2AnefBkwrtu7GhNfHxXn8bDB27I5tBKDRS0WkcunO4gyn0UtFJG4Z0WtIyqbRS0UkTioRiIjkOCUCEZEcp0SQA3RnsoiURW0EWU53JotIeVQiyHK6M1lEyqNEkOV0Z7KIlEeJIMvpzmQRKY8SQZYbOzbciZxIdyaLSCIlgiynO5NFpDxKBDlg0CBYuhS2bg3PO5sE1P1UJLup+6iUSd1PRbKfSgRSJnU/Fcl+SgRSJnU/Fcl+SgRSJnU/Fcl+SgRSJnU/Fcl+SgRSpsrofqpeRyKZTb2GpFzJTIyjXkcimS/WEoGZ9TKzRWa22MxGlPB+DzOba2abzax/nLFIeqjXkUjmiy0RmFl1YAJwEnAwMNDMDi622afAYODhuOKQ9FKvI5HMF2eJoCuw2N2XuPsPwGSgX+IG7r7U3ecBW2OMQ9JIvY5EMl+ciaAJ8FnCcmG0bqeZ2VAzyzez/BUrVlRKcJIaldHrSI3NIvGqEr2G3H2iu+e5e17jxo3THY7shGR7HRU1Ni9bBu7bG5uVDEQqT5yJYDnQLGG5abROckwyg96psVkkfnEmgjlAazNrZWa7AQOAqTEeT7KQGptF4hdbInD3zcAlwPPAh8Bj7r7AzMaYWV8AM+tiZoXAmcBdZrYgrnikaqqMxma1MYiULdYbytx9GjCt2LobE17PIVQZiZRo7Ngdb0iDnWts1g1tIuWrEo3FkruSbWxWG4NI+czd0x3DTsnLy/P8/Px0hyFVRLVqobdRcWah8VokV5jZO+6eV9J7KhFIVlMbg0j5lAgkqyV7Q5vuY5BcoEQgWU1tDCLlUyKQrJfMDW2VcR+DqpYk0ykRiJQh2TYGVS1JVaBEIFKGZNsYVLUkVYESgUgZkm1jqKwhMlS9JHHSVJUi5Uhmqs7mzUN1UEnrK0p3R0vcVCIQiVFlzMdQGdVLKlFIWZQIRGKUbNUSJF+9pAZrKY8SgUjMkum+Csn3XFKJQsqjRCCS4ZKtXsqEEoUSSWZTIhDJcMlWL6W7RKFEkvk0+qhIlive6whCiaKiySTZEVxbtiy551SLFqGqrDzJxi+BRh8VyWHpLlEkWzWVCW0cWV8icfcq9ejcubOLSOr885/udeq4h3JBeNSpE9ZXRIsWO3626NGiRcU+b1by581SE3+yny/aR4sWIeYWLXbus5UFyPdSrqtpv7Dv7EOJQCT1krmQpTuRpPvzmZJI0pYIgF7AImAxMKKE92sBj0bvvwW0LG+fSgQiVU86E0myJYpkP58JicQ9TYkAqA58DBwI7Aa8BxxcbJtfA3+PXg8AHi1vv0oEIrknmUSS7hJBuhNJkbISQZyNxV2Bxe6+xN1/ACYD/Ypt0w94IHr9BHCsmVmMMYlIFZTMTXnJ3oeR7OfT3dheEXEmgibAZwnLhdG6Erdx983AaqBh8R2Z2VAzyzez/BUrVsQUrohko2R7TSX7+XQnkoqoEt1H3X2iu+e5e17jxo3THY6IVDHJDvORzOfTnUgqIs5hqJcDzRKWm0brStqm0MxqAHsCK2OMSUQk5ZIZyrzocyNHhuqg5s1DEqjMm+niTARzgNZm1opwwR8AnF1sm6nA+cAbQH/gxahRQ0REIskkkoqILRG4+2YzuwR4ntCD6B/uvsDMxhBar6cC9wIPmdli4BtCshARkRSKdYYyd58GTCu27saE1xuAM+OMQUREylYlGotFRCQ+SgQiIjlOiUBEJMdVufkIzGwFUMLo5hmhEfB1uoMog+JLTqbHB5kfo+JLTjLxtXD3Em/EqnKJIJOZWb6XMvFDJlB8ycn0+CDzY1R8yYkrPlUNiYjkOCUCEZEcp0RQuSamO4ByKL7kZHp8kPkxKr7kxBKf2ghERHKcSgQiIjlOiUBEJMcpEewkM2tmZi+Z2QdmtsDMLi9hm55mttrMCqLHjSXtK8YYl5rZ+9Gx80t438xsvJktNrN5ZtYphbEdlHBeCsxsjZldUWyblJ8/M/uHmX1lZvMT1jUws3+b2UfR896lfPb8aJuPzOz8FMV2q5ktjH5/T5nZXqV8tszvQswxjjaz5Qm/x96lfLaXmS2Kvo8jUhjfowmxLTWzglI+G+s5LO2aktLvX2lzWOpR6lzM+wOdotf1gf/w47mYewLPpDHGpUCjMt7vDUwHDDgMeCtNcVYHviDc6JLW8wf0ADoB8xPW/QkYEb0eAfyxhM81AJZEz3tHr/dOQWwnADWi138sKbaKfBdijnE0cHUFvgNlzm0eV3zF3v8zcGM6zmFp15RUfv9UIthJ7v65u8+NXq8FPuTHU3Bmun7Agx68CexlZvunIY5jgY/dPe13irv7bMJQ6IkS59R+ADi1hI+eCPzb3b9x91XAv4Feccfm7jM8TO8K8CZh4qe0KeX8VURF5jZPWlnxRfOk/xJ4pLKPWxFlXFNS9v1TIkiCmbUEOgJvlfD24Wb2nplNN7NDUhoYODDDzN4xs6ElvF+R+aRTYQCl//Gl8/wV2dfdP49efwHsW8I2mXAuhxBKeCUp77sQt0ui6qt/lFK1kQnn7yjgS3f/qJT3U3YOi11TUvb9UyLYRWZWD3gSuMLd1xR7ey6huqM98DdgSorDO9LdOwEnAb8xsx4pPn65zGw3oC/weAlvp/v8/YiHcnjG9bU2s5HAZmBSKZuk87twJ/AToAPwOaH6JRMNpOzSQErOYVnXlLi/f0oEu8DMahJ+YZPc/f+Kv+/ua9z9u+j1NKCmmTVKVXzuvjx6/gp4ilD8TlSR+aTjdhIw192/LP5Gus9fgi+Lqsyi569K2CZt59LMBgN9gEHRheJHKvBdiI27f+nuW9x9K3B3KcdO63fRwlzppwOPlrZNKs5hKdeUlH3/lAh2UlSfeC/wobv/bynb7Bdth5l1JZznlSmKr66Z1S96TWhUnF9ss6nAeVHvocOA1QlF0FQp9b+wdJ6/Yorm1CZ6/lcJ2zwPnGBme0dVHydE62JlZr2Aa4G+7r6ulG0q8l2IM8bEdqfTSjn2trnNo1LiAMJ5T5XjgIXuXljSm6k4h2VcU1L3/YurJTxbH8CRhCLaPKAgevQGhgHDom0uARYQekC8CRyRwvgOjI77XhTDyGh9YnwGTCD01ngfyEvxOaxLuLDvmbAureePkJQ+BzYR6lkvABoCLwAfATOBBtG2ecA9CZ8dAiyOHr9KUWyLCXXDRd/Bv0fbHgBMK+u7kMLz91D0/ZpHuKjtXzzGaLk3oafMx3HFWFJ80fr7i753Cdum9ByWcU1J2fdPQ0yIiOQ4VQ2JiOQ4JQIRkRynRCAikuOUCEREcpwSgYhIjlMiEImY2RbbcWTUShsJ08xaJo58KZJJaqQ7AJEMst7dO6Q7CJFUU4lApBzRePR/isakf9vMfhqtb2lmL0aDqr1gZs2j9ftamCPgvehxRLSr6mZ2dzTm/Awz2z3a/rJoLPp5ZjY5TT+m5DAlApHtdi9WNXRWwnur3b0dcDswLlr3N+ABdz+UMOjb+Gj9eOBlD4PmdSLckQrQGpjg7ocA3wJnROtHAB2j/QyL64cTKY3uLBaJmNl37l6vhPVLgWPcfUk0ONgX7t7QzL4mDJuwKVr/ubs3MrMVQFN335iwj5aEceNbR8vXATXd/WYzew74jjDK6hSPBtwTSRWVCEQqxkt5vTM2JrzewvY2upMJYz91AuZEI2KKpIwSgUjFnJXw/Eb0+nXCaJkAg4BXotcvAMMBzKy6me1Z2k7NrBrQzN1fAq4D9gR+VCoRiZP+8xDZbnfbcQLz59y9qAvp3mY2j/Bf/cBo3aXAfWZ2DbAC+FW0/nJgopldQPjPfzhh5MuSVAf+GSULA8a7+7eV9hOJVIDaCETKEbUR5Ln71+mORSQOqhoSEclxKhGIiOQ4lQhERHKcEoGISI5TIhARyXFKBCIiOU6JQEQkx/1/je93aKqqgD4AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "guCw2RgXECTv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "d416b96e-1415-438d-9df9-4df45e8de0a6"
      },
      "source": [
        "plt.clf()   # 그래프를 초기화합니다\n",
        "acc = history_dict['accuracy']\n",
        "val_acc = history_dict['val_accuracy']\n",
        "\n",
        "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
        "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
        "plt.title('Training and validation accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9b3/8dcHBDHsiysRgguiFtkiFtxQaIvLhUK1grSCekVcW1vrtT/ayrWX3mvlqrUuLdatFou2vVLaYl0Qa1tqJSKiWJFF1OAGKIvsJJ/fH98zYRJOkklmy/J+Ph7zmDNnm8+cTM5nvsv5HnN3REREqmqR7wBERKRhUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISkzsyfNbGKm180nM1tjZiOysF83s6Oi6Z+Z2fdTWbce7zPBzJ6ub5wiNTFdB9G0mdlnSS8LgJ1AWfT6cneflfuoGg4zWwP8u7s/m+H9OnC0u6/M1LpmVgS8DbRy9z2ZiFOkJvvlOwDJLndvl5iu6WRoZvvppCMNhb6PDYOqmJopMxtmZqVm9h9m9iHwoJl1NrM/mtk6M/s0mi5M2uZ5M/v3aHqSmf3NzGZE675tZmfVc91eZvaCmW0xs2fN7G4z+1U1cacS4w/N7O/R/p42s25Jy79uZu+Y2QYzm1rD8TnJzD40s5ZJ88aY2dJoerCZ/cPMNprZB2Z2l5m1rmZfD5nZfyW9/k60zftmdkmVdc8xs1fMbLOZvWdm05IWvxA9bzSzz8xsSOLYJm0/1MwWmdmm6Hloqsemjse5i5k9GH2GT81sTtKy0Wa2JPoMq8xsZDS/UnWemU1L/J3NrCiqarvUzN4Fnovm/yb6O2yKviPHJ21/gJn9b/T33BR9xw4wsz+Z2TVVPs9SMxsT91mlekoQzdshQBegJzCZ8H14MHrdA9gO3FXD9icBy4FuwI+B+83M6rHuo8BLQFdgGvD1Gt4zlRgvBC4GDgJaA9cDmNlxwL3R/g+L3q+QGO7+T2ArcGaV/T4aTZcB10WfZwgwHLiyhriJYhgZxfMF4GigavvHVuAioBNwDnCFmX05WnZa9NzJ3du5+z+q7LsL8Cfgzuiz3Qb8ycy6VvkM+xybGLUd50cIVZbHR/u6PYphMPBL4DvRZzgNWFPd8YhxOnAs8KXo9ZOE43QQsBhIrhKdAQwChhK+xzcA5cDDwNcSK5lZP6A74dhIXbi7Hs3kQfhHHRFNDwN2AW1qWL8/8GnS6+cJVVQAk4CVScsKAAcOqcu6hJPPHqAgafmvgF+l+JniYvxe0usrgT9H0z8AZictaxsdgxHV7Pu/gAei6faEk3fPatb9JvBE0msHjoqmHwL+K5p+APifpPV6J68bs987gNuj6aJo3f2Slk8C/hZNfx14qcr2/wAm1XZs6nKcgUMJJ+LOMev9PBFvTd+/6PW0xN856bMdUUMMnaJ1OhIS2HagX8x6bYBPCe06EBLJPbn+f2sKD5Ugmrd17r4j8cLMCszs51GRfTOhSqNTcjVLFR8mJtx9WzTZro7rHgZ8kjQP4L3qAk4xxg+TprclxXRY8r7dfSuwobr3IpQWxprZ/sBYYLG7vxPF0TuqdvkwiuNHhNJEbSrFALxT5fOdZGYLoqqdTcCUFPeb2Pc7Vea9Q/j1nFDdsamkluN8OOFv9mnMpocDq1KMN07FsTGzlmb2P1E11Wb2lkS6RY82ce8VfacfA75mZi2A8YQSj9SREkTzVrUL27eBY4CT3L0De6s0qqs2yoQPgC5mVpA07/Aa1k8nxg+S9x29Z9fqVnb3Nwgn2LOoXL0EoarqTcKv1A7A/6tPDIQSVLJHgbnA4e7eEfhZ0n5r63L4PqFKKFkPYG0KcVVV03F+j/A36xSz3XvAkdXscyuh9JhwSMw6yZ/xQmA0oRquI6GUkYhhPbCjhvd6GJhAqPrb5lWq4yQ1ShCSrD2h2L4xqs++KdtvGP0iLwGmmVlrMxsC/FuWYvwtcK6ZnRI1KN9M7f8DjwLfIJwgf1Mljs3AZ2bWB7gixRgeByaZ2XFRgqoaf3vCr/MdUX3+hUnL1hGqdo6oZt/zgN5mdqGZ7WdmFwDHAX9MMbaqccQeZ3f/gNA2cE/UmN3KzBIJ5H7gYjMbbmYtzKx7dHwAlgDjovWLgfNSiGEnoZRXQCilJWIoJ1TX3WZmh0WljSFRaY8oIZQD/4tKD/WmBCHJ7gAOIPw6exH4c47edwKhoXcDod7/McKJIU69Y3T3ZcBVhJP+B4R66tJaNvs1oeH0OXdfnzT/esLJewtwXxRzKjE8GX2G54CV0XOyK4GbzWwLoc3k8aRttwHTgb9b6D31+Sr73gCcS/j1v4HQaHtulbhTVdtx/jqwm1CK+pjQBoO7v0RoBL8d2AT8hb2lmu8TfvF/CvwnlUtkcX5JKMGtBd6I4kh2PfAasAj4BLiFyue0XwJ9CW1aUg+6UE4aHDN7DHjT3bNegpGmy8wuAia7+yn5jqWxUglC8s7MTjSzI6MqiZGEeuc5tW0nUp2o+u5KYGa+Y2nMlCCkITiE0AXzM0If/ivc/ZW8RiSNlpl9idBe8xG1V2NJDVTFJCIisVSCEBGRWE1msL5u3bp5UVFRvsMQEWlUXn755fXufmDcsiaTIIqKiigpKcl3GCIijYqZVb36voKqmEREJJYShIiIxFKCEBGRWE2mDSLO7t27KS0tZceOHbWvLHnRpk0bCgsLadWqVb5DEZEqmnSCKC0tpX379hQVFVH9fWwkX9ydDRs2UFpaSq9evfIdjohUkbUqJjN7wMw+NrPXq1luZnanma2Mbgc4MGnZRDNbET0m1jeGHTt20LVrVyWHBsrM6Nq1q0p4IvU0axYUFUGLFuF51qzatqibbLZBPASMrGH5WYRbCR5NuN3lvVBx28SbCLeoHAzcZGad6xuEkkPDpr+P5FO6J9h8bj9rFkyeDO+8A+7hefLkzCaJrCUId3+BMARvdUYDv/TgRcLdqg4l3Iv2GXdP3LHqGWpONCIidZbuCTbf20+dCtu2VZ63bVuYnyn57MXUncq3XiyN5lU3fx9mNtnMSsysZN26dVkLtL42bNhA//796d+/P4cccgjdu3eveL1r164aty0pKeHaa6+t9T2GDh2aqXBFci7bVSQ1SfcEm+/t3323bvPro1F3c3X3me5e7O7FBx4Ye6V4nWT6y9q1a1eWLFnCkiVLmDJlCtddd13F69atW7Nnz55qty0uLubOO++s9T0WLlyYXpAieZKJKpJ0/mfTPcHme/seVW9WW8v8+shnglhL5XvzFkbzqpufVbmozwOYNGkSU6ZM4aSTTuKGG27gpZdeYsiQIQwYMIChQ4eyfPlyAJ5//nnOPfdcAKZNm8Yll1zCsGHDOOKIIyoljnbt2lWsP2zYMM477zz69OnDhAkTSIzUO2/ePPr06cOgQYO49tprK/abbM2aNZx66qkMHDiQgQMHVko8t9xyC3379qVfv37ceOONAKxcuZIRI0bQr18/Bg4cyKpV6dynXhqrdE7Q6f6CTvd/Nt0TbL63nz4dCgoqzysoCPMzxt2z9iDcZPz1apadQ7ivrQGfB16K5ncB3gY6R4+3gS61vdegQYO8qjfeeGOfedXp2dM9fM0qP3r2THkXNbrpppv81ltv9YkTJ/o555zje/bscXf3TZs2+e7du93d/ZlnnvGxY8e6u/uCBQv8nHPOqdh2yJAhvmPHDl+3bp136dLFd+3a5e7ubdu2rVi/Q4cO/t5773lZWZl//vOf97/+9a++fft2Lyws9NWrV7u7+7hx4yr2m2zr1q2+fft2d3d/6623PHE8582b50OGDPGtW7e6u/uGDRvc3X3w4MH+f//3f+7uvn379orl9VGXv5M0HL/6lXtBQeX/l4KCMD8VZvH/c2apbZ/u/2y68ed7+8Q+evYMx6xnz7ptmwCUeDXn1axdB2FmvwaGAd3MrJTQM6lVlJR+RrjB+tmE+/JuI9zHFnf/xMx+SLjPLMDN7l5TY3dG5KI+L+H888+nZcuWAGzatImJEyeyYsUKzIzdu3fHbnPOOeew//77s//++3PQQQfx0UcfUVhYWGmdwYMHV8zr378/a9asoV27dhxxxBEV1xmMHz+emTP3vcnW7t27ufrqq1myZAktW7bkrbfeAuDZZ5/l4osvpiD6qdKlSxe2bNnC2rVrGTNmDBAudpPmp6YSwIQJtW/fo0f41R83PxXp/s8mYpw6NWzTo0f49Z1K7A1h+8Q+6rJ+XWUtQbj7+FqWO+EG8nHLHgAeyEZc1Un3y1oXbdu2rZj+/ve/zxlnnMETTzzBmjVrGDZsWOw2+++/f8V0y5YtY9svUlmnOrfffjsHH3wwr776KuXl5TrpNxOzZtX/BJXuCXr69FAllJxk6lJFkon/2XRPsPnePtsadSN1JuWkPi/Gpk2b6N49dNJ66KGHMr7/Y445htWrV7NmzRoAHnvssWrjOPTQQ2nRogWPPPIIZWVlAHzhC1/gwQcfZFv0X/zJJ5/Qvn17CgsLmTMn3DZ6586dFcul8ch3Hf6ECTBzJvTsCWbheebM1E+Y+fqfbU6UICLpflnr64YbbuC73/0uAwYMqNMv/lQdcMAB3HPPPYwcOZJBgwbRvn17OnbsuM96V155JQ8//DD9+vXjzTffrCjljBw5klGjRlFcXEz//v2ZMWMGAI888gh33nknJ5xwAkOHDuXDDz/MeOxSu3w2EmfiBD1hAqxZA+Xl4bmu1Sv5+J9tVqprnGhsj3QbqZuyLVu2uLt7eXm5X3HFFX7bbbflOaLK9Heqn3w3EidiSLeRVPKLGhqpVYJoBu677z769+/P8ccfz6ZNm7j88svzHZJE8lkCyEQ/+nRKANLwNenRXCW47rrruO666/IdhlSRaANInOQTbQCQ2ok2343E0vSpBCGSJ/kuAagOX2qjBCGShnwO9ZDvRmJp+pQgROqpsXcTFamNEoRIPTX2bqIitVGCyKIzzjiDp556qtK8O+64gyuuuKLabYYNG0ZJSQkAZ599Nhs3btxnnWnTplVcj1CdOXPm8MYbb1S8/sEPfsCzzz5bl/ClFpkY6kElAGnIlCCyaPz48cyePbvSvNmzZzN+fI2jkFSYN28enTp1qtd7V00QN998MyNGjKjXviSeuolKU6cEkUXnnXcef/rTnypuDrRmzRref/99Tj31VK644gqKi4s5/vjjuemmm2K3LyoqYv369QBMnz6d3r17c8opp1QMCQ7hGocTTzyRfv368ZWvfIVt27axcOFC5s6dy3e+8x369+/PqlWrmDRpEr/97W8BmD9/PgMGDKBv375ccskl7Ny5s+L9brrpJgYOHEjfvn15880394mpqQ0Lnk4js4Z6kKau2VwH8c1vwpIlmd1n//5wxx3VL+/SpQuDBw/mySefZPTo0cyePZuvfvWrmBnTp0+nS5culJWVMXz4cJYuXcoJJ5wQu5+XX36Z2bNns2TJEvbs2cPAgQMZNGgQAGPHjuWyyy4D4Hvf+x73338/11xzDaNGjeLcc8/lvPPOq7SvHTt2MGnSJObPn0/v3r256KKLuPfee/nmN78JQLdu3Vi8eDH33HMPM2bM4Be/+EWl7Q866CCeeeYZ2rRpw4oVKxg/fjwlJSU8+eST/P73v+ef//wnBQUFfPJJGIB3woQJ3HjjjYwZM4YdO3ZQXl5er2OdDeleh5CJ0ThFGjKVILIsuZopuXrp8ccfZ+DAgQwYMIBly5ZVqg6q6q9//StjxoyhoKCADh06MGrUqIplr7/+Oqeeeip9+/Zl1qxZLFu2rMZ4li9fTq9evejduzcAEydO5IUXXqhYPnbsWAAGDRpUMcBfst27d3PZZZfRt29fzj///Iq4Ux0WvKDqT+48ysQ9fVVFJE1ZsylB1PRLP5tGjx7Nddddx+LFi9m2bRuDBg3i7bffZsaMGSxatIjOnTszadIkduzYUa/9T5o0iTlz5tCvXz8eeughnn/++bTiTQwZXt1w4U1pWPBc3gNEpDFSCSLL2rVrxxlnnMEll1xSUXrYvHkzbdu2pWPHjnz00Uc8+eSTNe7jtNNOY86cOWzfvp0tW7bwhz/8oWLZli1bOPTQQ9m9ezezkirQ27dvz5YtW/bZ1zHHHMOaNWtYuXIlEEZlPf3001P+PE1pWPBc3NNXpDFTgsiB8ePH8+qrr1YkiH79+jFgwAD69OnDhRdeyMknn1zj9gMHDuSCCy6gX79+nHXWWZx44okVy374wx9y0kkncfLJJ9OnT5+K+ePGjePWW29lwIABlRqG27Rpw4MPPsj5559P3759adGiBVOmTEn5szS0YcHVyCySPRZGe238iouLPXH9QMK//vUvjj322DxFJKmq79+paiMzhBN8Xa4lSOeOaiJNgZm97O7FcctUgpBGS43MItmlBCGNlhqZRbKrySeIplKF1lSl8/dRI7NIdjXpBNGmTRs2bNigJNFAuTsbNmyod1dZNTKLZFeTvg6isLCQ0tJS1q1bl+9QpBobNrThyisLef31ujcS60pmkexq0r2YpGHLRC8kEUmPejFJg5SJXkgikj1KEJI36oUk0rApQUjeqBeSSMOmBCF5o15IIg2bEoTkjW65KdKwNelurtLwTZighCDSUKkEISIisZQgJC3pDLctIg2bqpik3tK9p7OINGwqQUi96UI3kaZNCULqTRe6iTRtShBSb7rQTaRpU4KQetOFbiJNmxKE1JsudBNp2pQgmrl0u6nqns4iTVdWE4SZjTSz5Wa20sxujFne08zmm9lSM3vezAqTlpWZ2ZLoMTebcTZXiW6q77wD7nu7qepaBhGBLN4wyMxaAm8BXwBKgUXAeHd/I2md3wB/dPeHzexM4GJ3/3q07DN3b5fq++mGQXVXVBSSQlU9e4bSgIg0ffm6YdBgYKW7r3b3XcBsYHSVdY4DnoumF8QslyxSN1URqUk2E0R34L2k16XRvGSvAmOj6TFAezPrGr1uY2YlZvaimX057g3MbHK0TonuO1136qYqIjXJdyP19cDpZvYKcDqwFiiLlvWMij0XAneY2ZFVN3b3me5e7O7FBx54YM6CbirUTVVEapLNBLEWODzpdWE0r4K7v+/uY919ADA1mrcxel4bPa8GngcGZDHWZkndVEWkJtlMEIuAo82sl5m1BsYBlXojmVk3M0vE8F3ggWh+ZzPbP7EOcDLwBpJx6qYqItXJWoJw9z3A1cBTwL+Ax919mZndbGajotWGAcvN7C3gYCBRuXEsUGJmrxIar/8nufeTiIhkX9a6ueaaurmKiNRdvrq5iohII6YE0cjpjm4iki26o1wjpju6iUg2qQTRiOmObiKSTUoQjZiGyhCRbFKCaMQ0VIaIZJMSRCOmoTJEJJuUIBoxDZUhItmkXkyN3IQJSggikh0qQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCSLPNFy3iDRUulAujzRct4g0ZCpB5JGG6xaRhkwJIo80XLeINGRKEHmk4bpFpCFTgsgjDdctIg2ZEkQeabhuEWnI1IspzzRct4g0VCpBiIhIrFoThJn9m5kpkYiINDOpnPgvAFaY2Y/NrE+2AxIRkYah1gTh7l8DBgCrgIfM7B9mNtnM2mc9OhERyZuUqo7cfTPwW2A2cCgwBlhsZtdkMTYREcmjVNogRpnZE8DzQCtgsLufBfQDvp3d8EREJF9S6eb6FeB2d38heaa7bzOzS7MTloiI5FsqCWIa8EHihZkdABzs7mvcfX62AhMRkfxKpQ3iN0B50uuyaJ6IiDRhqSSI/dx9V+JFNN06eyGJiEhDkEqCWGdmoxIvzGw0sD57IYmISEOQShvEFGCWmd0FGPAecFFWoxIRkbyrNUG4+yrg82bWLnr9WdajEhGRvEtpNFczOwc4HmhjZgC4+81ZjEtERPIslQvlfkYYj+kaQhXT+UDPLMclIiJ5lkoj9VB3vwj41N3/ExgC9E5l52Y20syWm9lKM7sxZnlPM5tvZkvN7HkzK0xaNtHMVkSPial+IBERyYxUEsSO6HmbmR0G7CaMx1QjM2sJ3A2cBRwHjDez46qsNgP4pbufANwM/He0bRfgJuAkYDBwk5l1TiFWERHJkFQSxB/MrBNwK7AYWAM8msJ2g4GV7r46unZiNjC6yjrHAc9F0wuSln8JeMbdP3H3T4FngJEpvGfOzZoFRUXQokV4njUr3xGJiGRGjQkiulHQfHff6O6/I7Q99HH3H6Sw7+6ELrEJpdG8ZK8CY6PpMUB7M+ua4rZEw46XmFnJunXrUggps2bNgsmT4Z13wD08T56sJCEiTUONCcLdywnVRInXO919Uwbf/3rgdDN7BTgdWEsYyiMl7j7T3YvdvfjAAw/MYFipmToVtm2rPG/btjBfRKSxS6WKab6ZfcUS/VtTtxY4POl1YTSvgru/7+5j3X0AMDWatzGVbRuCd9+t23wRkcYklQRxOWFwvp1mttnMtpjZ5hS2WwQcbWa9zKw1MA6Ym7yCmXVLut/1d4EHoumngC+aWeeocfqL0bwGpUePus0XEWlMUrnlaHt3b+Hurd29Q/S6Qwrb7QGuJpzY/wU87u7LzOzmpLGdhgHLzewt4GBgerTtJ8APCUlmEXBzNK9BmT4dCgoqzysoCPNFRBo7c/eaVzA7LW5+1RsI5VtxcbGXlJTk/H1nzQptDu++G0oO06fDhAk5D0NEpF7M7GV3L45blspQG99Jmm5D6L76MnBmBmJr9CZMUEIQkaYplcH6/i35tZkdDtyRtYhERKRBSKWRuqpS4NhMByIiIg1LrSUIM/spkGioaAH0J1xRLc3crl3wwgvw9NNw8MEwYgT07RuuKheRxi+VNojklt89wK/d/e9ZiqfZcYetW6FtW6jzlSZ58Omn8OSTMHdueN68GVq1gt27w/Ju3eDMM0OyGD4cjjgiv/GKSP2lkiB+C+xw9zIIg/CZWYG7b6tlu2avrAw++gjWroXS0uqft2+HQYPghhtg7FjYL6W7dOTO6tUhIcydG0oMZWWhxPDVr8KoUSERfPopzJ8fHs8+C48/HrYtKtqbLM48Ew46KK8fRUTqIJVuri8CIxJ3kovuLPe0uw/NQXwpy1c3V4Dnn4eSkson/dJS+OCDcDJN1qoVdO8eHoWF4bljR/jVr2DFivCL+9vfhkmT9r3GIlfKy2HRIvj970NSWLYszP/c50JCGDUKTjyx+qokd1i+fG+yWLAANkUDtJxwQkgWI0bAaadBu3a5+UwiEq+mbq6pJIgl7t6/tnn5lq8EsWBB+GUMoZqosHDvIzkJJOZ16xZ/Yi0rCyfjW26Bf/4zrHfNNXDVVdC1a/Y/x7Zt4YQ+dy784Q+h5NOyZTiJJ5JCfauLyspg8eK9CeNvf4OdO0NJ6aSTQsK44AI4rupg8CKSdekmiL8D17j74uj1IOAudx+S8UjTkI8EsXs3DBgQTq6LFkGXLum3I7iHE+iPfwx//GMoRVxyCXzrW9CrV2bihnDSXro0JLgFC8LJe/t26NABzjorJISzzoLOWbgLx44dsHBhSBbz54fSV3l5KFVcey2cc44aukVypaYEgbvX+ABOBFYBfwX+BqwEBtW2Xa4fgwYN8ly7/XZ3cJ8zJzv7X7bM/eKL3Vu1cm/Rwn3cOPeXX67fvsrK3F95xf2229xHjXLv1CnEDu5HHeV+zTXuzzzjvnNnZj9DKj7+2P1HP3Lv3j3Ec+SR4dhu3Jj7WESaG6DEqzmv1lqCiDJMK+CY6OVyd9+dft7KrFyXID78EI45BoYOhXnzstsDae1a+MlP4Gc/gy1bwi/tG24Iz9W9b3k5vPZaaB9ZsCA0Ln/6aVh21FEwbNjeR/d97rSRH7t3w5w5cOedoRTVtm1oi7nmmnCss2XDhlByatUqe+8h0lClW4K4CuiU9LozcGVt2+X6kesSxMSJ4Zf98uW5e8+NG91vucX90EPDL+3+/d0ffdR99+5QQnj1Vfef/MT9y19279JlbwnhyCPdL73U/ZFH3N99N3fxpqOkJBzj1q3DZxg50n3evPA507Fnj/uSJe733OP+ta+5H3FE2P8hh7h///vupaUZCV+k0SCdEkQ1jdSveLiHQ4ORyxLEwoVw8slw443w3/+dk7esZOfOMEjgrbfCm2+GEsCOHeGXMIS2ijPOCKWD009v3MOPf/wxzJwJ99wTeoUdfXQoUUyaBO3b1779pk2h0X/hQvj738P0li1h2SGHhL9jcXEoscybF9o+vvzl0Dlg2LDGcW2KSDrSbaR+DTghyjSYWUtgqbsfn/FI05CrBFFWBoMHh14+b76Z326a5eXwpz+FE2i3biEpnH469OyZv5iyZdcu+N3vQvXTiy+G5HDJJXD11aHKDEJ5afXqkAgWLgyP118P81u0CF1shw7d+ygqqpwAVq8O1Xj33w+ffBJ6VV11FXz966klo7r65JO9nQTMQvI76qjwXFSkKi/JjXQTxK2Ee1H/PJp1OfCuu1+f0SjTlKsE8fOfw5QpMHt26JopuffSS/DTn8Jjj8GePaG3VevWISF8/HFYp0MHGDIkJIKTTw5JPdWT/Pbt4e97993w8sthu4suCsni2DRGIdu2LZRUEt19X3klJK927UKCSJRsIHQx7tVrb8JIfvTs2fAuppTGK90E0QKYDAyPZi0FDnH3qzIaZZpykSA2bIDevcMv0eeeU/VDvn3wQUjYv/gFHHDA3mQwdGj49Z9uV1n3kIzuvjsko127Qintqqtg9OjaT9J79oTuz4krzBcuDPto1Sokr8QFgyeeGPb18cewcmW4YDLxSLz+7LO9+91vv5A8EgnjqKPC/gYO1HdS6i6tBBHtYABwIfBVYDXwO3e/K6NRpikXCeKKK+C++2DJknBVsTQf69aFqqd77w03hyoshMsvh8suC8OOQEgob7yx9/qOv/wljFUF4XqZ4cPD49RTQw+tVLmHKs3khJGcQLZuDev16BHaT8aMgVNOUSlDUlOvBGFmvYHx0WM98Bhwvbs3yBrubCeIxYtDY+a118IduhtGs1VWFtp97r47jGLbqhV85SuhtPLcc6H7M8CRR+4dg+qMM0IbUTa4w/vvhzW8XPMAAA+FSURBVFieeCI879wZ3m/UqJAsRoyANm2y8/7S+NU3QZQTLo671N1XRvNWu3uDHJ8zmwmivDz8Ilu1Kowx1KlTVt5GGpm33goligcfhP3331tlNHx4/joKfPZZGGX3iSdCItu8ObRxnHVWSBZnnx3G/hJJqG+C+DIwDjgZ+DMwG/iFu2dwwIfMyWaCePjh0K3ywQfDs0iy8vJQ99/Q6v937QqlmieeCAMvfvRRKPEMHx6SxejRe6vHpPlKt5G6LTCaUNV0JvBL4Al3fzrTgaYjWwli06ZwFW+vXqH7pMYIksaorCx0D37iifBYvToktKFDQ7K44ILQrpJL7vDUU6Hr8q5de9toBg0KvbgkN9JupE7aUWfgfOACdx9e2/q5lK0E8a1vhTaHRYvCF1eksXMPAzUmksXSpeGEPHp06KF1xhnZLQ3t2gW//jXMmBGuUznssDBi8WuvheWdOoWLFBPVdccc0/BKZ01JxhJEQ5aNBLFsGfTrB//+7+ECKpGmaOXK0Dvv/vtDV+5jj4UrrwzXfnTokLn32bgxXNT5k5+EhvW+feH662HcuHAdy8cfhyqxRC+wNWvCdocdtjdZDB/ecMYOayqUIOrBPXwZlywJ3QlzcU8GkXzavj3cCfCuu8IQ7O3ahSRx5ZVwfBrjJrzzTkgK990XGtFHjAiJ4YtfrLlksHr13mTx3HOwfn2Y36fP3mQxbFhqQ9KXl4fPt3VruGBx69bK0wUFoZvw4YeHZNWcKEHUw+OPh3rZe+4J1z+INCfJFwju3BlOxIkLBFMdAuSVV0I10mOPhdfjxoXE0L8etxorLw9VYYmr0F94IZzcW7QIVb89e1Z/8t+6NSSHVJiFMbp69gyPHj32na5rLzD30Jts/frqHwUF4dqqz30uJONMltxqowRRR1u3hl8p3bqFX1JqMJPmav36vRcIvvNOqO65/HKYPDmcSKtKNDzPmBFO5u3bh3WvvTazg0bu2hUGXnz22VC62LAhnGTbtg2P5OnaXhcUhJLNu++Gz5h4Tkzv2lX5vTt02DdptG5d/cl/w4ZwVX2cVq1C7cSWLXsveISwz899LlTDJZ6POSZ0p840JYg6mjoVfvSjMG7OySdnZJcijVpZWRjt9q67wsV4++0H550XShUnnxzu5ZHc8Ny9O3zjGyE5NObrLsrLQ9tI1aSR/Jy4z0rLluFk361b9Y+qy9u3D6WW8vKwv9deC8cv8fzmm+HYJvbfu3flpPG5z4Uelun8iFWCqIMVK8JBHzcuXP8gIpUlXyC4aVM4UW3YEN/w3Bxs2RJKCB07Zr4b/O7d4XgnJ43XXw/tM4lT9wEHhAshf/e7+r2HEkSK3MP9kP/2t/BHiStCi0iwdSs8+mgYLLFTp9AlvLaGZ8mMrVvDuF+JxNGhA0ybVr99KUGk6A9/COPX3HYbXHddhgITEWnAakoQui44smNHqDM97rhwExoRkeZOAwJHbr0V3n479IrQnbxERFSCAEJPhB/9CM4/P1x8IyIiShAAfPvboffBjBn5jkREpOFo9gnirbfCgGVTp2b2Qh4Rkcau2bdB9O4dhgTo3TvfkYiINCzNPkEAnHBCviMQEWl4mn0Vk4iIxFOCEBGRWFlNEGY20syWm9lKM7sxZnkPM1tgZq+Y2VIzOzuaX2Rm281sSfTQ7XpERHIsa20QZtYSuBv4AlAKLDKzue7+RtJq3wMed/d7zew4YB5QFC1b5e71GDleREQyIZsliMHASndf7e67gNnA6CrrOJC4NUZH4P0sxiMiInWQzQTRHXgv6XVpNC/ZNOBrZlZKKD1ck7SsV1T19BczOzXuDcxsspmVmFnJunXrMhi6iIjku5F6PPCQuxcCZwOPmFkL4AOgh7sPAL4FPGpm+9yEz91nunuxuxcfeOCBOQ1cRKSpy2aCWAscnvS6MJqX7FLgcQB3/wfQBujm7jvdfUM0/2VgFaBL2UREciibCWIRcLSZ9TKz1sA4YG6Vdd4FhgOY2bGEBLHOzA6MGrkxsyOAo4HVWYxVRESqyFovJnffY2ZXA08BLYEH3H2Zmd0MlLj7XODbwH1mdh2hwXqSu7uZnQbcbGa7gXJgirt/kq1YRURkX7qjnIhIM6Y7yomISJ0pQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEymqCMLORZrbczFaa2Y0xy3uY2QIze8XMlprZ2UnLvhttt9zMvpTNOEVEZF/7ZWvHZtYSuBv4AlAKLDKzue7+RtJq3wMed/d7zew4YB5QFE2PA44HDgOeNbPe7l6WrXhFRKSybJYgBgMr3X21u+8CZgOjq6zjQIdouiPwfjQ9Gpjt7jvd/W1gZbQ/ERHJkWwmiO7Ae0mvS6N5yaYBXzOzUkLp4Zo6bCsiIlmU70bq8cBD7l4InA08YmYpx2Rmk82sxMxK1q1bl7UgRUSao2wmiLXA4UmvC6N5yS4FHgdw938AbYBuKW6Lu89092J3Lz7wwAMzGLqIiGQzQSwCjjazXmbWmtDoPLfKOu8CwwHM7FhCglgXrTfOzPY3s17A0cBLWYxVRESqyFqCcPc9wNXAU8C/CL2VlpnZzWY2Klrt28BlZvYq8GtgkgfLCCWLN4A/A1dlqwfTrFlQVAQtWoTnWbOy8S4iIo2PuXu+Y8iI4uJiLykpqdM2s2bB5MmwbdveeQUFMHMmTJiQ4QBFRBogM3vZ3YvjluW7kTqvpk6tnBwgvJ46NT/xiIg0JM06Qbz7bt3mi4g0J806QfToUbf5IiLNSbNOENOnhzaHZAUFYb6ISHPXrBPEhAmhQbpnTzALz2qgFhEJsjZYX2MxYYISgohInGZdghARkeopQYiISCwlCBERiaUEISIisZQgREQkVpMZi8nM1gHv5DuOGnQD1uc7iBoovvQovvQovvSkE19Pd4+9X0KTSRANnZmVVDcgVkOg+NKj+NKj+NKTrfhUxSQiIrGUIEREJJYSRO7MzHcAtVB86VF86VF86clKfGqDEBGRWCpBiIhILCUIERGJpQSRIWZ2uJktMLM3zGyZmX0jZp1hZrbJzJZEjx/kIc41ZvZa9P773MTbgjvNbKWZLTWzgTmM7ZikY7PEzDab2TerrJPTY2hmD5jZx2b2etK8Lmb2jJmtiJ47V7PtxGidFWY2MYfx3Wpmb0Z/vyfMrFM129b4XchifNPMbG3S3/DsarYdaWbLo+/ijTmM77Gk2NaY2ZJqts3F8Ys9r+TsO+juemTgARwKDIym2wNvAcdVWWcY8Mc8x7kG6FbD8rOBJwEDPg/8M09xtgQ+JFzEk7djCJwGDAReT5r3Y+DGaPpG4JaY7boAq6PnztF05xzF90Vgv2j6lrj4UvkuZDG+acD1Kfz9VwFHAK2BV6v+P2UrvirL/xf4QR6PX+x5JVffQZUgMsTdP3D3xdH0FuBfQPf8RlUvo4FfevAi0MnMDs1DHMOBVe6e16vj3f0F4JMqs0cDD0fTDwNfjtn0S8Az7v6Ju38KPAOMzEV87v60u++JXr4IFGb6fVNVzfFLxWBgpbuvdvddwGzCcc+omuIzMwO+Cvw60++bqhrOKzn5DipBZIGZFQEDgH/GLB5iZq+a2ZNmdnxOAwsceNrMXjazyTHLuwPvJb0uJT+JbhzV/2Pm+xge7O4fRNMfAgfHrNNQjuMlhBJhnNq+C9l0dVQF9kA11SMN4fidCnzk7iuqWZ7T41flvJKT76ASRIaZWTvgd8A33X1zlcWLCVUm/YCfAnNyHR9wirsPBM4CrjKz0/IQQ43MrDUwCvhNzOKGcAwreCjLN8i+4mY2FdgDzKpmlXx9F+4FjgT6Ax8QqnEaovHUXHrI2fGr6bySze+gEkQGmVkrwh9xlrv/X9Xl7r7Z3T+LpucBrcysWy5jdPe10fPHwBOEonyytcDhSa8Lo3m5dBaw2N0/qrqgIRxD4KNEtVv0/HHMOnk9jmY2CTgXmBCdQPaRwnchK9z9I3cvc/dy4L5q3jffx28/YCzwWHXr5Or4VXNeycl3UAkiQ6L6yvuBf7n7bdWsc0i0HmY2mHD8N+QwxrZm1j4xTWjMfL3KanOBi6LeTJ8HNiUVZXOl2l9u+T6GkblAokfIROD3Mes8BXzRzDpHVShfjOZlnZmNBG4ARrn7tmrWSeW7kK34ktu0xlTzvouAo82sV1SiHEc47rkyAnjT3UvjFubq+NVwXsnNdzCbLfDN6QGcQijmLQWWRI+zgSnAlGidq4FlhB4ZLwJDcxzjEdF7vxrFMTWanxyjAXcTepC8BhTnOMa2hBN+x6R5eTuGhET1AbCbUId7KdAVmA+sAJ4FukTrFgO/SNr2EmBl9Lg4h/GtJNQ9J76HP4vWPQyYV9N3IUfxPRJ9t5YSTnSHVo0ven02odfOqlzGF81/KPGdS1o3H8evuvNKTr6DGmpDRERiqYpJRERiKUGIiEgsJQgREYmlBCEiIrGUIEREJJYShEgtzKzMKo8ym7GRRc2sKHkkUZGGZL98ByDSCGx39/75DkIk11SCEKmn6H4AP47uCfCSmR0VzS8ys+eiwejmm1mPaP7BFu7P8Gr0GBrtqqWZ3ReN9/+0mR0QrX9tdB+ApWY2O08fU5oxJQiR2h1QpYrpgqRlm9y9L3AXcEc076fAw+5+AmGgvDuj+XcCf/Ew0OBAwhW4AEcDd7v78cBG4CvR/BuBAdF+pmTrw4lUR1dSi9TCzD5z93Yx89cAZ7r76mhAtQ/dvauZrScMH7E7mv+Bu3czs3VAobvvTNpHEWHM/qOj1/8BtHL3/zKzPwOfEUasnePRIIUiuaIShEh6vJrputiZNF3G3rbBcwjjYg0EFkUjjIrkjBKESHouSHr+RzS9kDD6KMAE4K/R9HzgCgAza2lmHavbqZm1AA539wXAfwAdgX1KMSLZpF8kIrU7wCrfuP7P7p7o6trZzJYSSgHjo3nXAA+a2XeAdcDF0fxvADPN7FJCSeEKwkiicVoCv4qSiAF3uvvGjH0ikRSoDUKknqI2iGJ3X5/vWESyQVVMIiISSyUIERGJpRKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISKz/D6d/jf+NswruAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bm-4SIXQECTz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "outputId": "78eef6d0-84ed-4184-e689-8e70aad28e3a"
      },
      "source": [
        "model = models.Sequential()\n",
        "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
        "model.add(layers.Dense(16, activation='relu'))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "model.compile(optimizer='rmsprop',\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
        "results = model.evaluate(x_test, y_test)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/4\n",
            "25000/25000 [==============================] - 1s 55us/step - loss: 0.4865 - accuracy: 0.8046\n",
            "Epoch 2/4\n",
            "25000/25000 [==============================] - 1s 51us/step - loss: 0.2776 - accuracy: 0.9085\n",
            "Epoch 3/4\n",
            "25000/25000 [==============================] - 1s 51us/step - loss: 0.2091 - accuracy: 0.9270\n",
            "Epoch 4/4\n",
            "25000/25000 [==============================] - 1s 51us/step - loss: 0.1744 - accuracy: 0.9397\n",
            "25000/25000 [==============================] - 2s 97us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LKDT8j4LECT2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "11bd99ce-d2c2-4b67-ca6a-c5bce3ce315d"
      },
      "source": [
        "results"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.28782773162841796, 0.8868399858474731]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RFs3q6zsECT5",
        "colab_type": "text"
      },
      "source": [
        "- 단순한 방식으로도 87%의 정확도를 달성"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BJ7yWJ16ECT7",
        "colab_type": "text"
      },
      "source": [
        "## 훈련된 모델로 새로운 데이터에 대해 예측하기"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V2ueU15lECT9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 140
        },
        "outputId": "cbf891d0-49e9-416e-86b6-d794aec2d355"
      },
      "source": [
        "model.predict(x_test)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.21431123],\n",
              "       [0.99819654],\n",
              "       [0.83573806],\n",
              "       ...,\n",
              "       [0.11186426],\n",
              "       [0.08854643],\n",
              "       [0.5961568 ]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yAc3atWNECUB",
        "colab_type": "text"
      },
      "source": [
        "## 추가 실험\n",
        "\n",
        "* 여기에서는 두 개의 은닉층을 사용했습니다. 한 개 또는 세 개의 은닉층을 사용하고 검증과 테스트 정확도에 어떤 영향을 미치는지 확인해 보세요.\n",
        "* 층의 은닉 유닛을 추가하거나 줄여 보세요: 32개 유닛, 64개 유닛 등\n",
        "* `binary_crossentropy` 대신에 `mse` 손실 함수를 사용해 보세요.\n",
        "* `relu` 대신에 `tanh` 활성화 함수(초창기 신경망에서 인기 있었던 함수입니다)를 사용해 보세요.\n",
        "\n",
        "다음 실험을 진행하면 여기에서 선택한 구조가 향상의 여지는 있지만 어느 정도 납득할 만한 수준이라는 것을 알게 것입니다!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z90HlzE_ECUC",
        "colab_type": "text"
      },
      "source": [
        "## 정리\n",
        "\n",
        "다음은 이 예제에서 배운 것들입니다:\n",
        "\n",
        "* 원본 데이터를 신경망에 텐서로 주입하기 위해서는 꽤 많은 전처리가 필요합니다. 단어 시퀀스는 이진 벡터로 인코딩될 수 있고 다른 인코딩 방식도 있습니다.\n",
        "* `relu` 활성화 함수와 함께 `Dense` 층을 쌓은 네트워크는 (감성 분류를 포함하여) 여러 종류의 문제에 적용할 수 있어서 앞으로 자주 사용하게 될 것입니다.\n",
        "* (출력 클래스가 두 개인) 이진 분류 문제에서 네트워크는 하나의 유닛과 `sigmoid` 활성화 함수를 가진 `Dense` 층으로 끝나야 합니다. 이 신경망의 출력은 확률을 나타내는 0과 1 사이의 스칼라 값입니다.\n",
        "* 이진 분류 문제에서 이런 스칼라 시그모이드 출력에 대해 사용할 손실 함수는 `binary_crossentropy`입니다.\n",
        "* `rmsprop` 옵티마이저는 문제에 상관없이 일반적으로 충분히 좋은 선택입니다. 걱정할 거리가 하나 줄은 셈입니다.\n",
        "* 훈련 데이터에 대해 성능이 향상됨에 따라 신경망은 과대적합되기 시작하고 이전에 본적 없는 데이터에서는 결과가 점점 나빠지게 됩니다. 항상 훈련 세트 이외의 데이터에서 성능을 모니터링해야 합니다."
      ]
    }
  ]
}