update week 5

Author: mhjensen

doc/pub/week5/html/week5-bs.html

@@ -296,10 +296,76 @@ input to a hidden layer, such that each neuron in the final pooling
 layer is connected to every single neuron in the hidden layer. This
 then serves as input to the output layer, e.g. a softmax output for
 classification.
-  
+
+!split
+===== Initialize TensorFlow =====
+
+!bc pycod
+from tensorflow.keras import datasets, layers, models
+from tensorflow.keras.layers import Input
+from tensorflow.keras.models import Sequential      #This allows appending layers to existing models
+from tensorflow.keras.layers import Dense           #This allows defining the characteristics of a particular layer
+from tensorflow.keras import optimizers             #This allows using whichever optimiser we want (sgd,adam,RMSprop)
+from tensorflow.keras import regularizers           #This allows using whichever regularizer we want (l1,l2,l1_l2)
+from tensorflow.keras.utils import to_categorical   #This allows using categorical cross entropy as the cost function
+!ec
+
+
+!split
+===== Example of how we can  up  a model (without a specific image)  =====
+
+The 6 lines of code below define the convolutional base using a common pattern: a stack of Conv2D and MaxPooling2D layers.
+
+As input, a CNN takes tensors of shape (image_height, image_width,
+color_channels), ignoring the batch size. If you are new to these
+dimensions, color_channels refers to (R,G,B). In this example, you
+will configure the CNN to process inputs of shape (32, 32, 3) as an
+example. You can do this by passing the argument input_shape to our
+first layer.
+
+!bc pycod
+model = models.Sequential()
+model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
+model.add(layers.MaxPooling2D((2, 2)))
+model.add(layers.Conv2D(64, (3, 3), activation='relu'))
+model.add(layers.MaxPooling2D((2, 2)))
+model.add(layers.Conv2D(64, (3, 3), activation='relu'))
+
+# Here we  display the architecture of our model so far.
+
+model.summary()
+!ec
+You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer.
+
+
+
+
+!split
+===== Add Dense layers on top =====
+
+To complete this model, you will feed the last output tensor from the
+convolutional base (of shape (4, 4, 64)) into one or more Dense layers
+to perform classification. Dense layers take vectors as input (which
+are 1D), while the current output is a 3D tensor. First, you will
+flatten (or unroll) the 3D output to 1D, then add one or more dense
+layers on top. The MNIST data discussed below  has 10 output classes, so we would use a final dense
+layer with 10 outputs and a softmax activation.
+
+!bc pycod
+model.add(layers.Flatten())
+model.add(layers.Dense(64, activation='relu'))
+model.add(layers.Dense(10))
+Here's the complete architecture of our model.
+model.summary()
+!ec
+As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers.
+
+
+
 
 !split
 ===== Prerequisites: Collect and pre-process data =====
+Now we switch to the MNIST data set.
 !bc pycod
 # import necessary packages
 import numpy as np
@@ -354,10 +420,6 @@ from tensorflow.keras.layers import Dense           #This allows defining the ch
 from tensorflow.keras import optimizers             #This allows using whichever optimiser we want (sgd,adam,RMSprop)
 from tensorflow.keras import regularizers           #This allows using whichever regularizer we want (l1,l2,l1_l2)
 from tensorflow.keras.utils import to_categorical   #This allows using categorical cross entropy as the cost function
-#from tensorflow.keras import Conv2D
-#from tensorflow.keras import MaxPooling2D
-#from tensorflow.keras import Flatten
-
 from sklearn.model_selection import train_test_split
 
 # representation of labels
@@ -372,7 +434,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
 !ec
 
 !split 
-===== Running with Keras =====
+===== Running with Keras and setting up the model =====
 
 !bc pycod
 def create_convolutional_neural_network_keras(input_shape, receptive_field,
@@ -386,7 +448,7 @@ def create_convolutional_neural_network_keras(input_shape, receptive_field,
     model.add(layers.Dense(n_neurons_connected, activation='relu', kernel_regularizer=regularizers.l2(lmbd)))
     model.add(layers.Dense(n_categories, activation='softmax', kernel_regularizer=regularizers.l2(lmbd)))
 
-    sgd = optimizers.SGD(lr=eta)
+    sgd = optimizers.SGD(learning_rate=eta)
     model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])
 
     return model
@@ -463,112 +525,6 @@ plt.show()
 
 
 
-
-!split
-===== The CIFAR01 data set =====
-
-The CIFAR10 dataset contains 60,000 color images in 10 classes, with
-6,000 images in each class. The dataset is divided into 50,000
-training images and 10,000 testing images. The classes are mutually
-exclusive and there is no overlap between them.
-
-!bc pycod
-import tensorflow as tf
-
-from tensorflow.keras import datasets, layers, models
-import matplotlib.pyplot as plt
-
-# We import the data set
-(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
-
-# Normalize pixel values to be between 0 and 1 by dividing by 255. 
-train_images, test_images = train_images / 255.0, test_images / 255.0
-
-!ec
-
-
-
-!split
-===== Verifying the data set =====
-
-To verify that the dataset looks correct, let's plot the first 25 images from the training set and display the class name below each image.
-
-!bc pycod
-class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer',
-               'dog', 'frog', 'horse', 'ship', 'truck']
-plt.figure(figsize=(10,10))
-for i in range(25):
-    plt.subplot(5,5,i+1)
-    plt.xticks([])
-    plt.yticks([])
-    plt.grid(False)
-    plt.imshow(train_images[i], cmap=plt.cm.binary)
-    # The CIFAR labels happen to be arrays, 
-    # which is why you need the extra index
-    plt.xlabel(class_names[train_labels[i][0]])
-plt.show()
-!ec
-
-!split
-===== Set up  the model =====
-
-The 6 lines of code below define the convolutional base using a common pattern: a stack of Conv2D and MaxPooling2D layers.
-
-As input, a CNN takes tensors of shape (image_height, image_width, color_channels), ignoring the batch size. If you are new to these dimensions, color_channels refers to (R,G,B). In this example, you will configure our CNN to process inputs of shape (32, 32, 3), which is the format of CIFAR images. You can do this by passing the argument input_shape to our first layer.
-
-!bc pycod
-model = models.Sequential()
-model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
-model.add(layers.MaxPooling2D((2, 2)))
-model.add(layers.Conv2D(64, (3, 3), activation='relu'))
-model.add(layers.MaxPooling2D((2, 2)))
-model.add(layers.Conv2D(64, (3, 3), activation='relu'))
-
-# Let's display the architecture of our model so far.
-
-model.summary()
-!ec
-
-You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer.
-
-
-
-
-!split
-===== Add Dense layers on top =====
-
-To complete our model, you will feed the last output tensor from the
-convolutional base (of shape (4, 4, 64)) into one or more Dense layers
-to perform classification. Dense layers take vectors as input (which
-are 1D), while the current output is a 3D tensor. First, you will
-flatten (or unroll) the 3D output to 1D, then add one or more Dense
-layers on top. CIFAR has 10 output classes, so you use a final Dense
-layer with 10 outputs and a softmax activation.
-
-!bc pycod
-model.add(layers.Flatten())
-model.add(layers.Dense(64, activation='relu'))
-model.add(layers.Dense(10))
-Here's the complete architecture of our model.
-
-model.summary()
-!ec
-As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers.
-
-!split
-===== Compile and train the model =====
-
-!bc pycod
-model.compile(optimizer='adam',
-              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
-              metrics=['accuracy'])
-
-history = model.fit(train_images, train_labels, epochs=10, 
-                    validation_data=(test_images, test_labels))
-
-!ec
-
-
 !split
 ===== Finally, evaluate the model =====
 
@@ -591,7 +547,7 @@ print(test_acc)
 !split
 ===== Building code using Pytorch =====
 
-This code loads and normalizes the MNIST dataset. Thereafter it defines  a CNN architecture with:
+This code loads and normalizes the MNIST dataset. Thereafter it defines  a CNN architecture using PyTorch with:
 o Two convolutional layers
 o Max pooling
 o Dropout for regularization