Using Logistic and Softmax Regression with TensorFlow

In this TensorFlow tutorial, we train a softmax regression model. The model should be able to look at the images of handwritten digits from the MNIST data set and classify them as digits from 0 to 9. For instance, our model might evaluate an image of a six and be 90% sure it is a six, give a 5% chance to it being an eight, and leave a bit of probability to all the other digits.

The data set is split into three parts: 55,000 data points of training data (mnist.train), 10,000 points of test data (mnist.test), and 5,000 points of validation data (mnist.validation).

Prerequisites

Before running the code from this tutorial, download the input_data.py file and place it into the same folder where softmax regression will be implemented. The script encapsulates the logic that is responsible for downloading the MNIST data set.

To import the data set, proceed as follows:

# Import MNIST data
import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

Building a computational graph

First of all, we should set up graph inputs and outputs. MNIST images are matrices of pixels (28×28). The simplest way to structure I/O data is just to flatten a matrix into a single column vector (28x28 = 784). In this case, an input tensor has the shape of [None, 784], where 784 is a flattened image and None indicates that the number of input images can be of any size.

The graph output is the tensor y with the vector size of 10 entries, where each entry expresses the probability for the image to be a particular digit (0–9) and None indicates that the number of output can be of any size.

# TF graph input
x = tf.placeholder("float", [None, 784]) # MNIST data image of shape 28*28=784
y = tf.placeholder("float", [None, 10]) # 0-9 digits recognition => 10 classes

Now let’s define and initialize the weights W and the biases b for our model.

# Set model weights
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))

Before Variables can be used, they must be initialized:

# Initializing the variables
init = tf.initialize_all_variables()

Model and cost function

For this example, we start with the simplest possible function—a linear mapping:

y = tf.nn.softmax(tf.matmul(x, W) + b)

As the cost function, we use the cross-entropy function:

cost_function = -tf.reduce_sum(y*tf.log(y))

Training a model

To minimize the cross-entropy function, we use the gradient descent method with the step length of 0.01.

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
for i in range(total_batch):
      batch_xs, batch_ys = mnist.train.next_batch(batch_size)
      # Fit training using batch data
      sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})

Evaluating the model

After training the model, we want to estimate the accuracy of this model. To get accuracy, we need to compare the predicted values with the actual ones in the training data set:

correct_predictions = tf.equal(tf.argmax(model, 1), tf.argmax(y, 1))

Then, compute accuracy:

accuracy = tf.reduce_mean(tf.cast(predictions, "float"))

Finally, we can evaluate our accuracy on the test data.

print "Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})

Running the code

Now, try to run the example source code:

# Import MINST data
import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)

import tensorflow as tf

# Set parameters
learning_rate = 0.01
training_iteration = 30
batch_size = 100
display_step = 2

# TF graph input
x = tf.placeholder("float", [None, 784]) # mnist data image of shape 28*28=784
y = tf.placeholder("float", [None, 10]) # 0-9 digits recognition => 10 classes

# Create a model

# Set model weights
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))

# Construct a linear model
model = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax

# Minimize error using cross entropy
# Cross entropy
cost_function = -tf.reduce_sum(y*tf.log(model)) 
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost_function)

# Initializing the variables
init = tf.initialize_all_variables()

# Launch the graph
with tf.Session() as sess:
    sess.run(init)

    # Training cycle
    for iteration in range(training_iteration):
        avg_cost = 0.
        total_batch = int(mnist.train.num_examples/batch_size)
        # Loop over all batches
        for i in range(total_batch):
            batch_xs, batch_ys = mnist.train.next_batch(batch_size)
            # Fit training using batch data
            sess.run(optimizer, feed_dict={x: batch_xs, y: batch_ys})
            # Compute average loss
            avg_cost += sess.run(cost_function, feed_dict={x: batch_xs, y: batch_ys})/total_batch
        # Display logs per eiteration step
        if iteration % display_step == 0:
            print "Iteration:", '%04d' % (iteration + 1), "cost=", "{:.9f}".format(avg_cost)

    print "Tuning completed!"

    # Test the model
    predictions = tf.equal(tf.argmax(model, 1), tf.argmax(y, 1))
    # Calculate accuracy
    accuracy = tf.reduce_mean(tf.cast(predictions, "float"))
    print "Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels})

As a result, we trained a softmax regression model that is able to look at images of handwritten digits and predict what digits they are.

Further reading

• Implementing Linear Regression with TensorFlow
• Basic Concepts and Manipulations with TensorFlow
• Using k-means Clustering with TensorFlow