Are you looking to solve regression problems using neural networks? Well, we will use neural networks for regression using TensorFlow. So, stay with us.
Computer models, especially Neural Networks models are much similar to the human brain. These models are used to carry out operations like picture identification and natural language processing that are challenging for traditional computer systems. A huge number of interconnected processing nodes, or neurons, make up ANNs, which can learn to carry out challenging tasks by adjusting the strength of their connections. Google Brain created the open-source machine learning package known as TensorFlow. It provides a range of resources, libraries, and tools that let programmers create and train machine learning and ANN models. Time series analysis, categorization, and natural language processing are just a few of the many tasks that TensorFlow is capable of. In this article, we will discuss how Neural Networks can be used to solve regression problems using TensorFlow. Moreover, we will also learn how we can tune the model in TensorFlow.
You may also be interested to learn how neural networks can be used to solve classification problems using TensorFlow.
Introduction to the TensorFlow for Regression in ANN
A Neural Network is a series of algorithms that detect basic patterns in a dataset. The Neural Network works similarly to the neural network in the human brain. In a neural network, a neuron is a mathematical function that seeks and categorizes patterns in accordance with a particular architecture. A very simple neural network consists of three most important layers.
- Input layer: This is the layer that takes the input data and has no activation function applied to the data. The number of nodes depends on the number of independent variables.
- Hidden layers: This is the layer of neutrons in between the input layer and output layer. There can be more than one hidden layer depending on the requirements and the complexity of the dataset. Also, the number of nodes is not specific. Moreover, any kind of activation function can be applied to the dataset in this layer except the softmax function.
- Output Layer: This is the last layer of the neural network. In the case of regression and binary classification, there is only one node in the output layer.
For example, the following figure shows a simple neural network for a regression problem.
In the case, of a regression problem, the linear identity activation function can be applied to the output layer.
Why TensorFlow is Popular for the Neural Network?
In this section, we will be using TensorFlow to create neural networks to solve regression problems. Let us first summarize why TensorFlow is one of the best modules for neural networks.
- TensorFlow is popular for its simple syntax and easy usability. TensorFlow provides an accessible and readable syntax which is essential for making these programming resources easier to use. The complex syntax is the last thing developers need to know given machine learning’s advanced nature.
- TensorFlow has a lot of useful built-in functions. It provides excellent functionalities and services when compared to other popular deep learning frameworks. These high-level operations are essential for carrying out complex parallel computations and for building advanced neural network models.
- TensorFlow is a low-level library that provides more flexibility. Thus we can define our own functionalities or services for our models.
- TensorFlow provides more network control. Thus allowing developers and researchers to understand how operations are implemented across the network.
These are some of the important points why TensorFlow is considered to be one of the popular modules for the ANN.
Building Neural Networks in TensorFlow for Regression
Now we will build the neural network for the regression problem using TensorFlow step by step. You can access all the data and source code from my GitHub account. But I will recommend following this article to understand it fully.
But before going to the implementation part, make sure that you have installed the following modules as we will be using them in the implementation part.
You can install these modules by using the pip command.
Creating and Splitting Sample Regression Dataset
For the simplification and understanding of the process of building a neural network, we will not use any real data. We will first create a random sample dataset and build a regression model on it using TensorFlow.
We will use the NumPy module to create some random numbers.
# Importing the numpy module
import numpy as np
# creating random dataset
Input = np.arange(-150, 150, 3)
output = np.arange(-150, 150, 3)As you can see, we have created a random dataset. Let us not visualize it to see, what it looks like on a graph.
# importing the matplotlib module
import matplotlib.pyplot as plt
# fixing the size of the plot
plt.figure( figsize = (8,6))
# plotting scattered plot
plt.scatter(Input, output, label = 'Dataset')
# labeling the plot
plt.legend()
plt.show()Output:
Now, we will use sklearn the module to split the above dataset into the testing and training parts.
# Splitting training and test data
X_train = Input[:80]
y_train = output[:80]
X_test = Input[80:]
y_test = output[80:]As you can see, we have split the dataset. Now, let us visualize again the testing part and the training part.
# size of the plot
plt.figure( figsize = (8,6))
# plotting input and output data
plt.scatter(X_train, y_train, c='b', label = 'Training data')
plt.scatter(X_test, y_test, c='g', label='Testing data')
# labeling
plt.legend()
plt.show()Output:
As you can see, we have split the dataset into testing and training parts.
Step by Step Creating a Neural Network for Regression Using TensorFlow
To understand the Neural Network fully, we will start from the very beginning. We will build the Neural Network using the following steps.
- We will first create a Neural network without any hidden layers
- Then we will create a Neural network with hidden layers and see how it affects the predictions
- We will also learn parameter tunning of Neural Networks to improve the performance of the model
- We will then learn about various evaluation matrices to evaluate the performance of the Neural Network Model
Keep in mind, that there are basically three steps to building a neural network using TensorFlow:
- Creating a model: The first step is to initialize and create a model. In our case, we will create a regression model.
- Compiling the model: Here, we define how to measure the model’s performance and specify the optimizer.
- Fitting the model: This is the training step of the model on the training dataset.
We will also follow these three simple steps to create and train the model.
Building Neural Networks for Regression Without Hidden Layer
Now, we will use TensorFlow to build Neural Network to solve a regression problem. So, the first step is to initialize the model.
# importing tensorflow module
import tensorflow as tf
# Tensorflow for Regression, input layer and output layer
model = tf.keras.Sequential([tf.keras.layers.InputLayer(
input_shape=1),
tf.keras.layers.Dense(1)])As you can see, the input shape is 1. This means that there is only one node in the input layer as our dataset has only one independent variable. And also the dense layer is 1 which means that the output layer has also only one node as it is a regression problem.
To understand the above code better we will visualize the neural network that the above code has created.
As you can see, we have a very simple neural network for now. The next step is to compile the model. We will use mean absolute error as a loss function and Adam as an optimizer.
# second step is to compile Tensorflow for Regression
model.compile(loss=tf.keras.losses.mae,
optimizer=tf.keras.optimizers.Adam(),
metrics=['mae'])The last step is to train the model. So, we will use the training dataset to train the model.
#traing model with 50 epochs
model.fit(tf.expand_dims(X_train, axis=-1), y_train, epochs=50)An epoch is a term used in machine learning and indicates the number of passes of the entire training dataset the machine learning algorithm has completed. So, we have fixed the epoch value to be 50 which means the model will train on the model 50 times to reduce the loss and learn the patterns.
Let us now use the testing dataset to find the predictions and calculate the r-square score of the model.
# making predictions using Tensorflow for Regression
y_preds = model.predict(X_test)
# Importing the required module
from sklearn.metrics import r2_score
# Evaluating the model
print('R score is :', r2_score(y_test, y_preds))Output:
R score is: -53.333
As you can see, we get a worse value for the r-square. Usually, the value of the r-square is between 0 and 1 but sometimes it can be negative as well. If the value of r-square is negative, that means the model has failed to learn and follow the patterns in the training dataset.
Let us now visualize the predictions to see how close they are to the actual values.
# size of the plot
plt.figure(figsize=(8,6))
# plotting training data
plt.scatter(X_train, y_train, c="m", label="Train data")
# plotting testing data
plt.scatter(X_test, y_test, c="g", label="Testing data")
# plotting predictions
plt.scatter(X_test, y_preds, c="r", label="Predictions")
# labeling
plt.legend()
plt.show()Output:
As you can see, the predictions are far away from the actual values.
Building a Neural Network for Regression With a Hidden Layer
This time we will build a model with a hidden layer and ReLU function on the output layer. Let us now understand how the hidden layer affects the performance of the model.
The capacity of a neural network model is defined by configuring the number of neurons (nodes) and the number of hidden layers. A model with less capacity may not be able to learn the training dataset sufficiently. In contrast, a model with too much capacity may memorize the training dataset, meaning it will overfit or may get stuck or lost during the optimization process.
A model with a single hidden layer and a sufficient number of nodes has the capability of learning any mapping function. The number of layers in a model is referred to as its depth. Increasing the depth increases the model’s capacity, but we have to make sure that our model is not overfitted. So, we have to come up with an optimum number of hidden layers and nodes in each layer.
Let us first initialize the model with one hidden layer and 10 nodes in the hidden layer.
# Model defining the Tensorflow for Regression
Model = tf.keras.Sequential([tf.keras.layers.InputLayer(input_shape=1,),
# building hiden layer with 10 nodes
tf.keras.layers.Dense(10, activation = tf.keras.activations.relu),
# adding the output layer
tf.keras.layers.Dense(1)
])As you can see, we have added one hidden layer with 10 nodes. To understand the above neural network, let us visualize the network.
Let us now compile the model.
# compiling the model
Model.compile(loss=tf.keras.losses.mae,
optimizer=tf.keras.optimizers.Adam(),
metrics=['mae'])As the last step of building a neural network is to train it on the training dataset.
# Training the model
Model.fit(tf.expand_dims(X_train, axis=-1), y_train, epochs=50)Once the training is complete, let us now find the r-square of the predictions.
# making predictions
y_preds = Model.predict(X_test)
# Evaluating the model
print('R score is :', r2_score(y_test, y_preds))Output:
R score is : 0.9999
As you can see, this time we get a really good r-square score for the model. Let us also visualize the predictions.
# size of the plot
plt.figure(figsize=(8,6))
# plotting training data
plt.scatter(X_train, y_train, c="m", label="Train data")
# plotting testing data
plt.scatter(X_test, y_test, c="g", label="Testing data")
# plotting predictions
plt.scatter(X_test, y_preds, c="r", label="Predictions")
# labeling
plt.legend()
plt.show()Output:
As you can see, this time the predictions are very much close to the actual values which means our model was able to learn the pattern of the training dataset.
Solving a Regression Problem Using Neural Networks
Now, we will use a real dataset to solve the regression problem. The dataset is about house prices in Dushanbe city. You can access the dataset and the code from my GitHub account. First, let us import the data set and explore it.
# importing the dataset
data = pd.read_csv('house.csv')
# printing the heading
data.head()Output:
As you can see, we have some null values in the dataset, let us first remove those null values.
# removing null values
data.dropna(axis=0, inplace=True)
# finding sum of null values
data.isnull().sum()Output:
As you can see now there are no null values. Let us now just plot the prices of houses on a line plot to see, how the price varies.
# setting the size of the plot
plt.figure(figsize=(8, 6))
# plotting line plot using matplotlib
plt.plot([i for i in range(len(data.price))], data.price, c='m')
# showing the plot
plt.show()Output:
As you can see, the price varies a lot but most of the prices are within a specific range.
The final step of the data preprocessing and exploring is to split the dataset into testing and training parts.
# input and output variables
x_data = data.drop('price', axis=1)
y_data = data.price
# importing the module
from sklearn.model_selection import train_test_split
# splitting the dataset into test data and training set
X_train, X_test, y_train, y_test = train_test_split(x_data, y_data, test_size = 0.3)Output:
As you can see, we have assigned 30% of the dataset to the testing part and 70% of the dataset to the training part.
Building a Neural Network for Housing Price
Let us now build a neural network with three different layers. We will not specify any activation function to the input layer and will build a hidden layer with 10 nodes.
# building Tensorflow for Regression
model_price = tf.keras.Sequential([tf.keras.layers.InputLayer(
#nodes in the input layers is 5
input_shape=5,),
#nodes in the hidden layer is 10
tf.keras.layers.Dense(10, activation = tf.keras.activations.relu),
tf.keras.layers.Dense(1)
])As you can see, the input layer has 5 nodes because our dataset contains 5 input variables. Let us now compile the model and train on the training dataset. We will specify the number of epochs to be 70.
# ccompiling the Tensorflow for Regression
model_price.compile(loss=tf.keras.losses.mae,
# optimizer is adam
optimizer=tf.keras.optimizers.Adam(),
metrics=['mae'])
# Training data neural networks for regression model
model_price.fit(X_train, y_train, epochs=70)Once the training of the model is complete, we can then move to the testing part.
Testing and Evaluating the Model
Let us first make predictions by using the testing dataset.
# making predictions Tensorflow for Regression
preds_price = model_price.predict(X_test)As the model had predicted the prices of the houses, we are not sure how well these predictions are. So in order to see how well the model is predicting, we will use some tools to evaluate the performance of the model.
Let us first visualize the actual values and the predicted values of the model.
# fitting the size of the plot
plt.figure(figsize=(10, 8))
# plotting training and test
plt.plot([i for i in range(len(y_test))],y_test, label="actual_values", c='g')
plt.plot([i for i in range(len(y_test))],preds_price, label="Predicted_values", c='m')
# showing the plotting
plt.legend()
plt.show()Output:
As you can see, the graph shows that the model failed to predict the values correctly. Let us also calculate the r-square score of the model.
# Evaluating the neural networks for regression
print('R score is :', r2_score(y_test, preds_price))Output:
R Score is : -1.033
As you can see, we get a negative value for the R-square score which means the model was not able to find the trend in the dataset.
Building a Neural Network With Multiple Hidden Layers
One of the reasons why the model couldn’t find the trend in the data set might be because of its complexity. When we have a complex dataset, it is always recommended to use at least two hidden layers. Let us now build a neural network with two hidden layers and train the model.
# building Tensorflow for Regression model
model_house = tf.keras.Sequential([tf.keras.layers.InputLayer(
#creating input layer with 5 nodes
input_shape=5,),
# first hidden layer with 10 nodes
tf.keras.layers.Dense(10, activation = tf.keras.activations.relu),
# second hidden layer with 20 nodes
tf.keras.layers.Dense(20, activation = tf.keras.activations.relu),
tf.keras.layers.Dense(1)
])
# compiling the Tensorflow for Regression model
model_house.compile(loss=tf.keras.losses.mae,
optimizer=tf.keras.optimizers.Adam(),
metrics=['mae'])
# Training the Tensorflow for Regression model
model_house.fit(X_train, y_train, epochs=70)Once the training is complete, we can make predictions and visualize the model.
# making prediction Tensorflow for Regression
preds_house = model_house.predict(X_test)
# fitting the size of the plot
plt.figure(figsize=(10, 8))
# plotting training and test
plt.plot([i for i in range(len(y_test))],y_test, label="actual_values", c='g')
plt.plot([i for i in range(len(y_test))],preds_house, label="Predicted_values", c='m')
# showing the plotting
plt.legend()
plt.show()Output:
As you can see, this time we get better predictions than last time. Let us also calculate the r-square score for the model.
# Finding the r-square score of Tensorflow for Regression
print('R score is :', r2_score(y_test, preds_house))Output:
R Score is : 0.3333
Although it is not perfect but much better than the previous model. You can even increase the R-square by finding the optimum values for the epoch, hidden layer, learning rate, and nodes by using the hyperparameter tuning method.
Summary
A neural network is a collection of algorithms that aims to identify underlying links in a set of data using a method that imitates how the human brain functions. It can be used for multi-purposes. In this article, we learned how we can use TensorFlow to build neural networks that can solve regression problems.