This SpringerBrief describes how to build a rigorous end-to-end mathematical framework for deep neural networks. In particular, the authors derive gradient descent algorithms in a unified way for several neural network structures, including multilayer perceptrons, convolutional neural networks, deep autoencoders and recurrent neural networks.