Architecture of Advanced Numerical Analysis Systems: Designing a Scientific Computing System Using Ocaml

ISBN-13: 9781484288528 / Angielski / Miękka / 2022 / 290 str.

Jianxin Zhao

Spis treści

Prologue

A Brief History

Reductionism vs. Holism

Key Features

Contact Me

PART 1: NUMERICAL TECHNIQUES

1. Introduction

What Is Scientific Computing

What is Functional Programming

Who Is This Book For

Structure of the Book

Installation

Option 1: Install from OPAM

Option 2: Pull from Docker Hub

Option 3: Pin the Dev-Repo

Option 4: Compile from Source

CBLAS/LAPACKE Dependency

Interacting with Owl

Using Toplevel

Using Notebook

Using Owl-Jupyter

Summary

2. Conventions

Pure vs. Impure

Ndarray vs. Scalar

Infix Operators

Operator Extension

Module Structures

Number and Precision

Polymorphic Functions

Module Shortcuts

Type Casting

3. Visualisation

Create Plots

Specification

Subplots

Multiple Lines

Legend

Drawing Patterns

Line Plot

Scatter Plot

Stairs Plot

Box Plot

Stem Plot

Area Plot

Histogram & CDF Plot

Log Plot

3D Plot

Advanced Statistical Plot

Summary

References

4. Mathematical Functions

Basic Functions

Basic Unary Math Functions

Basic Binary Functions

Exponential and Logarithmic Functions

Trigonometric Functions

Other Math Functions

Special Functions

Airy Functions

Bessel Functions

Elliptic Functions

Gamma Functions

Beta Functions

Struve Functions

Zeta Functions

Error Functions

Integral Functions

Factorials

Interpolation and Extrapolation

Integration

Utility Functions

Summary

5. Statistical Functions

Random Variables

Discrete Random Variables

Continuous Random Variables

Descriptive Statistics

Order Statistics

Special Distribution

Gamma Distribution

Beta Distribution

Chi-Square Distribution

Student-t Distribution

Cauchy Distribution

Multiple Variables

Sampling

Hypothesis Tests

Theory

Gaussian Distribution in Hypothesis Testing

Two-Sample Inferences

Goodness-of-fit Tests

Non-parametric Statistics

Covariance and Correlations

Analysis of Variance

Summary

6. N-Dimensional Arrays

Ndarray Types

Creation Functions

Properties Functions

Map Functions

Fold Functions

Scan Functions

Comparison Functions

Vectorised Functions

Iteration Functions

Manipulation Functions

Serialisation

Tensors

Summary

References

7. Slicing and Broadcasting

Slicing

Basic Slicing

Fancy Slicing

Conventions in Definition

Extended Operators

Advanced Usage

Broadcasting

What Is Broadcasting?

Shape Constraints

Supported Operations

Slicing in NumPy and Julia

Internal Mechanism

Summary

8. Linear Algebra

Vectors and Matrices

Creating Matrices

Accessing Elements

Iterate, Map, Fold, and Filter

Math Operations

Gaussian Elimination

LU Factorisation

Inverse and Transpose

Vector Spaces

Rank and Basis

Orthogonality

Solving Ax = b

Matrix Sensitivity

Determinants

Eigenvalues and Eigenvectors

Solving 

A

x

=

λ

 

x

Complex Matrices

Similarity Transformation and Diagonalisation

Positive Definite Matrices

Positive Definiteness

Singular Value Decomposition

Internal: CBLAS and LAPACKE

Low-level Interface to CBLAS & LAPACKE

Sparse Matrices

Summary

References

9. Ordinary Differential Equations

What Is An ODE

Exact Solutions

Linear Systems

Solving An ODE Numerically

Owl-ODE

Example: Linear Oscillator System

Solver Structure

Symplectic Solvers

Features and Limits

Examples of using Owl-ODE

Explicit ODE

Two Body Problem

Lorenz Attractor

Damped Oscillation

Stiffness

Solve Non-Stiff ODEs

Solve Stiff ODEs

Summary

References

10. Signal Processing

Discrete Fourier Transform

Fast Fourier Transform

Examples

Applications of FFT

Find period of sunspots

Decipher the Tone

Image Processing

Filtering

Example: Smoothing

Gaussian Filter

Signal Convolution

FFT and Image Convolution

Summary

References

11. Algorithmic Differentiation

Chain Rule

Differentiation Methods

How Algorithmic Differentiation Works

Forward Mode

Reverse Mode

Forward or Reverse?

A Strawman AD Engine

Simple Forward Implementation

Simple Reverse Implementation

Unified Implementations

Forward and Reverse Propagation API

Expressing Computation

Example: Forward Mode

Example: Reverse Mode

High-Level APIs

Derivative and Gradient

Jacobian

Hessian and Laplacian

Other APIs

Internal of Algorithmic Differentiation

Go Beyond Simple Implementation

Extend AD module

Lazy Evaluation

Summary

References

12. Optimisation

Introduction

Root Finding

Univariate Function Optimisation

Use Derivatives

Golden Section Search

Multivariate Function Optimisation

Nelder-Mead Simplex Method

Gradient Descent Methods

Conjugate Gradient Method

Newton and Quasi-Newton Methods

Global Optimisation and Constrained Optimisation

Summary

References

13. Regression

Linear Regression

Problem: Where to locate a new McDonald’s restaurant?

Cost Function

Solving Problem with Gradient Descent

Multiple Regression

Feature Normalisation

Analytical Solution

Non-linear regressions

Regularisation

Ols, Ridge, Lasso, and Elastic_net

Logistic Regression

Sigmoid Function

Cost Function

Example

Multi-class classification

Support Vector Machine

Kernel and Non-linear Boundary

Example

Model error and selection

Error Metrics

Model Selection

Summary

References

14. Deep Neural Networks

Perceptron

Yet Another Regression

Model Representation

Forward Propagation

Back propagation

Feed Forward Network

Layers

Activation Functions

Initialisation

Training

Test

Neural Network Module

Module Structure

Neurons

Neural Graph

Training Parameters

Convolutional Neural Network

Recurrent Neural Network

Long Short Term Memory (LSTM)

Generative Adversarial Network

Summary

References

15. Natural Language Processing

Introduction

Text Corpus

Step-by-step Operation

Use the Corpus Module

Vector Space Models

Bag of Words (BOW)

Term Frequency–Inverse Document Frequency (TF-IDF)

Latent Dirichlet Allocation (LDA)

Models

Dirichlet Distribution

Gibbs Sampling

Topic Modelling Example

Latent Semantic Analysis (LSA)

Search Relevant Documents

Euclidean and Cosine Similarity

Linear Searching

Summary

References

16. Dataframe for Tabular Data

Basic Concepts

Create Frames

Manipulate Frames

Query Frames

Iterate, Map, and Filter

Read/Write CSV Files

Infer Type and Separator

Summary

17. Symbolic Representation

Introduction

Design

Core abstraction

Engines

ONNX Engine

Example 1: Basic operations

Example 2: Variable Initialisation

Example 3: Neural network

LaTeX Engine

Owl Engine

Summary

18. Probabilistic Programming

Generative Model vs Discriminative Model

Bayesian Networks

Sampling Techniques

Inference

PART 2: SYSTEM ARCHITECTURE

19. Architecture Overview

Introduction

Architecture Overview

Core Implementation

N-dimensional Array

Interfaced Libraries

Advanced Functionality

Computation Graph

Algorithmic Differentiation

Regression

Neural Network

Parallel Computing

Actor Engine

GPU Computing

OpenMP

Community-Driven R&D

Summary

20. Core Optimisation

Background

Numerical Libraries

Optimisation of Numerical Computation

Interfacing to C Code

Ndarray Operations

From OCaml to C

Optimisation Techniques

Map Operations

Convolution Operations

Reduction Operations

Repeat Operations

Summary

References

21. Automatic Empirical Tuning

What is Parameter Tuning

Why Parameter Tuning in Owl

How to Tune OpenMP Parameters

Make a Difference

Summary

22. Computation Graph

Introduction

What is a Computation Graph?

From Dynamic to Static

Significance in Computing

Examples

Example 01: Basic CGraph

Example 02: CGraph with AD

Example 03: CGraph with DNN

Design Rationale

Optimisation of CGraph

Optimising memory with pebbles

Allocation Algorithm

As Intermediate Representations

Summary

23. Scripting and Zoo System

Introduction

Share Script with Zoo

Typical Scenario

Create a Script

Share via Gist

Import in Another Script

Select a Specific Version

Command Line Tool

More Examples

System Design

Services

Type Checking

Backend

Domain Specific Language

Service Discovery

Use Case

Summary

References

24. Compiler Backends

Base Library

Backend: JavaScript

Use Native OCaml

Use Facebook Reason

Backend: MirageOS

MirageOS and Unikernel

Example: Gradient Descent

Example: Neural Network

Evaluation

Summary

25. Distributed Computing

Actor System

Design

Actor Engines

Map-Reduce Engine

Parameter Server Engine

Peer-to-Peer Engine

Classic Synchronise Parallel

Bulk Synchronous Parallel

Asynchronous Parallel

Stale Synchronous Parallel

Probabilistic Synchronise Parallel

Basic idea: sampling

Compatibility

Barrier Trade-off Dimensions

Convergence

A Distributed Training Example

Step Progress

Accuracy

Summary

References

26. Testing Framework

Unit Test

Example

What Could Go Wrong

Corner Cases

Test Coverage

Use Functor

Summary

27. Constants and Metric System

What Is a Metric System

Four Metric Systems

SI Prefix

Example: Physics and Math constants

International System of Units

Time

Length

Area

Volume

Speed

Mass

Force

Energy

Power

Pressure

Viscosity

Luminance

Radioactivity

28. Internal Utility Modules

Dataset Module

MNIST

CIFAR-10

Graph Module

Stack and Heap Modules

Count-Min Sketch

Summary

PART 3: CASE STUDIES

29. Case - Image Recognition

Background

LeNet

AlexNet

VGG

ResNet

SqueezeNet

Capsule Network

Building InceptionV3 Network

InceptionV1 and InceptionV2

Factorisation

Grid Size Reduction

InceptionV3 Architecture

Preparing Weights

Processing Image

Running Inference

Applications

Summary

References

30. Case - Instance Segmentation

Introduction

Mask R-CNN Network

Building Mask R-CNN

Feature Extractor

Proposal Generation

Classification

Run the Code

Summary

References

31. Case - Neural Style Transfer

Content and Style

Content Reconstruction

Style Recreation

Combining Content and Style

Running NST

Extending NST

Fast Style Transfer

Building FST Network

Running FST

Summary

References

32. Case - Recommender System

Introduction

Architecture

Build Topic Models

Index Text Corpus

Random Projection

Optimising Vector Storage

Optimise Data Structure

Optimise Index Algorithm

Search Articles

Code Implementation

Make It Live

Summary

References

33. Case - Applications in Finance

Introduction

Bond Pricing

Black-Scholes Model

Mathematical Model

Option Pricing

Portfolio Optimisation

Mathematical Model

Efficient Frontier

Maximise Sharpe Ratio

O autorze

Liang Wang is the Chief AI Architect at Nokia, the Chief Scientific Officer at iKVA, a Senior Researcher at the University of Cambridge, and an Intel Software Innovator. He has a broad research interest in artificial intelligence, machine learning, operating systems, computer networks, optimization theory, and graph theory.

Jianxin Zhao is a PhD graduate from the University of Cambridge, supervised by Prof. Jon Crowcroft. His research interests include numerical computation, high-performance computing, machine learning, and their application in the real world.

Blurb

This unique open access book applies the functional OCaml programming language to numerical or computational weighted data science, engineering, and scientific applications. This book is based on the authors' first-hand experience building and maintaining Owl, an OCaml-based numerical computing library. 

You'll first learn the various components in a modern numerical computation library. Then, you will learn how these components are designed and built up and how to optimize their performance. After reading and using this book, you'll have the knowledge required to design and build real-world complex systems that effectively leverage the advantages of the OCaml functional programming language.

You will:

• Optimize core operations based on N-dimensional arrays
• Design and implement an industry-level algorithmic differentiation module
• Implement mathematical optimization, regression, and deep neural network functionalities based on algorithmic differentiation
• Design and optimize a computation graph module, and understand the benefits it brings to the numerical computing library
• Accommodate the growing number of hardware accelerators (e.g. GPU, TPU) and execution backends (e.g. web browser, unikernel) of numerical computation
• Use the Zoo system for efficient scripting, code sharing, service deployment, and composition
• Design and implement a distributed computing engine to work with a numerical computing library, providing convenient APIs and high performance