Machine Learning and Big Data with Kdb+/Q

ISBN-13: 9781119404750 / Angielski / Twarda / 2019 / 640 str.

Novotny, Jan

Spis treści

Preface xviiAbout the Authors xxiiiPart One Language FundamentalsChapter 1 Fundamentals of the q Programming Language 31.1 The (Not So Very) First Steps in q 31.2 Atoms and Lists 51.2.1 Casting Types 111.3 Basic Language Constructs 141.3.1 Assigning, Equality and Matching 141.3.2 Arithmetic Operations and Right-to-Left Evaluation: Introduction to q Philosophy 171.4 Basic Operators 191.5 Difference between Strings and Symbols 311.5.1 Enumeration 311.6 Matrices and Basic Linear Algebra in q 331.7 Launching the Session: Additional Options 351.8 Summary and How-To's 38Chapter 2 Dictionaries and Tables: The q Fundamentals 412.1 Dictionary 412.2 Table 442.3 The Truth about Tables 482.4 Keyed Tables are Dictionaries 502.5 From a Vector Language to an Algebraic Language 51Chapter 3 Functions 573.1 Namespace 593.1.0.1 .quantQ. Namespace 603.2 The Six Adverbs 603.2.1 Each 603.2.1.1 Each 613.2.1.2 Each-left : 613.2.1.3 Each-right /: 623.2.1.4 Cross Product /: : 623.2.1.5 Each-both ' 633.2.2 Each-prior ': 663.2.3 Compose (') 673.2.4 Over and Fold / 673.2.5 Scan 683.2.5.1 EMA: The Exponential Moving Average 693.2.6 Converge 703.2.6.1 Converge-repeat 703.2.6.2 Converge-iterate 713.3 Apply 723.3.1 @ (apply) 723.3.2 . (apply) 733.4 Protected Evaluations 753.5 Vector Operations 763.5.1 Aggregators 763.5.1.1 Simple Aggregators 763.5.1.2 Weighted Aggregators 773.5.2 Uniform Functions 773.5.2.1 Running Functions 773.5.2.2 Window Functions 783.6 Convention for User-Defined Functions 79Chapter 4 Editors and Other Tools 814.1 Console 814.2 Jupyter Notebook 824.3 GUIs 844.3.1 qStudio 854.3.2 Q Insight Pad 884.4 IDEs: IntelliJ IDEA 904.5 Conclusion 92Chapter 5 Debugging q Code 935.1 Introduction to Making It Wrong: Errors 935.1.1 Syntax Errors 945.1.2 Runtime Errors 945.1.2.1 The Type Error 955.1.2.2 Other Errors 985.2 Debugging the Code 1005.3 Debugging Server-Side 102Part Two Data OperationsChapter 6 Splayed and Partitioned Tables 1076.1 Introduction 1076.2 Saving a Table as a Single Binary File 1086.3 Splayed Tables 1106.4 Partitioned Tables 1136.5 Conclusion 119Chapter 7 Joins 1217.1 Comma Operator 1217.2 Join Functions 1257.2.1 ij 1257.2.2 ej 1267.2.3 lj 1267.2.4 pj 1277.2.5 upsert 1287.2.6 uj 1297.2.7 aj 1317.2.8 aj0 1347.2.8.1 The Next Valid Join 1357.2.9 asof 1387.2.10 wj 1407.3 Advanced Example: Running TWAP 144Chapter 8 Parallelisation 1518.1 Parallel Vector Operations 1528.2 Parallelisation over Processes 1558.3 Map-Reduce 1558.4 Advanced Topic: Parallel File/Directory Access 158Chapter 9 Data Cleaning and Filtering 1619.1 Predicate Filtering 1619.1.1 The Where Clause 1619.1.2 Aggregation Filtering 1639.2 Data Cleaning, Normalising and APIs 163Chapter 10 Parse Trees 16510.1 Definition 16610.1.1 Evaluation 16610.1.2 Parse Tree Creation 17010.1.3 Read-Only Evaluation 17010.2 Functional Queries 17110.2.1 Functional Select 17410.2.2 Functional Exec 17810.2.3 Functional Update 17910.2.4 Functional Delete 180Chapter 11 A Few Use Cases 18111.1 Rolling VWAP 18111.1.1 N Tick VWAP 18111.1.2 TimeWindow VWAP 18211.2 Weighted Mid for N Levels of an Order Book 18311.3 Consecutive Runs of a Rule 18511.4 Real-Time Signals and Alerts 186Part Three Data ScienceChapter 12 Basic Overview of Statistics 19112.1 Histogram 19112.2 First Moments 19612.3 Hypothesis Testing 19812.3.1 Normal p-values 19812.3.2 Correlation 20112.3.2.1 Implementation 20212.3.3 t-test: One Sample 20212.3.3.1 Implementation 20412.3.4 t-test: Two Samples 20412.3.4.1 Implementation 20512.3.5 Sign Test 20612.3.5.1 Implementation of the Test 20812.3.5.2 Median Test 21112.3.6 Wilcoxon Signed-Rank Test 21212.3.7 Rank Correlation and Somers' D 21412.3.7.1 Implementation 21612.3.8 Multiple Hypothesis Testing 22112.3.8.1 Bonferroni Correction 22412.3.8.2 Sidák's Correction 22412.3.8.3 Holm's Method 22512.3.8.4 Example 226Chapter 13 Linear Regression 22913.1 Linear Regression 23013.2 Ordinary Least Squares 23113.3 The Geometric Representation of Linear Regression 23313.3.1 Moore-Penrose Pseudoinverse 23513.3.2 Adding Intercept 23713.4 Implementation of the OLS 24013.5 Significance of Parameters 24313.6 How Good is the Fit: R² 24413.6.1 Adjusted R-squared 24713.7 Relationship with Maximum Likelihood Estimation and AIC with Small Sample Correction 24813.8 Estimation Suite 25213.9 Comparing Two Nested Models: Towards a Stopping Rule 25413.9.1 Comparing Two General Models 25613.10 In-/Out-of-Sample Operations 25713.11 Cross-validation 26213.12 Conclusion 264Chapter 14 Time Series Econometrics 26514.1 Autoregressive and Moving Average Processes 26514.1.1 Introduction 26514.1.2 AR(p) Process 26614.1.2.1 Simulation 26614.1.2.2 Estimation of AR(p) Parameters 26814.1.2.3 Least Square Method 26814.1.2.4 Example 26914.1.2.5 Maximum Likelihood Estimator 26914.1.2.6 Yule-Walker Technique 26914.1.3 MA(q) Process 27114.1.3.1 Estimation of MA(q) Parameters 27214.1.3.2 Simulation 27214.1.3.3 Example 27314.1.4 ARMA(p, q) Process 27314.1.4.1 Invertibility of the ARMA(p, q) Process 27414.1.4.2 Hannan-Rissanen Algorithm: Two-Step Regression Estimation 27414.1.4.3 Yule-Walker Estimation 27414.1.4.4 Maximum Likelihood Estimation 27514.1.4.5 Simulation 27514.1.4.6 Forecast 27614.1.5 ARIMA(p, d, q) Process 27614.1.6 Code 27614.1.6.1 Simulation 27714.1.6.2 Estimation 27814.1.6.3 Forecast 28214.2 Stationarity and Granger Causality 28514.2.1 Stationarity 28514.2.2 Test of Stationarity - Dickey-Fuller and Augmented Dickey-Fuller Tests 28614.2.3 Granger Causality 28614.3 Vector Autoregression 28714.3.1 VAR(p) Process 28814.3.1.1 Notation 28814.3.1.2 Estimator 28814.3.1.3 Example 28914.3.1.4 Code 29314.3.2 VARX(p, q) Process 29714.3.2.1 Estimator 29714.3.2.2 Code 298Chapter 15 Fourier Transform 30115.1 Complex Numbers 30115.1.1 Properties of Complex Numbers 30215.2 Discrete Fourier Transform 30815.3 Addendum: Quaternions 31415.4 Addendum: Fractals 321Chapter 16 Eigensystem and PCA 32516.1 Theory 32516.2 Algorithms 32716.2.1 QR Decomposition 32816.2.2 QR Algorithm for Eigenvalues 33016.2.3 Inverse Iteration 33116.3 Implementation of Eigensystem Calculation 33216.3.1 QR Decomposition 33316.3.2 Inverse Iteration 33716.4 The Data Matrix and the Principal Component Analysis 34116.4.1 The Data Matrix 34116.4.2 PCA: The First Principal Component 34416.4.3 Second Principal Component 34516.4.4 Terminology and Explained Variance 34716.4.5 Dimensionality Reduction 34916.4.6 PCA Regression (PCR) 35016.5 Implementation of PCA 35116.6 Appendix: Determinant 35416.6.1 Theory 35416.6.2 Techniques to Calculate a Determinant 35516.6.3 Implementation of the Determinant 356Chapter 17 Outlier Detection 35917.1 Local Outlier Factor 360Chapter 18 Simulating Asset Prices 36918.1 Stochastic Volatility Process with Price Jumps 36918.2 Towards the Numerical Example 37118.2.1 Numerical Preliminaries 37118.2.2 Implementing Stochastic Volatility Process with Jumps 37418.3 Conclusion 378Part Four Machine LearningChapter 19 Basic Principles of Machine Learning 38119.1 Non-Numeric Features and Normalisation 38119.1.1 Non-Numeric Features 38119.1.1.1 Ordinal Features 38219.1.1.2 Categorical Features 38319.1.2 Normalisation 38319.1.2.1 Normal Score 38419.1.2.2 Range Scaling 38519.2 Iteration: Constructing Machine Learning Algorithms 38619.2.1 Iteration 38619.2.2 Constructing Machine Learning Algorithms 389Chapter 20 Linear Regression with Regularisation 39120.1 Bias-Variance Trade-off 39220.2 Regularisation 39320.3 Ridge Regression 39420.4 Implementation of the Ridge Regression 39620.4.1 Optimisation of the Regularisation Parameter 40120.5 Lasso Regression 40320.6 Implementation of the Lasso Regression 405Chapter 21 Nearest Neighbours 41921.1 k-Nearest Neighbours Classifier 41921.2 Prototype Clustering 42321.3 Feature Selection: Local Nearest Neighbours Approach 42921.3.1 Implementation 430Chapter 22 Neural Networks 43722.1 Theoretical Introduction 43722.1.1 Calibration 44022.1.1.1 Backpropagation 44122.1.2 The Learning Rate Parameter 44322.1.3 Initialisation 44322.1.4 Overfitting 44422.1.5 Dimension of the Hidden Layer(s) 44422.2 Implementation of Neural Networks 44522.2.1 Multivariate Encoder 44522.2.2 Neurons 44622.2.3 Training the Neural Network 44822.3 Examples 45122.3.1 Binary Classification 45122.3.2 M-class Classification 45422.3.3 Regression 45722.4 Possible Suggestions 463Chapter 23 AdaBoost with Stumps 46523.1 Boosting 46523.2 Decision Stumps 46623.3 AdaBoost 46723.4 Implementation of AdaBoost 46823.5 Recommendation for Readers 474Chapter 24 Trees 47724.1 Introduction to Trees 47724.2 Regression Trees 47924.2.1 Cost-Complexity Pruning 48124.3 Classification Tree 48224.4 Miscellaneous 48424.5 Implementation of Trees 485Chapter 25 Forests 49525.1 Bootstrap 49525.2 Bagging 49825.2.1 Out-of-Bag 49925.3 Implementation 50025.3.1 Prediction 50325.3.2 Feature Selection 505Chapter 26 Unsupervised Machine Learning: The Apriori Algorithm 50926.1 Apriori Algorithm 51026.2 Implementation of the Apriori Algorithm 511Chapter 27 Processing Information 52327.1 Information Retrieval 52327.1.1 Corpus: Leonardo da Vinci 52327.1.2 Frequency Counting 52427.1.3 tf-idf 52827.2 Information as Features 53227.2.1 Sample: Simulated Proteins 53327.2.2 Kernels and Metrics for Proteins 53527.2.3 Implementation of Inner Products and Nearest Neighbours Principles 53527.2.4 Further Topics 539Chapter 28 Towards AI - Monte Carlo Tree Search 54128.1 Multi-Armed Bandit Problem 54128.1.1 Analytic Solutions 54328.1.2 Greedy Algorithms 54328.1.3 Confidence-Based Algorithms 54428.1.4 Bayesian Algorithms 54628.1.5 Online Gradient Descent Algorithms 54728.1.6 Implementation of Some Learning Algorithms 54728.2 Monte Carlo Tree Search 55828.2.1 Selection Step 56128.2.2 Expansion Step 56228.2.3 Simulation Step 56328.2.4 Back Propagation Step 56328.2.5 Finishing the Algorithm 56328.2.6 Remarks and Extensions 56428.3 Monte Carlo Tree Search Implementation - Tic-tac-toe 56528.3.1 Random Games 56628.3.2 Towards the MCTS 57028.3.3 Case Study 57928.4 Monte Carlo Tree Search - Additional Comments 57928.4.1 Policy and Value Networks 57928.4.2 Reinforcement Learning 581Chapter 29 Econophysics: The Agent-Based Computational Models 58329.1 Agent-Based Modelling 58429.1.1 Agent-Based Models in Society 58429.1.2 Agent-Based Models in Finance 58629.2 Ising Agent-Based Model for Financial Markets 58729.2.1 Ising Model in Physics 58729.2.2 Ising Model of Interacting Agents 58729.2.3 Numerical Implementation 58829.3 Conclusion 592Chapter 30 Epilogue: Art 595Bibliography 601Index 607

O autorze

JAN NOVOTNY is an eFX quant trader at Deutsche Bank. Previously, he worked at the Centre for Econometric Analysis on high-frequency econometric models. He holds a PhD from CERGE-EI, Charles University, Prague.PAUL A. BILOKON is CEO and founder of Thalesians Ltd and an expert in algorithmic trading. He previously worked at Nomura, Lehman Brothers, and Morgan Stanley. Paul was educated at Christ Church College, Oxford, and Imperial College.ARIS GALIOTOS is the global technical lead for the eFX kdb+ team at HSBC, where he helps develop a big data installation processing billions of real-time records per day. Aris holds an MSc in Financial Mathematics with Distinction from the University of Edinburgh.FRÉDÉRIC DÉLÈZE is an independent algorithm trader and consultant. He has designed automated trading strategies for hedge funds and developed quantitative risk models for investment banks. He holds a PhD in Finance from Hanken School of Economics, Helsinki.