Design for Embedded Image Processing on FPGAs

ISBN-13: 9781119819790 / Twarda / 2023 / 500 str.

Donald G. Bailey

Spis treści

Preface xiiiAcknowledgments xixAbout the Companion Website xxi1 Image Processing 11.1 Basic Definitions 11.2 Image Formation 31.2.1 Optics 31.2.2 Colour 51.3 Image Processing Operations 61.4 Real-time Image Processing 81.5 Embedded Image Processing 91.6 Computer Architecture 101.7 Parallelism 111.7.1 Temporal or Task Parallelism 121.7.2 Spatial or Data Parallelism 131.7.3 Logical Parallelism 141.7.4 Stream Processing 141.8 Summary 15References 162 Field-programmable Gate Arrays 192.1 Hardware Architecture of FPGAs 192.1.1 Logic 212.1.2 DSP Blocks 222.1.3 Memory 232.1.4 Embedded CPU 232.1.5 Interconnect 242.1.6 Input and Output 242.1.7 Clocking 262.1.8 Configuration 262.1.9 FPGAs vs. ASICs 272.2 Programming FPGAs 282.2.1 Register Transfer Level 302.2.2 Hardware Description Languages 322.2.3 High-level Synthesis 332.3 FPGAs and Image Processing 382.3.1 Choosing an FPGA or Development Board 392.4 Summary 40References 413 Design Process 453.1 Problem Specification 453.2 Algorithm Development 473.2.1 Algorithm Development Process 473.2.2 Algorithm Structure 483.2.3 FPGA Development Issues 513.3 Architecture Selection 513.3.1 System Architecture 523.3.2 Partitioning Between Hardware and Software 533.3.3 Computational Architecture 553.4 System Implementation 603.4.1 Mapping to FPGA Resources 603.4.2 Algorithm Mapping Issues 623.5 Testing and Debugging 633.5.1 Design 633.5.2 Implementation 643.5.3 Common Implementation Bugs 643.5.4 Timing 663.5.5 System Debugging 683.5.6 Algorithm Tuning 703.5.7 In-field Diagnosis 713.6 Summary 72References 734 Design Constraints 774.1 Timing Constraints 774.1.1 Low-level Pipelining 784.1.2 Process Synchronisation 804.1.3 Synchronising Between Clock Domains 824.1.4 I/O Constraints 834.2 Memory Bandwidth Constraints 844.2.1 Memory Architectures 844.2.2 Caching 864.2.3 Row Buffering 874.3 Resource Constraints 884.3.1 Bit-serial Computation 894.3.2 Resource Multiplexing 894.3.3 Arbitration 924.3.4 Resource Controllers 944.3.5 Reconfigurability 954.4 Power Constraints 974.5 Performance Metrics 984.5.1 Speed 994.5.2 Resources 994.5.3 Power 994.5.4 Cost 1004.5.5 Application Metrics 1004.6 Summary 101References 1025 Computational Techniques 1055.1 Number Systems 1055.1.1 Binary Integers 1055.1.2 Residue Systems 1065.1.3 Redundant Representations 1075.1.4 Fixed-point Numbers 1075.1.5 Floating-point Numbers 1085.1.6 Logarithmic Number System 1105.1.7 Posit Numbers 1105.2 Elementary Functions 1115.2.1 Square Root 1115.2.2 Trigonometric Functions 1125.2.3 Linear CORDIC 1165.2.4 Hyperbolic Functions 1175.2.5 Logarithms and Exponentials 1185.2.6 Lookup Tables 1185.2.7 Polynomial Approximations 1225.2.8 Iterative Techniques 1235.3 Other Computation Techniques 1245.3.1 Incremental Update 1245.3.2 Separability 1245.4 Memory Structures 1245.4.1 FIFO Buffer 1245.4.2 Zigzag Buffers 1265.4.3 Stacks 1265.4.4 Linked Lists 1275.4.5 Trees 1285.4.6 Graphs 1295.4.7 Hash Tables 1295.5 Summary 130References 1316 Interfacing 1356.1 Camera Input 1356.1.1 Analogue Video 1366.1.2 Direct Digital Interface 1376.1.3 MIPI Camera Serial Interface 1386.1.4 Camera Link 1396.1.5 USB Cameras 1396.1.6 GigE Vision 1396.1.7 Camera Processing Pipeline 1406.2 Display Output 1436.2.1 Display Driver 1436.2.2 Display Content 1466.3 Serial Communication 1476.3.1 Rs- 232 1476.3.2 I 2 c 1486.3.3 Serial Peripheral Interface (SPI) 1496.3.4 Universal Serial Bus (USB) 1506.3.5 Ethernet 1506.3.6 PCI Express 1516.4 Off-chip Memory 1516.4.1 Static RAM 1526.4.2 Dynamic RAM 1526.4.3 Flash Memory 1556.5 Processors 1556.5.1 AXI Interface 1556.5.2 Avalon Bus 1566.5.3 Operating Systems 1576.5.4 Implications for System Design 1576.6 Summary 157References 1587 Point Operations 1617.1 Point Operations on a Single Image 1617.1.1 Contrast and Brightness Adjustment 1617.1.2 Global Thresholding and Contouring 1647.1.3 Lookup Table Implementation 1667.2 Point Operations on Multiple Images 1677.2.1 Image Averaging 1687.2.2 Image Subtraction 1707.2.3 Background Modelling 1727.2.4 Intensity Scaling 1757.2.5 Masking 1757.2.6 High Dynamic Range (HDR) Imaging 1777.3 Colour 1797.3.1 False Colour 1797.3.2 Colour Space Conversion 1807.3.3 Colour Thresholding 1927.3.4 Colour Enhancement 1937.3.5 White Balance 1947.4 Multi-spectral and Hyperspectral Imaging 1977.4.1 Hyperspectral Image Acquisition 1977.4.2 Processing Steps 1987.5 Summary 199References 1998 Histogram Operations 2038.1 Greyscale Histogram 2038.1.1 Building the Histogram 2038.1.2 Data Gathering 2058.1.3 Histogram Equalisation 2098.1.4 Automatic Exposure 2148.1.5 Threshold Selection 2158.1.6 Histogram Similarity 2208.2 Multidimensional Histograms 2208.2.1 Triangular Arrays 2218.2.2 Multidimensional Statistics 2228.2.3 Colour Segmentation 2258.2.4 Colour Indexing 2288.2.5 Texture Analysis 2298.3 Summary 231References 2319 Local Filters 2359.1 Window Caching 2359.1.1 Border Handling 2379.1.2 Filter Latency 2399.2 Linear Filters 2399.2.1 Filter Techniques 2409.2.2 Noise Smoothing 2439.2.3 Edge Detection 2469.2.4 Edge Enhancement 2489.3 Nonlinear Filters 2499.3.1 Gradient Magnitude 2499.3.2 Edge Orientation 2509.3.3 Peak Detection and Non-maximal Suppression 2519.3.4 Zero-crossing Detection 2529.3.5 Bilateral Filter 2529.3.6 Adaptive Thresholding 2539.3.7 High Dynamic Range Tone Mapping 2559.4 Rank Filters 2569.4.1 Sorting Networks 2589.5 Adaptive Histogram Equalisation 2629.6 Morphological Filters 2629.6.1 Binary Morphology 2629.6.2 Greyscale Morphology 2669.7 Colour Filtering 2689.7.1 Colour Morphology and Vector Median 2699.7.2 Edge Enhancement 2699.7.3 Bayer Pattern Demosaicing 2719.7.4 White Balancing 2729.8 Summary 273References 27410 Geometric Transformations 28110.1 Reverse Mapping 28210.1.1 Anti-alias Filtering 28310.1.2 Interpolation 28410.2 Forward Mapping 29110.2.1 Separable Mapping 29210.2.2 Hybrid Approach 29610.3 Common Mappings 29710.3.1 Affine Transformation 29710.3.2 Perspective Mapping 29710.3.3 Polynomial Mapping 29810.3.4 Lens Distortion 29910.3.5 Non-parametric Mappings 30210.4 Image Registration 30210.4.1 Feature-based Methods 30310.4.2 Area-based Methods 30710.4.3 Applications 31410.5 Summary 315References 31511 Linear Transforms 32111.1 Discrete Fourier Transform 32211.1.1 Fast Fourier Transform (FFT) 32311.1.2 Goertzel's Algorithm 33111.1.3 Applications 33211.2 Discrete Cosine Transform (DCT) 33611.3 Wavelet Transform 33811.3.1 Filter Implementations 34011.3.2 Applications 34411.4 Summary 345References 34512 Image and Video Coding 34912.1 Compression Techniques 35012.1.1 Colour Conversion 35012.1.2 Prediction and Transformation 35012.1.3 Motion Estimation and Compensation 35112.1.4 Quantisation 35212.1.5 Run-length Coding 35312.1.6 Entropy Coding 35412.2 DCT-based Codecs 35712.2.1 DCT Block Processing 35712.2.2 Jpeg 35712.2.3 Video Codecs 35812.3 Wavelet-based Codecs 35912.4 Lossless Compression 36012.5 Perceptual Coding 36112.6 Coding Hyperspectral Images 36212.7 Summary 362References 36313 Blob Detection and Labelling 36713.1 Bounding Box 36713.2 Run-length Coding 36913.3 Chain Coding 36913.3.1 Sequential Implementation 37013.3.2 Single-pass Stream Processing Algorithms 37013.3.3 Feature Extraction 37213.4 Connected Component Labelling (CCL) 37413.4.1 Random Access Algorithms 37413.4.2 Multiple Pass Algorithms 37413.4.3 Two-pass Algorithms 37513.4.4 Parallel Algorithms 37713.4.5 Hysteresis Thresholding 37713.5 Connected Component Analysis (CCA) 37713.5.1 Basic Single-pass Algorithm 37813.5.2 Reducing Memory Requirements 37913.5.3 Eliminating End-of-row Overheads 37913.5.4 Parallel Algorithms 38013.5.5 Further Considerations and Optimisations 38113.6 Distance Transform 38113.6.1 Morphological Approaches 38113.6.2 Chamfer Distance 38213.6.3 Euclidean Distance 38413.6.4 Applications 38613.6.5 Geodesic Distance Transform 38613.7 Watershed Transform 38713.7.1 Flow Algorithms 38813.7.2 Immersion Algorithms 38913.8 Hough Transform 39113.8.1 Line Hough Transform 39113.8.2 Circle Hough Transform 39413.8.3 Generalised Hough Transform 39513.9 Summary 396References 39614 Machine Learning 40314.1 Training 40314.1.1 Loss and Cost Functions 40414.1.2 Model Optimisation 40514.1.3 Considerations 40614.2 Regression 40914.2.1 Linear Regression 40914.2.2 Nonlinear Regression 40914.2.3 Neural Networks 40914.3 Classification 41114.3.1 Decision Trees 41114.3.2 Random Forests 41214.3.3 Bayesian Classification 41214.3.4 Quadratic Discriminant Analysis 41414.3.5 Linear Discriminant Analysis 41414.3.6 Support Vector Machines 41514.3.7 Neural Networks 41614.3.8 Clustering 41714.4 Deep Learning 41814.4.1 Building Blocks 41914.4.2 Architectures and Applications 42114.4.3 Training 42714.4.4 Implementation Issues 42814.5 Summary 433References 43315 Example Applications 44115.1 Coloured Region Tracking 44115.2 Foveal Sensor 44315.2.1 Foveal Mapping 44415.2.2 Using the Sensor 44715.3 Real-time Produce Grading 44815.3.1 Software Algorithm 44815.3.2 Hardware Implementation 45015.4 Stereo Imaging 45315.4.1 Rectification 45415.4.2 Calculating the Depth 45615.4.3 Stereo Matching Design 45715.5 Face Detection 45915.5.1 Design 46015.6 Summary 461References 461Index 465

O autorze

Donald G. Bailey, PhD, is Professor of Imaging Systems in the Department of Mechanical and Electrical Engineering, Massey University, New Zealand. He is an internationally recognized expert on FPGA technology in image processing and has published widely on FPGAs and related subjects.