Lecture 1 Introduction of Deep Learning

Supervised Learning

1. Training with Labeled Data: The training data provided to the algorithm is labeled, meaning that each input is paired with the correct output label. This allows the algorithm to learn from the examples provided.

2. Prediction of Outputs: Once trained, the algorithm can predict output labels for new input data that it has not seen before. These predictions are based on the learned mapping from the training data.

3. Types of Supervised Learning:
• Classification: In classification tasks, the output labels are discrete categories or classes. The algorithm learns to classify inputs into one of these predefined classes. Example: predicting whether an email is spam or not spam.
• Regression: In regression tasks, the output labels are continuous numerical values. The algorithm learns to predict a value based on input features. Example: predicting house prices based on features like size, number of bedrooms, etc.

4. Evaluation: The performance of a supervised learning algorithm is typically evaluated using metrics such as accuracy (for classification tasks) or mean squared error (for regression tasks) on a separate validation or test dataset.

Forward Propagation and Backward Propagation

1. Forward Propagation:
• During forward propagation, input data passes through the neural network, undergoing a series of operations such as weighted summation and activation functions at each layer, ultimately producing the network's output.
• The purpose of forward propagation is to compute the network's predictions or outputs, which are then compared with the true labels to calculate the loss function.

2. Backward Propagation:
• In backward propagation, the gradient of the loss function is computed with respect to the parameters of the network, and this gradient is propagated backward through the network, from the output layer to the input layer.
• The goal of backward propagation is to compute the gradients of the loss function with respect to the network parameters, enabling the optimization algorithm (e.g., gradient descent) to update the parameters in a way that minimizes the loss function.

Convolutional Kernel

Convolution operation , with example

1. Place the filter on top of the input signal:
• Initially, the leftmost element of the filter aligns with the leftmost element of the input signal.

2. Compute the dot product:
• At each position, we compute the dot product between the filter and the overlapping portion of the input signal.
• For example, at the first position: f1⋅g1+f2⋅g2+f3⋅g3=1⋅0.5+2⋅1+3⋅0.5=6

3. Move the filter:
• After computing the dot product at the current position, we shift the filter to the right by one position.

4. Repeat:
• We repeat steps 2 and 3 until the filter has passed over the entire input signal.

Pooling Layer

CNN Samoyed Example:

1. Convolutional Layer:
• The first layer mentioned is a convolutional layer. This layer applies a set of learnable filters (also known as kernels) to the input image.
• Each filter convolves over the input image, computing a dot product between the filter weights and a small region of the input image.
• The output of this operation is a feature map that represents the presence of different patterns or features in the input image.
• The Rectified Linear Unit (ReLU) activation function is then applied element-wise to the feature map, introducing non-linearity to the network.

2. Max Pooling Layer:
• The next layer mentioned is a max pooling layer. Max pooling is a downsampling operation that reduces the spatial dimensions of the feature maps while retaining the most important information.
• In a max pooling operation, a small window (typically 2x2 or 3x3) slides over the feature map, and only the maximum value within each window is retained.
• This helps in reducing the computational complexity of the network and making the learned features more robust to small translations and distortions in the input image.

3. Repetition:
• The process of convolution followed by ReLU activation and max pooling is repeated multiple times in the network.
• Each repetition allows the network to learn increasingly complex and abstract features from the input image.
• As the network progresses deeper, the spatial dimensions of the feature maps decrease while the number of channels (or depth) typically increases.

Stochastic Gradient Descent (SGD):

1. Initialization: The algorithm starts with an initial guess for the parameters of the model. These parameters could be randomly initialized or set to some predefined values.

2. Compute Gradient: At each iteration, the gradient of the objective function with respect to the parameters is computed. This gradient represents the direction of the steepest increase of the function.

3. Update Parameters: The parameters are then updated in the direction opposite to the gradient, scaled by a factor known as the learning rate. The learning rate determines the size of the steps taken during each iteration.

4. Convergence: Steps 2 and 3 are repeated iteratively until a stopping criterion is met. This could be a fixed number of iterations, convergence of the objective function, or reaching a predefined threshold for the change in parameters.

linear classifiers are applied on top of hand-engineered features

1. Linear Classifiers: These are algorithms used for classification tasks where the decision boundary separating different classes is linear. Examples include logistic regression and linear support vector machines (SVM). Linear classifiers compute a weighted sum of the feature vector components, which is then used to make predictions.

2. Hand-Engineered Features: These are features extracted from the input data using domain knowledge or manual feature engineering techniques. Hand-engineered features are designed to capture relevant information that is useful for the classification task. Examples of hand-engineered features include color histograms for image classification or word frequencies for text classification.

3. Two-Class Classification: In many practical applications, the task involves classifying inputs into one of two classes or categories. For example, classifying emails as spam or not spam, or classifying images as containing cats or dogs. A two-class linear classifier computes the weighted sum of the feature vector components and compares it to a threshold. If the weighted sum is above the threshold, the input is classified as belonging to a particular category; otherwise, it is classified as belonging to the other category.

CNN vs RNN, summarized by chatgpt 3.5

• CNNs are deep learning models commonly used for tasks such as image classification, object detection, and image segmentation. They consist of multiple layers of convolutional and pooling operations, followed by fully connected layers for classification.

• RNNs are deep learning models commonly used for tasks involving sequential data, such as natural language processing, time series analysis, and speech recognition. They have recurrent connections that allow them to maintain information over time and capture temporal dependencies in the data.

• Convolutional Neural Networks (CNNs):
• LeNet
• AlexNet
• VGG
• GoogLeNet (Inception)
• ResNet
• MobileNet
• EfficientNet
• YOLO (You Only Look Once)
• SSD (Single Shot Multibox Detector)

• Recurrent Neural Networks (RNNs):
• Vanilla RNN
• Long Short-Term Memory (LSTM)
• Gated Recurrent Unit (GRU)
• Bidirectional RNN
• Seq2Seq (Sequence-to-Sequence)
• Transformer
• BERT (Bidirectional Encoder Representations from Transformers)
• WaveNet

Related Mathematics Knowledge

1. Derivative:
• The derivative of a function represents the rate of change of that function with respect to a single variable.
• It measures how the function's output changes as the input variable changes.
• Symbolically, if y=f(x), then the derivative of f(x) with respect to x is denoted as dy/dx or f′(x).
• Geometrically, the derivative gives the slope of the tangent line to the curve of the function at a specific point.

2. Partial Derivative:
• The partial derivative of a function represents the rate of change of that function with respect to one of its several variables, while holding all other variables constant.
• It is used for functions with multiple variables, where the rate of change may vary in different directions.
• Symbolically, if a function f depends on multiple variables, say x and y, then the partial derivative of f with respect to x is denoted as ∂f/∂x.
• Geometrically, the partial derivative gives the slope of the tangent line to the curve of the function in the direction of the specified variable.

Reference

Question 3: What are the concepts: Loop, iteration, and recursion in programming?

Question 4: i++

Question 5: What is the norm of a vector? What is p-norm?

Question 6: How to write 𝑓(𝑥) = 𝑎𝑥2 +𝑏𝑥 + 𝑐, 𝑥 ∈ [𝑥1, 𝑥2] in a matrix way?

Semantic Segmetation

1. Labeling Your Data:
• Semantic segmentation requires labeled data where each pixel in an image is annotated with the class it belongs to (e.g., road, car, pedestrian). This labeling process is often done manually by humans, although there are some automated methods available.

2. Create Two Datastores:
• Image Datastore: This contains the input images for training and testing.
• Pixel Label Datastore: This contains the corresponding labeled images where each pixel is labeled with its class.

3. Partition Datastores:
• After creating the datastores, you split them into separate subsets for training and testing. This ensures that the model is trained on one set of data and evaluated on another set to assess its generalization performance.

4. Import Pre-trained Model and Modify to SegNet:
• Pre-trained models, especially in deep learning, are often used as a starting point due to their learned representations. These models are imported and adapted for semantic segmentation. SegNet is one such architecture specifically designed for semantic segmentation.
• The pre-trained model's architecture is adjusted to suit the task of semantic segmentation. This typically involves modifying the last few layers of the network to output pixel-wise class predictions instead of some other task like image classification.

5. Train and Evaluate:
• With the modified model and partitioned data, the training process begins. During training, the model learns to predict the correct class labels for each pixel in the input images.
• After training, the model's performance is evaluated using the test data. This evaluation usually involves metrics such as pixel accuracy, mean Intersection over Union (IoU), or F1-score to quantify how well the model performs at segmenting objects in unseen images.

Lecture 3: Artificial Neural Network

Vocabulories

1. 神经网络直觉 (Neural networks intuition)

2. 随机梯度下降 (Stochastic gradient descent)

3. 人工神经元 (Artificial neuron)

4. 感知器 (Perceptron)

5. Sigmoid 神经元 (Sigmoid neuron)

6. 做出相当微妙的决定 (Make quite subtle decisions)

7. 更抽象的层面 (More abstract level)

8. 复杂的决策制定 (Sophisticated decision making)

9. 偏差 (Bias)

10. 繁琐的 (Cumbersome)

11. 进行两次符号变更 (Make two notational changes)

12. 点乘 (Dot product)

13. 与非门 (NAND gate)

14. 明确表示乘法 (Multiplications explicit)

15. 按位求和 (Bitwise sum)

16. 代数形式 (Algebraic form)

17. 灰度图像(greyscale image)

18. 最终的理由是经验主义(The ultimate justification is empirical)

1. What is ANN, CNN and DNN in this course

1. ANN: Basic neural network model that simulates the human brain's neuron connections. It's composed of interconnected nodes, used for tasks like classification and regression.

2. CNN: Specialized for grid-like data like images, it consists of convolutional and pooling layers. Great for computer vision tasks due to its ability to extract features efficiently.

3. DNN: Refers to neural networks with multiple hidden layers. It's a broader term encompassing various architectures, including CNN. DNNs can learn complex features for various tasks.

2. What is Perceptron, Sigmoid, and MLP

1. Perceptron:
• Perceptron is one of the simplest artificial neuron models, proposed by American psychologist Frank Rosenblatt in 1957.
• The perceptron takes inputs and produces an output, which is the result of the weighted sum of inputs passed through a threshold function (such as a step function).

2. Sigmoid Function:
• The sigmoid function is a commonly used activation function, typically employed in the hidden layers and output layer of neural networks.
• Its output ranges from 0 to 1, exhibiting continuity and differentiability, enabling effective parameter updates using optimization algorithms like gradient descent.
• The sigmoid function can map the output of perceptrons to the (0, 1) range, providing a nonlinear transformation.

3. MLP (Multi-Layer Perceptron):
• MLP is a neural network structure consisting of multiple layers of neurons, typically including input layer, multiple hidden layers, and output layer.
• In MLP, each neuron is connected to all neurons in the previous layer, and each connection has a weight.
• The sigmoid function is commonly used as the activation function in the hidden layers and output layer of MLP, enabling the network to learn nonlinear features and relationships.

3. Perceptrons

4. Sigmoid Neuron

5. Multilayer Perceptrons

6. What’s in the input layer

Q & A From Lecture 3

1. Sigmoid: It squashes the output to a range between 0 and 1. It's useful in binary classification problems where the output needs to be interpreted as a probability. But because its derivative is between0-0.25, in the backpropagation processing, when the layers are too many，it's easy to fail named as Vanishing Gradient, and the intial input can hardly affect the final output.

2. Hyperbolic Tangent (tanh): Similar to the sigmoid function, but squashes the output to a range between -1 and 1. Prefer to be used more than Sigmoid, It tends to make the distribution of outputs more centered around zero.

3. Rectified Linear Unit (ReLU): It returns zero for negative inputs and linearly increases for positive inputs. It's one of the most popular activation functions because of its simplicity and effectiveness in training deep neural networks.

4. Leaky ReLU: Similar to ReLU, but it allows a small, positive gradient for negative inputs, which helps with the vanishing gradient problem. For the negative inputs, it simply get the absolute value and multiply by a tiny value like 0.01.

ANN Toolbox Examples:

Wine Classification

1. dataset preparation: The 178 wine samples are automatically divided into training, validation and test sets.

2. The input X is a 13 X 178 matrix, it has 178 columns as 178 wine samples, 13 rows as 13 elements, I guess these 13 elements are the features of the wines, like nose, palate, finish, color, spicy level, aroma, … ,. The value I guess, for example, for nose, it may be 1 - 5, representing 5 levels. This need expert information or so called auxiliary information.

3. The output T is 3 X 178 matrix, it has 178 columns as results of 178 wine samples, 3 rows meaning which winery from the 3 wineries the wine belong to. The value is 1 if it belongs to the winery, and 0 if not.

4. The neural network has a single hidden layer of 10 neurons, it means:
• there are 10 sets of “13 weight values and 13 bias values” from the input to the hidden layer; each set of the “13 w and 13 b” can be different.
• there is 1 set of “10 weight values and 10 bias values” from the hidden layer to the output

5. Training and validation; 这个怎么运行的？？？等能上matlab了试一下

6. performance evaluation:
1. PLOTPERFORM没搞懂，需要matlab上跑一下
2. plotconfusion(testT,testY)
3. plotroc(testT,testY)

Cancer Detection

The general neural network design process:

1. Collect Data: Gather relevant data that will be used to train and test the neural network. This data should be representative of the problem domain and should include both input features and corresponding target labels or outcomes.

2. Create the Network: Design the architecture of the neural network, including the number of layers, the types of layers (e.g., fully connected layers, convolutional layers, recurrent layers), and the activation functions used in each layer. This step involves deciding on the overall structure of the network based on the characteristics of the data and the complexity of the problem.

3. Configure the Network: Specify the hyperparameters of the neural network, such as the learning rate, batch size, optimization algorithm, and regularization techniques. These hyperparameters control the training process and can significantly impact the performance of the network.

4. Initialize the Weights and Biases: Initialize the weights and biases of the network with small random values or using predefined initialization techniques. Proper initialization of these parameters is crucial for ensuring that the network can learn effectively during training.

5. Train the Network: Train the neural network using the training data collected in step 1. During training, the network iteratively adjusts its weights and biases based on the input-output pairs in the training data to minimize a chosen loss function. This process typically involves forward propagation to compute predictions, backpropagation to compute gradients, and optimization algorithms to update parameters.

6. Validate the Network (Post-training Analysis): Evaluate the performance of the trained network on a separate validation dataset to assess its generalization ability. This step involves analyzing metrics such as accuracy, loss, precision, recall, and F1-score to determine how well the network performs on unseen data and whether it has overfit or underfit the training data.

7. Use the Network: Once the network has been trained and validated, it can be deployed for making predictions or performing tasks on new, unseen data. The trained network can be integrated into applications, systems, or pipelines to provide solutions to real-world problems. Additionally, the network's performance can be continuously monitored and improved over time as more data becomes available or as the problem requirements change.

Wine Classification with Pre-defined Neural Network

Lecture 4: Convolution Neural Network

Layers in CNN

• Convolution puts the input images through a set of convolutional filters, each of which activates certain features from the images.

• Rectified linear unit (ReLU) allows for faster and more effective training by mapping negative values to zero and maintaining positive values. This is sometimes referred to as activation, because only the activated features are carried forward into the next layer.

• Pooling simplifies the output by performing nonlinear downsampling, reducing the number of parameters that the network needs to learn.

• Unlike a traditional neural network, a CNN has shared weights and bias values, which are the same for all hidden neurons in a given layer.

• This makes the network tolerant to translation of objects in an image. For example, a network trained to recognize cars will be able to do so wherever the car is in the image.

R-CNN (Region-based CNN) is a two-stage detection algorithm:

• The first stage identifies a subset of regions in an image that might contain an object.

• The second stage classifies the object in each region.

• Faster R-CNN detector adds a region proposal network (RPN) to generate region proposals directly in the net instead of using an external algorithm like Edge Boxes

SSD: Single Shot MultiBox Detector

• SSD is similar to the Faster R-CNN and simultaneously produces a score for each object in each box.

• SSD skips the proposal step and predicts bounding boxes & confidences scores for multiple classes.

• SSD uses default bounding boxes of different aspect ratios on each location from multiple feature maps.

YOLO

• YOLO sees the entire image during training and testing time so that it encodes contextual information of all classes.

• YOLO predicts what objects present and where they are.

• A single convolutional network simultaneously predicts multiple bounding boxes and class probabilities for those boxes.

Instance Segmentation

Lecture 5

Vocabulary

1. residual connections: 剩余链接，Resnet中也称为shortcut connections

2. saturated：饱和的

3. degradation problem：降解问题

Degradation Problem

DenseNets

• To preserve the feedforward nature, DenseNets use direct connections from any layer to all subsequent layers.

• DenseNets alleviate the vanishing gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters

• blocks: Each layer takes all preceding feature maps as input

ResNets

• ResNets are easy to be optimized for degradation problem

• ResNets can easily enjoy accuracy gains from greatly increased depth.

• Insert shortcut connections to convert a plain network to ResNet.

• The identity shortcuts can be directly used if the input and output have the same dimensions.

Matlab Deep Neural Network

• AlexNet is a convolutional neural network that is trained based on more than a million images from the ImageNet

• AlexNet is eight layers deep and can classify images into 1,000 object classes, such as keyboard, mouse, pencil, and many animals.

• VGG-19 is a convolutional neural network that is trained based on more than a million images from the ImageNet

• GoogLeNet is a pretrained convolutional neural network that is 22 layers deep

• Inception-v3 is a CNN network that is trained based on more than a million images from the ImageNet

Siamese Neural Network

• pairwise similarity: loss function = sim(x1-x2) - 1, two same class’ similarity is 1.

• Triplet loss: anchor - positive, the smaller the better; anchor - negative the bigger the better;

Supervised Learning

• Test samples are never seen during the training

• Test samples are from the known class of the training

Few-Shot Learning

• Test samples are never seen during the training

• Test samples are not from the known class of the training

• The class of the ‘query set’ is in the ‘support set’

• k-way means class number in support set, accuracy decreases when k increase

• n-shot means sample number in the class, 0 shot, 1 shot, 2 shot; accuracy increase when n increase

Lecture 6

In-context learning

Multi-domain Few-shot Learning:

Lecture 7 LSTM

1. Capabilities of LSTM Networks:

1. Learning Long-Term Dependencies: Unlike standard RNNs, which struggle with vanishing and exploding gradients, LSTMs are designed to remember information for long durations. This makes them excellent at learning from data where past information is key to understanding the future.

2. Handling Sequences of Variable Length: LSTMs can process input sequences of varying lengths, which is beneficial for many types of data such as text, speech, and time series.

3. Working With Time-Series Data: They can predict future values in a sequence, making them ideal for forecasting tasks in finance, weather, and other fields.

4. Sequence Classification: LSTMs can classify entire sequences into categories. This is useful in sentiment analysis, video classification, and other applications where the entire input sequence is used to determine a single output.

5. Sequence Generation: They can generate new sequences that are similar to those they have been trained on. This ability is used in text generation, music composition, and chatbots.

6. Sequence to Sequence Learning (Seq2Seq): LSTMs can be used for tasks that require mapping input sequences to output sequences. This is common in machine translation, speech recognition, and other natural language processing tasks.

2. Common Applications of LSTM Networks:

1. Natural Language Processing (NLP):
• Text Generation: Generating coherent and contextually relevant text based on a given input.
• Machine Translation: Translating text or speech from one language to another.
• Sentiment Analysis: Determining the sentiment expressed in a piece of text.

2. Speech Recognition: Transcribing spoken language into text. LSTMs can handle the temporal dynamics of speech which makes them suitable for this task.

3. Music Generation: Creating new pieces of music that mimic the style of the training data.

4. Financial Forecasting:
• Predicting stock prices, market trends, and economic indicators based on historical data.
• Risk assessment and anomaly detection in financial transactions.

5. Healthcare:
• Patient monitoring systems that predict changes in patient condition based on medical records.
• Analysis of medical imaging data.

6. Video Processing:
• Action recognition in videos where understanding the sequence of frames is crucial.
• Anomaly detection in surveillance video.

3. LSTM Practice: Sequence Classification Using Deep Learning https://au.mathworks.com/help/deeplearning/ug/classify-sequence-data-using-lstm-networks.html

1. Why Use LSTM to Classify Waveforms

• Retain information from earlier in the sequence, which is essential for recognizing entire patterns.

• Handle variations in the length of sequences effectively, a common characteristic in waveform data.

2. Necessity of Data Sorting

• Minimize Padding: Sorting reduces the variability of sequence lengths within each batch, minimizing the amount of padding required and hence reducing computational waste.

• Enhance Computational Efficiency: With less padding, each training step is more efficient, as the network spends less time processing zero-padded data.

3. Mini-Batch and Epoch Explanation

• Mini-Batch: Refers to processing a subset of the training data at one go. This approach balances the training speed and memory efficiency, and can help the model generalize better by providing a stochastic estimation of the gradient.

• Epoch: Refers to one complete pass through the entire training dataset, including all batches.. Using multiple epochs allows the model to learn incrementally, refining its weights as it sees the data multiple times.

4. Choosing Different Output Modes

• "last": Used when only the final state is needed for a prediction, such as in sentiment analysis or any other form where the entire sequence is summarized into a single output.

• "sequence": Used when an output is required for each time step of the input, like in time-series forecasting or tagging tasks (e.g., part-of-speech tagging).

• "none": This might be used in complex architectures where the LSTM outputs are not directly used but are passed through other transformations or layers.

5. When to Use BiLSTM vs. LSTM

• BiLSTM (Bidirectional LSTM): Should be used when the context from both past and future is important for understanding the sequence at any point. This is particularly useful in tasks like speech recognition or text processing where the subsequent inputs can provide relevant context for interpreting the current input.

• LSTM: More suited when only the previous context is necessary, or when the future input should not influence the current output. This is typical in real-time applications such as live audio processing or online prediction tasks where future inputs are not available.

4. LSTM Practice: Classify Videos Using Deep Learning

1. Which Layer should be used to extract features:

• Convolutional Neural Networks (CNNs): For image and video data, features from different layers of a CNN capture various aspects of the data:
• Early Layers: Typically capture basic features such as edges, colors, and textures. These features are more generic.
• Middle Layers: Start to assemble more complex features that represent parts of objects in images, such as corners or specific textures.
• Deep Layers: Capture high-level features that often correspond to specific objects or scenes. These features are more abstract and task-specific.

• Simple Tasks: For less complex tasks or when the objects of interest are common and not deeply embedded within the scene, features from earlier or middle layers may suffice as they provide more generic visual descriptors.

• Complex Tasks: For more complex tasks, such as distinguishing between highly similar categories or detailed scene analysis, deeper layers are preferable as they encode more abstract and detailed aspects of the input data.

• Empirical Testing: Often, the best way to determine the most effective layer for feature extraction is through empirical testing. This involves experimenting with features from different layers and evaluating their impact on the performance of the specific task.

• Cross-Validation: Use cross-validation to assess the robustness of the chosen features across different subsets of your data to ensure generalizability.

• Fully Connected Layers: In many networks, fully connected layers (towards the end of the network) are used to make final predictions and may represent too specific features that are closely tailored to the original training task of the network.

• Pooling Layers: Features from pooling layers are often more robust and invariant to the exact positioning of features within the image, which can be useful for tasks requiring recognition of objects regardless of their specific location in the image.

• Model-Specific Literature: Look for academic papers, benchmarks, and community forums discussing the specific model you are using. Insights from these sources can provide valuable information on which layers have been found most useful for tasks similar to yours.

• Frameworks and Tools: Use tools and frameworks that allow for easy experimentation with different layers. Many deep learning frameworks, like TensorFlow and PyTorch, provide utilities to access intermediate layers of pre-trained models effortlessly.

• Image Classification: Deeper convolutional layers (just before the fully connected layers) might be best as they capture the essence of the images that are crucial for distinguishing between complex categories.

• Object Detection and Localization: Features from the last convolutional layers are often used because they provide a good balance between high-level semantics and detailed spatial information, which is essential for locating objects.

2. How to use temp dir and data

Explanation of Key Parts:

• Temporary File Storage: The tempFile is set to a path in the tempdir, which is the system's designated temporary directory. This ensures that the data does not permanently occupy disk space and can be cleared easily.

• Existence Check: The script checks if the file exists using exist(tempFile, 'file'). If it does, it loads the data directly, skipping the processing steps, which saves time especially when working with large datasets.

• Data Processing and Saving: If the data does not exist, the videos are read, cropped, processed through the neural network to extract features, and the resulting data is saved in the temporary file for future use.

• Usage: After loading or processing, the sequences data, which contains feature vectors for each video, is ready for further analysis or training tasks.

3. Assemble CNN and the LSTM

1. Sequence Input: A sequence of video frames enters the network.

2. Initial CNN Processing: Early CNN layers process each frame for basic feature extraction.

3. Sequence Folding: These features are then "folded," allowing deeper CNN layers (which don’t handle sequences) to process each frame's features as if they were part of a batch of unrelated images.

4. Further CNN Processing: Additional CNN layers apply more complex transformations to these features.

5. Sequence Unfolding: The unfolded operation is then performed to rearrange these features back into a sequence format.

6. LSTM Processing: Finally, LSTM layers analyze the sequence of features to capture and interpret temporal patterns and relationships.

• 目的：折叠层的作用是将一系列特征图（每一帧图像通过 CNN 层处理后的输出）转换成可以作为独立图像处理的一批特征图。基本上，它重新组织数据，暂时折叠时间维度（时间或序列顺序），以便 CNN 层可以像处理单独图像一样处理每一帧。

• 过程：它接收一系列图像的输出（每个图像已通过 CNN 的初步层处理）并将这些输出重新排列成二维格式。这允许后续的 CNN 层（不原生处理序列的层）处理数据。在这一阶段，每一帧的特征都被独立处理。

• 目的：展开层通过撤销折叠层所做的变换，恢复时间维度。这种序列格式的重新建立对于 LSTM 层非常必要，因为 LSTM 需要以序列的形式处理数据，以捕捉时间关系和动态。

• 过程：它接收处理后的特征图批次（来自 CNN）并将它们重新组织回原始的序列格式。现在，序列中的每个元素代表经过 CNN 处理的一帧的特征，适合输入到 LSTM 层。