Depth learning SSD learning record -

Reprinted from: http://blog.csdn.net/u010167269/article/details/52563573

Preface

This is an ECCV 2016 article this year. It is a new work by Wei Liu from UNC Chapel Hill (University of North Carolina at Chapel Hill). The paper code: https://github.com/weiliu89 /caffe/tree/ssd
Obeject detection summary link:Object Detection
Current voc 2012 list:

Abstract

This article guarantees both speed and accuracy Under the circumstances, the SSD object detection model is proposed, which, like the current popular detection model, integrates the detection process into a single deep neural network. It is convenient for training and optimization, and at the same time improves the detection speed. SSD will output a series of bounding boxes of discretization, these bounding boxes are in different levels ( layers) are generated on the feature maps and have different aspect ratio.

At the prediction stage:

To calculate each An object in default box, the possibility of belonging to each category, namely score ,Score. For example, for the PASCAL VOC data set, there are a total of 20 categories, then the possibility of objects in each bounding box belonging to each of these 20 categories is obtained.
At the same time, the shape of these bounding boxes should be fine-tuned to make it fit the bounding rectangle of the object.
Also, in order to deal with different sizes of the same object, SSD combines the predictions of feature maps of different resolutions.

Compared to those detection models that require object proposals, the SSD method in this article completely cancels proposals generation, pixel resampling or feature resampling these stages. This makes it easier for SSDs to optimize training and to integrate the detection model into the system.

In PASCAL VOC, MS COCO, Experiments on the ILSVRC data set show that while ensuring accuracy, SSD is much faster than using region proposals.

SSD compared to other single structure Model (YOLO), SSD achieves higher accuracy, that is, when the input image is small. For example, input the size of the PASCAL VOC 2007 test image, on Titan X, the SSD at the rate of 58 frames, at the same time obtained mAP.

If the input image is , SSD has achieved a mAP, which is higher than the current The most state-of-art Faster R-CNN is much better.

Introduction

The state-of-art detection system that is popular in cash generally has the following steps, First generate some hypothetical bounding boxes, then extract features from these bounding boxes, and then go through a classifier to determine whether it is an object or what it is.

This type of pipeline has been since IJCV 2013, Selective Search for Object Recognition started, and now it has achieved the leading Faster R-CNN-based ResNet on the PASCAL VOC, MS COCO, and ILSVRC data sets. However, this kind of method requires too long calculation time for embedded systems, and it is not enough for real-time detection. Of course, there is also a lot of work towards real-time detection , but so far, detection accuracy has been sacrificed in exchange for time.

The real-time detection method proposed in this article, Eliminate the middle the process of bounding boxes, pixel or feature resampling. Although this article is not the first article to do so (YOLO), this article has done some improvement work to ensure both the speed and the detection accuracy.

There is a very crucial sentence , Which basically summarizes the core idea of this article:

Our improvements include using a small convolutional filter to predict object categories and offsets in bounding box locations, using separate predictors (filters) for different aspect ratio detections, and applying these filters to multiple feature maps from the later stages of a network in order to perform detection at multiple scales.

The main contributions of this article are summarized as follows:

Proposed a new object detection method: SSD, which is faster than the original fastest YOLO: You Only Look Once method. Be precise. While ensuring speed, the resulting mAP can be used with the region proposals technical method ( Such as Faster R-CNN) is comparable.
The core of the SSD method is predict object (object), and its score (score) belonging to the category; at the same time, use a small convolution kernel on the feature map to predict box offsets of a series of bounding boxes.
In this article, in order to obtain high-precision detection results, predict objectobject, box offsets, at the same time, different predictions of aspect ratio are also obtained.
These improved designs in this article can ensure the accuracy of detection when inputting images with lower resolution. At the same time, this overall end-to-end design makes training easier. A better trade-off is achieved between detection speed and detection accuracy.
The model proposed in this article has been tested on different data sets, such as PASCAL VOC, MS COCO, and ILSVRC. The detection time (timing) and detection accuracy (accuracy) are compared with the current state-of-art detection methods in the field of object detection.

The Single Shot Detector (SSD)

This part is detailed Explained the SSD object detection framework and SSD training methods.

Here, first figure out what follows What do you mean by default box and feature map cell. See the picture below:

feature map cell is to divide the feature map into or One by one after Grid;
And the default box is on each grid, A series of fixed-size boxes, that is, a series of boxes formed by the dotted line in the figure.

Model

SSD is based on a forward propagation CNN network, which produces a series of fixed-size The bounding boxes, and the possibility of object instances contained in each box, namely score. After that, perform a non-maximum suppression (Non-maximum suppression) to get the final predictions.

The very beginning part of the SSD model, This article is called base network, which is a standard framework for image classification. After the base network, this article adds an additional auxiliary network structure:

Multi-scale feature maps for detection
Add additional volumes after the basic network structure Layers, the size of these convolutional layers is decreasing layer by layer, and predictions can be performed at multiple scales.
Convolutional predictors for detection
Each feature layer added (or in the basic network structure A series of convolutional filters can be used to generate a series of predictions of fixed size, see Fig.2 for details. For a size of , with a channel of In the feature layer, the convolutional filters used are the kernels of . The generated predictions are then a score of the attribution category, or the shape offsets relative to the default box coordinate.
In each feature map position of , use the above The kernel will produce an output value. The bounding box offset value is the relative distance between the output default box and the feature map location at this time (the YOLO architecture uses a fully connected layer instead of the convolutional layer here).
Default boxes and aspect ratios
The ratio of each box to its corresponding feature map cell The location is fixed. In each feature map cell, we have to predict the the obtained box and the default box offsets and the score of the object contained in each box (the probability of each category must be calculated).
Therefore, for each box in in a position, we need to calculate A category, the score of each category, and the box relative to its default box 4 offsets (offsets). Therefore, on each feature map cell in the feature map, there needs to be a one filters. For a feature map of size ,

The default box here is very similar to the Anchor boxes in Faster R-CNN. For the Anchor boxes here, please refer to the original paper for details. But different from Faster R-CNN, the Anchor boxes in this article are used on feature maps of different resolutions.

Training

During training, the difference between the SSD in this article and those using region proposals + pooling is that the groundtruth in the SSD training image needs to be given To those boxes with fixed output. As mentioned earlier, the output of SSD is a series of bounding boxes with a fixed size defined in advance.

The dog’s The groundtruth is a red bounding box, but when labeling, the red groundtruth box should be assigned to one of a series of fixed output boxes in Figure (c), that is, the red dashed box in Figure (c).

In fact, the article pointed out that the groundtruth boxes defined like this are not only used in this article. In YOLO, it is used in the region proposal stage of Faster R-CNN and in MultiBox.

When this kind of training image After the groundtruth corresponding to the fixed output boxes, the calculation of the loss function and the calculation of the back-propagation can be updated end-to-end.

There will be some problems in training :

Select a series of default boxes
Select the scales mentioned above
hard negative mining

< /li>
Data augmentation strategy

The following will discuss the ways of solving these problems in this article, which is divided into the following parts.

Matching strategy:

How to pair groundtruth boxes with default boxes to form a label?

At the beginning, use MultiBox In best jaccard overlap to match each ground truth box with the default box, so as to ensure that each ground truth box corresponds to a unique default box.

but different from MultiBox, this article Then pair the default box with any groundtruth box, as long as the jaccard overlap between the two is greater than a threshold, here the threshold is 0.5.

Training objective:

SSD training objective function (training objective) is derived from the objective function of MultiBox, but this article expands it so that it can handle multiple target categories. Use to indicate the th a default box and category matches the th ground truth box, otherwise If it does not match, then .

According to the above matching strategy, we must Yes , which means that for the th A ground truth box, there may be multiple default box matching it.

Total objective loss function (objective loss function) is weighted by localization loss(loc) and confidence loss(conf) Sum:

where:

is the number of default boxes matching the ground truth box
localization loss(loc) It is the Smooth L1 Loss in Fast R-CNN, used in predict box() and ground truth box() parameters (That is, the center coordinate position, width, height), return to the center position of bounding boxes, as well as width and height
confidence loss(conf) is Softmax Loss, input is the confidence level of each category

< li>

Weight item , set to 1

Choosing scales and aspect ratios for default boxes:

most CNN net The deeper the network, the smaller the size of the feature map. This is not only to reduce the calculation and memory requirements, but also has the advantage that the finally extracted feature map will have a certain degree of translation and Scale invariance.

At the same time for processing Objects of different scales, some articles, such as ICLR 2014, Overfeat: Integrated recognition, localization and detection using convolutional networks, and ECCV 2014, Spatial pyramid pooling in deep convolutional networks for visual recognition, they convert images into different scales, These images are processed independently through the CNN network, and then the image results of these different scales are integrated.

But actually, If you use feature maps on different layers in the same network, you can also achieve the same effect, at the same time on all objects Shared parameters in the scale.

Previous work, such as CVPR 2015 , Fully convolutional networks for semantic segmentation, as well as CVPR 2015, Hypercolumns for object segmentation and fine-grained localization, use the layers in front of CNN to improve the effect of image segmentation, because the lower the layers, the more image details are retained. The article ICLR 2016, ParseNet: Looking wider to see better also proves that the above ideas are feasible.

Therefore, this article also uses lower feature Maps, upper feature maps are used to predict detections. The following figure shows the feature map of two different scales used in this article, the feature map of and the ’s feature map:

Generally speaking, different layers in a CNN network have different sizes of receptive fields. The receptive field here refers to a node on the output feature map, which corresponds to the size of the input image. For the calculation of the specific receptive field, see two blogs:

http://www.voidcn.com/article/p-biogqgvx-hy.html
http://blog.cvmarcher.com/posts/2015/05/ 17/cnn-trick/

Fortunately, in the SSD structure, default boxes do not necessarily correspond to the receptive fields of each layer. In the design of this article, the specific position in the feature map is responsible for the specific area in the image and the specific size of the object. Join us and use feature maps to do predictions. The size of the default box in each feature map is calculated as follows:

where, Take Value , Take Value , which means that the scale of the lowest layer is , the scale of the highest level is , then use a default boxe with a different aspect ratio s, use to mean: , the width and height of each default box can be calculated:

For the aspect ratio of 1, this article also adds a default box, the scale of this box is . So in the end, there are 6 default boxes on each feature map location.

The center of each default box is set to: , where is the size of the feature map, and at the same time , .

On the combination of feature maps, all The default boxes of different scales and different aspect ratios are after their predicted predictions. As you can imagine, we have many predictions, including different sizes and shapes of objects. As shown in the figure below, the ground truth box of the dog is consistent with the red box in the feature map, so the rest of the boxes are regarded as negative samples.

Hard negative mining

在生成一系列的predictions 之后，会产生很多个符合ground truth box 的predictions boxes，但同时，不符合ground truth boxes 也很多，而且这个negative boxes，远多于positive boxes。这会造成 negative boxes、positive boxes 之间的不均衡。训练时难以收敛。

因此，本文采取，先将每一个物体位置上对应 predictions（default boxes）是 negative 的 boxes 进行排序，按照 default boxes 的 confidence 的大小。选择最高的几个，保证最后 negatives、positives 的比例在。

本文通过实验发现，这样的比例可以更快的优化，训练也更稳定。

Data augmentation

本文同时对训练数据做了 data augmentation，数据增广。关于数据增广，推荐一篇文章：Must Know Tips/Tricks in Deep Neural Networks，其中的 section 1 就讲了 data augmentation 技术。

每一张训练图像，随机的进行如下几种选择：

使用原始的图像
采样一个 patch，与物体之间最小的 jaccard overlap 为：，，，与
随机的采样一个 patch

采样的 patch 是原始图像大小比例是，aspect ratio 在与之间。

当 groundtruth box 的中心（center）在采样的 patch 中时，我们保留重叠部分。

在这些采样步骤之后，每一个采样的 patch 被 resize 到固定的大小，并且以的概率随机的水平翻转（horizontally flipped）

Experimental Results

Base network and hole filling algorithm

本文的 Base network 是基于 ICLR 2015, VGG16 来做的，在 ILSVRC CLS-LOC 数据集上进行了预训练。

与 ICLR 2015, DeepLab-LargeFOV 的工作类似，本文将 VGG 中的 FC6 layer、FC7 layer 转成为卷积层，并从模型的 FC6、FC7 上的参数，进行采样得到这两个卷积层的 parameters。

还将 Pool5 layer 的参数，从转变成，外加一个 pad（1），如下图：

但是这样变化后，会改变感受野（receptive field）的大小。因此，采用了 atrous algorithm 的技术，这里所谓的 atrous algorithm ，我查阅了资料，就是 hole filling algorithm 。

在 DeepLab 的主页上：http://liangchiehchen.com/projects/DeepLab.html，有一张如下的图：

博客 1：http://www.cnblogs.com/jianyingzhou/p/5386222.html

最早用的就是 deeplab 的文章了，Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFS 这篇文章和 fcn 不同的是，在最后产生 score map 时，不是进行upsampling，而是采用了 hole algorithm，就是在 pool4 和 pool 5层，步长由 2 变成 1，必然输出的 score map 变大了，但是 receptive field 也变小了，为了不降低 receptive field，怎么做呢？利用 hole algorithm，将卷积 weights 膨胀扩大，即原来卷积核是 3×3，膨胀后，可能变成 7×7 了，这样 receptive field 变大了，而 score map 也很大，即输出变成 dense 的了。

这么做的好处是，输出的 score map 变大了，即是 dense 的输出了，而且 receptive field 不会变小，而且可以变大。这对做分割、检测等工作非常重要。

博客 2：http://www.voidcn.com/article/p-qsojqref-vy.html

既想利用已经训练好的模型进行 fine-tuning，又想改变网络结构得到更加 dense 的 score map.

这个解决办法就是采用 Hole 算法。如下图 (a) (b) 所示，在以往的卷积或者 pooling 中，一个 filter 中相邻的权重作用在 feature map 上的位置都是物理上连续的。如下图 (c) 所示，为了保证感受野不发生变化，某一层的 stride 由 2 变为 1 以后，后面的层需要采用 hole 算法，具体来讲就是将连续的连接关系是根据 hole size 大小变成 skip 连接的（图 (c) 为了显示方便直接画在本层上了）。不要被 (c) 中的 padding 为 2 吓着了，其实 2 个 padding 不会同时和一个 filter 相连。

pool4 的 stride 由 2 变为 1，则紧接着的 conv5_1, conv5_2 和 conv5_3 中 hole size 为 2。接着 pool5 由 2 变为 1 , 则后面的 fc6 中 hole size 为 4。

本文还将 fully convolutional reduced (atrous) VGGNet 中的所有的 dropout layers、fc8 layer 移除掉了。

本文在 fine-tuning 预训练的 VGG model 时，初始 learning rate 为，momentum 为，weight decay 为，batch size 为，learning rate decay 的策略随数据集的不同而变化。

PASCAL VOC 2007

在这个数据集中，与 Fast R-CNN、Faster R-CNN 进行了比较，几种检测网络都用相同的训练数据集，以及预训练模型（VGG16）。

本文训练图像是 VOC 2007 train + VOC 2007 validation + VOC 2012 train + VOC 2012 validation，共计 16551 张图像；

测试集选取的是 VOC 2007 test，共计 4952 张图像。

下图展示了 SSD300 model 的结构：

我们用 conv4_3，conv7（原先的 FC7），conv8_2，conv9_2，conv10_2，以及 pool11，这些 layer 来predict location、 confidence。

在 VGG16 上新加的 convolutional layers，其参数初始化都用 JMLR 2010, Understanding the difficulty of training deep feedforward neural networks 提出的 xavier 方法。

因为 conv4_3 的尺寸比较大，size 为的大小，我们只在上面放置 3 个 default boxes，一个 box 的 scale 为，另外两个 boxes 的 aspect ratio 分别为、。但对于其他的用来做 predictions 的 layers，本文都放了个 default boxes。

文献 ICLR 2016, ParseNet: Looking wider to see better 指出，conv4_3 相比较于其他的 layers，有着不同的 feature scale，我们使用 ParseNet 中的 L2 normalization 技术将 conv4_3 feature map 中每一个位置的 feature norm scale 到 20，并且在 back-propagation 中学习这个 scale。

在最开始的 40K 次迭代中，本文使用的 learning rate 是，之后将其减小到，再接着迭代 20K 次。

下面 Table 1 显示了，我们的 SSD300 model 的精度已经超过了 Fast R-CNN，当我们用 SSD 在更大的图像尺寸上，训练得到的 model，甚至要比 Faster R-CNN 还要高出的 mAP。

为了更细节的了解本文的两个 SSD model，我们使用了 ECCV 2012, Diagnosing error in object detectors 的检测分析工具。下图 Figure 3 显示了 SSD 可以高质量的检测不同种类的物体。

下图 Figure 4 展示了 SSD 模型对 bounding box 的 size 非常的敏感。也就是说，SSD 对小物体目标较为敏感，在检测小物体目标上表现较差。其实这也算情理之中，因为对于小目标而言，经过多层卷积之后，就没剩多少信息了。虽然提高输入图像的 size 可以提高对小目标的检测效果，但是对于小目标检测问题，还是有很多提升空间的。

同时，积极的看，SSD 对大目标检测效果非常好。同时，因为本文使用了不同 aspect ratios 的 default boxes，SSD 对于不同 aspect ratios 的物体检测效果也很好。

Model analysis

为了更好的理解 SSD，本文还使用控制变量法来验证 SSD 中的每一部分对最终结果性能的影响。测试如下表 Table 2 所示：

从上表可以看出一下几点：

数据增广（Data augmentation）对于结果的提升非常明显
Fast R-CNN 与 Faster R-CNN 使用原始图像，以及 0.5 的概率对原始图像进行水平翻转（horizontal flip），进行训练。如上面写的，本文还使用了额外的 sampling 策略，YOLO 中还使用了亮度扭曲（photometric distortions），但是本文中没有使用。
做了数据增广，将 mAP 从提升到了，提升了。
我们还不清楚，本文的 sampling 策略会对 Fast R-CNN、Faster R-CNN 有多少好处。但是估计不会很多，因为 Fast R-CNN、Faster R-CNN 使用了 feature pooling，这比人为的对数据进行增广扩充，还要更 robust。
使用更多的 feature maps 对结果提升更大
类似于 FCN，使用含图像信息更多的低 layer 来提升图像分割效果。我们也使用了 lower layer feature maps 来进行 predict bounding boxes。
我们比较了，当 SSD 不使用 conv4_3 来 predict boxes 的结果。当不使用 conv4_3，mAP 下降到了。
可以看见，低层的 feature map 蕴含更多的信息，对于图像分割、物体检测性能提升帮助很大的。
使用更多的 default boxes，结果也越好
如 Table 2 所示，SSD 中我们默认使用 6 个 default boxes（除了 conv4_3 因为大小问题使用了 3 个 default boxes）。如果将 aspect ratios 为、的 boxes 移除，performance 下降了。如果再进一步的，将、的 default boxes 移除，那么 performance 下降了近。
Atrous 使得 SSD 又好又快
如前面所描述，我们根据 ICLR 2015, DeepLab-LargeFOV，使用结合 atrous algorithm 的 VGG16 版本。
如果我们使用原始的 VGG16 版本，即保留 pool5 的参数为：，且不从 FC6，FC7 上采集 parameters，同时添加 conv5_3 来做 prediction，结果反而会下降。同时最关键的，速度慢了。

PASCAL VOC 2012

本文又在 VOC 2012 test 上进行的实验，比较结果如下：

MS COCO

为了进一步的验证本文的 SSD 模型，我们将 SSD300、SSD500 在 MS COCO 数据集上进行训练检测。

因为 COCO 数据集中的检测目标更小，我们在所有的 layers 上，使用更小的 default boxes。

这里，还跟 ION 检测方法进行了比较。

总的结果如下：

Inference time

本文的方法一开始会生成大量的 bounding boxes，所以有必要用 Non-maximum suppression（NMS）来去除大量重复的 boxes。

通过设置 confidence 的阈值为，我们可以过滤掉大多数的 boxes。

之后，我们再用 Thrust CUDA library 进行排序，用 GPU 版本的实现来计算剩下的 boxes 两两之间的 overlap。然后，进行 NMS，每一张图像保留 top 200 detections。这一步 SSD300 在 VOC 20 类的每张图像上，需要耗时。

下面是在 PASCAL VOC 2007 test 上的速度统计：

Related work and result images

这篇文章居然把相关工作总结放在最后面，我还是第一次见到。

具体的看原文吧。

最后放几张结果图：

转自:http://blog.csdn.net/u010167269/article/details/52563573

Preface

这是今年 ECCV 2016 的一篇文章，是 UNC Chapel Hill（北卡罗来纳大学教堂山分校）的 Wei Liu 大神的新作，论文代码：https://github.com/weiliu89/caffe/tree/ssd
Obeject detection summary link: Object Detection
目前 voc 2012 的榜单:

Abs tract

这篇文章在既保证速度，又要保证精度的情况下，提出了 SSD 物体检测模型，与现在流行的检测模型一样，将检测过程整个成一个 single deep neural network。便于训练与优化，同时提高检测速度。 SSD 将输出一系列离散化（discretization）的 bounding boxes，这些 bounding boxes 是在不同层次（layers）上的 feature maps 上生成的，并且有着不同的 aspect ratio。

在 prediction 阶段：

要计算出每一个 default box 中的物体，其属于每个类别的可能性，即 score，得分。如对于 PASCAL VOC 数据集，总共有 20 类，那么得出每一个 bounding box 中物体属于这 20 个类别的每一种的可能性。
同时，要对这些 bounding boxes 的 shape 进行微调，以使得其符合物体的外接矩形。
还有就是，为了处理相同物体的不同尺寸的情况，SSD 结合了不同分辨率的 feature maps 的 predictions。

相对于那些需要 object proposals 的检测模型，本文的 SSD 方法完全取消了 proposals generation、pixel resampling 或者 feature resampling 这些阶段。这样使得 SSD 更容易去优化训练，也更容易地将检测模型融合进系统之中。

在 PASCAL VOC、MS COCO、ILSVRC 数据集上的实验显示，SSD 在保证精度的同时，其速度要比用 region proposals 的方法要快很多。

SSD 相比较于其他单结构模型（YOLO），SSD 取得更高的精度，即是是在输入图像较小的情况下。如输入大小的 PASCAL VOC 2007 test 图像，在 Titan X 上，SSD 以 58 帧的速率，同时取得了的 mAP。

如果输入的图像是，SSD 则取得了的 mAP，比目前最 state-of-art 的 Faster R-CNN 要好很多。

Introduction

现金流行的 state-of-art 的检测系统大致都是如下步骤，先生成一些假设的 bounding boxes，然后在这些 bounding boxes 中提取特征，之后再经过一个分类器，来判断里面是不是物体，是什么物体。

这类 pipeline 自从 IJCV 2013, Selective Search for Object Recognition 开始，到如今在 PASCAL VOC、MS COCO、ILSVRC 数据集上取得领先的基于 Faster R-CNN 的 ResNet 。但这类方法对于嵌入式系统，所需要的计算时间太久了，不足以实时的进行检测。当然也有很多工作是朝着实时检测迈进，但目前为止，都是牺牲检测精度来换取时间。

本文提出的实时检测方法，消除了中间的 bounding boxes、pixel or feature resampling 的过程。虽然本文不是第一篇这样做的文章（YOLO），但是本文做了一些提升性的工作，既保证了速度，也保证了检测精度。

这里面有一句非常关键的话，基本概括了本文的核心思想：

Our improvements include using a small convolutional filter to predict object categories and offsets in bounding box locations, using separate predictors (filters) for different aspect ratio detections, and applying these filters to multiple feature maps from the later stages of a network in order to perform detection at multiple scales.

本文的主要贡献总结如下：

提出了新的物体检测方法：SSD，比原先最快的 YOLO: You Only Look Once 方法，还要快，还要精确。保证速度的同时，其结果的 mAP 可与使用 region proposals 技术的方法（如 Faster R-CNN）相媲美。
SSD 方法的核心就是 predict object（物体），以及其归属类别的 score（得分）；同时，在 feature map 上使用小的卷积核，去 predict 一系列 bounding boxes 的 box offsets。
本文中为了得到高精度的检测结果，在不同层次的 feature maps 上去 predict object、box offsets，同时，还得到不同 aspect ratio 的 predictions。
本文的这些改进设计，能够在当输入分辨率较低的图像时，保证检测的精度。同时，这个整体 end-to-end 的设计，训练也变得简单。在检测速度、检测精度之间取得较好的 trade-off。
本文提出的模型（model）在不同的数据集上，如 PASCAL VOC、MS COCO、ILSVRC，都进行了测试。在检测时间（timing）、检测精度（accuracy）上，均与目前物体检测领域 state-of-art 的检测方法进行了比较。