What does classifying more than 10,000 image categories tell us?

J. Deng, A. C. Berg, K. Li, and L. Fei-Fei

Introduction

Image classification has been an important problem in MIR for a long time, and there are several works proposed to solve this problem. However, as the number of images and classes increasing rapidly, these old methods gradually become unfeasible. This paper investigated some famous works, and do some comparison on different phases.

Comparison

Datasets : 

(1) ImageNet10K

(2) ImageNet7K

(3) ImageNet1K

(4) Rand200

(5) Ungulate183

(6) Fungus134

(7) Vehicle262

(8) CalNet200

Evaluation : 

(1) Mean accuracy

 (2) Mean misclassification cost

Method : 

(1) GIST+NN 

(2) BOW+NN 

(3) BOW+SVM

(4)SPM+SVM

Consideration :

(1) Computation

(2) Size

(3) Density

(4) Hierarchy

(1) Computation

They regarded that it takes several CPU years to train these classifier. They used 66 CPU cluster and parallel algorithm to do these training, but it still took several weeks to finish their experiments. 

(2) Size

Experiments showed that as the number of classes increases, the accuracy becomes lower. A technique that significantly outperforms others on small datasets may actually underperform them on large number of categories.

(3) Density

If the data is more sparse, the performance of classification is better. More sparse means the longer distance between each data.

(4) Hierarchy

The author claimed that the cost of different error type should be different. For instance, the error of classifying redshank to bird should be lower than classifying redshank to microwave. They call this "Hierarchical Cost".