ZHAO Jian

Training a deep neural network usually requires sufficient annotated samples. The scarcity of supervision samples in practice thus becomes the major bottleneck on performance of the network. In this work, we propose a principled method to circumvent this difficulty through marginalizing all the possible transformations over samples, termed as Marginalized Convolutional Neural Network (mCNN). mCNN implicitly considers in- finitely many transformed copies of the training data in every training epoch and therefore is able to learn representations invariant for transformation in an end-to-end way. We prove that such marginalization can be understood as a classic CNN with a special form of regularization and thus is efficient for implementation. Experimental results on the MNIST and affNIST digit number datasets demonstrate that mCNN can match or outperform the original CNN with much fewer training samples. Moreover, mCNN also performs well for face recognition on the recently released largescale MS-Cele-1M dataset and outperforms stateof-the-arts. Moreover, compared with the traditional CNNs which use data augmentation to improve their performance, the computational cost of mCNN is reduced by a factor of 25.

ZHAO Jian (NUS), LI Jianshu (NUS), ZHAO Fang (NUS), YAN Shuicheng (Qihoo/360 AI Institute & NUS), and FENG Jiashi (NUS)

BMVC2017 PDF

In this paper, we consider the challenging unconstrainedset-based face recognition problem where each subject faceis instantiated by a set of media (images and videos) in-stead of a single image. Traditional face recognition ap-proaches based on single image may not perform well insuch scenarios since they do not exploit media-set informa-tion effectively. But naively aggregating information fromall the media within a set would suffer from the large intra-set variance caused by heterogeneous factors (e.g., varyingmedia modalities, poses and illuminations) and fail to learndiscriminative face representations. To address this chal-lenging problem, we propose a novel Multi-Prototype Net-work (MPNet) model that adaptively learns multiple proto-type face representations from sets of media. Each learnedprototype is representative for the subject face under certainconditions in terms of pose, illumination and media modal-ity. Instead of handcrafting the set partition for prototypelearning, MPNet introduces a new Dense SubGraph (DSG)learning sub-net that implicitly untangles inconsistent me-dia and learns a number of prototypes for unconstrainedset-based face recognition. The proposed MPNet with theDSG sub-net is end-to-end trainable. Comprehensive evalu-ations on the challenging IJB-A and large-scale MS-Celeb-1M benchmark datasets clearly demonstrate the superiorityof our proposed MPNet over state-of-the-arts.

ZHAO Jian (NUS), ZHAO Jiaojiao (Northumbria University, UK), LI Jianshu (NUS), ZHAO Fang (NUS), Jayashree Karlekar (Panasonic), FENG Jiashi (NUS), and YAN Shuicheng (Qihoo/360 AI Institute & NUS)

ICCV2017 (under review) 

Free-form textual phrase localization in images is extremely challenging when only image-level supervision is available. In this paper, we propose a novel weakly supervised localization model, namely Multi-scale Anchored Transformer Network (MATN) that can accurately localize textual phrases. Through taking region proposals of an image as localization anchors and computing multi-scale correspondence maps between a given phrase and the image spatial feature map, MATN learns to predict phrase location referring to the anchors. These anchors provide useful cues for MATN to reliably reason about the regions where objects most likely appear given only image-level supervision. MATN is trained by a novel strategy that simultaneously minimizes a contrastive reconstruction loss between different phrases from a single image and a set of triplet losses among multiple images with the similar phrases. Compared with other region proposal based methods, MATN searches for the optimal bounding box over the entire feature map instead of selecting a sub-optimal one from discrete region proposals and thus is more resistant to errors in the proposals. Besides, MATN explicitly leverages the shared knowledge across multiple images containing the similar objects and the discriminative information across different phrases from a single image in the learning process, which are absent in previous methods. We evaluate MATN on the Flickr30K Entities and ReferItGame datasets and the experimental results show that MATN significantly outperforms the state-of-the-art methods.

ZHAO Fang (NUS), LI Jianshu (NUS), ZHAO Jian (NUS), FENG Jiashi (NUS), and YAN Shuicheng (Qihoo/360 AI Institute & NUS)

ICCV2017 (under review) 

Face attribute prediction in the wild is important for many facial analysis applications yet it is very challenging due to ubiquitous face variations. In this paper, we address face attribute prediction in the wild by proposing a novel method, lAndmark Free Face AttrIbute pRediction (AFFAIR). Unlike traditional face attribute prediction methods that require facial landmark detection and face alignment, AFFAIR uses an end-to-end learning pipeline to jointly learn spatial transformations and attribute localizations that optimize facial attribute prediction with no reliance on landmark annotations or pre-trained landmark detectors. AFFAIR achieves this through simultaneously 1) learning a global transformation which effectively alleviates negative effect of global face variation for the following attribute prediction tailored for each face, 2) locating the most relevant facial part for attribute prediction and 3) aggregating the global and local features for robust attribute prediction. Within AFFAIR, a new competitive learning strategy is developed that effectively enhances global transformation learning for better attribute prediction. We show that with zero information about landmarks, AFFAIR achieves state-of-the-art performance on three face attribute prediction benchmarks, which also simultaneously learns the face-level transformation and attribute-level localization within a unified framework.

LI Jianshu (NUS), ZHAO Fang (NUS), ZHAO Jian (NUS), FENG Jiashi (NUS), YAN Shuicheng (Qihoo/360 AI Institute & NUS), and Terence Sim (NUS)

ICCV2017 (under review) 

This paper describes our proposed method targeting at the MSR Image Recognition Challenge MS-Celeb-1M. The challenge is to recognize one million celebrities from their face images captured in the real world. The challenge provides a large scale dataset crawled from the Web, which contains a large number of celebrities with many images for each subject. Given a new testing image, the challenge requires an identify for the image and the corresponding confidence score. To complete the challenge, we propose a two-stage approach consisting of data cleaning and multi-view deep representation learning. The data cleaning can effectively reduce the noise level of training data and thus improves the performance of deep learning based face recognition models. The multi-view representation learning enables the learned face representations to be more specific and discriminative. Thus the difficulties of recognizing faces out of a huge number of subjects are substantially relieved. Our proposed method achieves a coverage of 46.1% at 95% precision on the random set and a coverage of 33.0% at 95% precision on the hard set of this challenge.

LI Jianshu (NUS), ZHAO Jian (NUS), ZHAO Fang (NUS), LIU Hao (HeFei University of Technology)， LI Jing (NUS), SHEN Shengmei (Panasonic), FENG Jiashi (NUS), and Terence Sim (NUS)

ACM MM16 PDF BibTeX

Face recognition techniques have been developed significantly in recent years. However, recognizing faces with partial occlusion is still challenging for existing face recognizers which is heavily desired in real-world applications concerning surveillance and security. Although much research effort has been devoted to developing face de-occlusion methods, most of them can only work well under constrained conditions, such as all the faces are from a pre-defined closed set. In this paper, we propose a robust LSTM-Autoencoders (RLA) model to effectively restore partially occluded faces even in the wild. The RLA model consists of two LSTM components, which aims at occlusion-robust face encoding and recurrent occlusion removal respectively. The first one, named multi-scale spatial LSTM encoder, reads facial patches of various scales sequentially to output a latent representation, and occlusion-robustness is achieved owing to the fact that the influence of occlusion is only upon some of the patches. Receiving the representation learned by the encoder, the LSTM decoder with a dual channel architecture reconstructs the overall face and detects occlusion simultaneously, and by feat of LSTM, the decoder breaks down the task of face de-occlusion into restoring the occluded part step by step. Moreover, to minimize identify information loss and guarantee face recognition accuracy over recovered faces, we introduce an identity-preserving adversarial training scheme to further improve RLA. Extensive experiments on both synthetic and real datasets of faces with occlusion clearly demonstrate the effectiveness of our proposed RLA in removing different types of facial occlusion at various locations. The proposed method also provides significantly larger performance gain than other deocclusion methods in promoting recognition performance over partially-occluded faces.

ZHAO Fang (NUS), FENG Jiashi (NUS), ZHAO Jian (NUS), YANG Wenhan (Peking University), and YAN Shuicheng (Qihoo/360 AI Institute & NUS)

TIP (under review) PDFBibTeX

ZHAO Jian (NUDT), LIU Hengzhu (NUDT), FENG Yiliu (NUDT), YUAN Shandong (NUDT), and CAI Wanzeng (NUDT)

IEEE PICOM2015;PDF BibTeX

Jian ZHAO (NUDT), Hengzhu LIU (NUDT), Xucan CHEN (NUDT), and Zhengfa LIANGANG (NUDT)

AIAA2014 

Jian ZHAO (NUDT), Hengzhu LIU (NUDT), Xucan CHEN (NUDT), Botao ZHANG (NUDT), and Li ZHOU (NUDT)

ICT2014 

Jian ZHAO (NUDT), Hengzhu LIU (NUDT), Xucan CHEN (NUDT), and Ting CHEN (NUDT)

ACA2014 

Jian ZHAO (NUDT), Hengzhu LIU (NUDT), Xucan CHEN (NUDT), and Shandong YUAN (NUDT)

ACA2014 

Jian ZHAO (NUDT), Hengzhu LIU (NUDT), Xucan CHEN (NUDT), Botao ZHANG (NUDT), and Ting CHEN (NUDT)

JOURNAL OF NORTHEASTERN UNIVERSITY