Research Projects
Facial Micro-Expression Analysis in Interview Videos for Autism DiagnosisOct 2022 - May 2023
- Focused on analyzing face micro-expressions of the participants for diagnosis in Autism Diagnostic Observation Schedule (ADOS) hour-long videos.
- Designed a pipeline of classifying autism/control groups by spotting micro-expression segments and extracting discriminative feature of subtle facial movement of these spotted segments via optical flow/attention mechanism/local patch of interest.
- Obtained 97.32% classification accuracy showing the efficiency of capturing subtle facial muscle changes for autism diagnosis.
- Language & Packages: Python, PyTorch, OpenCV, Dlib, Scikit-learn, Pandas, NumPy, Scipy, Pillow, Matplotlib, ImgAug.
- Model: CNN, Autoencoder, Transformer, SVM; Unsupervised learning, Regression, classification.
Face Dynamics Analysis for Autism Diagnosis on Interview VideosJul 2019 - Aug 2020
- Constructed a 3-category discriminative few shot learning system to classify participants in hour-long videos into Autism, Autism Spectrum, and Non-Spectrum by leveraging well-established computer vision tools from 3D spatio-temporal dynamic texture feature (LPQ-TOP) extraction, sparse coding (K-SVD), marginal fisher analysis (MFA) to few-shot learning (FSL) and scene-level fusion of facial dynamics.
- Demonstrated the potential of the proposed automatic ASD trait classification system (achieving 91.72% accuracy on the ADOS video dataset) and the benefits of few-shot learning and scene-level fusion strategy by extensive ablation studies.
- Language & Packages: Python, PyTorch, OpenCV, NumPy, Scikit-learn, Pillow, Matplotlib; MatLab
- Model: CNN, Attention, SVM, sparse coding, few-shot learning; classification
Facial Traits Rating Prediction and Analysis on Autism ParticipantsDec 2020 - Jun 2021
- Designed a deep regression model (VGG16-based) for facial traits rating prediction via transfer learning.
- Investigated the difference between autism and normal groups on how they make facial trait judgement by visually checking face images.
- Demonstrated different facial areas are involved for autism and normal individuals on facial traits judgment.
- Language & Packages: Python, Tensorflow, Keras, OpenCV, NumPy, Scipy, Scikit-learn, Matplotlib
- Model: CNN, Regression, Transfer Learning
Transformer-based Face Morphing and De-Morphing Using GAN ModelDec 2021 - Sep 2022
- Developed a transformer-based semantic image manipulation GAN scheme to do face morphing.
- Designed special loss functions (Wing loss, biometric loss, perceptual loss, and pixel-wise mean square
error) to support the optimization in latent code learning of an given face image.
- Demonstrated its superiority to StyleGAN-based morphing methods.
- Extended the study of transformer-based face morphing to face de-morphing task by presenting an effective defense strategy with access to a reference image using the same generator of MorphGANFormer.
- Such demorphing is conceptually similar to unmixing of hyperspectral images but operates in the latent (instead of pixel) space.
- Addressed a fundamental issue of vulnerability-detectability trade-off for face morphing studies.
- Language & Packages: Python, PyTorch, Dlib, OpenCV, NumPy, Scipy, Scikit-Image, Pillow
- Model: GAN, Transformer, Attention Mechanism; StyleGAN2
Fusion-based Few-Shot Morphing Attack Detection and FingerprintingSep 2020 - Nov 2021
- Extended Morphing Attack Detection (MAD) from supervised learning to few-shot learning and from binary detection to multiclass morphing attack fingerprinting (MAF).
- Proposed a few-shot learning (FSL) method to learn discriminative fusion features of two sensor pattern noise (handcrafted PRNU feature, deep learning based Noiseprint feature) by factorized bilinear coding (FBC), which can generalize to unseen morphing attack types from predefined presentation attacks.
- Generated a high-resolution database named Doppelgänger composing of look-alike face pairs without biological connections, and corresponding different morphing attacks result.
- Extensive experimental results show the outstanding performance performance of our fusion-based FSL-MAF.
- Language & Packages: Python, Tensorflow, Keras, PyTorch, Dlib, OpenCV, ImgAug, NumPy, Scipy, Scikit-learn, ImageIO, Pillow, Pandas, Scikit-Image, Matplotlib, bob.measure; MatLab
- Model: CNN, Few-shot Learning, Sparse coding, Attention Mechanism, Siamese network; Classification
Face Image Quality AnalysisAug 2017 - Nov 2018
- Investigated the relationship between face image quality and performance of deep learning based cross-quality face recognition quantitatively.
- Showed that deep models can allow quality changes to some degrees, but not too large, for the test face images.
- Proposed one promising direction for deep learning is to further improve its capability in building the relations between face images with large quality gaps.
- Language & Packages: MatLab; Python, PyVision, Numpy, Scipy, Matplotlib, Pandas
- Model: CNN, lightCNN, CenterLoss, FaceNet, VGGFace; Face Identification/Verification
An overview of Deep Learning based Face Recognition (FR)Aug 2016 - Jun 2019
- Presented a thorough overview of existing deep learning based face recognition algorithms and face databases by summarizing about 330 contributions.
- Reviewed major deep learning concepts pertinent to face image analysis and face recognition, and provided a concise overview of studies on specific face recognition problems, such as handling variations in pose, age, illumination, expression, and heterogeneous face matching.
- Gave a summary of static image/video/heterogeneous (visible, thermal, sketch, RGB-D, etc.) databases used for deep face recognition.
- Discussed some open challenges and directions for future research, e.g., network design, architecture optimization, alignment necessity, face database related issues, and cross-quality matching.
Asian Celebrities Face Image Database ConstructionMay 2017 - Jan 2019
- Gathered a face image dataset with 356.4K images of 7,676 Chinese celebrities.
- Face images are crawled online automatically from versatile sources, presenting different sizes, various pose angles, a range of different light conditions, large age variations, make-ups, partial occlusion, etc.
- Extended to collect a number of famous people of Korea (552 subjects) and Japan (470 subjects).
- Language & Packages: Python
Facial Landmark Detection Evaluation on MOBIO DatabaseMay 2017 - Jan 2019
- Performed facial landmark detection evaluation on face images extracted from multi-module audio-visual face video database MOBIO.
- Manually labelled 20,600 face images with 22 landmarks as the groundtruth using a self-developed facial point labelling tool.
- Seven state-of-the-art facial landmark detection methods (Tweaked CNN, WingLoss, DAC-CSR, PA-CNN, OpenPose, ECT, and TCDCN) are adopted to evaluate their performance on the data.
- Conducted a thorough analysis and comparison to show that MOBIO is challenging for facial landmark detection.
- Language & Packages: Python, PyTorch, Matplotlib, Numpy, OpenCV, caffe; MatLab
Face Image Quality Enhancement Study for Face Recognition (FR)Nov 2018 - Jun 2019
- Explored the face recognition performance on low face quality (large pose angles, low illumination conditions, motion blur, etc.) photos of IJB-A.
- Try to improve the accuracy in dealing with low quality face images by pose correction, motion debluring, normalizing illumination variations.
- Examine the performance of face recognition algorithms for three different quality sets (high, middle, low) after face enhancement.
- Language & Packages: MatLab
Deepfake Face Image DetectionJan 2019 - July 2019
- Explored different types of image features for deepfake face images detection, including Fast Fourier Transform (FFT) based feature, image quality measure (IQM) based feature, steganalysis feature, natural scene statistics based features, and learning-based features, on three different deepfake face datasets, 100k-generated dataset from Flickr-Face-HQ, FaceForensics++, DeepfakeTIMIT generated from VidTIMIT.
- Achieved a high 99.9% binary classification accuracy using FFT based feature.
- Language & Packages: MatLab, Python
Defect Measure Metrics for Software Development ProcessNov 2010 - Dec 2011
Institute of Software Engineering (SEI), Beihang University
- Designed and implemented Defect Measurement Tool to gather, analyze, and measure defects generated during the whole software development life cycle (SDLC), include planning, analysis, design, development & implementation, testing & deployment and maintenance.
- Constructed a series of defect measure metrics to evaluate individuals' performance during the life cycle.
- Language & Platform: C#, .Net framework, MySQL.
Software Project Management System (SPMS)Dec 2009 - Jun 2010
- Designed and implemented system modules for managing the process of entire software development life cycle (SDLC).
- Wrote corresponding technical documents during life cycles, such as Software Requirements Specification (SRS), System Design Document (SDD), Technical Design Document (TDD), Architecture Diagrams (UML diagrams, flowcharts, entity-relationship diagrams, etc.), User Interface (UI) Design Documents, Code Documentation, Test Plan, Test Case and Test Script Documents, User Manuals and Help Guides, Installation and Deployment Guides, and so on.
- Collected each individual's process data based on Personal Software Process framework for further evaluation.
- Language & Platform: C#, .Net framework, MySQL.
Office Automation System (OAS)Jan 2009 - May 2009
Beijing Information Science and Technology University
- Designed a website to implement basic functions of office automation.
- Constructed the website based on the framework of Struts2, Spring and Hibernate (SSH2).
- Applied JSP to implement information display, and utilized JavaScript and Ajax to improve user interaction.
- Language & Platform: Java, JSP, JavaScript, Ajax, SQL Server.