publication

2021

2021

1. SMA 2021

   Facial Emotion Recognition in video clips using 3D Face Reconstruction

   Do, Nhu-Tai, Yang, Hyung-Jeong, Lee, Guee-Sang, and Kim, Soo-Hyung

   In Proceedings of the 2021 International Conference Smart Media and Applications, Gunsan, Korea 2021
2. Diagnostics

   Multi-Level Seg-Unet Model with Global and Patch-Based X-ray Images for Knee Bone Tumor Detection

   Do, Nhu-Tai, Jung, Sung-Taek, Yang, Hyung-Jeong, and Kim, Soo-Hyung

   Diagnostics 2021

   Abs

   Tumor classification and segmentation problems have attracted interest in recent years. In contrast to the abundance of studies examining brain, lung, and liver cancers, there has been a lack of studies using deep learning to classify and segment knee bone tumors. In this study, our objective is to assist physicians in radiographic interpretation to detect and classify knee bone regions in terms of whether they are normal, begin-tumor, or malignant-tumor regions. We proposed the Seg-Unet model with global and patched-based approaches to deal with challenges involving the small size, appearance variety, and uncommon nature of bone lesions. Our model contains classification, tumor segmentation, and high-risk region segmentation branches to learn mutual benefits among the global context on the whole image and the local texture at every pixel. The patch-based model improves our performance in malignant-tumor detection. We built the knee bone tumor dataset supported by the physicians of Chonnam National University Hospital (CNUH). Experiments on the dataset demonstrate that our method achieves better performance than other methods with an accuracy of 99.05% for the classification and an average Mean IoU of 84.84% for segmentation. Our results showed a significant contribution to help the physicians in knee bone tumor detection.

3. Sensors

   Context-Aware Emotion Recognition in the Wild Using Spatio-Temporal and Temporal-Pyramid Models

   Do, Nhu-Tai, Kim, Soo-Hyung, Yang, Hyung-Jeong, Lee, Guee-Sang, and Yeom, Soonja

   Sensors 2021

   Abs

   Emotion recognition plays an important role in human–computer interactions. Recent studies have focused on video emotion recognition in the wild and have run into difficulties related to occlusion, illumination, complex behavior over time, and auditory cues. State-of-the-art methods use multiple modalities, such as frame-level, spatiotemporal, and audio approaches. However, such methods have difficulties in exploiting long-term dependencies in temporal information, capturing contextual information, and integrating multi-modal information. In this paper, we introduce a multi-modal flexible system for video-based emotion recognition in the wild. Our system tracks and votes on significant faces corresponding to persons of interest in a video to classify seven basic emotions. The key contribution of this study is that it proposes the use of face feature extraction with context-aware and statistical information for emotion recognition. We also build two model architectures to effectively exploit long-term dependencies in temporal information with a temporal-pyramid model and a spatiotemporal model with “Conv2D+LSTM+3DCNN+Classify” architecture. Finally, we propose the best selection ensemble to improve the accuracy of multi-modal fusion. The best selection ensemble selects the best combination from spatiotemporal and temporal-pyramid models to achieve the best accuracy for classifying the seven basic emotions. In our experiment, we take benchmark measurement on the AFEW dataset with high accuracy.

4. AACR 2021

   Abstract PO-024: Boosting up knee bone tumor detection from radiology and magnetic resonance imaging by using deep learning techniques

   Do, Nhu-Tai, Jung, Sung-Taek, and Kim, Soo-Hyung

   Clinical Cancer Research 2021

   Abs

   Abstracts: AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; January 13-14, 2021Bone tumors can occur at all ages and also appear around skeletal positions of the human knee. Since the types and expression patterns are extremely diverse, some cases are non-specific symptoms or the onset of symptoms is delayed, and the diagnosis is delayed by low suspicion of physicians. However, timely diagnosis is very important for treatment to reduce extreme pain for patients. Therefore, detecting tumors from radiography and MRI images based on deep learning has recently emerged with the advantage of remarkably shortening time and a completely automatic diagnosis. In this study, we developed a radiology and MRI dataset for knee bone tumor and contributed a robust deep learning-based method to detect tumors. Through an association with Chonnam National University Hospital, we built the radiology knee bone tumor dataset with 1695 high-resolution images including normal, benign and malignant. It is the first dataset for knee bone tumor detection with five categories for classification, and tumor masks, bone regions for segmentation. We proposed multi-level distance features to exploit the tumors based on neighboring regions dealing with challenges from small size, appearance variety, and uncommon. It led to interpreting high-risk regions occurring tumors. Multi-task learning with classification and segmentation helped to learn the mutual benefit from the pixel-wise segmentation and image classification. Our model is fined-tuned in image patches to boost up the accuracy in malignant classification. In difficult cases, MRI has been proved to be a preeminent option to be capable of presenting bone marrow and soft tissue. Therefore, we built MRI knee bone tumor dataset with 104 study cases over 900 series. We used Slicer3D for making the ground-truth in T1-, T2-, and PD- weighted MRI. We generated tumor and bone segmentation masks and classified the study case into three groups: normal, benign, and malignant. We made the ablation study experiments to our radiology dataset. For multi-task learning, we increased Mean IoU from 69.5% to 77.3%. Multi-level distance features helped to increase to 78.8%. For classification, multi-task learning boosted the accuracy from 93% to 95.3%, and multi-level distance features helped the increase of 3%. Besides, the patching training improved malignant classification from 66.7% to 81.5%. The results demonstrated that our deep learning-based method made an important contribution to help doctors in the clinical diagnosis process using a radiology image. Furthermore, our method produced the high-risk neighbor tumor regions by multi-level distance features instead of extracting exactly tumor regions. To improve more accuracy in automatic detection, we hope that MRI images with high contrast resolution can describe clearly soft tissue components and help to recognize the attributes of the tumors clearly. Nonetheless, the dataset of knee bone tumor on MRI is raw and not standard with high complexity. Preprocessing these MRI images is also our initial success to the steppingstone of further research.Citation Format: Nhu-Tai Do, Sung-Taek Jung, Soo-Hyung Kim. Boosting up knee bone tumor detection from radiology and magnetic resonance imaging by using deep learning techniques [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-024.

2020

2020

1. FG 2020

   Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model

   Do, Nhu-Tai, Nguyen-Quynh, Tram-Tran, and Kim, Soo-Hyung

   In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) 2020
2. IEEE Access

   Image Colorization Using the Global Scene-Context Style and Pixel-Wise Semantic Segmentation

   Nguyen-Quynh, Tram-Tran, Kim, Soo-Hyung, and Do, Nhu-Tai

   IEEE Access 2020
3. Applied Sciences

   Robust hand shape features for dynamic hand gesture recognition using multi-level feature LSTM

   Do, Nhu-Tai, Kim, Soo-Hyung, Yang, Hyung-Jeong, and Lee, Guee-Sang

   Applied Sciences 2020
4. KSII

   HSFE Network and Fusion Model based Dynamic Hand Gesture Recognition

   Tai, Do Nhu, Na, In Seop, and Kim, Soo Hyung

   KSII Transactions on Internet and Information Systems (TIIS) 2020

2019

2019

1. IMAVIS

   A novel 2D and 3D multimodal approach for in-the-wild facial expression recognition

   Ly, Thai Son, Do, Nhu-Tai, Kim, Soo-Hyung, Yang, Hyung-Jeong, and Lee, Guee-Sang

   Image and Vision Computing 2019
2. CVPRW 2019

   Multimodal 2D and 3D for In-The-Wild Facial Expression Recognition.

   Ly, Son Thai, Do, Nhu-Tai, Lee, Gueesang, Kim, Soo-Hyung, and Yang, Hyung-Jeong

   In CVPR Workshops 2019

2018

2018

1. ISITC 2018

   Forensics face detection from GANs using convolutional neural network

   Do, Nhu-Tai, Na, In-Seop, and Kim, Soo-Hyung

   In 2018 International Symposium on Information Technology Convergence (ISITC 2018), South Korea 2018
2. KISM

   Tracking by Detection of Multiple Faces using SSD and CNN Features

   Tai, Do Nhu, Kim, Soo-Hyung, Lee, Guee-Sang, Yang, Hyung-Jeong, Na, In-Seop, and Oh, A-Ran

   Smart Media Journal 2018
3. ICMLSC 2018

   Face tracking with convolutional neural network heat-map

   Do, Nhu-Tai, Kim, Soo-Hyung, Yang, Hyung-Jeong, Lee, Guee-Sang, and Na, In-Seop

   In Proceedings of the 2nd International Conference on Machine Learning and Soft Computing 2018

2016

2016

1. RIVF 2016

   Speed up temporal median filter and its application in background estimation

   Le, Thanh-Sach, Do, Nhu-Tai, and Hamamoto, Kazuhiko

   In 2016 IEEE RIVF International Conference on Computing & Communication Technologies, Research, Innovation, and Vision for the Future (RIVF) 2016