I am a 2nd-year Ph.D. student in the Robotics Institute at Carnegie Mellon University, working with Prof. Simon Lucey on computer vision and deep learning.
My research is focused on spatial alignment, 3D vision, and generative modeling.
I'm also interested in understanding neural networks and exploiting geometry as to improve learning efficiency.
11/2017I have two papers accepted to AAAI 2018 (as an oral) and WACV 2018.
07/2017My oral presentation at CVPR 2017 is online here.
02/2017I have two papers accepted to CVPR 2017 (with one oral)!
Research
ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 paper |
arXiv preprint |
website |
code |
BibTex(show)
@inproceedings{lin2018stgan,
title={ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing},
author={Lin, Chen-Hsuan and Yumer, Ersin and Wang, Oliver and Shechtman, Eli and Lucey, Simon},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2018}
}
We propose a novel GAN architecture leveraging Spatial Transformer Networks to learn realistic geometric corrections for image compositing. We demonstrate the efficacy of our method on various applications, where ground-truth supervision is unavailable and geometric predictions are driven purely by appearance realism.
Deep-LK for Efficient Adaptive Object Tracking
Chaoyang Wang, Hamed Kiani Galoogahi, Chen-Hsuan Lin, and Simon Lucey IEEE International Conference on Robotics and Automation (ICRA), 2018 paper |
arXiv preprint |
BibTex(show)
@inproceedings{wang2018deeplk,
title={Deep-LK for Efficient Adaptive Object Tracking},
author={Wang, Chaoyang and Galoogahi, Hamed Kiani and Lin, Chen-Hsuan and Lucey, Simon},
booktitle={IEEE International Conference on Robotics and Automation ({ICRA})},
year={2018}
}
We demonstrate a theoretical relationship between Siamese regression networks and the Lucas-Kanade algorithm. We propose a novel framework for object tracking to learn the feature representation for adaptation of regression parameters online with respect to the tracked template images.
Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
Chen-Hsuan Lin, Chen Kong, and Simon Lucey AAAI Conference on Artificial Intelligence (AAAI), 2018(oral presentation) paper |
arXiv preprint |
website |
code |
BibTex(show)
@inproceedings{lin2018learning,
title={Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction},
author={Lin, Chen-Hsuan and Kong, Chen and Lucey, Simon},
booktitle={AAAI Conference on Artificial Intelligence ({AAAI})},
year={2018}
}
We propose to use 2D convolutional operations to efficiently generate 3D object shapes in the form of dense point clouds. We jointly apply geometric reasoning with 2D projection optimization using the pseudo-renderer, a differentiable module to approximate the true rendering operation of novel depth images.
Object-Centric Photometric Bundle Adjustment with Deep Shape Prior
Rui Zhu, Chaoyang Wang, Chen-Hsuan Lin, Ziyan Wang, and Simon Lucey IEEE Winter Conference on Applications of Computer Vision (WACV), 2018 paper |
arXiv preprint |
BibTex(show)
@inproceedings{zhu2017object,
title={Object-Centric Photometric Bundle Adjustment with Deep Shape Prior},
author={Zhu, Rui and Wang, Chaoyang and Lin, Chen-Hsuan and Wang, Ziyan and Lucey, Simon},
booktitle={IEEE Winter Conference on Applications of Computer Vision ({WACV})},
year={2018}
}
We introduce learned shape prior in the form of deep shape generators into Photometric Bundle Adjustment (PBA) and propose to accommodate full 3D shape generated by the shape prior within the optimization-based inference framework, demonstrating impressive results.
@inproceedings{lin2017inverse,
title={Inverse Compositional Spatial Transformer Networks},
author={Lin, Chen-Hsuan and Lucey, Simon},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2017}
}
Inspired by the classical Lucas-Kanade algorithm for alignment, we propose a neural network module capable of resolving large geometric variations in data through recurrent spatial transformations. Superior performance is demonstrated in various pure image alignment and joint alignment/classification tasks.
Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image
Chen Kong, Chen-Hsuan Lin, and Simon Lucey IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 paper |
BibTex(show)
@inproceedings{kong2017using,
title={Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image},
author={Kong, Chen and Lin, Chen-Hsuan and Lucey, Simon},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2017}
}
We estimate the dense 3D shape of an object given a set of 2D landmarks and silhouette in a single image by choosing the closest single CAD model to the projected image. We design a novel graph embedding based on local dense correspondence to allow for sparse linear combinations of CAD models.
The Conditional Lucas & Kanade Algorithm
Chen-Hsuan Lin, Rui Zhu, and Simon Lucey European Conference on Computer Vision (ECCV), 2016 paper |
arXiv preprint |
website |
code |
BibTex(show)
@inproceedings{lin2016conditional,
title={The Conditional Lucas \& Kanade Algorithm},
author={Lin, Chen-Hsuan and Zhu, Rui and Lucey, Simon},
booktitle={European Conference on Computer Vision (ECCV)},
pages={793--808},
year={2016},
organization={Springer International Publishing}
}
We propose an efficient alignment algorithm inspired by the Lucas-Kanade Algorithm and the Supervised Descent Method, achieving significant improvement over the two learned with little training data. Superior performance is also achieved in applications including template tracking and facial landmark alignment.
Teaching
Head Teaching Assistant
Computer Vision (CMU 16-720 A/B), Fall 2017
Instructor: Prof. Srinivasa Narasimhan, Prof. Simon Lucey, Prof. Yaser Sheikh
Teaching Assistant
Designing Computer Vision Apps (CMU 16-423), Fall 2015
We formulate an agent's curiosity using the distribution on observations with efficient causality modeling for policy networks. Our algorithm can solve navigational problems with very sparse rewards in VizDoom.
We designed a 3D metric reconstruction system of subject-specific faces given self-recorded 2D videos, achieved by solving for the 3D structure from facial landmark tracking with bundle adjustment.
Perceptual Rate-Distortion Optimization of Motion Estimation
I designed a rate-distortion optimization framework in video coding for motion estimation using perceptual quality metrics, achieving 12.2% bitrate reduction over H.264/AVC conventional encoders.
Disentangler Networks with Absolute and Relative Attributes
We design a deep generative network to disentangle image representations into distinct, controllable attributes for image generation. Our network learns from both absolute and relative supervision.
Video Summarization via Convolutional Neural Networks
We designed a novel loss function for deep networks to learn image representations specifically for video summarization, achieved through K-means clustering in the feature space during test time.
We developed a virtual touch instrumental system with only a paper keyboard using real-time fingertip detection and keyboard pattern recognition algorithms for raw images captured by CCD image sensors.
CV
GitHub