I am interested in solving 3D vision problems using spatial alignment and self-supervised learning techniques.
My research goal is to enable geometric factorization and reconstruction of the 3D world from visual data, in turn to improve learning efficiency in visual recognition.
My research has been supported by the NVIDIA Graduate Fellowship.
@article{wang2020deep,
title={Deep NRSfM++: Towards 3D Reconstruction in the Wild},
author={Wang, Chaoyang and Lin, Chen-Hsuan and Lucey, Simon},
journal={arXiv preprint arXiv:2001.10090},
year={2020}
}
We propose a framework that learns to solve for the 3D structure from 2D landmarks, which models both weak and strong perspective cameras and tolerates potential missing data. We show that our formulation is closely related to hierarchical block-sparse coding in NRSfM and achieves state-of-the-art performance across benchmarks.
Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction
Chen-Hsuan Lin, Oliver Wang, Bryan C. Russell, Eli Shechtman, Vladimir G. Kim, Matthew Fisher, and Simon Lucey IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019 paper•arXiv preprint•project page•code•BibTex(show)
@inproceedings{lin2019photometric,
title={Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction},
author={Lin, Chen-Hsuan and Wang, Oliver and Russell, Bryan C and Shechtman, Eli and Kim, Vladimir G and Fisher, Matthew and Lucey, Simon},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2019}
}
We propose to combine multi-view geometric methods with data-driven shape priors for 3D mesh reconstruction from RGB video frames. We pose the photometric mesh optimization as a piece-wise 2D image alignment problem, which enables 3D meshes to deform and align against 2D video sequences without any depth or mask information.
ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018 paper•arXiv preprint•project page•code•BibTex(show)
@inproceedings{lin2018stgan,
title={ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing},
author={Lin, Chen-Hsuan and Yumer, Ersin and Wang, Oliver and Shechtman, Eli and Lucey, Simon},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2018}
}
We propose a novel GAN architecture leveraging Spatial Transformer Networks to predict geometric corrections on objects, creating realistic composite images. We demonstrate the efficacy of our method on various applications, where ground-truth supervision is unavailable and the geometric predictions are driven by appearance realism.
Deep-LK for Efficient Adaptive Object Tracking
Chaoyang Wang, Hamed Kiani Galoogahi, Chen-Hsuan Lin, and Simon Lucey IEEE International Conference on Robotics and Automation (ICRA), 2018 paper•arXiv preprint•BibTex(show)
@inproceedings{wang2018deeplk,
title={Deep-LK for Efficient Adaptive Object Tracking},
author={Wang, Chaoyang and Galoogahi, Hamed Kiani and Lin, Chen-Hsuan and Lucey, Simon},
booktitle={IEEE International Conference on Robotics and Automation ({ICRA})},
year={2018}
}
We demonstrate a theoretical relationship between Siamese regression networks and the Lucas-Kanade algorithm for object tracking. To take the best of both worlds, we propose a novel framework for object tracking to learn the feature representation for adaptation of regression parameters online with respect to the tracked template images.
Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
Chen-Hsuan Lin, Chen Kong, and Simon Lucey AAAI Conference on Artificial Intelligence (AAAI), 2018(oral presentation) paper•arXiv preprint•project page•code•BibTex(show)
@inproceedings{lin2018learning,
title={Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction},
author={Lin, Chen-Hsuan and Kong, Chen and Lucey, Simon},
booktitle={AAAI Conference on Artificial Intelligence ({AAAI})},
year={2018}
}
We propose a simple generator network using 2D convolutional operations to efficiently reconstruct 3D shapes in the form of dense point clouds. During optimization, we jointly apply geometric reasoning with 2D projection using the pseudo-renderer, a novel differentiable module to approximate the rasterization operation of depth images.
Object-Centric Photometric Bundle Adjustment with Deep Shape Prior
Rui Zhu, Chaoyang Wang, Chen-Hsuan Lin, Ziyan Wang, and Simon Lucey IEEE Winter Conference on Applications of Computer Vision (WACV), 2018 paper•arXiv preprint•BibTex(show)
@inproceedings{zhu2017object,
title={Object-Centric Photometric Bundle Adjustment with Deep Shape Prior},
author={Zhu, Rui and Wang, Chaoyang and Lin, Chen-Hsuan and Wang, Ziyan and Lucey, Simon},
booktitle={IEEE Winter Conference on Applications of Computer Vision ({WACV})},
year={2018}
}
We introduce learned shape priors in the form of deep neural networks into the photometric bundle adjustment framework. We propose to accommodate 3D point clouds generated by the shape prior within the optimization-based inference framework, which allows adaptation of the generated shapes to the given video sequences.
@inproceedings{lin2017inverse,
title={Inverse Compositional Spatial Transformer Networks},
author={Lin, Chen-Hsuan and Lucey, Simon},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2017}
}
Inspired by the classical Lucas-Kanade algorithm for alignment, we improve upon Spatial Transformer Networks for efficient visual recognition while resolving large geometric redundancies through recurrent spatial transformations. Superior performance is demonstrated in various pure image alignment and joint alignment/classification tasks.
Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image
Chen Kong, Chen-Hsuan Lin, and Simon Lucey IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 paper•BibTex(show)
@inproceedings{kong2017using,
title={Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image},
author={Kong, Chen and Lin, Chen-Hsuan and Lucey, Simon},
booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2017}
}
We propose to estimate the dense 3D shape of an object given a set of 2D landmarks and silhouette in a single image by choosing the closest single CAD model to the projected image. We propose a novel graph embedding based on local landmark correspondences to allow for sparse linear combinations of a CAD model dictionary.
@inproceedings{lin2016conditional,
title={The Conditional Lucas \& Kanade Algorithm},
author={Lin, Chen-Hsuan and Zhu, Rui and Lucey, Simon},
booktitle={European Conference on Computer Vision (ECCV)},
pages={793--808},
year={2016},
organization={Springer International Publishing}
}
We propose an alignment algorithm inspired by the Lucas-Kanade algorithm and Supervised Descent Method by learning from a conditional objective function, achieving significant improvement learned from little training data. Superior performance is also achieved in applications including template tracking and facial landmark alignment.
Experience
Carnegie Mellon University, 2017 - present
Graduate Research Assistant (with Prof. Simon Lucey) asdfas
We formulate an agent's intrinsic reward using the statistical distribution of visual observations with convolutional causality modeling for policy networks. Our algorithm can solve navigational problems with very sparse rewards in VizDoom.
We design a dense 3D reconstruction system of human faces in metric scale given self-recorded 2D videos. We achieve this by solving for the sparse 3D structure from sequential tracking of facial landmarks followed by dense mesh fitting.
Perceptual Rate-Distortion Optimization of Motion Estimation
We design an optimization framework in video compression to use perceptual metrics to find an optimal balance between encoding bitrate and decoding image distortion, achieving 12.2% bitrate reduction over H.264/AVC encoders.
Disentangler Networks with Absolute and Relative Attributes
We design a generative network to disentangle image embeddings into distinct and controllable attributes for image manipulation. Our network is able to learn from strong absolute supervision and relative pairwise rankings of attributes.
Video Summarization via Convolutional Neural Networks
We propose a novel objective function for deep networks to learn latent representations for video summarization, which is achieved through K-means clustering of the input video frames in the representation space during test time.
We develop a sensor-based virtual instrumental system using only a paper keyboard. To emulate this, we design real-time fingertip detection and keyboard pattern recognition algorithms directly on raw images captured by CCD image sensors.
CV (updated: 01/2020)
GitHub
Google Scholar
arXiv preprint •
BibTex (show)
