Chen-Hsuan Lin

Research

ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing

Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
paper | arXiv preprint | website | code | BibTex (show)

@inproceedings{lin2018stgan,
  title={ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing},
  author={Lin, Chen-Hsuan and Yumer, Ersin and Wang, Oliver and Shechtman, Eli and Lucey, Simon},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2018}
}

We propose a novel GAN architecture leveraging Spatial Transformer Networks to learn realistic geometric corrections for image compositing. We demonstrate the efficacy of our method on various applications, where ground-truth supervision is unavailable and geometric predictions are driven purely by appearance realism.

Deep-LK for Efficient Adaptive Object Tracking

Chaoyang Wang, Hamed Kiani Galoogahi, Chen-Hsuan Lin, and Simon Lucey
IEEE International Conference on Robotics and Automation (ICRA), 2018
paper | arXiv preprint | BibTex (show)

@inproceedings{wang2018deeplk,
  title={Deep-LK for Efficient Adaptive Object Tracking},
  author={Wang, Chaoyang and Galoogahi, Hamed Kiani and Lin, Chen-Hsuan and Lucey, Simon},
  booktitle={IEEE International Conference on Robotics and Automation ({ICRA})},
  year={2018}
}

We demonstrate a theoretical relationship between Siamese regression networks and the Lucas-Kanade algorithm. We propose a novel framework for object tracking to learn the feature representation for adaptation of regression parameters online with respect to the tracked template images.

Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

Chen-Hsuan Lin, Chen Kong, and Simon Lucey
AAAI Conference on Artificial Intelligence (AAAI), 2018 (oral presentation)
paper | arXiv preprint | website | code | BibTex (show)

@inproceedings{lin2018learning,
  title={Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction},
  author={Lin, Chen-Hsuan and Kong, Chen and Lucey, Simon},
  booktitle={AAAI Conference on Artificial Intelligence ({AAAI})},
  year={2018}
}

We propose to use 2D convolutional operations to efficiently generate 3D object shapes in the form of dense point clouds. We jointly apply geometric reasoning with 2D projection optimization using the pseudo-renderer, a differentiable module to approximate the true rendering operation of novel depth images.

Object-Centric Photometric Bundle Adjustment with Deep Shape Prior

Rui Zhu, Chaoyang Wang, Chen-Hsuan Lin, Ziyan Wang, and Simon Lucey
IEEE Winter Conference on Applications of Computer Vision (WACV), 2018
paper | arXiv preprint | BibTex (show)

@inproceedings{zhu2017object,
  title={Object-Centric Photometric Bundle Adjustment with Deep Shape Prior},
  author={Zhu, Rui and Wang, Chaoyang and Lin, Chen-Hsuan and Wang, Ziyan and Lucey, Simon},
  booktitle={IEEE Winter Conference on Applications of Computer Vision ({WACV})},
  year={2018}
}

We introduce learned shape prior in the form of deep shape generators into Photometric Bundle Adjustment (PBA) and propose to accommodate full 3D shape generated by the shape prior within the optimization-based inference framework, demonstrating impressive results.

Inverse Compositional Spatial Transformer Networks

@inproceedings{lin2017inverse,
  title={Inverse Compositional Spatial Transformer Networks},
  author={Lin, Chen-Hsuan and Lucey, Simon},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2017}
}

Inspired by the classical Lucas-Kanade algorithm for alignment, we propose a neural network module capable of resolving large geometric variations in data through recurrent spatial transformations. Superior performance is demonstrated in various pure image alignment and joint alignment/classification tasks.

Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image

Chen Kong, Chen-Hsuan Lin, and Simon Lucey
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017
paper | BibTex (show)

@inproceedings{kong2017using,
  title={Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image},
  author={Kong, Chen and Lin, Chen-Hsuan and Lucey, Simon},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2017}
}

We estimate the dense 3D shape of an object given a set of 2D landmarks and silhouette in a single image by choosing the "closest" single CAD model to the projected image. We design a novel graph embedding based on local dense correspondence to allow for sparse linear combinations of CAD models.

The Conditional Lucas & Kanade Algorithm

Chen-Hsuan Lin, Rui Zhu, and Simon Lucey
European Conference on Computer Vision (ECCV), 2016
paper | arXiv preprint | website | code | BibTex (show)

@inproceedings{lin2016conditional,
  title={The Conditional Lucas \& Kanade Algorithm},
  author={Lin, Chen-Hsuan and Zhu, Rui and Lucey, Simon},
  booktitle={European Conference on Computer Vision (ECCV)},
  pages={793--808},
  year={2016},
  organization={Springer International Publishing}
}

We propose an efficient alignment algorithm inspired by the Lucas-Kanade Algorithm and the Supervised Descent Method, achieving significant improvement over the two learned with little training data. Superior performance is also achieved in applications including template tracking and facial landmark alignment.

Academic Projects

Disentangler Networks with Absolute and Relative Attributes

CMU 16-824 Visual Learning & Recognition

paper

We design a deep generative network to disentangle image representations into distinct, controllable attributes for image generation. Our network learns from both absolute and relative supervision.

Video Summarization via Convolutional Neural Networks

CMU 10-701 Machine Learning

report

We designed a novel loss function for deep networks to learn image representations specifically for video summarization, achieved through K-means clustering in the feature space during test time.

Virtual Piano Keyboard System

NTU Digital Circuit Design Lab

video

We developed a virtual touch instrumental system with only a paper keyboard using real-time fingertip detection and keyboard pattern recognition algorithms for raw images captured by CCD image sensors.

3D Facial Model Fitting from 2D Videos

LuSee LLC. internship

video (show)

We designed a 3D metric reconstruction system of subject-specific faces given self-recorded 2D videos, achieved by solving for the 3D structure from facial landmark tracking with bundle adjustment.

Perceptual Rate-Distortion Optimization of Motion Estimation

NTU Multimedia Processing & Communications Lab

paper

I designed a rate-distortion optimization framework in video coding for motion estimation using perceptual quality metrics, achieving 12.2% bitrate reduction over H.264/AVC conventional encoders.

Chen-Hsuan Lin

Updates

02/2018 I have one paper accepted to CVPR 2018!

01/2018 I have one paper accepted to ICRA 2018.

11/2017 I have two papers accepted to AAAI 2018 (as an oral) and WACV 2018.

07/2017 My oral presentation at CVPR 2017 is online here.

02/2017 I have two papers accepted to CVPR 2017 (with one oral)!