Chen-Hsuan Lin

I am a 4th-year Ph.D. student in the Robotics Institute at Carnegie Mellon University, working with Prof. Simon Lucey on computer vision & artificial intelligence research. Before that, I received my M.S. in Robotics from CMU and B.S. in Electrical Engineering from National Taiwan University.

I am interested in solving 3D reconstruction problems using self-supervised learning techniques. My research is primarily focused on the following aspects:

Practicality — utilizing practical sources of image/video datasets with minimal annotations.
Interpretability — exploiting geometric priors for learning interpretable 3D representations.
Generalizability — learning in-the-wild 3D reconstruction for arbitrary objects and scenes.

My long-term research goal is to automate geometric factorization and reconstruction of the 3D world from image/video data, in turn to improve learning efficiency in visual recognition systems. My research has been supported by the NVIDIA Graduate Fellowship. During my graduate years at CMU, I have also spent wonderful internships at Facebook AI Research and Adobe Research.

Email: chlin (at) cmu (dot) edu

CV (updated: 01/2021) Google Scholar GitHub Twitter Linkedin

Updates

04/2021

We released our latest work on training NeRF from unknown camera poses!

10/2020

The code and short talk of our NeurIPS 2020 paper are online! Check out the project page.

09/2020

I have two papers accepted to NeurIPS 2020 and 3DV 2020!

05/2019

I have joined Facebook AI Research (FAIR) for an internship this summer.

02/2019

I have one paper accepted to CVPR 2019!

12/2018

I received the NVIDIA Graduate Fellowship (10 selected worldwide). Thanks NVIDIA!

older updates... (show)

Research

BARF: Bundle-Adjusting Neural Radiance Fields

Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey

paper • arXiv preprint • project page • code • BibTex (show)

@article{lin2021barf,
  title={BARF: Bundle-Adjusting Neural Radiance Fields},
  author={Lin, Chen-Hsuan and Ma, Wei-Chiu and Torralba, Antonio and Lucey, Simon},
  journal={arXiv preprint arXiv:2104.06405},
  year={2021}
}

Neural Radiance Fields can be trained from unknown camera poses! Inspired by classical image alignment, we show that coarse-to-fine optimization is simple yet effective for joint registration and reconstruction on 3D scene representations.

SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images

Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey
Advances in Neural Information Processing Systems (NeurIPS), 2020

paper • arXiv preprint • project page • code • BibTex (show)

@inproceedings{lin2020sdfsrn,
  title={SDF-SRN: Learning Signed Distance 3D Object Reconstruction from Static Images},
  author={Lin, Chen-Hsuan and Wang, Chaoyang and Lucey, Simon},
  booktitle={Advances in Neural Information Processing Systems ({NeurIPS})},
  year={2020}
}

Implicit 3D shape reconstruction can be trained from static image collections with silhouette supervision! We establish the geometric connection of 2D silhouettes to 3D SDF shapes for single-view training without 3D or multi-view supervision.

Deep NRSfM++: Towards Unsupervised 2D-3D Lifting in the Wild

Chaoyang Wang, Chen-Hsuan Lin, and Simon Lucey
IEEE International Conference on 3D Vision (3DV), 2020 (oral presentation)

paper (arXiv) • BibTex (show)

@inproceedings{wang2020deep,
  title={Deep NRSfM++: Towards Unsupervised 2D-3D Lifting in the Wild},
  author={Wang, Chaoyang and Lin, Chen-Hsuan and Lucey, Simon},
  booktitle={IEEE International Conference on 3D Vision ({3DV})},
  year={2020}
}

A self-supervised framework for recovering the 3D structure from annotated 2D landmarks, closely related to hierarchical block-sparse coding in non-rigid structure from motion. Our method handles perspective camera models and missing data.

Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction

Chen-Hsuan Lin, Oliver Wang, Bryan C. Russell, Eli Shechtman, Vladimir G. Kim, Matthew Fisher, and Simon Lucey
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019

paper • arXiv preprint • project page • code • BibTex (show)

@inproceedings{lin2019photometric,
  title={Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction},
  author={Lin, Chen-Hsuan and Wang, Oliver and Russell, Bryan C and Shechtman, Eli and Kim, Vladimir G and Fisher, Matthew and Lucey, Simon},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2019}
}

3D mesh reconstruction from RGB videos using photometric optimization with learned shape priors. We cast the problem as piece-wise 2D image registration, enabling 3D meshes to be aligned against RGB videos without depth or silhouettes.

ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing

Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018

paper • arXiv preprint • project page • code • BibTex (show)

@inproceedings{lin2018stgan,
  title={ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing},
  author={Lin, Chen-Hsuan and Yumer, Ersin and Wang, Oliver and Shechtman, Eli and Lucey, Simon},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2018}
}

A GAN architecture that iteratively corrects the geometry of objects and creates realistic image composites. Our method predicts geometric corrections driven solely by appearance realism where ground-truth supervision are not available.

Deep-LK for Efficient Adaptive Object Tracking

Chaoyang Wang, Hamed Kiani Galoogahi, Chen-Hsuan Lin, and Simon Lucey
IEEE International Conference on Robotics and Automation (ICRA), 2018

paper • arXiv preprint • BibTex (show)

@inproceedings{wang2018deeplk,
  title={Deep-LK for Efficient Adaptive Object Tracking},
  author={Wang, Chaoyang and Galoogahi, Hamed Kiani and Lin, Chen-Hsuan and Lucey, Simon},
  booktitle={IEEE International Conference on Robotics and Automation ({ICRA})},
  year={2018}
}

A Siamese regression neural network with structured optimization based on registration for object tracking. We learn the feature representation that adapts to the regression parameters online with respect to the tracked template images.

Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

Chen-Hsuan Lin, Chen Kong, and Simon Lucey
AAAI Conference on Artificial Intelligence (AAAI), 2018 (oral presentation)

paper • arXiv preprint • project page • code • BibTex (show)

@inproceedings{lin2018learning,
  title={Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction},
  author={Lin, Chen-Hsuan and Kong, Chen and Lucey, Simon},
  booktitle={AAAI Conference on Artificial Intelligence ({AAAI})},
  year={2018}
}

A 2D convolutional neural network to efficiently reconstruct dense 3D point clouds. We design a novel differentiable point cloud renderer to approximate the rasterization of predicted point clouds to supervise with the given depth images.

Object-Centric Photometric Bundle Adjustment with Deep Shape Prior

Rui Zhu, Chaoyang Wang, Chen-Hsuan Lin, Ziyan Wang, and Simon Lucey
IEEE Winter Conference on Applications of Computer Vision (WACV), 2018

paper • arXiv preprint • extension paper • BibTex (show)

@inproceedings{zhu2017object,
  title={Object-Centric Photometric Bundle Adjustment with Deep Shape Prior},
  author={Zhu, Rui and Wang, Chaoyang and Lin, Chen-Hsuan and Wang, Ziyan and Lucey, Simon},
  booktitle={IEEE Winter Conference on Applications of Computer Vision ({WACV})},
  year={2018}
}

Trained 3D shape prediction networks can be utilized as shape priors for photometric bundle adjustment. We demonstrate the use of learned 3D point cloud generators to align shapes to videos within the optimization-based inference framework.

Inverse Compositional Spatial Transformer Networks

Chen-Hsuan Lin and Simon Lucey
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017 (oral presentation)

paper • arXiv preprint • project page • presentation • code • BibTex (show)

@inproceedings{lin2017inverse,
  title={Inverse Compositional Spatial Transformer Networks},
  author={Lin, Chen-Hsuan and Lucey, Simon},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2017}
}

A redesign of Spatial Transformer Networks inspired by the seminal Lucas-Kanade algorithm. With the same architecture, our method learns recurrent spatial transformations to resolve geometric redundancies for efficient visual recognition.

Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image

Chen Kong, Chen-Hsuan Lin, and Simon Lucey
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

paper • BibTex (show)

@inproceedings{kong2017using,
  title={Using Locally Corresponding CAD Models for Dense 3D Reconstructions from a Single Image},
  author={Kong, Chen and Lin, Chen-Hsuan and Lucey, Simon},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2017}
}

A 3D shape reconstruction method based on CAD model retrieval by matching keypoint projections from the given image, where local landmark correspondences allow one to solve for sparse linear combinations of a CAD model dictionary.

The Conditional Lucas & Kanade Algorithm

Chen-Hsuan Lin, Rui Zhu, and Simon Lucey
European Conference on Computer Vision (ECCV), 2016

paper • arXiv preprint • project page • code • BibTex (show)

@inproceedings{lin2016conditional,
  title={The Conditional Lucas \& Kanade Algorithm},
  author={Lin, Chen-Hsuan and Zhu, Rui and Lucey, Simon},
  booktitle={European Conference on Computer Vision (ECCV)},
  pages={793--808},
  year={2016},
  organization={Springer International Publishing}
}

An image registration method inspired by the Lucas-Kanade algorithm, internally learned with structured optimization and a conditional loss function. Our method significantly improves over classical synthesis-based optimization objectives.

Experience

Carnegie Mellon University, 2014 - present
Graduate Research Assistant (with Prof. Simon Lucey)
Research in geometric image registration, dense 3D reconstruction, and self-supervised learning.

Facebook AI Research, 2019
Research Intern (with Dr. Kaiming He, Dr. Georgia Gkioxari, and Prof. Justin Johnson)
Learning 3D-aware feature representations for improving standard 2D object detection systems.

Adobe Research, 2018
Research Intern (with Dr. Oliver Wang, Dr. Bryan Russell, Dr. Eli Shechtman, Dr. Vladimir Kim, and Dr. Matthew Fisher)
Photometric optimization of 3D object meshes for shape reconstruction aligned to RGB videos.

Adobe Research, 2017
Research Intern (with Dr. Eli Shechtman, Dr. Oliver Wang, and Dr. Ersin Yumer)
Learning geometric corrections of composited objects in images driven by appearance realism.

National Taiwan University, 2011 - 2013
Undergraduate Research Assistant (with Prof. Homer H. Chen)
Designing rate-distortion optimization for video compression based on perceptual quality metrics.

Teaching

Visual Learning and Recognition (CMU 16-824), Spring 2019
Teaching Assistant / Graduate Student Instructor (with Prof. Abhinav Gupta)
(Lectures: 3D Vision & 3D Reasoning, Semantic Segmentation & Pixel Labeling)

Computer Vision (CMU 16-720 A/B), Fall 2017
Head Teaching Assistant (with Prof. Srinivasa Narasimhan, Prof. Simon Lucey, Prof. Yaser Sheikh)

Designing Computer Vision Apps (CMU 16-423), Fall 2015
Teaching Assistant (with Prof. Simon Lucey)

Academic Projects

Towards a More Curious Agent

CMU 10-703 Deep Reinforcement Learning & Control

paper

We model the intrinsic reward of agents as the causal distribution of visual observations for policy networks, solving navigation problems with very sparse rewards.

Disentangler Networks with Absolute and Relative Attributes

CMU 16-824 Visual Learning & Recognition

paper

A neural network that disentangles image embeddings into controllable attributes for image manipulation. Our network learns from relative rankings of attributes.

3D Facial Model Fitting from 2D Videos

CMU CI2CV Lab

video (show)

A 3D reconstruction system of metric-scale faces from self-captured 2D videos by solving for the sparse 3D facial landmarks followed by dense mesh fitting.

Video Summarization via Convolutional Neural Networks

CMU 10-701 Machine Learning

report

We use a new loss function to learn image features for video summarization, achieved via K-means clustering of input video frames in the latent space at test time.

Perceptual Rate-Distortion Optimization of Motion Estimation

NTU Multimedia Processing & Communications Lab

paper

An optimization framework for video coding to find the optimal tradeoff between the encoding bitrate and the decoding image distortion using perceptual metrics.

Virtual Piano Keyboard System

NTU Digital Circuit Design Lab

presentation • demo

A sensor-based virtual instrumental system with only a paper keyboard, using real-time fingertip and keyboard pattern recognition on images from CCD sensors.