Yutong (Kelly) He 何雨桐

Contact: yutonghe [at] cs [dot] cmu [dot] edu

kelly2024.JPG

PhD Student @ MLD, SCS, CMU

yutonghe [at] cs [dot] cmu [dot] edu

I am a second-year PhD student in the Machine Learning Department, School of Computer Science at Carnegie Mellon University, advised by Prof. Zico Kolter and Prof. Ruslan Salakhutdinov.

Before coming to CMU, I was a master’s student at Stanford Computer Science with distinction in research. I was advised by Prof. Stefano Ermon, and closely worked with Prof. Christopher Manning, Prof. David Lobell and Prof. Marshall Burke. I completed my B.S in Mathematics and B.S. in Data Science with highest distinction at University of Rochester, where I worked with Prof. Henry Kautz and Prof. Jiebo Luo.

I was selected as a Siebel Scholar and a Xerox Engineering Research Fellow. I was also awarded an Outstanding Paper Award at ICLR 2022 and Doris Ermine Smith Award for Achievement in Mathematics.

My research interests include generative models (more specifically controllable generation), representation learning, computational sustainability and broad deep learning topics in general.

News

01/2024 Two of our papers Manifold Preserving Guided Diffusion and Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion are accepted by ICLR 2024.
08/2023 I am invited to give several talks at University of Tokyo and RIKEN about diffusion models and their controllable generation.
06/2023 I start my internship as a student research scientist at Sony.
04/2023 Our paper CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations is accepted by ICML 2023.
04/2023 I am invited to give a guest lecture talks at CMU 10707 about diffusion models and their applications.
11/2022 I am invited to give a guest lecture talks at CMU 10417 about diffusion models and their applications.
09/2022 Our paper SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery is accepted by NeurIPS 2022.
08/2022 I start my PhD in the Machine Learning Department, School of Computer Science at Carnegie Mellon University.
07/2022 Our paper Neural Generation Meets Real People: Building a Social, Informative Open-Domain Dialogue Agent is accepted by SIGDial 2022.
04/2022 Our paper Comparing Distributions by Measuring Differences that Affect Decision Making is awarded as an Outstanding Paper at ICLR 2022 (Top 7/4492).
03/2022 I graduate from Stanford University with a Master of Science in Computer Science with distinction in reasearch.
01/2022 Two of our papers Comparing Distributions by Measuring Differences that Affect Decision Making (oral presentation, top 1.6%) and SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations are accepted by ICLR 2022.
01/2022 I am TAing CS 228 in Winter 2022. If you are in the class, please feel free to come to my office hours!
09/2021 Our paper Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis is accepted by NeurIPS 2021.
09/2021 I am TAing CS 236 in Fall 2021. If you are in the class, please feel free to come to my office hours!
08/2021 Our team Chirpy Cardinal stands 2nd place in Alexa Prize 2021.
06/2021 I am an AI4ALL computer vision mentor in summer 2021 at Stanford.
01/2021 I am TAing CS 228 in Winter 2021. If you are in the class, please feel free to come to my office hours!
09/2020 I am awarded as a Siebel Scholar for Class of 2021.
06/2020 I start my internship as a machine learning engineer at Adobe Inc. . I am featured in one of the Adobe Intern Insider videos, check it out here.
02/2020 Our paper Fine-grained Image-to-Image Transformation towards Visual Recognition is accepted by CVPR 2020.
09/2019 I start my Master of Science program in computer science at Stanford University.
05/2019 I graduate from University of Rochester as a magna cum laude with double Bachelor of Science degrees in Data Science and Mathematics.

Publications

  1. arXiv
    Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation
    Yutong He, Alexander Robey, Naoki Murata, Yiding Jiang, Joshua Williams, George J Pappas, Hamed Hassani, Yuki Mitsufuji, Ruslan Salakhutdinov, and J Zico Kolter
    2024
  2. ICLR
    Manifold Preserving Guided Diffusion
    Yutong He*, Naoki Murata*, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J. Zico Kolter, Ruslan Salakhutdinov, and Stefano Ermon
    In International Conference on Learning Representations (ICLR) 2024
  3. ICLR
    Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion
    Dongjun Kim*, Chieh-Hsin Lai*, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon
    In International Conference on Learning Representations (ICLR) 2024
  4. arXiv
    Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association
    Qiyu Wu, Mengjie Zhao, Yutong He, Lang Huang, Junya Ono, Hiromi Wakaki, and Yuki Mitsufuji
    arXiv preprint 2023
  5. arXiv
    Localized Text-to-Image Generation for Free via Cross Attention Control
    Yutong He, Ruslan Salakhutdinov, and J. Zico Kolter
    arXiv preprint 2023
  6. ISPRS
    Sphere2Vec: A general-purpose location representation learning over a spherical surface for large-scale geospatial predictions
    Gengchen Mai, Yao Xuan, Wenyun Zuo, Yutong He, Jiaming Song, Stefano Ermon, Krzysztof Janowicz, and Ni Lao
    ISPRS Journal of Photogrammetry and Remote Sensing 2023
  7. ICML
    CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations
    Gengchen Mai, Ni Lao, Yutong He, Jiaming Song, and Stefano Ermon
    In International Conference on Machine Learning (ICML) 2023
  8. NeurIPS
    SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery
    Yezhen Cong*, Samar Khanna*, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, and Stefano Ermon
    In Neural Information Processing Systems (NeurIPS) 2022
  9. SIGSPATIAL
    Understanding Economic Development in Rural Africa using Satellite Imagery, Building footprints and Deep Models
    Amna Elmustafa, Erik Rozi, Yutong He, Gengchen Mai, Stefano Ermon, Marshall Burke, and David Lobell
    In International Conference on Advances in Geographic Information Systems (ACM SIGSPATIAL) 2022
  10. SIGDial
    Neural Generation Meets Real People: Building a Social, Informative Open-Domain Dialogue Agent
    Ethan A. Chi, Caleb Chiam, Trenton Chang, Swee Kiat Lim, Chetanya Rastogi, Alexander Iyabor, Yutong He, Hari Sowrirajan, Avanika Narayan, Jillian Tang, Haojun Li, Ashwin Paranjape, and Christopher D. Manning
    In The 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDial) 2022
  11. ICLR Oral Award
    Comparing Distributions by Measuring Differences that Affect Decision Making
    Shengjia Zhao*, Abhishek Sinha*, Yutong He*, Aidan Perreault, Jiaming Song, and Stefano Ermon
    In International Conference on Learning Representations (ICLR) 2022
    Outstanding Paper Award [Top 0.15%]
    Oral Presentation [Top 1.6%]
  12. ICLR
    SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon
    In International Conference on Learning Representations (ICLR) 2022
  13. NeurIPS
    Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis
    Yutong He, Dingjie Wang, Nicholas Lai, William Zhang, Chenlin Meng, Marshall Burke, David B. Lobell, and Stefano Ermon
    In Neural Information Processing Systems (NeurIPS) 2021
  14. NeurIPS Workshop
    Tracking Urbanization in Developing Regions with Remote Sensing Spatial-Temporal Super-Resolution
    Yutong He*, William Zhang*, Chenlin Meng, Marshall Burke, David B. Lobell, and Stefano Ermon
    In Neural Information Processing Systems (NeurIPS) workshop on Machine Learning for the Developing World (ML4D) 2021
  15. Alexa Prize
    Neural Neural Everywhere: Controlled Generation Meets Scaffolded and Structured Dialogue
    Ethan A. Chi, Caleb Chiam, Trenton Chang, Swee Kiat Lim, Chetanya Rastogi, Alexander Iya-bor, Yutong He, Hari Sowrirajan, Avanika Narayan, Jillian Tang, Haojun Li, Ashwin Paranjape, and Christopher D. Manning
    In Alexa Prize Proceedings 2021
  16. CVPR
    Fine-grained Image-to-Image Transformation towards Visual Recognition
    Wei Xiong, Yutong He, Yixuan Zhang, Wenhan Luo, Lin Ma, and  Jiebo Luo
    In International Conference on Computer Vision and Pattern Recognition (CVPR) 2020
  17. arXiv
    Motion-Based Handwriting Recognition and Word Reconstruction
    Junshen Kevin Chen*, Wanze Xie*, and Yutong He*
    arXiv preprint 2020