Josh Urban Davis — Curriculum Vitae Download PDF

RESEARCH INTERESTS

I am a research-driven engineer with 6 years of industry experience in machine learning and computer vision, currently an Applied AI Engineer at Apple building system software and on-device AI/ML experiences for Apple Intelligence, following roles as the founding Computer Vision/Machine Learning Engineer at Overlay Robots and two years advancing multimodal interactive systems for AR/VR at Meta Reality Labs. My work focuses on aligning AI capabilities with human needs through scalable, real-world applications, particularly in telepresence, and embodiment. During my PhD in Computer Science at Dartmouth, I developed 7 multimodal systems integrating vision language models and computer vision to support people with disabilities, earning multiple research awards. This work was grounded in human- computer interaction principles and explored how data-driven systems can extend human potential, especially in high- stakes or assistive contexts. I've held research internships at Microsoft Research, Adobe Research, and Autodesk, where I contributed to projects spanning explainability, user feedback loops, and ML-powered creative tools. My approach is both research and impact oriented: I move fluidly between system prototyping, user studies, and large-model experimentation to deployment (primarily in PyTorch). Before graduate school, I worked at several software startups including the DALI Lab, Bregmann Media Labs, and Hex Plexus between 2013 and 2017.

PEER-REVIEWED PUBLICATIONS

C1.
M. Sturdee, M. Lewis, Josh Urban Davis, A. Strohmayer, K. Spiel, N. Koulidou, S. Fdili Alaoui. 2026. Plurality, Revisited. In Seeing Interaction Design. Springer Nature Switzerland, Cham, 191–223.
C2.
Z. Zhang, Joshua Davis, J. P. A. Vu, J. Kuang, J-F. Lalonde. 2025. Improving the Color Accuracy of Lighting Estimation Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR '25).
C3.
Jun Miao, Alex Shin, Jeanne Vu, Takanori Miki, Guodong Rong, Joshua Davis, Zilong Li, Wenbin Wang, Jinglun Gao, and Jiangtao Kuang. 2024. Bridging Reality and the Virtual Environment: Perceptual Consistency and Visual Adaptation. In SIGGRAPH Asia 2024 (SA '24). Association for Computing Machinery, New York, NY, USA, Article 32, 1–3.
C4.
Makayla Lewis, Miriam Sturdee, Denise Lengyel, Mauro Toselli, John Miers, Violet Owen, Josh Urban Davis, et al. 2024. Traveling Arts x HCI Sketchbook: Exploring the Intersection Between Artistic Expression and Human-Computer Interaction. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA '24). Association for Computing Machinery, New York, NY, USA, Article 568, 1–14.
C5.
Josh Urban Davis, Hongwei Wang, Parmit Chilana, Xing-Dong Yang. "it's not an issue of malice, but of ignorance": Designing Inclusive Video Conferencing for Presenter who are d/Deaf or Hard of Hearing. Proceedings of ACM Interactive, Wearable, and Ubiquitous Computing (IMWUT/UbiComp'23). Cancun, MX. 2023. [Acceptance Rate: 23.4%]
C6.
Josh Urban Davis, Paul Asente, Xing-Dong Yang. "Multimodal Direct Manipulation in Video Conferencing: Challenges and Opportunities". Proceedings of Designing Interactive Systems (DIS'23). Pittsburg, Pennsylvania. 2023. [Acceptance Rate: 24.6%] [Completed During Internship at Adobe Research]
C7.
Makayla Lewis, Miriam Sturdee, Josh Urban Davis, Mafalda Gamboa, Sarah Fdili Alaoui, Claire Elisabeth Ohlenschlager, William Gaver, Eli Blevis, and Lian Loke. 2023. Is it Art, is it HCI? Exploring Tensions Between Practice and Research. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23). Association for Computing Machinery, New York, NY, USA, Article 525, 1–4.
C8.
Kelly Mack, Maitraye Das, Dhruv Jain, Danielle Bragg, John Tang, Andrew Begel, Erin Beneteau, Josh Urban Davis, Abraham Glasser, Joon Sung Park, Venkatesh Potluri. Mixed Abilities and Varied Experiences: A Group Autoethnography of a Virtual Summer Internship. Communications of the ACM 66 (8), 105-113 [ACM Research Highlight] [Completed During Internship at Microsoft Research]
C9.
Makayla Lewis, Miriam Sturdee, Thuong Hoang, John Miers, Josh Urban Davis. "Exploring AltNarrative in HCI Imagery and Comics". Proceedings of Human Factors in Computing (CHI'22). New Orleans, LA. 2022 [Acceptance Rate: 24.8%]
C10.
Josh Urban Davis, John Tang, Edward Cutrell, Teddy Seyed. PokerFace Mask: Exploring Augmenting Masks through an Interactive, Mixed-Reality Prototype. Proceedings of HICSS'22. [Completed During Internship at Microsoft Research]
C11.
Josh Urban Davis, Fraser Anderson, Merten Stroetzel, Tovi Grossman, George Fitzmaurice. "Designing Co-Creative Artificial Intelligence for Virtual Environments". Proceedings of the ACM International Conference on Creativity and Cognition (C&C '21) [Acceptance Rate: 23.1%] [Completed During Internship at Autodesk Research]
C12.
Josh Urban Davis, Miriam Sturdee, Makayla Lewis, Angelika Strohmayer, Katta Spiel, Nantia Koulidou, Sarah Fdili Alaoui. The Art in HCI: Human Creative Interactions. Proceedings of the ACM International Conference on Creativity and Cognition (C&C '21) [Acceptance Rate: 23.1%] [Best Paper: Top 1%] 🏆
C13.
Kelly Mack, Maitraye Das, Dhruv Jain, Danielle Bragg, John Tang, Andrew Begel, Erin Beneteau, Josh Urban Davis, Abraham Glasser, Joon Sung Park, Venkatesh Potluri. Mixed abilities and varied experiences: a group auto-ethnography of a virtual summer internship. Proceedings of Computing and Accessibility (ASSETS '21) [Acceptance Rate: 23.20%] [Completed During Internship at Microsoft Research] [Best Paper: Top 5%] 🏆
C14.
Josh Urban Davis, Johann Wentzel. "Font Your Friends and Loved Ones: In Defense of Ugly Interfaces". Proceedings of Human Factors in Computing (CHI'21). Yokohama, Japan. 2021. [Acceptance Rate: 23.8%]
C15.
Josh Urban Davis, Te-Yen Wu, Bo Shi, Hanyi Lu, Athina Panotopoulou, Emily Whiting, Xing-Dong Yang. "TangibleCircuits: An Interactive 3D Printed Circuit Education Tool for People with Visual Impairments". Proceedings of Human Factors in Computing (CHI'20). Honolulu, Hawaii. 2020. [Acceptance Rate: 24.31%] [Best Paper: Top 5%] [Neukom Institute Award for Outstanding Graduate Research] 🏆
C16.
Josh Urban Davis, Jun Gong, Yunxin Sun, Parmit Chilana, Xing-Dong Yang. "CircuitStyle: A System for Peripherally Reinforcing Best Practices in Hardware Computing". Proceedings of User Interfaces Software Technology (UIST'19) New Orleans, LA. 2019. [Acceptance Rate: 20.6%]
C17.
Josh Urban Davis. "IllumiWear: A Bendable Interactive Fiber-Optic eTextile for Audio and Visual Interactions." Proceedings of New Interfaces in Musical Expression (NIME'19). Porto Alegre, Brazil. 2019. [Acceptance Rate: 31%]
C18.
Jun Gong, Xin Yang, Teddy Sayed, Josh Urban Davis, Xing-Dong Yang. "Indutivo: Contact-Based Object-Driven Interactions with Inductive Sensing." Proceedings of User Interface Software Technology (UIST'18), Berlin, Germany. 2018. [Acceptance Rate: 21%]
C19.
Josh Urban Davis. "Postcards from the Electric Void: Interactive Generation of Animations, Images, and Sound Using Adversarial Learning". Second Workshop on Machine Learning for Creativity and Design, Neuro Information Systems Processing (NeruIPS'18). Montreal, QC. 2018. [Acceptance Rate: 21%]
C20.
Josh Urban Davis. "The Gender Generator: Towards a Brain-Computer Interface for Evocation of Gender Dysphoria Symptoms." Proceedings of the 8th Annual Workshop on Creative Applications of Brain-Computer Interfacing, Human Factors in Computing (CHI'18). Montreal, QC. 2018. [Acceptance Rate: 27%]

PATENTS

P1.
Calliope: A System for Supporting Human-AI Collaboration in Virtual Environments. Josh Urban Davis, Fraser Anderson, George Fitzmaurice. (Pending)
P2.
Circuit Style: A System for Peripherally Reinforcing Best Practices in Hardware Computing. Josh Urban Davis, Jun Gong, Parmit Chilana, Xing-Dong Yang (No. 62/916,977)

RESEARCH EXPERIENCE

Apple, Applied AI Engineer
Cupertino, CA
June 2026 – Present
  • Build system software, frameworks, and end-user interfaces in C++ to deliver Apple Intelligence features across the stack, from low-level infrastructure to user-facing experiences.
  • Evaluate, iterate, and integrate AI/ML algorithms, including on-device inference, for production deployment across Apple's ecosystem.
  • Collaborate cross-functionally with design, algorithms, privacy, security, performance, and hardware/silicon teams to engineer end-to-end solutions for features used by millions of users.
  • Apply UX intuition to identify experience opportunities, iterate with design, and translate customer needs into technical requirements.
  • Debug and triage complex cross-device interactions under strict privacy and performance constraints.
Overlay Robots, Founding Computer Vision and Machine Learning Engineer (Lead)
Pre-seed, stealth startup — Oakland, CA
Sept 2025 – June 2026
  • Architected and built the end-to-end perception stack as the founding CV/ML engineer for the world's first intelligent makeup robot.
  • Drove sensor selection, sensor fusion, and camera calibration for real-time high-precision 3D reconstruction, pose estimation, eye tracking, alignment, and skin surface mapping deployed on embedded hardware (NVIDIA Jetson/Orin) under strict latency and safety constraints.
  • Built data collection, labeling, and evaluation infrastructure from scratch, including automated capture pipelines for face scans.
  • Defined the perception roadmap and prioritized technical investments across 3D reconstruction, tracking, color perception, and safety monitoring, balancing prototype speed with production scalability as the system moved from lab demos to manufacturing-ready hardware.
Meta, Research Scientist and Software Engineer
Reality Labs
XR AI Sensing and Signal Processing Algorithms Lab
March 2024 – Sept 2025
  • Research, design, and develop computer vision and machine learning algorithms for light estimation and virtual object rendering applied to AR, VR, and MR.
  • Research, design, and develop applied color perception algorithms for camera and AR applications including novel metric development and key performance indicator evaluation pipelines
  • Develop camera calibration systems and algorithms for 3D surface reconstruction and semantic scene understanding.
  • Implement and support large dataset generation and processing for algorithm development (AI/ML) in new technology explorations. Integrate SLAM, pix2pix, and SIFT algorithms for novel lighting estimation.
Adobe Research, Research Intern
GILL Lab
Mentor: Paul Asente
  • Led the development of a cutting-edge media augmented video conferencing creativity tool that leveraged body pose recognition, gesture and speech detection to deliver immersive experiences.
  • Utilized a powerful stack including Python, Javascript, MediaPipe, and OpenCV to independently prototype the system.
  • Designed and executed comprehensive mixed-method qualitative and quantitative studies involving 46 users to gather valuable insights and user feedback. Effectively presented the study results to key stakeholders within the company, translating research findings into actionable recommendations.
  • Optimized models for on-device object detection and 3D human understanding
Microsoft Research, Research Intern
Future Wearables Lab, RiSE Group, and Ability Team
Mentor: John Tang, Teddy Seyed, Edward Cutrell,
  • Developed and implemented an interactive mask using a smartphone to display real-time video of the user's mouth and nose on the mask's surface.
  • Created real-time mapping techniques to ensure accurate and distortion-free visualization and optimized for on-device inference.
  • Fostered collaborations between research team and commercial partners to transition the prototype into a marketable product. Published the mask design in peer-reviewed proceedings (http://hdl.handle.net/10125/79732)
Autodesk Research, User Interface Research Intern
HCI and Graphics Research Group
Mentor: Fraser Anderson, Tovi Grossman, George Fitzmaurice
  • Spearheaded collaboration with the Machine Learning, HCI/Graphics, and Generative Design research teams to pioneer cutting-edge techniques for authoring 3D objects using generative AI in VR. Leveraged state-of-the-art technologies such as GANs, Transformers, and Autoencoders to develop innovative solutions.
  • Prototyped and deployed interactive generative adversarial networks (GANs) specifically tailored for 3D design tasks in virtual reality. Utilized a powerful tech stack including pyTorch, Pandas, Numpy, and Unity to create immersive and user-friendly generative AI authoring experiences.
  • Recognized for intellectual contributions and innovation by securing a patent for virtual reality interaction techniques and system design. Published technical paper in prestigious, peer-reviewed science proceedings. (doi.org/10.1145/3450741.3465260)
XDiscovery Lab, Dartmouth College, Graduate Student Researcher
Mentor: Xing-Dong Yang
March 2019 - 2024
  • Developed advanced inductive sensing systems capable of accurately distinguishing between various objects by utilizing ensemble learning techniques.
  • Established strategic partnerships with the Kelley Center for the Blind to implement technologies which enabled visually impaired users to access electrical engineering and STEAM education.
  • Played an integral role in grant writing efforts and contributed to the publication of 14 technical papers in peer-reviewed journals and proceedings. Received 4 best paper awards for outstanding research.
  • Served as a mentor to undergraduate and master's students from diverse disciplines, providing guidance and support in cross-departmental projects, fostering collaborative learning and professional development.
Bregmann Media Labs, Media Arts and Sciences Researcher
Mentor: Michael Casey, Grace Leslie
February 2017 – February 2019
  • Analyzed fMRI data for signal reconstruction of auditory stimuli
  • Developed novel brain-computer interaction methods for creating paintings and music
  • Led experiment team in collection of EEG and user experience data
  • Presented and demonstrated brain-computer interaction techniques at multiple venues (Human Factors in Computing CHI and New Interfaces for Musical Expression NIME)
DALI Lab, Developer II
August 2016 – February 2017
  • Led team of designers and developers to create virtual reality and biofeedback systems
  • Collaborated with NASA and the Space Medicine Laboratory at Geisel Medical School to create virtual reality content for space flights
  • Prototyped biofeedback systems for virtual reality to provide assistive health and well-being
  • Developed and edited 360 video and 3D modeled VR environments with Unity and Adobe Creative Suite

MANUSCRIPTS IN PREPARATION

M1.
Josh Urban Davis, Farshid Salemi Parizi, John Tang, Edward Cutrell, Teddy Seyed. Make It Or Break It: Design and Accessibility Considerations when Making during a Crisis.
M2.
Josh Urban Davis, Yizhe Zu, Xing-Dong Yang. "PantoTouch: Enabling Precision Gestural Input on Smart Watches Using a Pantograph"
M3.
Jun Gong, Teddy Sayed, Josh Urban Davis, Songlin Xu, Yao Zou, Yuke Wang, Xin Lui, Da-Yuan Huang, Xing-Dong Yang. "SpringBoard: A Haptic Feedback System for Breadboards Using Virtual Springs."

AWARDS

A1.
Gary Marsden Grant for Early Career Researchers 2023
A2.
Best Paper Award Creativity and Cognition (C&C'21)
A3.
Best Paper Nomination Accessibility and Computing (ASSETS'21)
A4.
Neukom Institute Award for Outstanding Graduate Research 2020
A5.
Best Paper Honorable Mention Human Factors in Computing (CHI'20)
A6.
Neukom Institute Grant 2019
A7.
Dartmouth Graduate Student Fellowship

INVITED TALKS

  • Autodesk Research Connection Speaker Series: From Intent to Experience, Autodesk Research. "Multimodal Systems to Support Accessible and Flexible Creativity" (Virtual 2023)
  • UC Berkeley BiD Seminar, University of California Berkeley. "Multimodal Systems to Support Accessible and Flexible Creativity" (Berkeley, CA 2023)
  • CU Boulder ATLAS Seminar, University of Colorado Boulder "Accessible and Flexible Creativity" (Virtual 2023)
  • Stanford HAI Seminar, Stanford University. "Towards Omni-Modality" (Palo Alto, CA 2023)
  • MIT HCI Engineering Group, Massachusetts Institute of Technology (Cambridge, MA) "Systems for Democratizing Creativity" (Virtual 2022)
  • Art and Computation Reading Group, Adobe Research, (San Francisco, CA) "A Brief History of Generative Art and Design" (Virtual 2021)
  • Center for Accessibility and Inclusion Research (CAIR), iSchool of the Golisano College of Computing and Information Sciences at RIT, (Rochester, NY) "Accessible Creativity: Wearables, Generative Design, and Inclusive Human-AI Co-Creation" (Virtual 2021)
  • HCI Reading Group, Adobe Research, (San Jose, CA) "Making on Your Feet, Semi-extemporaneous Presentations in Mixed-Reality"
  • Harvard Graduate School of Design, Harvard University, (Cambridge, MA) "Accessible Creativity: Wearables, Generative Design, and Inclusive Human-AI Co-Creation" (Virtual '20)
  • Summer HCI Talks, Microsoft Research, (Redmond, WA) "Exploring the Accessibility of Personal Protective Equipment (PPE), Wearables, and Beyond" (Virtual) 2020
  • Taste of Science Houston, (Houston, TX) "Sonnets and Science, A Brief History and Future of Computational Poetry" (Virtual) 2020
  • Ability Team, Microsoft Research, (Redmond, WA) "Project PokerFace: Designing an Interactive Mixed-Reality Mask" (Virtual) 2020
  • RiSE Group, Microsoft Research, (Redmond, WA) "Make It or Break It: Design Considerations When Making During a Crisis" (Virtual) 2020
  • MIT Computer Science and Artificial Intelligence Laboratory, (Boton, MA) Fabrication at CHI, "Tangible Circuits: An Interactive 3D Printed Circuit Education Tool for People with Visual Impairments" (Virtual) 2020
  • Thayer School of Engineering (Hanover, NH) "Creating Tools for Accessible STEAM Education" (Virtual) 2020
  • Autodesk Research (Toronto, ON) "Creative Support Tools in Virtual Environments with Generative Design" 2020

SERVICE and VOLUNTEERING

  • CHI' 23 Papers Reviewed: 2
  • DIS' 22 Papers Reviewed: 2
  • C&C' 21 Pictorials Chair
  • C&C' Papers Reviewed: 8
  • CHI' 22 Papers Reviewed: 2
  • IEEE VR' 22 Papers Reviewed: 1
  • Total Papers Reviewed 2022: 13
  • MobileHCI 2021 Late Breaking Work Chair
  • MobileHCI 2021 Papers Reviewed: 8
  • DIS' 21 Papers Reviewed: 2
  • CHI' 21 Late Breaking Work Chair
  • CHI' 21 Papers Reviewed: 6
  • TEI' 21 Papers Reviewed: 1
  • Total Papers Reviewed 2021: 17
  • UIST' 20 Papers Reviewed: 2
  • DIS' 20 Papers Reviewed: 1
  • CHI' 20 Papers Reviews: 2
  • Total Papers Reviewed 2020: 5

EDUCATION AND TRAINING

Dartmouth College, Hanover, NH. 2018-2024

Ph.D in Computer Science

Department of Computer Science Human Computer Interaction Lab

Advisors: Prof. Xing-Dong Yang, Prof. Devin Balkcom, Prof. Elizabeth Murnane, Dr. Paul Asente

Dartmouth College, Hanover, NH. 2016-2018

M.S. in Computer Science and Digital Arts

Bregmann Media Labs and Department of Computer Science

Advisors: Prof. Michael Casey, Prof. Xing-Dong Yang, Prof. Grace Leslie

REFERENCES

Prof. Xing-Dong Yang

Department of Computer Science

Dartmouth College

HB 6211

xing-dong.yang@dartmouth.edu

Dr. John Tang

Ability Team, Department of Mechanical Engineering

Microsoft Research, Stanford University

San Francisco, CA

johntang@microsoft.com

Dr. Paul Asente

Adobe Research

San Jose, CA

research@asente.com

650-454-4258

Dr. Fraser Anderson

HCI and Visualization Research Group

Autodesk Research, Toronto

Fraser.anderson@autodesk.com

Prof. Grace Leslie

Department of Music Technology

Georgia Institute of Technology

grace.leslie@gatech.edu