Education History

Ph.D
August 2019 - Present, Georgia Institute of Technology
I recently completed my PhD in the CORE Lab at Georgia Tech under Dr. Matthew Gombolay, exploring interactive and interpretable machine learning applied to personalization, language generation, game playing, and more.

Master’s
January 2015 - May 2017, Georgia Institute of Technology
I graduated from the Georgia Institute of Technology with a Master's in Computer Science, specializing in Computational Perception and Robotics. I worked as a research assistant in the RAIL Lab and as a teaching assistant in CS 6476 Intro to Computer Vision.

Bacherlor’s
August 2010 - May 2014, Georgia Institute of Technology
I graduated from the Georgia Institute of Technology with a Bachelor of Science in Computational Media, focusing on "People" and "Media" threads of computer science and on "Film" and "Theater" threads in the humanities. I also graduated with a Certificate in Business Entrepreneurship.

Professional History

Graduate Research Assistant
August 2019 - Present, Georgia Institute of Technology
I worked as a GRA in the CORE lab under Dr. Matthew Gombolay at Georgia Tech. In this time, I have:

Been awarded the Apple Scholars Fellowship in AI/ML
Been named a DAAD AInet Fellow
Developed new approach for personalizing LLMs to match speaker styles (EACL 2023)
Introduced new influence-function view for explaining neural networks (AISTATS 2022)
Studied humans and their reactions to different types of explainable AI (IJHCI 2022)
Introduced new approach to multi-task learning with language (RAL-ICRA 2022)
Measured and reported societal biases in pre-trained LLMs (NAACL 2021)
Introduced a new mechanism to inject domain expertise into neural networks (AAAI 2021)
Developed approach to convert a neural network into a decision-tree (AISTATS 2020)
Enabled unsupervised role discovery in multi-agent systems (AAMAS 2019)
Developed a computer vision system for robots to estimate human workload (THRI 2018)
Applied deep networks to creating video embeddings for zero-shot learning
Conducted and piloted several user studies to validate my work

Research Intern
May 2022 - Present, Google Brain Robotics
I worked as a research intern with Pannag Sanketi and Avi Singh to develop robust and agile robot policies that work with a diverse set of human users. While at Google I:

Implemented and experimented with various architectures for agile robot task learning
Developed standardized evaluation protocols for diverse robot policies
Evaluated policy performance across different data regimes and styles of interaction

Research Intern
May 2020 - September 2020, & May 2021 - September 2021, Apple, Inc.
I worked as a research intern in the Interactive Intelligence group with Nick Apostoloff and Barry Theobald to develop punctuation prediction models for unstructured text from a speech recognition system, and later developing a new approach to private & personalized federated learning. While working at Apple, I:

Implemented and experimented with multimodal approaches to language understanding
Developed a new approach to personalized federated learning even with differential privacy
Presented findings to the SVP of AI/ML within Apple
Submitted methods and results to ICASSP 2021 and ICML 2022

Research Scientist
May 2017 - August 2019, Georgia Institute of Technology
I worked as a research scientist in the CORE lab, previously in the RAIL lab, primarily on differentiable decision tree architectures, formerly on bringing visual perception to robots. While in the labs, I:

Developed a differentiable decision tree architecture to enable intelligent initialization, and went to present this work at the Naval Applications of Machine Learning workshop.
Wrote and presented a workshop paper at HRI ‘18.
Wrapped several state-of-the-art deep networks in ROS for deployment to an autonomous robot.
Published a new ROS package for real time face detection using a deep network, and improved an existing ROS package for real time object detection.
Conducted two user studies to evaluate the performance of an experimental algorithm against a random baseline

Graduate Teaching Assistant
August 2016 - September 2017, Georgia Institute of Technology
I worked as a GTA at Georgia Tech for CS 6476, Intro to Computer Vision, helping about 200 students through the OMSCS program each semester. As a TA for the course, I would host office hours, answer student questions online, and grade assignments.

Applications Developer
June 2014 - December 2015, AT&T
Within AT&T I worked primarily on two separate teams, CDI (Concept Development Innovation) and TDP (Technology Development Program). The CDI team focuses on working with new technologies and experimenting, and the TDP team focuses on developing new employees with new technologies. While there, I:

Author on Patent US10189479B2 for monitoring vehicle driver status
Wrote a library for Android devices to work with Apple iBeacons
Oversaw the Atlanta internship program, helping to manage 40 summer interns
Presented and developed a job search platform for employees seeking new opportunities within the company.
Assisted in development of data-visualization project for third-party advertisers
Worked as a developer on various web projects and Android projects within the Agile methodology.

Web Development Intern
May 2013 - August 2013
I was an intern at a web development startup, working on custom WordPress themes for various small business clients.

Publications

Journals

Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction.
Andrew Silva, Mariah Schrum, Erin Hedlund-Botti, Nakul Gopalan and Matthew Gombolay, IJHCI 2022
In this work, we present the first comprehensive (n=286) user study testing a wide range of approaches for explainable machine learning, including feature importance, probability scores, decision trees, counterfactual reasoning, natural language explanations, and case-based reasoning, as well as a baseline condition with no explanations. We provide the first large-scale empirical evidence of the effects of explainability on human-agent teaming. Our results will help to guide the future of explainability research by highlighting the benefits of counterfactual explanations and the shortcomings of confidence scores for explainability. We also propose a novel questionnaire to measure explainability with human participants, inspired by relevant prior work and correlated with human-agent teaming metrics

LanCon-Learn: Learning With Language to Enable Generalization in Multi-Task Manipulation
Andrew Silva, Nina Moorman, William Silva, Zulfiqar Zaidi, Nakul Gopalan and Matthew Gombolay, RA-L 2022
We present LanCon-Learn , a novel attention-based approach to language-conditioned multi-task learning in manipulation domains to enable learning agents to reason about relationships between skills and task objectives through natural language and interaction. We evaluate LanCon-Learn for both reinforcement learning and imitation learning, across multiple virtual robot domains along with a demonstration on a physical robot. LanCon-Learn achieves up to a 200% improvement in zero-shot task success rate and transfers known skills to novel tasks faster than non-language-based baselines, demonstrating the utility of language for goal specification.

Robot Classification of Human Interruptibility and a Study of Its Effects
Siddhartha Banerjee, Andrew Silva, and Sonia Chernova, THRI 2018
As robots become increasingly prevalent in human environments, there will inevitably be times when the robot needs to interrupt a human to initiate an interaction. We introduce the first interruptibility-aware mobile-robot system and deploy our system in a large-scale user study to understand the effects of interruptibility-awareness on human-task performance, robot-task performance, and on human interpretation of the robot’s social aptitude. Our results show that while participants are able to maintain task performance, even in the presence of interruptions, interruptibility-awareness improves the robot’s task performance and improves participant social perceptions of the robot.

Conferences

FedPerC: Federated Learning for Language Generation with Personal and Context Preference Embeddings
Andrew Silva*, Pradyumna Tambwekar*, and Matthew Gombolay, EACL 2023
We propose a new direction for personalization research within federated learning, leveraging both personal embeddings and shared context embeddings.We also present an approach to predict these “preference” embeddings, enabling personalization without backpropagation. Compared to state-of-the-art personalization baselines, our approach achieves a 50% improvement in test-time perplexity using 0.001% of the memory required by baseline approaches, and achieving greater sample- and compute-efficiency.

Using Cross-Loss Influence Functions to Explain Deep Network Representations
Andrew Silva, Rohit Chopra, and Matthew Gombolay, AISTATS 2022
We propose a relaxation to the assumptions of influence functions, allowing for rigorous interrogation of datasets for arbitrary properties of learned neural network embeddings. On a synthetic dataset, we show that our relaxation does not adversely affect influence estimation, and that our approach closely approximates true influence of data points on learned network properties. We then show examples on real-world datasets, where we are able to leverage cross-loss influence functions to understand which samples in a large dataset are responsible for different word-embedding properties.

Multimodal Punctuation Prediction with Contextual Dropout
Andrew Silva, Barry Theobald, and Nicholas Apostoloff, ICASSP 2021
We first present a transformer-based approach for punctuation prediction that achieves 8% improvement on the IWSLT 2012 TED Task, beating the previous state of the art [1]. We next describe our multimodal model that learns from both text and audio, which achieves 8% improvement over the text-only algorithm on an internal dataset for which we have both the audio and transcriptions. Finally, we present an approach to learning a model using contextual dropout that allows us to handle variable amounts of future context at test time.

Towards a comprehensive understanding and accurate evaluation of societal biases in pre-trained transformers
Andrew Silva*, Pradyumna Tambwekar, and Matthew Gombolay, NAACL 2020
We investigate gender and racial bias across ubiquitous pre-trained language models, including GPT-2, XLNet, BERT, RoBERTa, ALBERT and DistilBERT. We evaluate bias within pre-trained transformers using three metrics: WEAT, sequence likelihood, and pronoun ranking. We conclude with an experiment demonstrating the ineffectiveness of word-embedding techniques, such as WEAT, signaling the need for more robust bias testing in transformers.

Encoding Human Domain Knowledge to Warm Start Reinforcement Learning
Andrew Silva, Matthew Gombolay, AAAI 2021
The modern practice of attempting to learn tabula rasa disregards the logical structure of many domains and the wealth of readily available domain experts' knowledge that could help "warm start" the learning process. We present a new reinforcement learning architecture that can encode expert knowledge directly into a neural, tree-like structure of fuzzy propositions amenable to gradient descent and show that our novel architecture is able to outperform reinforcement and imitation learning techniques across an array of reinforcement learning challenges.

Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrations
Rohan Paleja, Andrew Silva, Letian Chen, Matthew Gombolay, NeurIPS 2020
Inferring strategies for task completion from humans is inherently challenging in that humans exhibit heterogeneity in their latent decision-making criteria. To overcome this, we propose a personalized apprenticeship learning framework that automatically infers a representation of all human task demonstrators by extracting a human-specific embedding. Our framework is built on a propositional architecture that allows for distilling an interpretable representation of each human demonstrator's decision-making.

Optimization Methods for Interpretable Differentiable Decision Trees in Reinforcement Learning
Andrew Silva, Taylor Killian, Ivan Rodriguez, Sung-Hyun Son, Matthew Gombolay, AISTATS 2020
We provide background on differentiable decision trees applied to reinforcement learning, and introduce a framework which produces discrete, interpretable decision trees. We validate our approach through a user study which shows that our approach is more inherently interpretable than other neural network approaches, without significant sacrifices in performance.

Unsupervised Role Discovery Using Temporal Observations of Agents
Andrew Silva, Sonia Chernova, AAMAS 2019
We propose an unsupervised approach to discovering common roles by observing agents over time, allowing us to construct a role-based representation of multi-agent systems that aids in understanding and interpreting the state of the system. We validate our approach on both a soccer and a StarCraft dataset, and show that unsupervised role discovery through observation can provide meaningful insight into the state of a multi-agent system, aiding or even replacing game state data for interpretation or understanding of the system.

FedEmbed: Personalized Private Federated Learning
Andrew Silva, Katherine Metcalf, Barry Theobald, and Nicholas Apostoloff, arXiv 2022
We present FedEmbed, a new approach to private federated learning for personalizing a global model that uses (1) sub-populations of similar users, and (2) personal embeddings. We demonstrate that current approaches to federated learning are inadequate for handling data with conflicting labels, and we show that FedEmbed achieves up to 45% improvement over baseline approaches to personalized private federated learning.

Safe Coordination of Human-Robot Firefighting Teams
Esmaeil Seraj, Andrew Silva, Matthew Gombolay, arXiv 2019
Researchers have proposed unmanned aerial vehicles (UAVs) aid firefighters in fighting wildfires with real-time tracking information. We propose a model-predictive, probabilistically safe distributed control algorithm for human-robot collaboration in wildfire fighting. We derive a novel, analytical bound that enables UAVs to distribute their resources and provides a probabilistic guarantee of the humans' safety while preserving the UAVs' ability to cover an entire fire.

Effects of Interruptibility-Aware Robot Behavior
Siddhartha Banerjee, Andrew Silva, Sonia Chernova, arXiv 2018
As robots become increasingly prevalent in human environments, there will inevitably be times when the robot needs to interrupt a human to initiate an interaction. We introduce the first interruptibility-aware mobile-robot system and deploy our system in a large-scale user study to understand the effects of interruptibility-awareness on human-task performance, robot-task performance, and on human interpretation of the robot’s social aptitude. Our results show that while participants are able to maintain task performance, even in the presence of interruptions, interruptibility-awareness improves the robot’s task performance and improves participant social perceptions of the robot.

Action2Vec: A Crossmodal Embedding Approach to Action Learning
Meera Hahn, Andrew Silva, James M Rehg, arXiv 2019
We describe a novel cross-modal embedding space for actions, named Action2Vec, which combines linguistic cues from class labels with spatio-temporal features derived from video clips. We evaluate Action2Vec by performing zero shot action recognition and obtain state of the art results on three standard datasets. In addition, we present two novel analogy tests which quantify the extent to which our joint embedding captures distributional semantics. This is the first joint embedding space to combine verbs and action videos, and the first to be thoroughly evaluated with respect to its distributional semantics.

Workshops

Excuse Me, Could You Please Assemble These Blocks For Me?
Andrew Silva, Siddhartha Banerjee, Sonia Chernova, What Could Go Wrong? Workshop at HRI 2018
We designed and conducted a user study in which we had to collect data, train a model, and analyze the effects of the model on specific performance metrics, all while obscuring the true nature of our study. We examine the importance of (1) model evaluation and selection, (2) proper participant motivation and instruction, and (3) active control of confounding factors. We present an account of our experiences, some of our ad hoc solutions, and the lessons that we think are valuable.

Reviewing Experience

AAAI Conference on Artificial Intelligence (AAAI)
ACM/IEEE International Conference on Human-Robot Interaction (HRI)
International Conference on Machine Learning (ICML)
International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS)
Robotics: Science and Systems (RSS)
Neural Information Processing Systems (NeurIPS)
Transactions on Human Robot Interaction (THRI)
Robotics and Automation Letters (RA-L)
Empirical Methods in Natural Language Processing (EMNLP)