Ramit Pahwa

I'm a Senior Research Engineer at Rivian and Volkswagen Group Technologies in Palo Alto, working on speech, multimodal AI and post-training. I lead a team of four engineers and scientists on production wake word detection across Rivian, Porsche, Bentley, and Audi, and architect an internal full-duplex speech-to-speech system. I also lead post-training of a 30B omni foundation model for in-vehicle spoken dialogue, with further contributions to on-device multilingual ASR and production text-to-speech.

Before that I was a Machine Learning Scientist at Secta.ai, working on diffusion models and LLM-based image quality assessment, and a Software Engineer (ML) at Adobe on Acrobat's mobile ML platform. I did my M.S. in Computer Science at UT Austin, where I worked on video quality assessment in the LIVE Lab with Prof. Alan Bovik, and my integrated B.S./M.S. in Mathematics and Computing at IIT Kharagpur.

Email  /  Resume  /  Scholar  /  Twitter  /  Github  /  LinkedIn

Rivian & Volkswagen Group Technologies

Selected industry work in speech and multimodal AI, spanning production wake words, full-duplex conversation, multilingual speech recognition, and decision-making models.

I lead a team of four engineers and scientists to deliver production-ready keyword spotting models across Rivian, Porsche, Bentley, and Audi. This work powers in-vehicle wake word experiences and has led to several patent filings.

I architect and lead development of an internal full-duplex speech-to-speech system end to end, including cloud infrastructure, model training, conversation dataset curation, and data pipelines. My speech model work also includes native tool calling and leading post-training of a 30B omni foundation model for spoken dialogue.

I developed the team’s first on-device, LLM-enhanced ASR model for multilingual speech recognition, bringing language model context into speech recognition in the vehicle.

I am a core contributor to the development of an internal decision-making model.

Ongoing Research

I also lead the following research efforts:

• COTIQA — Investigating the use of registers to improve reasoning for image quality assessment. Targeting CVPR.
• Full-Duplex Gym — Adapting Speech Gym into an automated pipeline for training full-duplex speech models. Targeting ICML.
• Vocabulary expansion for tool generation — Developing vocabulary expansion methods to help models generate tool calls. Targeting ACL.

Research

I work on speech and multimodal AI: post-training voice-language models for spoken dialogue, evaluating what they actually hear, and making speech models small enough to run on-device. Earlier I worked on perceptual video quality assessment, and on compressing and distilling neural networks.

A long-horizon conversational memory benchmark in which every answer depends on paralinguistic metadata (emotion, prosody, voice events) attached to the turns, filtered by a three-stage adversarial gate so that a strong language model fails given the transcript alone. Frontier models show a pervasive affect gap: adding the paralinguistic track lifts accuracy by 0.09 to 0.38, while standard ASR pipelines discard the signal entirely.

A reinforcement learning method for aligning flow-matching image generators. It uses predicted path variance to control importance-ratio instability and allocate gradient effort across denoising steps, improving text rendering and preference rewards in the paper’s experiments.

About 30,000 spoken tool-calling queries across smart car, smart home and wearables, organized in a reasoning hierarchy from single-intent calls to needle-in-a-haystack extraction, with realistic acoustic variation from zero-shot voice-cloning TTS and noise. SpeechLM and ASR-plus-LLM baselines expose the gap between text and audio tool calling.

A public video quality database of 358 compression-distorted American Sign Language videos, built from 149 signed messages by 21 signers and rated for both intelligibility and quality by 42 Deaf and Hard of Hearing volunteers. Existing VQA models leave significant room for improvement on sign language video.

A survey of perceptual video quality assessment: full-, reduced- and no-reference models, the subjective databases behind them, and emerging topics such as HDR, high frame rate, cloud gaming and AR/VR.

Blending in complementary knowledge from an LSTM teacher lets us train CNN students for text classification that are up to 15x faster at inference, while being more accurate than the same CNN trained directly on the data.

Reinforcement learning over architecture search, combined with knowledge distillation, compresses trained CNNs such as VGG and ResNet automatically, exposing the trade-off between speed, memory and accuracy and exploiting the fact that a deployed model may only need a subset of the classes.

An LSTM with an attention unit for the shared task on aggression identification in Facebook posts and comments, ranking 6th and 4th on the two Hindi subtasks.

An end-to-end training regime for infant hip ultrasound that first localizes the femoral head with a Faster R-CNN and then segments it with a U-Net, built on a collected and cleaned set of high-dimensional MRI images.

Design and source code from Jon Barron's website.