Ramit Pahwa

I'm a Senior Research Engineer at Rivian and Volkswagen Group Technologies in Palo Alto, working on speech, multimodal AI and post-training. I lead post-training of a 30B omni foundation model for in-vehicle spoken dialogue, and have worked on full-duplex speech models with native tool calling, on-device wake word detection, streaming multilingual ASR, and production text-to-speech.

Before that I was a Machine Learning Scientist at Secta.ai, working on diffusion models and LLM-based image quality assessment, and a Software Engineer (ML) at Adobe on Acrobat's mobile ML platform. I did my M.S. in Computer Science at UT Austin, where I worked on video quality assessment in the LIVE Lab with Prof. Alan Bovik, and my integrated B.S./M.S. in Mathematics and Computing at IIT Kharagpur.

Email  /  Resume  /  Scholar  /  Twitter  /  Github  /  LinkedIn

Research

I work on speech and multimodal AI: post-training voice-language models for spoken dialogue, evaluating what they actually hear, and making speech models small enough to run on-device. Earlier I worked on perceptual video quality assessment, and on compressing and distilling neural networks.

A long-horizon conversational memory benchmark in which every answer depends on paralinguistic metadata (emotion, prosody, voice events) attached to the turns, filtered by a three-stage adversarial gate so that a strong language model fails given the transcript alone. Frontier models show a pervasive affect gap: adding the paralinguistic track lifts accuracy by 0.09 to 0.38, while standard ASR pipelines discard the signal entirely.

About 30,000 spoken tool-calling queries across smart car, smart home and wearables, organized in a reasoning hierarchy from single-intent calls to needle-in-a-haystack extraction, with realistic acoustic variation from zero-shot voice-cloning TTS and noise. SpeechLM and ASR-plus-LLM baselines expose the gap between text and audio tool calling.

A public video quality database of 358 compression-distorted American Sign Language videos, built from 149 signed messages by 21 signers and rated for both intelligibility and quality by 42 Deaf and Hard of Hearing volunteers. Existing VQA models leave significant room for improvement on sign language video.

A survey of perceptual video quality assessment: full-, reduced- and no-reference models, the subjective databases behind them, and emerging topics such as HDR, high frame rate, cloud gaming and AR/VR.

Blending in complementary knowledge from an LSTM teacher lets us train CNN students for text classification that are up to 15x faster at inference, while being more accurate than the same CNN trained directly on the data.

Reinforcement learning over architecture search, combined with knowledge distillation, compresses trained CNNs such as VGG and ResNet automatically, exposing the trade-off between speed, memory and accuracy and exploiting the fact that a deployed model may only need a subset of the classes.

An LSTM with an attention unit for the shared task on aggression identification in Facebook posts and comments, ranking 6th and 4th on the two Hindi subtasks.

An end-to-end training regime for infant hip ultrasound that first localizes the femoral head with a Faster R-CNN and then segments it with a U-Net, built on a collected and cleaned set of high-dimensional MRI images.

Design and source code from Jon Barron's website.