Jai Shankar

$ cat bio.txt

I build things that listen. LLM agents in production by day; by night, the same systems from the ground up: VAD, ASR, TTS and the real-time pipeline between them written in PyTorch rather than called over HTTP, a video retrieval stack where the encoder and the index are mine, and an assistant that runs on one machine and keeps running after you close the window. Same instinct as my IIT Delhi thesis, which taught a cheap simulation to correct its own numerical error: open the box, learn every layer, then make it fast.

# this is a real shell — type help to explore.

view work cv

find me

$ open ./avatar.png

$ git log --oneline --recent

Recently shipped

Production LLM agents, speech models built from the ground up, local-first AI tooling, and research code.

feat(prod)

Support-debug AI agent

An LLM agent running in production that triages incoming support issues, reproduces them against logs, and root-causes bugs, cutting time-to-diagnosis for the team.

feat(agent)

Vela, a resident local assistant

A process that keeps running after you close the window. She sees the room through the camera, hears you and answers out loud in about 2 seconds end to end, with everything except the language model on the machine: Silero VAD, whisper-base.en, Kokoro-82M at 13.2x realtime on Metal, and a three.js body.

view source

feat(video)

Video intelligence engine from scratch

Ask a two-hour video when backpropagation was explained and get the answer back with the timestamps and frames that prove it. A mini-CLIP, an ANN index, a shot detector and a spatiotemporal encoder all written explicitly, then benchmarked against the production alternative behind the same interface.

view source

feat(voice)

Voice agent from scratch

VAD, ASR, TTS and the real-time pipeline connecting them, built layer by layer in PyTorch. A Conformer with BPE and dynamic chunk streaming replaced the BiGRU+CTC baseline and halved word error on identical audio at 2.7x the speed, then an ISTFT vocoder, voice cloning and speaker embeddings on top.

view source

feat(proto)

Tonecall, an inter-agent voice handshake

Two AI agents on the same phone call detect each other with in-band DTMF and drop the speech-to-text to LLM to text-to-speech loop for a JSON side-channel, about 3 seconds against 4 to 6 per voice turn, while keeping the call open for a human.

view source

feat(local)

Fully local voice notetaker

Talk to a Mac, get organised Obsidian notes. Whisper large-v3-turbo on MLX and a local Qwen through Ollama, so the audio and the notes never leave the machine.

view source

research(iit-d)

Reducing numerical errors with deep learning

My B.Tech thesis at IIT Delhi: embedding neural networks inside differentiable physics solvers so a cheap, low-fidelity simulation learns to correct its own numerical error toward high-fidelity ground truth.

view source

$ cat ~/.interests

Things I keep thinking about

Outside the day job — the areas I read about, tinker with, and lose weekends to.

Multimodal AI

Models that read, see and listen at once. The interesting part is the seam — how a system grounds a sentence in a single frame, and the specific ways it fails when it can't.

Voice AI, end to end

Speech in, speech out, under a latency budget. Endpointing and barge-in are where a voice agent stops feeling like a demo and starts feeling like a conversation.

Local & on-device models

Apple Silicon is quietly fast enough. Running Whisper and a 9B model on a laptop means the private things never have to leave it.

Astronomy

The oldest data problem there is. Light that left its source before the Earth existed, and instruments patient enough to make sense of it.

Systems & computers

How the machine actually works — compiler, kernel, cache line. Abstractions get more useful once you've looked underneath them at least once.

Maths & physics

Differentiable solvers and numerical error. My thesis taught a cheap simulation to correct its own drift toward the truth, and I've been hooked since.

$ stat --summary --live