Bhaumik Mistry

Real-time computer use at human speeds sounds impossible until you rethink data from the ground up. I architected a five-stage Synthetic Data Factory (Specify, Generate, Execute, Validate, Publish) that ran across roughly 400,000 CPU instances and produced hundreds of gigabytes of high-quality training data for Vision-Language Models performing live screen interactions, including playing Duck Hunt at normal speed.

I recognised early that open-source datasets and manual human annotation were severely bottlenecking research velocity, so I pivoted the team strategy to a synthetic-first generation paradigm. The key architectural call was choosing deterministic rule-based execution over autonomous agents during the bootstrap phase, cutting inference latency, lowering infrastructure spend, and guaranteeing 100% reproducibility at scale. Diversity wasn't left to chance: task categories, user personas, UI states, skill constraints, and edge cases were sampled along structured axes so the resulting distribution was genuinely comprehensive.

Validation ran on two tracks simultaneously: fast rule-based state checkers (DOM, file system, OS artifacts) paired with semantic VLM judges for overall trajectory plausibility. I also built the Eval Grid and Eval Image & Prompt Viewer dashboards that gave researchers real-time latency metrics and step-by-step visual inspection of every trajectory, which directly uncovered a critical image encoding and resizing misconfiguration between training and evaluation pipelines.

After Nova Act launched, the question became: how do you give an AI coding agent actual eyes on the frontend it's building? I conceptualised and co-built the UI Verification Skill, a native Kiro skill backed by a Model Context Protocol server that runs two parallel verification modes over the same nova-act-mcp connection.

I led the cross-org architecture across Kiro backend, platform, and research science teams to embed Nova Act behind Kiro's existing paywall without fragmenting authentication or billing. Visual Verification is deterministic: it reads computed DOM styles directly via getComputedStyle and reconciles them against design tokens and accessibility rules in a design.md spec, generating annotated screenshots with failure bounding boxes. Flow Verification is agentic: it executes Gherkin-formatted user journeys through Nova Act's act() and act_get() methods, testing real interactions and state transitions end to end.

Keeping tests honest required system invariants: an immutable .integrity.json ledger and write-once assertion state, so intentional design changes could be distinguished from genuine regressions. The skill became the blueprint for handing core Nova Act integrations back to the Kiro platform.

SDK documentation is a high-friction onboarding path. I prototyped the official Nova Act IDE extension in two days, then led a team of three engineers to take it to enterprise production across VS Code, Cursor, and Kiro in one month, navigating Amazon Legal, IP, Security, and penetration testing reviews along the way.

I invented Assisted Workflow Building by integrating GitHub Copilot Chat API (@novaAct) so developers could describe what they want in natural language and get execution-ready Python browser automation scripts back, with no manual SDK reading required. The extension also unifies the full development lifecycle: interactive notebook-style cell execution with live side-by-side browser debugging, an Action Viewer for step analysis, and one-click cloud deployment through Docker, ECR, and AWS Bedrock AgentCore Runtime.

I also partnered with Solutions Architects on discovery calls with major enterprise accounts including PGA Tour and Samsung, translating client feedback directly into engineering execution and product capabilities.

When testing browser agents on live e-commerce sites created unpredictable failures and security risks, I immediately whiteboarded a solution with lead designer James Powderly: a controlled, isolated sandbox where agents could be evaluated against realistic multi-step tasks. I then spearheaded the effort to take that concept to production at nova.amazon.com/act/gym.

Getting a new public-facing service out of Amazon meant independently driving alignment across IP, Legal, PR, Security, and Open Source governance teams. I navigated all of it to secure clearances for both the website and the open-source repository release. Without this work, the Gym would have remained an internal tool.

The flagship environment, NextDot Travels, is a five-step interstellar booking simulator with mock payment gateways, medical clearance steps, and full state persistence. Under the hood, a hash-seeded FlightService generates identical DOM structures from identical inputs, a deliberate architectural choice that makes every evaluation reproducible. The application uses React 19, TypeScript 5.9, and Vite 7 with strict CSS, state, and route isolation across all gym environments.

Nova Act's free tier was an existing product bet: let developers prototype with a standard amazon.com login instead of an AWS account, IAM configuration, and billing setup. I owned the authentication layer that made it work, scoping, designing, and implementing it end to end.

The Interceptor Service sits at the entry point of every free-tier request (web playground, IDE extension, and Python SDK), validating keys, enforcing quotas, and routing commands to the backend orchestrator. Behind it, an ephemeral data management layer handles session state, execution traces, DOM screenshots, and telemetry without requiring customers to manage their own S3 buckets.

I also co-designed the Python SDK authentication handshake so developers can switch seamlessly between free-tier API keys and production AWS IAM credentials, making the upgrade path to paid tiers frictionless.

Getting a trained Alexa voice model onto millions of devices involved handoffs across Engine, Science, and Release teams that took two weeks per cycle. I solved this by establishing modular component contracts and building a 24/7 automated MLOps pipeline that collapsed release turnaround from two weeks to two days.

Each model leaving the secure training environment went through hardware-targeted compression tuned to the memory and DSP constraints of different Echo and Fire TV devices, then ran against automated engine tests on physical test benches to catch firmware regressions before any public exposure. Rollout used a percentage-based canary system with interactive slider controls that let science owners adjust deployment scale, model temperature, and sensitivity parameters in real time. I also built automated rule engines that let science teams dynamically scale rollouts on low-traffic hardware variants without waiting for engineering release cycles.

The closed-loop feedback architecture was the most significant design. Production device telemetry flowed into an abstracted data layer with a hook system where science teams could inject custom business logic. Automated email and portal reports surfaced actionable decision prompts (dial up, dial down, roll back), keeping humans in the loop without manual dashboard digging.

When the team needed to build AI datasets from hundreds of gigabytes of raw music and acoustic data, the existing approach (ad-hoc single-node Python scripts) wasn't going to scale and was creating real business delays in AI dataset availability. I modernised the architecture end-to-end, taking full ownership from S3 landing bucket triggers through to SageMaker on-demand inference.

Ingest started with automated S3 landing triggers that ran validation, deduplication via payload hashing, and schema enforcement before anything moved downstream. Processing moved to Apache Spark distributed jobs, with datasets standardised into Parquet, JSONL, and TFRecords formats for high-IOPS model consumption. Apache Airflow DAGs orchestrated batch execution and dynamically triggered Amazon SageMaker on-demand hosting endpoints to run Whisper-AT, an OpenAI Whisper extension that adds 527-class AudioSet event labels, across the entire corpus concurrently.

Using SageMaker on-demand hosting triggered via Airflow DAGs meant inference clusters scaled up only when needed, eliminating idle compute spend while maintaining throughput.