Real-Time Vector Search at Streaming Scale
A proof of concept bringing a major streaming company's search onto Google Vector Search V2, ingesting real-time changes from MongoDB.
I led a team of 4 (developer, data scientist, data engineer, DevOps) on a PoC evaluating Google Vector Search V2 as a vector database for a major media & entertainment company's search engine. The pipeline captures real-time changes from MongoDB via Debezium (CDC), publishes them to an AWS Kafka cluster, and continuously ingests events into Vector Search V2. The FastAPI consumption API deployed on Cloud Run integrated directly with the client's production search flow.
- Kafka + Debezium CDC
- Google Vector Search V2
- FastAPI on Cloud Run
- Production Search Integration
- Unit Tests & CI/CD
- Terraform IaC
AI/ML
Media & Entertainment (Streaming)
Remote
2026
Project Summary
This PoC validated Google Vector Search V2 as a viable alternative to MongoDB for real-time semantic search, proving the vector store could stay continuously in sync without impacting the search experience for the client's users. Beyond the core CDC pipeline, the project shipped with unit tests, a CI/CD pipeline for GCP deployments, and infrastructure managed entirely as code with Terraform — turning what started as an evaluation exercise into a production-ready reference architecture the client's engineering team could adopt directly.
