Enhancing Content Discovery Using Embedding-Based AI Agents on a Data Engineering Lakehouse Architecture

The rapid growth of digital content platforms has significantly increased the complexity of content discovery and personalization. Traditional recommendation systems rely heavily on rule-based filtering, metadata tagging, or collaborative filtering techniques, which often fail to capture the semantic intent and emotional context of user preferences. This research presents FMD-AI (Future of Movie Discovery), an embedding-based AI agent designed on a Data Engineering Lakehouse architecture to enhance content discovery through semantic understanding, conversational interaction, and mood-based personalization. The proposed system integrates structured and unstructured content data, leverages vector embeddings for semantic similarity, and employs scalable data pipelines built on a modern lakehouse platform. Unlike conventional dashboard-driven analytics, the system enables users to interact with data using natural language queries, making discovery intuitive and context-aware. The solution was developed and validated as part of the Databricks Free Edition Hackathon, where it received an Honored Mention for innovation and technical execution. This paper details the architectural design, data engineering workflows, AI techniques, and evaluation strategy used to implement FMD-AI. The findings demonstrate how embedding-based AI agents, when combined with robust data engineering foundations, can significantly improve user engagement, relevance of recommendations, and scalability of content discovery systems.