Enterprise Recommendation System (Real-Time, AWS-Based)
Designed and deployed a real-time, scalable recommendation system on AWS using a hybrid retrieval + ranking architecture, enabling personalised product discovery and improved conversion across web and mobile channels.
1The Problem
The existing system relied on static rules and popularity-based recommendations, resulting in low relevance and poor personalization. Key challenges: - Weak personalization for returning and anonymous users - No real-time adaptation to user behavior - Poor utilization of clickstream and session data - Difficulty promoting long-tail and new products - Lack of feedback loop for continuous learning This led to reduced CTR, conversion, and missed revenue opportunities.
2The Approach
I designed a hybrid recommendation system using a two-stage architecture: candidate retrieval and ranking. Key decisions: - Use matrix factorization for scalable candidate retrieval - Use XGBoost ranking model for personalized ordering - Separate offline training and online inference pipelines - Build real-time serving using AWS Lambda and API Gateway - Store and retrieve live features using DynamoDB - Implement continuous feedback loop for retraining and optimization
Technical Architecture
Data Ingestion: User events and item catalog processed via AWS Glue
Feature Store: Offline (S3) + Online (DynamoDB) feature storage
Retrieval Model: Matrix factorization trained in SageMaker to generate candidate items
Candidate Store: DynamoDB for low-latency retrieval
Ranking Model: XGBoost model trained in SageMaker
Model Serving: SageMaker endpoints for real-time inference
API Layer: AWS API Gateway + Lambda for orchestration
Application Layer: Web/mobile apps consuming recommendations
Monitoring & Governance: AWS CloudWatch and IAM
Results
Improved click-through rates on recommended products
Increased conversion rates across product and cart pages
Boosted average order value through better cross-sell
Enabled real-time personalised recommendations
Reduced reliance on manual merchandising rules
Key Insights
“Two-stage retrieval + ranking architecture is critical for scalability”
“Real-time feature serving significantly improves recommendation relevance”
“Hybrid models outperform single-model systems in production”
“Separating offline training and online inference enables system robustness”
“Feedback loops are essential for continuous improvement”
Tech Stack
Interested in a similar project?
I'd love to discuss how we can apply this kind of approach to your specific business challenge.
Let's Talk