Comparative Evaluation of Hyperparameter Tuning Strategies in Apache Spark MLlib

. Hyperparameter tuning is a critical yet computationally intensive step in building machine learning models for large-scale datasets. While Apache Spark MLlib offers distributed computing capabilities, its native hyperparameter tuning methods face efficiency and scalability challenges. This study investigates the viability of traditional methods (grid search, random search) and advanced techniques (Bayesian optimization) for optimizing hyperparameter tuning in Spark MLlib. Using the NYC Taxi Trip Duration dataset, that design a systematic evaluation framework to compare these strategies in terms of predictive accuracy (RMSE) and computational efficiency (execution time) within a distributed Spark environment. The experimental analysis highlights methodological trade-offs and provides empirical insights into scalable hyperparameter tuning for regression tasks in distributed machine learning workflows.