Benchmarking Analytical Throughput between Impala and DB2 in Massive Insurance Data Lakes

This research has presented a rigorous benchmarking analysis of Apache Impala and IBM Db2 within a massive insurance data lake environment. The empirical findings underscore a significant architectural divergence: while IBM Db2 maintains high precision and sophisticated optimization for complex, low-concurrency relational workloads, Apache Impala offers a decisive advantage in analytical throughput and horizontal scalability. For insurance providers managing petabyte-scale HDFS repositories, Impala’s MPP architecture achieved a 3.2x reduction in scan latency and nearly doubled the resource efficiency compared to traditional RDBMS-on-Hadoop integrations. The study demonstrates that the shift toward decoupled storage and compute is a performance necessity for the modern enterprise. By reaching a peak throughput of 420 Queries Per Hour, Impala proved capable of supporting the high-concurrency demands of actuarial departments and real-time usage-based insurance (UBI) platforms. Conversely, the throughput degradation observed in Db2 under heavy concurrent load highlights the inherent challenges of adapting traditional relational engines to the "schema-on-read" dynamics of a distributed data lake.

[1] R. Liu, H. Isah, and F. Zulkernine, “A big data lake for multilevel streaming analytics,” arXiv preprint arXiv:2009.12415, Sep. 2020. [Online]. Available: https://arxiv.org/abs/2009.12415 [2] S. Munirathinam, S. Sun, J. Rosin, H. Sirigibathina, and A. Chinthakindi, “Design and implementation of manufacturing data lake in Hadoop,” in Proc. IEEE Int. Conf. Smart Manufacturing, Industrial & Logistics Engineering (SMILE), Apr. 2019. [Online]. Available: https://ieeexplore.ieee.org/document/8965302 [3] S. Rangarajan, H. Liu, H. Wang, and C.-L. Wang, “Scalable architecture for personalized healthcare service recommendation using big data lake,” arXiv preprint arXiv:1802.04105, Feb. 2018. [Online]. Available: https://arxiv.org/abs/1802.04105 [4] Y. Liu, J. Peng, and Z. Yu, “Big data platform architecture under the background of financial technology,” arXiv preprint arXiv:1901.10527, Jan. 2019. [Online]. Available: https://arxiv.org/abs/1901.10527 [5] W. Roediger, T. Muehlbauer, A. Kemper, and T. Neumann, “High-speed query processing over high-speed networks,” arXiv preprint arXiv:1502.07169, Nov. 2015. [Online]. Available: https://arxiv.org/abs/1502.07169 [6] M. Kornacker et al., “Impala: A modern, open-source SQL engine for Hadoop,” in Proc. Conf. Innovative Data Systems Research (CIDR), 2015. [Online]. Available: https://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper28.pdf [7] A. Tapdiya and D. Fabbri, “A comparative analysis of state-of-the-art SQL-on-Hadoop systems for interactive analytics,” arXiv preprint arXiv:1804.00224, Apr. 2018. [Online]. Available: https://arxiv.org/pdf/1804.00224 [8] M. Armbrust et al., “Spark SQL: Relational data processing in Spark,” in Proc. ACM SIGMOD Int. Conf. Management of Data, 2015. [Online]. Available: https://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf [9] T. Vinçon, A. Koch, and I. Petrov, “Moving processing to data: On the influence of processing in memory on data management,” arXiv preprint arXiv:1905.04767, May 2019. [Online]. Available: https://arxiv.org/pdf/1905.04767 [10] A. Kolychev and K. Zaytsev, “Research of the effectiveness of SQL engines working in HDFS,” J. Theoretical and Applied Information Technology, vol. 95, no. 20. [Online]. Available: https://www.jatit.org/volumes/Vol95No20/6Vol95No20.pdf