Resilient Data Analytics Pipelines for Fault-Tolerant Smart Manufacturing Systems
Keywords:
Resilient data analytics pipeline; Smart manufacturing; Fault tolerance; Industrial Internet of Things; Edge-fog-cloud computing.Abstract
This review examines resilient data analytics pipelines as critical infrastructure for fault-tolerant smart manufacturing systems. It addresses the need for reliable, low-latency, and integrity-preserving data flows in industrial environments where sensor failures, network disruptions, data corruption, platform faults, and cyber-physical disturbances can compromise real-time analytics and autonomous decision-making. The study synthesizes recent literature on Industrial Internet of Things-enabled manufacturing, distributed stream processing, edge-fog-cloud computing, fault-tolerant architectures, and data governance. It analyses pipeline layers covering data sources, edge preprocessing, fault-tolerant ingestion, stream analytics, distributed storage, security, governance, and decision-support applications. Core resilience mechanisms examined include redundancy, replication, monitoring, checkpointing, rollback recovery, graceful degradation, secure aggregation, audit logging, and adaptive recovery. The review also evaluates technologies such as Apache Kafka, Apache Flink, Apache Spark, Storm, HDFS, Cassandra, MongoDB, cloud-agnostic platforms, microservices, and digital twins. Findings show that resilient analytics pipelines support predictive maintenance, real-time process monitoring, quality assurance, supply-chain optimization, and autonomous manufacturing by preserving data continuity and analytical reliability during faults. However, major challenges remain in balancing scalability with performance, maintaining data integrity across heterogeneous edge-fog-cloud layers, achieving low-latency recovery, integrating legacy systems, securing distributed data flows, and establishing standardized design frameworks. The review identifies future directions in AI-driven fault detection, digital-twin-based resilience testing, scalable distributed architectures, autonomous data management, and benchmarking protocols. Resilient pipelines are therefore essential operational assets for sustaining reliable, secure, and fault-tolerant smart manufacturing intelligence.
References
AlSuwaidan, L. (2020). The role of data management in the Industrial Internet of Things. Concurrency and Computation Practice and Experience, 33(23). https://doi.org/10.1002/cpe.6031
Andronie, M., Lăzăroiu, G., Iatagan, M., Uţă, C., Ștefănescu, R., & Cocoşatu, M. (2021). Artificial Intelligence-Based Decision-Making Algorithms, Internet of Things Sensing Networks, and Deep Learning-Assisted Smart Process Management in Cyber-Physical Production Systems. Electronics, 10(20), 2497. https://doi.org/10.3390/electronics10202497
Angelopoulos, A., Michailidis, E., Νομικός, Ν., Trakadas, P., Hatziefremidis, A., Voliotis, S., … & Zahariadis, T. (2019). Tackling Faults in the Industry 4.0 Era—A Survey of Machine-Learning Solutions and Key Aspects. Sensors, 20(1), 109. https://doi.org/10.3390/s20010109
Asaithambi, S., Venkatraman, R., & Venkatraman, S. (2020). MOBDA: Microservice-Oriented Big Data Architecture for Smart City Transport Systems. Big Data and Cognitive Computing, 4(3), 17. https://doi.org/10.3390/bdcc4030017
Assunção, M., Veith, A., & Buyya, R. (2018). Distributed data stream processing and edge computing: A survey on resource elasticity and future directions. Journal of Network and Computer Applications, 103, 1-17. https://doi.org/10.1016/j.jnca.2017.12.001
Caiazzo, B., Murino, T., Petrillo, A., Piccirillo, G., & Santini, S. (2022). An IoT-based and cloud-assisted AI-driven monitoring platform for smart manufacturing: design architecture and experimental validation. Journal of Manufacturing Technology Management, 34(4), 507-534. https://doi.org/10.1108/jmtm-02-2022-0092
Çakır, A., Akın, Ö., Deniz, H., & Yılmaz, A. (2022). Enabling real time big data solutions for manufacturing at scale. Journal of Big Data, 9(1). https://doi.org/10.1186/s40537-022-00672-6
Cañizo, M., Conde, Á., Charramendieta, S., Miñón, R., Cid-Fuentes, R., & Onieva, E. (2019). Implementation of a Large-Scale Platform for Cyber-Physical System Real-Time Monitoring. Ieee Access, 7, 52455-52466. https://doi.org/10.1109/access.2019.2911979
Cardellini, V., Presti, F., Nardelli, M., & Russo, G. (2022). Runtime Adaptation of Data Stream Processing Systems: The State of the Art. Acm Computing Surveys, 54(11s), 1-36. https://doi.org/10.1145/3514496
Chae, J., Lee, S., Jang, J., Hong, S., & Park, K. (2023). A Survey and Perspective on Industrial Cyber-Physical Systems (ICPS): From ICPS to AI-Augmented ICPS. Ieee Transactions on Industrial Cyber-Physical Systems, 1, 257-272. https://doi.org/10.1109/ticps.2023.3323600
Cheng, Z., Huang, Q., & Lee, P. (2019). On the performance and convergence of distributed stream processing via approximate fault tolerance. The VLDB Journal, 28(5), 821-846. https://doi.org/10.1007/s00778-019-00565-w
Chidiebube, I. N., Nwamekwe, C. O., Chukwuemeka, G. H., and Wilfred, M. (2025). Optimization Of Overall Equipment Effectiveness Factors in a Food Manufacturing Small and Medium Enterprise. Journal of Research in Engineering and Applied Sciences, 10(1), 836-845.
Chidiebube, I.N., Onyeka, N.C., Sunday, A.P., et al. (2025a) ‘A comparative analysis of machine learning models for inventory demand forecasting in a food manufacturing SME’, Indonesian Journal of Innovation Science and Knowledge, 2(3), pp. 35-48.
Chidiebube, I.N., Uzochukwu, M.G., Nwamekwe, C.O., et al. (2025b) ‘Evaluating machine learning models for optimizing overall equipment effectiveness in food manufacturing SMEs’, Jurnal Inovasi Teknologi Dan Edukasi Teknik, 5(2). https://hal.science/hal-05149408v1/file/igbokwe-nkemakonam-chidiebube-layout-jitet.pdf
Davoudian, A. and Liu, M. (2020). Big Data Systems. Acm Computing Surveys, 53(5), 1-39. https://doi.org/10.1145/3408314
Dongen, G. and Poel, D. (2021). A Performance Analysis of Fault Recovery in Stream Processing Frameworks. Ieee Access, 9, 93745-93763. https://doi.org/10.1109/access.2021.3093208
Dubuc, T., Stahl, F., & Roesch, E. (2021). Mapping the Big Data Landscape: Technologies, Platforms and Paradigms for Real-Time Analytics of Data Streams. Ieee Access, 9, 15351-15374. https://doi.org/10.1109/access.2020.3046132
Emeka, U. C., Chikwendu, O. C., & Onyeka, N. C. (2025). Human-Centric Design Integration in Industry 5.0: A Framework for Resilient Smart Manufacturing. INTERNATIONAL JOURNAL, 3(4).
Emeka, U. C., Okpala, C., and Nwamekwe, C. O. (2025a). Circular Economy Principles'implementation in Electronics Manufacturing: Waste Reduction Strategies in Chemical Management. International journal of industrial and production engineering, 3(2), 29-42.
Ezeanyim, O. C., Ewuzie, N. V., Aguh, P. S., Nwabueze, C. V., and Nwamekwe, C. O. (2025). Effective Maintenance of Industrial 5-Stage Compressor: A Machine Learning Approach. Gazi University Journal of Science Part A: Engineering and Innovation, 12(1), 96-118. https://dergipark.org.tr/en/pub/gujsa/issue/90827/1646993
Ezeanyim, O.C., Nwabunwanne, E.C., Igbokwe, N.C. and Nwamekwe, C.O. (2025a) ‘Patient flow and service efficiency in public hospitals: data-driven approaches, strategies, challenges, and future directions’, Journal Health of Indonesian, 3(02), pp. 104–124. https://doi.org/10.58471/health.v3i02.228
Geldenhuys, M., Pfister, B., Scheinert, D., Thamsen, L., & Kao, O. (2022). Khaos: Dynamically Optimizing Checkpointing for Dependable Distributed Stream Processing., 30, 553-561. https://doi.org/10.15439/2022f225
Huang, Z., Shen, Y., Li, J., Fey, M., & Brecher, C. (2021). A Survey on AI-Driven Digital Twins in Industry 4.0: Smart Manufacturing and Advanced Robotics. Sensors, 21(19), 6340. https://doi.org/10.3390/s21196340
Igbokwe, N. C., and Nwamekwe, C. O. (2025). Application of Machine Learning in Predicting Emergency Obstetric Cases in Sub-Saharan Africa: An Early Appraisal. International Journal of Industrial Engineering, Technology and Operations Management, 3(1), 13-22.
Igbokwe, N. C., Christiana, C., Nweke, C. O. N., and Onyeka, C. (2025). Data-Driven Solutions for Shuttle Bus Travel Time Prediction: Machine Learning Model Evaluation at Nnamdi Azikiwe University. African Journal of Computing, Data Science and Informatics (AJCDSI), 1(1), 31-55.
Igbokwe, N. C., Nwamekwe, C. O., Ono, C. G., Nwabunwanne, E. C., & Aguh, P. S. (2024). The role of digital twins in optimizing renewable energy utilization and energy efficiency in manufacturing. Siber International Journal of Digital Business, 1(4), 93-111.
Igbokwe, N. C., Okeagu, F. N., Onyeka, N. C., Onwuliri, J. B., and Godfrey, O. C. (2024a). Machine Learning-Driven Maintenance Cost Optimization: Insights from a Local Industrial Compressor Case Study. Jurnal Inovasi Teknologi dan Edukasi Teknik, 4(11), 2.
Igbokwe, N.C., Emmanuel, U.N. and Nwamekwe, C.O. (2025a) ‘Advances in post-harvest fish processing: an appraisal of traditional and modern smoking techniques for improved quality and efficiency’, Jurnal Integrasi Dan Harmoni Inovatif Ilmu-Ilmu Sosial’, 5 (9), pp. 1-13. https://philarchive.org/rec/IGBAIP
Igbokwe, N.C., Nwamekwe, C.O. and Aguh, P.S. (2025b) ‘Predictive modeling of manufacturing defects using machine learning: A comparative performance study in a manufacturing SME’, African Journal of Advances in Engineering and Technology (AJAET), 1(02), pp. 93-115.
Isah, H., Abughofa, T., Mahfuz, S., Ajerla, D., Zulkernine, F., & Khan, S. (2019). A Survey of Distributed Data Stream Processing Frameworks. Ieee Access, 7, 154300-154316. https://doi.org/10.1109/access.2019.2946884
Javed, A., Robert, J., Heljanko, K., & Främling, K. (2020). IoTEF: A Federated Edge-Cloud Architecture for Fault-Tolerant IoT Applications. Journal of Grid Computing, 18(1), 57-80. https://doi.org/10.1007/s10723-019-09498-8
Jayasekara, S., Harwood, A., & Karunasekera, S. (2020). A utilization model for optimization of checkpoint intervals in distributed stream processing systems. Future Generation Computer Systems, 110, 68-79. https://doi.org/10.1016/j.future.2020.04.019
Kang, S., Jin, R., Deng, X., & Kenett, R. (2021). Challenges of modelling and analysis in cybermanufacturing: a review from a machine learning and computation perspective. Journal of Intelligent Manufacturing, 34(2), 415-428. https://doi.org/10.1007/s10845-021-01817-9
Khalid, M. and Yousaf, M. (2021). A Comparative Analysis of Big Data Frameworks: An Adoption Perspective. Applied Sciences, 11(22), 11033. https://doi.org/10.3390/app112211033
Khan, H., Jabeen, F., Khan, A., Waqar, M., & Kim, A. (2025). IoT-Enabled Fog-Based Secure Aggregation in Smart Grids Supporting Data Analytics. Sensors, 25(19), 6240. https://doi.org/10.3390/s25196240
Khattach, O., Moussaoui, O., & Hassine, M. (2025). End-to-End Architecture for Real-Time IoT Analytics and Predictive Maintenance Using Stream Processing and ML Pipelines. Sensors, 25(9), 2945. https://doi.org/10.3390/s25092945
Marosi, A., Emődi, M., Farkas, A., Lovas, R., Beregi, R., Pedone, G., … & Gáspár, P. (2022). Toward Reference Architectures: A Cloud-Agnostic Data Analytics Platform Empowering Autonomous Systems. Ieee Access, 10, 60658-60673. https://doi.org/10.1109/access.2022.3180365
Mehmood, E. and Anees, T. (2020). Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review. Ieee Access, 8, 119123-119143. https://doi.org/10.1109/access.2020.3005268
Nasiri, H., Nasehi, S., & Goudarzi, M. (2019). Evaluation of distributed stream processing frameworks for IoT applications in Smart Cities. Journal of Big Data, 6(1). https://doi.org/10.1186/s40537-019-0215-2
Nwamekwe, C. O., and Igbokwe, N. C. (2024). Supply Chain Risk Management: Leveraging AI for Risk Identification, Mitigation, and Resilience Planning. International Journal of Industrial Engineering, Technology and Operations Management, 2(2), 41–51. https://doi.org/10.62157/ijietom.v2i2.38
Nwamekwe, C. O., and Nwabunwanne, E. C. (2025). Immersive Digital Twin Integration in the Metaverse for Supply Chain Resilience and Disruption Management. Journal of Engineering Research and Applied Science, 14(1), 95-105.
Nwamekwe, C. O., Chidiebube, I. N., Godfrey, O. C., Celestine, N. E., and Sunday, A. P. (2025). Resilience and Risk Management in Social Robot Systems: An Industrial Engineering Perspective. Culture education and technology research (Cetera), 2(2), 1-12.
Nwamekwe, C. O., Chidiebube, I. N., Godfrey, O. C., Celestine, N. E., and Aguh, P. S. (2025a). Human-Robot Collaboration in Industrial Engineering: Enhancing Productivity and Safety. Journal of Industrial Engineering and Management Research, 6(5), 1-20.
Nwamekwe, C. O., Chinwuko, C. E. and Mgbemena, C. E. (2020). Development and Implementation of a Computerised Production Planning and Control System. UNIZIK Journal of Engineering and Applied Sciences, 17(1), 168-187. https://journals.unizik.edu.ng/ujeas/article/view/1771
Nwamekwe, C. O., Edokpia, R. O., & Eboigbe, C. I. (2026). Integration of Machine Learning into Lean Six Sigma: A Systematic Review for Enhancing Predictive Analytics in the Pharmaceutical Industry. Siber Journal of Advanced Multidisciplinary, 3(4), 133-151.
Nwamekwe, C. O., Edokpia, R. O., and Igbinosa, E. C. (2025b). Exploring the Role of Artificial Intelligence in Enhancing Lean Manufacturing and Six Sigma for Smart Factories. International Journal of Industrial Engineering, Technology and Operations Management, 3(1), 1-12.
Nwamekwe, C. O., Ewuzie, N.V., Igbokwe, N. C., Nwabunwanne, E. C., and Ono, C. G. (2025c). Digital Twin-Driven Lean Manufacturing: Optimizing Value Stream Flow. Letters in Information Technology Education (LITE), 8 (1), pp.1-13. https://hal.science/hal-05127340/
Nwamekwe, C. O., Nwabunwanne, E. C., Okeagu, F. N., and Ono, C. G. (2025d). Lean Manufacturing Principles in the Design and Production of Social Robots. International Journal of Industrial Engineering, Technology and Operations Management, 3(1), 23-34.
Nwamekwe, C. O., Okpala, C. C., and Nwabunwanne, E. C. (2025e). Design Principles and Challenges in Achieving Zero-Energy Manufacturing Facilities. Journal of Engineering Research and Applied Science, 14(1), 1-21.
Nwamekwe, C. O., Uchenna, P. C., Onyedika, S. C. (2026a). Leveraging Emerging Technologies to Enhance Business Processes in Blue Economy Sectors: A Case Study of Anambra State’s Industrial Landscape. International Journal of Technology, Health and Sustainability, 2(2), pp. 559-572. https://ijths.com/wp-content/uploads/IJTHS-0202024.pdf
Okeagu, F., Nwamekwe, C., and Nnamani, B. (2024). Challenges and Solutions of Industrial Development in Anambra State, Nigeria. Iconic Research and Engineering Journals, 7(11), 467-472. https://www.irejournals.com/formatedpaper/1705825.pdf
Okpala C. C., Chukwudi Emeka Udu, and Charles Onyeka Nwamekwe. (2025). Sustainable HVAC Project Management: Strategies for Green Building Certification. International Journal of Industrial and Production Engineering, 3(2), 14-28. https://journals.unizik.edu.ng/ijipe/article/view/5595.
Okpala, C. C., Ezeanyim, O. C., and Nwamekwe, C. O. (2024). The Implementation of Kaizen Principles in Manufacturing Processes: A Pathway to Continuous Improvement. International Journal of Engineering Inventions, 13(7), 116-124. https://www.ijeijournal.com/papers/Vol13-Issue7/1307116124.pdf
Okpala, C. C., Udu, C. E., and Nwamekwe, C. O. (2025a). Artificial Intelligence-Driven Total Productive Maintenance: The Future of Maintenance in Smart Factories. International Journal of Engineering Research and Development (IJERD), (21)1, 68-74. https://www.ijerd.com/paper/vol21-issue1/21016874.pdf
Okpala, C., Onyeka, C. and Igbokwe, N.C. (2024a) ‘The implementation of Internet of Things in the manufacturing industry: An appraisal’, International Journal of Engineering Research and Development, 20(7), pp. 510-516.
Onyeka, N. C., and Emeka, N. (2025). Circular Economy and Zero-Energy Factories: A Synergistic Approach to Sustainable Manufacturing. Journal of Research in Engineering and Applied Sciences, 10(1), 829-835.
Onyeka, N. C., Vitalis, E. N., Chidiebube, I. N., U-Dominic, C. M., and Chibuzo, N. (2024). Adoption of Smart Factories in Nigeria: Problems, Obstacles, Remedies and Opportunities. International journal of industrial and production engineering, 2(2), 68-81. https://journals.unizik.edu.ng/ijipe/article/view/4167
Oza, J., Patil, A., Maniyath, C., More, R., Kambli, G., & Maity, A. (2024). Harnessing Insights from Streams: Unlocking Real-Time Data Flow with Docker and Cassandra in the Apache Ecosystem. https://doi.org/10.36227/techrxiv.170475337.78884732/v1
Panetto, H., Iung, B., Ivanov, D., Weichhart, G., & Wang, X. (2019). Challenges for the cyber-physical manufacturing enterprises of the future. Annual Reviews in Control, 47, 200-213. https://doi.org/10.1016/j.arcontrol.2019.02.002
Parri, J., Patara, F., Sampietro, S., & Vicario, E. (2020). A framework for Model-Driven Engineering of resilient software-controlled systems. Computing, 103(4), 589-612. https://doi.org/10.1007/s00607-020-00841-6
Peres, R., Rocha, A., Leitão, P., & Barata, J. (2018). IDARTS – Towards intelligent data analysis and real-time supervision for industry 4.0. Computers in Industry, 101, 138-146. https://doi.org/10.1016/j.compind.2018.07.004
Power, A. and Kotonya, G. (2018). A Microservices Architecture for Reactive and Proactive Fault Tolerance in IoT Systems., 588-599. https://doi.org/10.1109/wowmom.2018.8449789
Rehman, M., Yaqoob, I., Salah, K., Imran, M., Jayaraman, P., & Perera, C. (2019). The role of big data analytics in industrial Internet of Things. Future Generation Computer Systems, 99, 247-259. https://doi.org/10.1016/j.future.2019.04.020
Sahal, R., Alsamhi, S., Brown, K., O’Shea, D., McCarthy, C., & Guizani, M. (2021). Blockchain-Empowered Digital Twins Collaboration: Smart Transportation Use Case. Machines, 9(9), 193. https://doi.org/10.3390/machines9090193
Srirama, S. (2024). Distributed edge analytics in edge‐fog‐cloud continuum. Internet Technology Letters, 8(3). https://doi.org/10.1002/itl2.562
Suvarna, M., Büth, L., Hejny, J., Mennenga, M., Li, J., Ng, Y., … & Wang, X. (2020). Smart Manufacturing for Smart Cities—Overview, Insights, and Future Directions. Advanced Intelligent Systems, 2(10). https://doi.org/10.1002/aisy.202000043
Syafrudin, M., Alfian, G., Fitriyani, N., & Rhee, J. (2018). Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing. Sensors, 18(9), 2946. https://doi.org/10.3390/s18092946
Vergilio, T., Kor, A., & Mullier, D. (2022). A Unified Vendor-Agnostic Solution for Big Data Stream Processing in a Multi-Cloud Environment. https://doi.org/10.21203/rs.3.rs-1253161/v1
Vitalis, E. N., Nwamekwe, C. O., Chidiebube, I. N., Chibuzo, N., Nwabunwanne, E. C., and Ono, C. G. (2024). Application Of Machine-Learning-Based Hybrid Algorithm for Production Forecast in Textile Company. Jurnal Inovasi Teknologi dan Edukasi Teknik, 4(12), 1-9.
Vital-Soto, A. and Olivares-Aguila, J. (2023). Manufacturing Systems for Unexpected Events: An Exploratory Review for Operational and Disruption Risks. Ieee Access, 11, 96297-96316. https://doi.org/10.1109/access.2023.3311362
Yang, C., Lan, S., Wang, L., Shen, W., & Huang, G. (2020). Big Data Driven Edge-Cloud Collaboration Architecture for Cloud Manufacturing: A Software Defined Perspective. Ieee Access, 8, 45938-45950. https://doi.org/10.1109/access.2020.2977846
Zeadally, S., Sanislav, T., & Moiş, G. (2019). Self-Adaptation Techniques in Cyber-Physical Systems (CPSs). Ieee Access, 7, 171126-171139. https://doi.org/10.1109/access.2019.2956124
