Toward Generalizable and Extensible Workflow Automation for Multi-Source Data Processing

Modern data-driven applications increasingly demand flexible, scalable, and generalizable solutions to manage complex, heterogeneous workflows. Traditional Workflow Management Systems (WMS) often fall short in dynamic, high-throughput contexts due to their reliance on static configurations and limited adaptability. This paper introduces a modular and extensible framework for building dynamic data processing pipelines capable of integrating multiple data sources and evolving analytical tasks. The proposed system employs a three-layer architecture to facilitate pipeline construction, validation, and deployment through reusable components and standardized metadata. The framework simplifies the creation and connection of processing steps by using a standardized, modular approach that ensures data compatibility and logical consistency, making it accessible and useful for both expert and non-expert users. A real-world use case involving IoT data from the GAIA Metaplatform validates the system’s effectiveness, showcasing features such as runtime block isolation, robust input validation, and dynamic workflow execution. This framework aims to significantly reduce integration overhead while enhancing reusability and adaptability across diverse domains.