Combining fault-tolerant persistence and low-latency streaming access to binary data for AI models

In many AI-enabled scenarios, such as video surveillance systems, besides requiring the data to be stored safely, human operators and AI models must also be able to access audio and video streams continuously while media files are still being collected. However, system throughput and latency are often limited by the use of transactionality to guarantee data persistence. This paper presents a solution providing both high ingestion rates with transactional data persistence and low-latency access to the stream during collection in near real-time. This enables the AI algorithms to be immediately applied as soon as the data is received. The binary data sources fit well with the audio and video capture of surveillance or similar systems, but the proposed solution can be extended through well-defined general interfaces. The scalability of the proposed approach is based on the microservice architecture. Using Apache Kafka and MongoDB replica sets, preliminary results show that the proposed solution provides up to 6 times larger throughput and 4.5 times lower latency than current standard multi-document transactions.