Member-only story
Modern Hurdles in Data Engineering
The key challenges in data engineering modernization.
In my recent discussions with customers it is clear that modernizing their data pipelines requires a new approach to operationalizing data, and these are as follows:
Building agile data pipelining
Data engineers today are increasingly required to build and operate complex data pipelines for multiple purposes.
At a deeper level, operations teams want their engineers to be agile in operations, operate in a manner that is decoupled from infrastructure code, and respond to production changes quickly and predictably.
To meet these demands, modern infrastructure code needs to support data teams to operate as independent contract teams operating their own environment with independent teams responsible for various data pipelines (data processing, storage, caching, staging, replaying, etc.).
The infrastructure code should then support the operationalization of these pipelines, such as lifecycle management, event-driven automation, and separation of concerns (such as “Datasets”, “Files”, “Files and Datasets”).
