Member-only story
Data Engineer — Consultant’s Eye
Engineers design and build things. We like to think of ourselves as builders and creators. We want to deliver apps and services that last for years, that scale gracefully for millions of users. Sometimes it feels as if we build web services that enable us to live longer as a species. When I think about the projects that I’ve worked on, the ones that become technical landmarks, the projects that I cherish, the ones that make me proud. Those projects usually involve big datasets, processing them in real-time at scale.
Data engineers design and build pipelines that transform and transport data into a format, wherein by the time it reaches to data scientists or other end users, it is in a highly usable state.
Many engineers are busy building complex systems that manage huge quantities of data, well that’s the engineer in me. Of course, it is really obvious that data engineers are building roads, planes and cars stuck in data traffic jam! Now my worry statement is: how the heck will data engineers build the amazing products of the future?
These pipelines must take data from many disparate sources and collect them into a single warehouse that represents the data uniformly as a single source of truth. Sounds simple enough, but a lot of data literacy skills goes into this role.
