1. Data engineers are hard to find.
Data engineers are new and rare. Just 18% of companies have a data team and less than 6% of companies have a data engineer or team (Data Engineer Hiring Report from LinkedIn).
2. Data engineers aren’t considered “real engineers.”
Data engineers aren’t considered “real engineers” because many companies reporting to a product lead (for example) don’t consider building analytics to be an engineering problem. They instead view it as a marketing problem.
3. Data engineers are experts in a narrow field.
Most data engineers are experts in big data systems, building data pipelines, and open-source tools (such as Hadoop, Hive, and Spark). They might not be familiar with business problems or frameworks such as SQL.
4. Data engineers are new to product engineering.
Most data engineers are new to both product and engineering roles. Kubernetes, CI/CD, and data science are often new to them.
5. Data engineer education is limited.
Only one US university offers a bachelor’s degree in Data Engineering. The number of universities offering a bachelor’s degree in Data Engineering is very limited and is predicted to increase in the future, but very slowly.
6. Data engineers are paid less than software engineers.
The median pay for a data engineer is $92,000 per year, while the median pay for a data scientist is $110,000 per year (I may be wrong about the salary please correct it in the comments only — LOL).
7. Data engineers don’t receive development or engineering resources.
Most data engineers receive no development resources or engineering support which is why data engineers are self-motivated problem-solvers who often work as solo-engineers.
8. Most data engineers have no mentors.
Data engineers often look for mentors online or sometimes offline but not often. I have been a self-taught data engineer, I relied mostly on books, articles and cheat sheets to learn. So if I would like to become a data professional, where can I turn to for help?
9. 70% of data engineers are not confident in using SQL.
Though SQL is not as popular as Python and R among data scientists, yet SQL is still a game changer in SQL is easy to learn and extremely effective to solve complex data analysis problems.
10. Data engineers need to be fluent in both R and Python.
Since R and Python are common languages for data professionals, most companies prefer to hire data professionals who have at least 1 year of experience using them.
Hope you found this post helpful if you have any comments or questions about this post or in general just say hi, please find me on Twitter or post a question or drop your thoughts in the comments just do whatever you find enticing!