The Rise of Data Engineering: Common Skills and Tools
This post is contributed byBurtch Works’ data engineering recruiting team.
As data teams have increased in size, it’s now become more common to see data engineers working alongside data scientists and other analytics professionals. Whereas once it was expected that data scientists be responsible for every aspect of the data life cycle, we’re now seeing data engineers in charge of pulling, cleaning and loading the data into databases for other modelers to work with.With these roles continuing to evolve, as a recruiter in this space I thought it might be helpful to look at some of the common tools and skills I’m seeing in high demand more recently.
Key Data Engineering Tools
Data engineering is a specialty that relies very heavily on tool knowledge. Obviously the exact tools required will vary from role to role, but below are the most common ones I usually see requested by employers. Often the attitude is “the more the merrier”, but luckily there are plenty of resources like Coursera or EDX that you can use to pick up new tools if your current employer isn’t pursuing them or giving you the resources to learn them at work.
- Hadoop
- Spark
- Python
- Scala
- Java
- C++
- SQL
- AWS/Redshift
- Azure
Skills/Tools that Set Data Engineers Apart
Whether it’s wanting data engineers that can better interact with the rest of the data science team, or looking for professionals that can actually assemble models in Tensorflow, there are several skills that tend to make a resume pop including:
- An understanding of machine learning and artificial intelligence concepts and techniques
- Hands-on experience developing reports and dashboards
For senior-level data engineer roles, the desire for business intelligence experience is even more common, since these jobs are likely to be more strategic or have BI professionals reporting to them. And especially as its use becomes more widespread, data engineers that have experience with machine learning, in any capacity, will usually have a leg up on their competition during the interview process.Often, companies will have substantial amounts of data that needs to be transferred from legacy systems, or they’ll want to make data more accessible via dashboards or other visualization methods.
Industry Variations in Tool Requirements
One area that I wanted to highlight is that required skills and tools can vary significantly between industries. One particularly stark contrast is between the Financial Services industry, which tends to focus on older tools, and Consulting firms, which often require a broad base of tool knowledge since their data engineers will be working with many different companies.Financial ServicesConsulting
- Java
- C++
- SQL
- Sybase
- Oracle
- Hadoop
- Spark
- Scala
- Java
- AWS/cloud
Some financial services companies are beginning to move to cloud platforms like AWS and Azure, but many are still storing their data on legacy systems. This reflects a trend that we found in our annual SAS, R, or Python flash survey, which noted that many analytics and data science professionals in financial services still prefer older tools like SAS.By contrast, a data engineer working with a consulting firm might be expected to be fluent in tools and systems ranging from Spark and Hadoop to Teradata and Oracle. Working with different clients will often mean exposure to a wide variety of tools, so the old and new are still very important to these roles.
Learn more about data engineering job titles and specializations in this post about skills, backgrounds, and job responsibilities.
Overall, I’m finding that the data engineers with the strongest skillsets are always those who are constantly evolving with the latest trends in technology. Even if your current employer isn’t giving you the opportunity to explore Big Data tools like Hadoop, Spark, Scala, etc., staying up to date is more important than ever, even if that means taking classes online or switching jobs to get the experience you need to stay marketable.
Other Helpful Resources for Data Engineers:
- 12 Do’s and Don’ts for Data Engineer Resumes
- 3 Essential Soft Skills for Data Engineers
- 2020 Data Engineer Job Search and COVID-19 Hiring Trends
- 9 Predictions for Data Engineering in 2021 and Beyond
We hope you found this information interesting, and if you’re looking for opportunities or to hire professionals in data engineering, be sure to connect with us.