Case Studies

This blog is contributed by Burtch Works’ data engineering recruiting team.The growth of digital data continues to accelerate, fueling the need for deliberate digital transformation strategy in order to stay competitive. Every organization is feeling the pressure to reform and streamline internal processes to match growing expectations of customers, both internal and external to the organization.Digital native organizations have continued to raise the bar for seamless user experience and digital leadership and strategy, and many companies are reaching the tipping point where their legacy solutions and architectures are no longer up to the task of managing today’s data. As more organizations are realizing that digital transformation is essential to their survival, it has spurred on competition for top notch data engineering talent to execute this transition.

The Critical Role of Data Engineering in Digital Transformation

Data-driven organizations must commit to constant improvements and to modernizing their data architecture, tools, and technologies. Since there will be a heavy emphasis on improving everything from internal reporting, operations, and analytics, this means that there is a dire need for updated infrastructure and systems. Legacy systems that have been in place for years (and sometimes decades) may find themselves being completely overhauled or abandoned for newer technologies.The data engineering team is a key factor in facilitating these types of technology changes. Effective modernization efforts require a thorough understanding of legacy systems as well as how to implement newer technologies that will better serve the specific needs of the business. Organizations need architects and engineers with a broad range of technical skills who can migrate existing systems, integrate older systems with newer ones, or even build new systems.Data engineering has evolved immensely over the past decade, folding in sophisticated data science practices like machine learning and artificial intelligence. These modern tools and responsibilities now support cloud computing, data infrastructure, data warehousing, data mining, data modeling, data crunching, metadata management, data testing, and governance, among others. Digital transformation requires strong leadership in Data Science, Data Analytics, and Data Engineering to push the organization forward. Digital transformation typical requires improvements in data quality as well as the integration of many data sources, and these requirements are an ideal match for the skills of the data engineering team.With the proliferation of so many newer tools and technologies, there is an integrations crisis as companies look to connect disparate tools and data sources, and also leverage their internal data for competitive advantage. In the past, sometimes certain job responsibilities like data cleaning or architecture, among many others, may have been assigned to data scientists. However, we’ve seen that as teams grow, there is a growing importance placed on hiring more data engineers, since these professionals are experts at building data infrastructure and pipelines, which is not typically a data scientist’s primary area of expertise. Investing heavily into a data science team without hiring data engineers to support them means the skills of your data science team will go underutilized, and it will be incredibly difficult to execute any data-forward strategy.

How to Identify Data Engineering Talent

In our 2021 Data Engineer Salary Report, we found that data engineers typically hold a Bachelor’s or Master’s degree in Computer Science, Information Systems, or Computer Engineering, with the most common degree being a Master’s degree (62% of the sample), followed by Bachelor’s degrees (32%), and PhDs were rare (5%). By comparison, PhDs are a lot more common among data scientists, who will also have more math and statistics education in their background to focus more on data modeling and analysis.Strong data engineering talent will have strong experience with a wide range of technical tools and technologies and will often work alongside data scientists to prepare data for analysis and put data products into production. Because of the wide variety of their job responsibilities, there is no singular tool that makes someone a data engineer, and so we find that most data engineers will have a very broad set of experience with many tools. Check out this post for more information on how we define data engineers, typical job responsibilities, and technical tools.As the needs of data teams continue to shift with team growth and new technologies, we’ve seen an increasing prevalence of hybrid or blended-type roles, that may mix skillsets from data engineering, data science, analytics, and other areas (like DevOps). This may include professionals like Machine Learning Engineers, Data Science Engineers, and Analytics Engineers, among others, which you can read more about in this post. As the needs of data teams continue to evolve, we’re continuing to see more shifts in the market with regards to hybrid-type roles.

Hiring the Right Data Engineering Talent for Your Team’s Needs

Finding the right talent in a field like data engineering, where there are so many tools and technologies that may be required for a role on your team, can be very challenging. From our many conversations with hiring managers, we wanted to share some of our advice for streamlining your talent search:

• Consider what is required for talent on day one vs. what you can train
• Prioritize your needs: what is required vs. nice to have
• Work with your hiring manager to think beyond keyword searches
• Choose an appropriate salary range based on required experience
• Consider talent that can be trained up to fit your role
• Support training opportunities to widen your potential talent
• Provide benchmark candidates to guide your recruiters’ searches
• Consider whether remote talent might widen your talent pool

As many employers are well aware, the Great Resignation continues to impact the current talent pool, as well as the work priorities for available talent. Since many data professionals changed jobs in 2021, there have been less data engineers on the job market in 2022, and with so many teams still looking to hire, this has made competition fierce. Additionally, the conversation around hybrid schedules and remote work continues to shift, and there is sometimes a disconnect between employers and their teams about which approach will satisfy both parties. For more information about how data teams can mitigate impacts from the Great Resignation on hiring and retention efforts, check out this post.

Building a Supercharged Data Engineering Team

So how are organizations approaching their needs for building out a strong data engineering team? For companies where data processes are not built out or well-developed yet, this often requires a senior data engineering hire first. A data engineering leader can help to establish strategy and develop a roadmap, as well assist in hiring the right team to support their strategy. Typically, data scientist hires would come after the data engineering team has already been established. In cases where there is an offshore data engineering team, we’d recommend having at least some onshore talent who can act as the liaison or lead. Typical reporting structures and team organization often varies significantly between companies, and often depends on job responsibilities as well as the structure of the rest of the organization.Data engineering as a profession continues to grow in importance, and as more companies push towards revamping legacy systems and overhauling their data strategy, we’ve seen the need for this talent increase dramatically. In some cases, it may make sense to bring some of this talent on board to train them on your own systems to push along digital transformation efforts along with your data engineering talent. We’ve seen many organizations that have built out their own internal training programs to capitalize on high potential candidates that may not have necessarily had enough experience with every tool they desire. This can be an easier hiring solution than searching for unicorn candidates that must already possess the skillset to cover every area at once.

Additional Resources when Considering Compensation for a Data Engineering Team

As with many other fields, salaries remain a top priority for data engineering talent, and that includes both hiring plans as well as retention efforts. Our research has also found that data engineers tend to receive strong bonuses, and for many this is considered an expected part of their compensation plan. In our full industry report you can find a plethora of data and insights that are crucial to managing talent strategy, including WFH, industry trends, salaries examined by various factors, attrition insights, and more.While we’re still working on our 2022 data engineering salary report, a few months ago we did examine the typical base salary increases that data engineers were receiving when changing jobs, and found that 51% of our sample got a base salary increase of 20% or more. Salaries can be impacted by a number of factors, so we recommend consulting our full report for more information on how this might impact your talent strategy.