Industry Insights

This post contains segments of the Q&A from our 2021 Data Engineering Trends & Predictions video. Answers have been lightly edited for length and clarity.

Is industry experience becoming more of a requirement in data engineering?

We're used to seeing this in data science and analytics, but it's always been a little different with engineers. Most data engineers would probably tell you that data is data at the end of the day. And that as long as you have the experience working with a certain volume of data and the tools that go along with it, it shouldn't be as much of a requirement to have industry experience.But this idea might be changing if what we’re seeing in the market continues. And to give you an idea of some of the industries requiring this experience, it would be healthcare, and also advertising.

With the increasing prevalence of hybrid roles, are these types of positions beneficial to a data engineer’s career?

The good news is we love data engineers who have had this experience, especially early on in their career. While we wouldn't expect someone to remain in this type of situation for the long term, it can really help you grow your career early on. You get a lot more exposure to tools and different problems, so it really can be beneficial.However, from what we've heard from engineers, most companies who attempt to fill positions like this are doing themselves a disservice. Sometimes it also can give engineers the impression that the company doesn't really know what they want or need.Not only is it in very difficult to find someone who's an expert in many fields at once, but also hiring data engineers for these roles can be genuinely misleading and frustrating. If they find themselves often working on different tasks that are not exactly in their wheelhouse or don't fall under what they prefer to do most of the time.

Are more data engineering jobs are going 100% remote?

Over the past year, there has been a noticeable increase in remote working opportunities. And this has a variety of advantages. For the talent market, this opens doors for both data engineers looking for opportunities in a variety of locations, as well as for employers who are looking to broaden their talent reach.It also can reduce the cost of hiring substantially, especially if companies are able to access talent in less expensive areas, and don't have to pay for relocation expenses. Many companies who might not have been open to remote work prior to the pandemic, have now been able to see firsthand the many benefits of working remotely through the success of their team over the last year.And many of the data engineers we work with now prefer remote work or at least more flexibility. The truth is, commuting takes up a lot of our time. So, remote work always has been something that people tend to like, but it also helps them to put more time in at their desk for better or for worse.

Will remote working reduce salaries? What about other long-term challenges to WFH?

The answer to this is maybe. To give you an example, Mark Zuckerberg said that Facebook employees will have salary adjustments if they choose to live away from the headquarters in Silicon Valley.Another big impact that we have that we've seen is onboarding challenges. For professionals with experience, it might not be as difficult, but for professionals in the first few years of their career, it could be a real challenge.More work from home questions are based on collaboration, sense of communication, and work culture. How do you do well in a long-term, work from home environment?Childcare is also still a big issue, with remote learning becoming the norm, at least for now. It's a really tough balancing act for parents. Many parents have been asking their employers if they can go part-time or have more flexible hours to balance all of this out, and some have altogether decided to quit their jobs to take care of their kids.So, the big question is, what will some of the long-term impacts be from working from home? It will certainly widen the talent pool, and companies will have greater options. On the other hand, job seekers will also have more employment choices, so that can make the market even more competitive than it was before.

What about the migration to the cloud? Are Hadoop skills still in demand?

Cloud is everywhere right now, and strong talent are drawn into organizations that are using the latest tools and technologies to push the business forward. We've had many calls with data engineers and hiring leaders who are trying to get away from Hadoop. This isn’t a huge surprise, especially with the rise of Apache Spark and AWS. Things have been changing very quickly, and a lack of cloud experience has become very limiting for data engineers seeking new opportunities.In terms of in-demand cloud platforms, such as Amazon Web Services, known as AWS, Microsoft Azure, and Google Cloud Platform, otherwise known as GCP, are always in demand. Obviously, there are many, many different tools that go along with all of these products and serve different functions, so it's becoming increasingly common for employers to target their searches around the specific tool.

How can I add tools and skills if my current employer isn’t investing in training?

I have this conversation with early career data engineers quite often, and even with more experienced engineers who are trying to transition into the Big Data world. We usually discuss different options on how to increase their skillset when they don't have the opportunity through their current employer.This is such a common concern, but we’re happy to say that we've seen a number of data engineers have success by working really hard on their own time, gaining professional certifications in specific tools and technologies, and creating an online portfolio of their work.It not only showcases your work, but it actually, in our experience, has a really positive impact on hiring managers. Especially when you include that on your resume, it shows your passion and your drive for the work you're doing and also the work you want to be doing professionally.

Are bootcamps worth it for adding skills?

There's a lot of questions around bootcamps: are they worth it? And, to be honest, we don't think it always takes a boot camp to get where you need to go in data engineering, but online courses can be very valuable if you want to be more competitive in the market. We also can't stress enough that, in addition to taking a course, it's really important to start actually using the programming languages and tools that you're learning, to cement that knowledge.

Have you seen requests for experience with Snowflake becoming more prevalent?

That is a great question, and the answer is yes. This is a highly popular technology, especially with the cloud becoming more of the norm. Snowflake is very flexible, and it can support all three major cloud platforms.So, from what we hear, it's really user friendly and makes building data pipelines a lot easier. The companies we work with as well, it's typically on the job description. We've been seeing that a lot over the last year or year and a half. So, yes, we highly recommend learning it if you have not had any exposure yet.

Any tips for new graduates that are looking to break into data engineering?

Our advice would be to keep learning and practicing your craft even while you're on the job hunt. If you have like a break in-between jobs, stay active. Tools are constantly evolving, so it’s essential to keep up with your skills.For someone just getting out of school, if you don't have work experience, we suggest getting an internship, if you can, to get some work experience. Having that under your belt will be really valuable, and the other part of it is you can start meeting people. Networking will help you meet more people in this industry and this will help you really grow and with that there’s more potential that the right opportunity will present itself.And of course the other thing too is connect with recruiters. Recruiters like us at Burtch Works, we're here to help you. We are happy to give you our perspective on what’s happening in the industry, or even advice on your resume if you want us to take a quick look.

What are some of the most common reasons for a data engineer to be rejected during the hiring process?

This is a tough question, because there's a lot of potential reasons. One of the most overlooked reasons would be communication. Something as simple as not being prepared to describe your work in a clear and concise way, can really negatively impact your job search moving forward. Because communication skills are something that can be really important, depending on the job. If you, say, have to work across different organizations within the company, you have to be able to communicate to a non-technical audience as well.We’ve seen this problem with early career data engineers who maybe aren't aware of this, but even with more experienced engineers who could have 15 or 20 years of experience and maybe never got this advice. You should always practice the way you’re going to present your skills and experience before an interview.Another reason might be if your coding skills aren't up to the level that’s needed for that particular role. Maybe you use Python, but they need Scala. Or, maybe you haven't had the opportunity to work with large volumes of data. Or you don't have cloud experience. The cloud experience issue comes up a lot for our team.

What would you say, from your experience, and expertise, are the most valuable skills to have as a data engineer?

With data engineering, there's a lot of tools that come into play. So we always recommend staying up to date on the latest and greatest technologies that are coming out. To give you an example, earlier, we were talking about Snowflake, but Airflow is also extremely popular, as well as Kubernetes and Docker.We also get a lot of questions from data engineers wanting to get into data science and work on a data scientist team: how do I move out of IT and onto the data science team? Our advice is to make sure you’re familiar with what the data scientists are doing at your company, and be able to talk knowledgably about their work. That will be valuable in making a strong case for what you can bring to the table there.