Data engineer

A large enterprise team is seeking a data engineer to support the development and evolution of data‑driven rule‑based platforms. This role focuses on building, maintaining, and optimizing data pipelines and analytical systems that support advanced logic, automation, and machine learning use cases. The ideal candidate is hands‑on, adaptable, and able to deliver high‑quality work quickly in a collaborative environment.

Experience in healthcare or similarly complex, regulated domains is a plus, but not required.

Responsibilities

• Design, build, and maintain scalable data pipelines and data processing workflows
• Develop and optimize SQL and Python code to support analytics, automation, and rule‑based systems
• Work with large and complex datasets to ensure data accuracy, reliability, and performance
• Leverage distributed processing frameworks to support high‑volume data workloads
• Apply data engineering best practices to support AI and machine learning initiatives
• Collaborate with cross‑functional teams to translate data requirements into technical solutions
• Support ongoing enhancements to data platforms through testing, optimization, and monitoring
• Contribute to continuous improvement of data quality, performance, and reliability

Required experience and skills

• Strong proficiency in SQL and Python
• Experience working with cloud‑based data platforms, including GCP
• Hands‑on experience with Spark for distributed data processing
• Experience using Pandas for data manipulation and analysis
• Exposure to AI and machine learning concepts or workflows
• Ability to learn quickly, adapt to new tools and technologies, and deliver results efficiently
• Approximately three to five years of relevant experience, or equivalent hands‑on capability demonstrating the ability to perform the role effectively

FAQ

1. What are the core responsibilities of a Data Engineer?
A Data Engineer designs, builds, and maintains data pipelines that move and transform data across systems. The role focuses on ensuring data is reliable, scalable, and accessible for analytics and business use. It also involves integrating multiple data sources into centralized platforms like data warehouses or data lakes.

2. What types of data systems does a Data Engineer typically work with?
Data Engineers work with data warehouses, data lakes, and streaming systems. These systems store structured and unstructured data from various sources such as applications, APIs, and databases. The goal is to create a unified and efficient data ecosystem.

3. What tools and technologies are commonly used in this role?
Common tools include SQL, Python, and ETL/ELT frameworks for data processing. Cloud platforms such as AWS, Azure, or Google Cloud are widely used for storage and compute. Technologies like Spark, Kafka, and Airflow are often part of modern data stacks.

4. How is data quality ensured in this role?
Data quality is maintained through validation checks, monitoring pipelines, and implementing governance standards. Automated testing and alerts help identify issues early. Ensuring accuracy, completeness, and consistency of data is a key responsibility.

5. How does a Data Engineer collaborate with other teams?
Data Engineers work closely with data scientists, analysts, and software engineers. Collaboration ensures that data pipelines meet analytical and operational needs. Clear communication helps align technical solutions with business requirements.