Job description

A leading organization is seeking a Lead Data Engineer to deliver automation and data integration solutions for HEDIS workflows. The Lead Data Engineer will own technical architecture, build real-time data pipelines, and coordinate delivery across three workstreams.

The position requires hands-on data engineering expertise and the ability to establish scalable, reusable automation capabilities that reduce manual work.

Responsibilities

• Design and own the architecture for HEDIS data automation solutions.
• Build and maintain real-time data pipelines.
• Develop automated pipelines that reduce manual processing.
• Complete data integration and automation development work.
• Coordinate technical delivery across three workstreams.
• Use Python, PySpark, and SQL to develop data processing solutions.
• Build and support cloud-based data workflows using GCP.
• Partner with technical teams to align architecture, integrations, and pipeline delivery.
• Create reusable automation capabilities that support current and future HEDIS workflows.
• Improve the scalability and efficiency of data processing operations.

Required experience and skills

• At least six years of relevant data engineering experience
• Advanced SQL skills
• Python development experience
• PySpark experience
• GCP experience
• Experience building real-time data pipelines
• Data pipeline automation experience
• Data integration and automation development experience
• Experience coordinating delivery across multiple workstreams
• Experience owning data architecture decisions
• Experience supporting HEDIS-related data workflows

FAQs

1. What does a Lead Data Engineer do?

A Lead Data Engineer designs and leads the development of data platforms, pipelines, and infrastructure that support analytics, applications, and data-driven products. The role combines hands-on engineering with technical leadership, architecture decisions, delivery planning, and mentoring of other engineers.

2. What types of data pipelines does a Lead Data Engineer build?

Data pipelines may ingest, transform, validate, and deliver data from databases, APIs, files, streaming systems, and other sources. Lead Data Engineers design pipelines for reliability, scalability, maintainability, and efficient downstream consumption.

3. Which technologies are commonly used by Lead Data Engineers?

Lead Data Engineers may work with Python, SQL, Spark, Kafka, Airflow, dbt, and cloud-native data services depending on the organization’s architecture. Cloud platforms such as AWS, Azure, or Google Cloud may provide storage, compute, orchestration, databases, and data warehouse capabilities.

4. How does a Lead Data Engineer design scalable data architecture?

Architecture decisions consider data volume, processing requirements, latency, reliability, security, cost, and future growth. Lead engineers select appropriate storage, processing, orchestration, and integration patterns while establishing standards that allow data systems to scale without unnecessary complexity.

5. How does a Lead Data Engineer ensure data quality?

Data quality is supported through validation rules, schema checks, monitoring, reconciliation, automated tests, and clearly defined data contracts. Lead engineers also establish processes for detecting and resolving pipeline failures, missing data, duplicates, and unexpected changes in source systems.

6. What role does SQL play in data engineering?

SQL is fundamental for querying, transforming, validating, and modeling structured data in databases and data warehouses. Lead Data Engineers use advanced SQL to develop efficient transformations, optimize queries, investigate data issues, and support analytical data models.

7. How does a Lead Data Engineer work with data scientists and analysts?

The role works closely with data scientists, analysts, product teams, and business stakeholders to understand data requirements and provide reliable datasets. Collaboration may involve defining schemas, developing reusable data models, improving data accessibility, and supporting machine learning or analytics workflows.

8. How does a Lead Data Engineer handle pipeline reliability and monitoring?

Reliable pipelines require automated monitoring, logging, alerting, retries, failure handling, and operational visibility. Lead engineers establish service-level expectations and observability practices so teams can identify failures quickly and understand their impact on downstream systems.