Job description
A leading organization is seeking experienced data engineering contractors to support the modernization of a high-volume healthcare data platform. This initiative focuses on rebuilding a critical event processing system into a scalable, cloud-based architecture using modern data engineering tools and frameworks.
The role involves working on a phased migration from a legacy system to a standardized ingestion and processing platform. Engineers will contribute across multiple areas, including legacy system stabilization, development of scalable ingestion pipelines, and integration of downstream data models and business logic.
Responsibilities
- Design, build, and enhance scalable data pipelines within a modern cloud-based data platform
- Support migration of legacy data processing systems to standardized ingestion architectures
- Create and maintain testing environments to stabilize and validate legacy systems
- Troubleshoot complex data pipeline issues and support system reliability during transition periods
- Develop and optimize ingestion pipelines to ensure alignment with standardized data models and processing layers
- Implement high-volume data processing workflows using streaming and event-driven architectures
- Build and maintain data pipelines that integrate with messaging systems for real-time data distribution
- Develop logic to handle entity matching, system events, and data state transitions
- Translate ingestion-layer outputs into downstream data models and business logic processes
- Support implementation of complex transformation logic, including state management and event triggering
- Participate in integration testing, data validation, and system troubleshooting across environments
- Collaborate with cross-functional engineering teams to ensure seamless execution and deployment
- Ensure smooth transition of data traffic through dual-processing systems during migration phases
Required experience and skills
- Proven experience in large-scale data engineering and distributed data processing environments
- Strong hands-on experience with Databricks and Apache Kafka
- Experience working with relational databases such as PostgreSQL
- Proven ability to analyze, refactor, and modernize legacy data pipelines
- Experience designing systems that handle high-volume event processing and stateful data tracking
- Strong debugging and troubleshooting skills in complex data environments
- Experience supporting phased migrations, including dual-write systems and controlled traffic cutovers
- Familiarity with data modeling, ingestion frameworks, and scalable pipeline design
- Ability to work within collaborative engineering teams and contribute to system design and delivery
FAQ
1. What are the primary responsibilities of a Senior Data Engineer specializing in Databricks and streaming pipelines?
A Senior Data Engineer designs, develops, and maintains scalable data pipelines that process large volumes of data in real time and batch environments. The role focuses on modernizing enterprise data platforms, building cloud-native solutions, optimizing streaming architectures, and ensuring reliable data delivery for analytics, reporting, and AI applications.
2. How does this role support healthcare data platform modernization?
The engineer helps migrate legacy data processing systems to modern cloud-based architectures by designing standardized ingestion pipelines, improving data reliability, and supporting scalable event-driven processing. This modernization enables organizations to manage growing healthcare data volumes while improving performance and operational efficiency.
3. What is the role of Databricks in this position?
Databricks serves as the primary platform for large-scale data processing, engineering, and analytics. The engineer uses it to develop data pipelines, process streaming and batch workloads, optimize distributed computing, and support downstream analytics and machine learning initiatives.
4. Why are streaming pipelines important for this role?
Streaming pipelines enable organizations to process and analyze data as events occur rather than waiting for scheduled batch jobs. This supports real-time monitoring, event processing, operational reporting, and time-sensitive business decisions while improving responsiveness across enterprise systems.
5. How does Apache Kafka contribute to data engineering workflows?
Apache Kafka facilitates reliable, high-throughput event streaming between systems. The Senior Data Engineer designs and maintains messaging architectures that enable scalable event ingestion, data synchronization, and real-time communication across distributed applications.
6. What role does legacy system migration play in this position?
A key responsibility is analyzing existing data pipelines, stabilizing legacy systems, and implementing phased migration strategies that minimize operational risk. Engineers often support parallel processing, controlled traffic transitions, and validation processes to ensure successful modernization.