Senior databricks data engineer (embedded platform practitioner)

Job description

A leading organization is seeking a senior Databricks data engineer to operate as an embedded technical practitioner within one or two vertical delivery pods. This position functions as a player-coach, working directly inside the pod to elevate delivery quality and reinforce established data engineering operating standards. The role is fully hands-on and delivery‑accountable, participating in the same cadence, ceremonies, and expectations as internal engineers.

The practitioner is expected to guide architecture and implementation in real time, pairing with engineers when helpful and intervening directly in delivery when patterns drift from defined standards.

Responsibilities

Embedded pod delivery

• Participate fully in agile pod ceremonies, including standups, sprint planning, retrospectives, and demos
• Act as an active member of the delivery team, accountable to sprint commitments and shared outcomes
• Pair with engineers on active workstreams to accelerate learning and unblock complex implementation challenges

Architecture guidance and pattern enforcement

• Provide real-time architectural guidance and pull request feedback on data pipelines, transformations, and data product development
• Ensure ingestion, transformation, and governance work aligns with Databricks best practices and established medallion architecture standards
• Enforce quality and completeness requirements so Gold-layer data products meet defined standards for downstream consumption

Hands-on data engineering

• Build and refine Bronze, Silver, and Gold medallion pipelines using Databricks-native tooling
• Implement ingestion patterns across multiple source types, including databases, files, and APIs
• Apply data quality controls and validation logic between medallion layers
• Support both batch and scheduled workflows using Databricks orchestration capabilities

Operating model enablement

• Identify where delivery patterns diverge from the intended operating model and correct course directly within the work
• Contribute to lightweight documentation and playbooks that capture architectural rationale and repeatable patterns
• Enable internal engineers to independently extend and maintain platform standards after the engagement

Collaboration and delivery alignment

• Partner closely with pod leads and engineers to evaluate trade-offs and make pragmatic architecture decisions
• Balance tactical delivery needs with long-term platform consistency and maintainability

Required experience and skills

• Five to eight years of hands-on experience in data engineering within modern cloud data platforms
• Deep working knowledge of Databricks, including PySpark, Delta Lake, Databricks Workflows, and Unity Catalog
• Proven experience building production-grade medallion architecture pipelines from Bronze through Gold
• Experience implementing ingestion patterns across varied source systems such as databases, files, and APIs
• Strong understanding of data quality practices and governance controls within analytical pipelines
• Familiarity with CI and CD practices for data pipelines, including Git-based workflows, environment promotion, and observability
• Experience working within agile pod or squad-based delivery models
• Ability to coach and collaborate across skill levels, from pairing with junior engineers to advising senior technical leads

This role is ideal for a senior data engineer who prefers deep, hands-on involvement, thrives in collaborative delivery environments, and excels at raising engineering standards through direct participation rather than detached guidance.

FAQ

1. What is the primary focus of a Senior Databricks Data Engineer in an embedded platform role?
This role focuses on building and optimizing data pipelines within a Databricks-based ecosystem while being embedded in a product or business team. It combines hands-on engineering with platform enablement, ensuring data solutions are scalable and aligned with real use cases. The engineer acts as a bridge between centralized data platforms and domain-specific needs.

2. What does “embedded platform practitioner” mean in this context?
An embedded practitioner works directly within a specific business or product team rather than a centralized data team. This enables faster delivery, better context on data requirements, and tighter collaboration with stakeholders. The role ensures platform best practices are applied while meeting domain-specific goals.

3. What tools and technologies are commonly used in this role?
Core technologies include Databricks, Apache Spark, Python, and SQL for data processing and transformation. Cloud platforms like AWS, Azure, or GCP are typically used for storage and compute. Tools such as Delta Lake, Airflow, and dbt may also be part of the stack.

4. What types of data pipelines are built in this role?
The role involves building batch and real-time data pipelines for ingestion, transformation, and consumption. Pipelines often support analytics, machine learning, and operational use cases. Ensuring reliability, scalability, and performance is a key responsibility.

5. How does this role collaborate with data scientists and analysts?
The engineer works closely with data scientists and analysts to provide clean, well-structured datasets. Collaboration ensures that pipelines support modeling, reporting, and experimentation needs. The role also helps optimize data access and usability.