Senior databricks data engineer (embedded platform practitioner)

Job description

A leading organization is seeking a senior Databricks data engineer to operate as an embedded technical practitioner within one or two vertical delivery pods. This position functions as a player-coach, working directly inside the pod to elevate delivery quality and reinforce established data engineering operating standards. The role is fully hands-on and delivery‑accountable, participating in the same cadence, ceremonies, and expectations as internal engineers.

The practitioner is expected to guide architecture and implementation in real time, pairing with engineers when helpful and intervening directly in delivery when patterns drift from defined standards. Success in this role comes from leading by example through active contribution, not through detached advisory work.

Responsibilities

Embedded pod delivery

• Participate fully in agile pod ceremonies, including standups, sprint planning, retrospectives, and demos
• Act as an active member of the delivery team, accountable to sprint commitments and shared outcomes
• Pair with engineers on active workstreams to accelerate learning and unblock complex implementation challenges

Architecture guidance and pattern enforcement

• Provide real-time architectural guidance and pull request feedback on data pipelines, transformations, and data product development
• Ensure ingestion, transformation, and governance work aligns with Databricks best practices and established medallion architecture standards
• Enforce quality and completeness requirements so Gold-layer data products meet defined standards for downstream consumption

Hands-on data engineering

• Build and refine Bronze, Silver, and Gold medallion pipelines using Databricks-native tooling
• Implement ingestion patterns across multiple source types, including databases, files, and APIs
• Apply data quality controls and validation logic between medallion layers
• Support both batch and scheduled workflows using Databricks orchestration capabilities

Operating model enablement

• Identify where delivery patterns diverge from the intended operating model and correct course directly within the work
• Contribute to lightweight documentation and playbooks that capture architectural rationale and repeatable patterns
• Enable internal engineers to independently extend and maintain platform standards after the engagement

Collaboration and delivery alignment

• Partner closely with pod leads and engineers to evaluate trade-offs and make pragmatic architecture decisions
• Balance tactical delivery needs with long-term platform consistency and maintainability

Required experience and skills

• Five to eight years of hands-on experience in data engineering within modern cloud data platforms
• Deep working knowledge of Databricks, including PySpark, Delta Lake, Databricks Workflows, and Unity Catalog
• Proven experience building production-grade medallion architecture pipelines from Bronze through Gold
• Experience implementing ingestion patterns across varied source systems such as databases, files, and APIs
• Strong understanding of data quality practices and governance controls within analytical pipelines
• Familiarity with CI and CD practices for data pipelines, including Git-based workflows, environment promotion, and observability
• Experience working within agile pod or squad-based delivery models
• Ability to coach and collaborate across skill levels, from pairing with junior engineers to advising senior technical leads

This role is ideal for a senior data engineer who prefers deep, hands-on involvement, thrives in collaborative delivery environments, and excels at raising engineering standards through direct participation rather than detached guidance.