Senior Data Engineer (GCP, BigQuery, Python)

Job description

A leading organization is seeking a senior data engineer to help design, build, and operate scalable, cloud-native data platforms and production APIs on Google Cloud Platform. This position focuses on BigQuery-based analytics, containerized services, and reliable data pipelines that support both event-driven and batch processing. The senior data engineer will contribute to architecture decisions, lead design reviews, and promote engineering best practices across data and service platforms while partnering closely with product, platform, and downstream consumer teams.

Responsibilities

• Build and maintain cloud-native data platforms and backend services on Google Cloud Platform
• Design, develop, and operate containerized workloads using Google Kubernetes Engine
• Develop and support data pipelines using Google Cloud Composer, including batch and event-driven processing patterns
• Implement and optimize BigQuery-based analytics solutions, including data modeling and query performance tuning
• Create and maintain production-grade APIs and Python-based microservices that expose data and platform capabilities
• Drive architecture decisions, participate in design reviews, and establish best practices for data and service platform reliability
• Partner with product, platform, and downstream teams to define requirements, align on interfaces, and support adoption
• Contribute to operational excellence through monitoring, incident response, and continuous improvement of production deployments
• Support modern software delivery practices including CI/CD workflows, version control, and repeatable release processes

Required experience and skills

• Eight or more years of experience in data engineering, backend engineering, or platform engineering
• Hands-on experience building cloud-native platforms and services on Google Cloud Platform
• Strong experience with Google Kubernetes Engine, Google Cloud Composer, and BigQuery
• Proficiency in Python and SQL
• Experience building and operating microservices and production APIs
• Experience designing and supporting event-driven and batch data pipelines
• Experience with CI/CD pipelines, version control, and production deployments

Preferred qualifications

• Experience working in regulated domains such as healthcare or finance
• Knowledge of security and authentication patterns, including OAuth2 and service-to-service authentication

FAQ

1. What are the core responsibilities of a Senior Data Engineer in this role?
This role focuses on designing, building, and maintaining scalable data pipelines on Google Cloud Platform. It involves ingesting, transforming, and optimizing data for analytics and business use. The engineer also ensures data reliability, performance, and availability across systems.

2. How is Google Cloud Platform used in this position?
GCP services such as BigQuery, Cloud Storage, Dataflow, and Pub/Sub are used to manage and process data. The engineer designs cloud-native architectures for efficient data handling. Cost optimization and performance tuning within GCP are also key responsibilities.

3. What role does BigQuery play in this job?
BigQuery is used as the primary data warehouse for storing and querying large datasets. The engineer designs schemas, optimizes queries, and manages partitioning and clustering for performance. It supports analytics, reporting, and downstream data use cases.

4. How is Python used in this role?
Python is used for building data pipelines, automating workflows, and performing data transformations. Libraries such as pandas or PySpark may be used for processing data. Python also supports integration with GCP services and APIs.

5. What types of data pipelines are built in this role?
The role involves building both batch and real-time data pipelines. These pipelines handle data ingestion, transformation, and delivery to data warehouses or analytics tools. Ensuring scalability and reliability is a key focus.

6. How does this role collaborate with other teams?
The engineer works closely with data analysts, data scientists, and business stakeholders. Collaboration ensures that data is accessible, accurate, and aligned with business needs. The role also coordinates with DevOps and platform teams for deployment and monitoring.