Senior Data Engineer (GCP, BigQuery, Python)

Job description

A leading organization is seeking a senior data engineer to help design, build, and operate scalable, cloud-native data platforms and production APIs on Google Cloud Platform. This position focuses on BigQuery-based analytics, containerized services, and reliable data pipelines that support both event-driven and batch processing. The senior data engineer will contribute to architecture decisions, lead design reviews, and promote engineering best practices across data and service platforms while partnering closely with product, platform, and downstream consumer teams.

Start date: ASAP

Location: Remote

Responsibilities

• Build and maintain cloud-native data platforms and backend services on Google Cloud Platform
• Design, develop, and operate containerized workloads using Google Kubernetes Engine
• Develop and support data pipelines using Google Cloud Composer, including batch and event-driven processing patterns
• Implement and optimize BigQuery-based analytics solutions, including data modeling and query performance tuning
• Create and maintain production-grade APIs and Python-based microservices that expose data and platform capabilities
• Drive architecture decisions, participate in design reviews, and establish best practices for data and service platform reliability
• Partner with product, platform, and downstream teams to define requirements, align on interfaces, and support adoption
• Contribute to operational excellence through monitoring, incident response, and continuous improvement of production deployments
• Support modern software delivery practices including CI/CD workflows, version control, and repeatable release processes

Required experience and skills

• Eight or more years of experience in data engineering, backend engineering, or platform engineering
• Hands-on experience building cloud-native platforms and services on Google Cloud Platform
• Strong experience with Google Kubernetes Engine, Google Cloud Composer, and BigQuery
• Proficiency in Python and SQL
• Experience building and operating microservices and production APIs
• Experience designing and supporting event-driven and batch data pipelines
• Experience with CI/CD pipelines, version control, and production deployments

Preferred qualifications

• Experience working in regulated domains such as healthcare or finance
• Knowledge of security and authentication patterns, including OAuth2 and service-to-service authentication

FAQ

1. What are the core responsibilities of a Senior Data Engineer in this role?
This role focuses on designing, building, and optimizing scalable data pipelines and architectures on cloud platforms. It includes integrating data from multiple sources, ensuring data quality, and enabling analytics and reporting. The engineer also drives best practices in data engineering and mentors junior team members.

2. How is Google Cloud Platform used in this position?
The role leverages Google Cloud Platform to build and manage data infrastructure. Services such as BigQuery are used for large-scale data storage and analytics. The engineer designs cloud-native solutions that are scalable, secure, and cost-efficient.

3. What role does Python play in this job?
Python is used for building data pipelines, performing transformations, and automating workflows. It supports integration with various data sources and cloud services. Strong programming skills are essential for efficient and maintainable solutions.

4. What types of data pipelines are typically built?
The engineer builds batch and real-time data pipelines to process structured and unstructured data. These pipelines may involve ETL/ELT processes, streaming data, and event-driven architectures. The goal is to deliver reliable and timely data for business use.

5. How is data quality ensured in this role?
Data quality is maintained through validation checks, monitoring systems, and automated testing. The engineer implements governance standards to ensure consistency and accuracy. Early detection of issues helps maintain trust in data.

6. What tools and technologies are commonly used alongside BigQuery?
Tools such as Dataflow, Pub/Sub, and Cloud Storage are often used within the GCP ecosystem. Workflow orchestration tools like Airflow may also be used. These technologies support scalable and efficient data processing.