Senior Data Engineer – Python and Cloud Data Platforms

Job Description

A leading organization is seeking a Senior Data Engineer to build and enhance large-scale data processing solutions that support forecasting, analytics, and business decision-making. This position combines data engineering, backend software development, cloud technologies, and API development to deliver scalable data platforms capable of handling high-volume datasets.

The ideal candidate will have experience developing data transformation pipelines, building backend services, working with distributed data processing frameworks, and supporting cloud-native analytics environments.

Responsibilities

Data Engineering and Processing

• Design, develop, and maintain large-scale data transformation and processing solutions.
• Build and optimize data pipelines that handle high-volume and complex datasets.
• Develop solutions for integrating and processing data from multiple sources.
• Support data ingestion, transformation, validation, and delivery workflows.
• Improve performance, scalability, and reliability across data platforms.

Cloud Data Platform Development

• Develop and support cloud-based data solutions using Google Cloud Platform and Azure technologies.
• Work with services such as BigQuery, Cloud Composer, Dataproc, Dataflow, Azure Databricks, Azure Data Factory, and Delta Lake.
• Build scalable solutions for analytics and forecasting workloads.
• Support cloud-native data architecture and processing frameworks.

Backend Development and API Engineering

• Design and develop backend APIs using Python.
• Build integrations that connect data platforms, applications, and external systems.
• Contribute to software development efforts supporting data-driven applications.
• Develop reusable and maintainable services that support business operations.

Distributed Data Processing

• Utilize Spark and related technologies to process large datasets efficiently.
• Develop and optimize high-volume data workflows for performance and scalability.
• Support batch and workflow orchestration using tools such as Apache Airflow, Azure Data Factory, and Databricks Workflows.

Collaboration and Problem Solving

• Work closely with engineering, analytics, and business teams to deliver technical solutions.
• Analyze complex technical challenges and recommend scalable approaches.
• Participate in design discussions, code reviews, and solution planning.
• Communicate technical concepts effectively to both technical and non-technical stakeholders.

Required Experience and Skills

• 5+ years of professional experience in data engineering, software engineering, or a related field.
• Professional experience building data transformation and processing solutions using Python and SQL.
• Experience developing backend APIs using Python or similar programming languages.
• Strong SQL development and data analysis experience.
• Experience working with large-volume datasets and distributed processing technologies.
• Hands-on experience with Spark.
• Experience developing solutions within Google Cloud Platform and/or Azure environments.
• Experience with data platform technologies including:
  • BigQuery
  • Cloud Composer
  • Dataproc
  • Dataflow
  • Azure Databricks
  • Azure Data Factory
  • Delta Lake
• Experience building large-scale applications and high-volume data pipelines.
• Experience using orchestration tools such as Apache Airflow, Azure Data Factory, or Databricks Workflows.
• Strong problem-solving, analytical thinking, and troubleshooting skills.
• Excellent collaboration and communication abilities.

Required Experience and Skills

• Minimum of 2–3 years of professional experience beyond completion of a master's degree, or equivalent hands-on industry experience.
• Experience combining data engineering expertise with backend application development.
• Ability to work across multiple technologies and data environments.
• Experience integrating APIs and data services within enterprise-scale platforms.
• Familiarity with both data processing and application development concepts.