Responsibilities
- Design and develop end-to-end data pipelines using Python and SQL
- Build and maintain scalable data processing solutions to support underwriting and analytics needs
- Work with large datasets to ensure data quality, consistency, and reliability
- Collaborate with cross-functional teams to gather requirements and translate them into technical solutions
- Implement and manage data workflows using cloud-based platforms and modern engineering practices
- Contribute to version control and deployment processes using standard CI and CD tools
Required Experience and Skills
- At least 3 years of experience in data engineering
- Strong programming skills in Python and advanced proficiency in SQL
- Experience developing data pipelines from the ground up
- Hands-on experience with Azure as a cloud platform, including Azure Databricks
- Familiarity with version control systems such as Git
- Experience with CI and CD tools such as Jenkins, GitLab CI or GitHub Actions
- Working knowledge of Spark for distributed data processing
- Experience with cloud-based data engineering across platforms such as Azure, AWS or GCP
FAQ
1. What are the core responsibilities of a Data Engineer working with Azure and Python?
This role focuses on designing, building, and maintaining scalable data pipelines and cloud-based data platforms. Responsibilities include integrating data sources, transforming datasets, and optimizing data workflows for analytics and reporting. The engineer also supports data reliability, governance, and platform performance.
2. What types of data systems are typically managed in this role?
Systems may include cloud data warehouses, data lakes, ETL pipelines, streaming platforms, and analytics environments. These systems support business intelligence, machine learning, and operational reporting. Scalability and automation are key priorities.
3. How is Microsoft Azure commonly used in this position?
Azure is used for cloud storage, data processing, orchestration, and analytics services. Common services may include Azure Data Factory, Azure Synapse Analytics, Azure Databricks, and Azure Storage. Cloud-native architecture and deployment practices are often involved.
4. What role does Python play in this job?
Python is used for data transformation, automation, scripting, API integrations, and pipeline development. It helps process large datasets efficiently and supports workflow orchestration. Python libraries are also commonly used for data validation and analytics tasks.
5. What tools and technologies are commonly used alongside Azure and Python?
Common technologies include SQL, Spark, Databricks, Airflow, and version control systems such as Git. CI/CD pipelines and monitoring tools are often part of the environment. Data visualization and reporting platforms may also be integrated.
Apply for this position
**If you have already submitted your resume for another Job Opening please do not re-apply to a different role. You can email through Contact Us about your interest in other roles.