Senior Data Engineer, MongoDB and BigQuery

Job description

A leading organization is seeking a Senior Data Engineer to lead a proof of concept evaluating MongoDB for a business-critical production application. The evaluation will determine whether MongoDB can provide the required application performance, scalability, and architectural compatibility.

The Senior Data Engineer will own the technical evaluation from design through validation. Key areas of focus include MongoDB architecture and performance optimization, BigQuery integration, and data pipeline development using Cloud Composer or Apache Airflow. The position requires an experienced individual contributor who can work independently and communicate effectively with engineering leaders, architects, data engineers, and data scientists.

Following the proof of concept, the Senior Data Engineer may support additional data engineering and platform development initiatives.

Responsibilities

• Define and execute a MongoDB proof of concept that evaluates performance, scalability, and suitability for a production application.
• Review current system behavior and identify database or application performance constraints.
• Design MongoDB document models, collections, indexes, and query patterns based on application requirements.
• Tune MongoDB queries and indexing strategies to improve performance and resource utilization.
• Establish evaluation criteria, test scenarios, and repeatable performance benchmarks.
• Connect MongoDB components with existing BigQuery and data platform environments.
• Design, build, maintain, and optimize data pipelines using Cloud Composer, Apache Airflow, or similar orchestration tools.
• Apply appropriate data integration patterns across cloud platforms and distributed systems.
• Compare proof-of-concept results against technical and operational requirements.
• Document the proposed architecture, implementation approach, test methodology, benchmark results, technical limitations, and recommendations.
• Present findings and architecture recommendations to engineering leaders and solution architects.
• Collaborate with engineering teams to align the proposed solution with long-term platform requirements.
• Identify opportunities to improve application performance, platform scalability, and operational efficiency.
• Contribute to ongoing data engineering and platform initiatives following the initial evaluation.

Required experience and skills

• 7+ years of experience in data engineering, software engineering, or a related technical discipline.
• Advanced hands-on experience with MongoDB architecture and performance optimization.
• Strong knowledge of MongoDB document modeling, collection design, indexing, query optimization, and performance tuning.
• Experience using Google BigQuery in large-scale data environments.
• Experience developing and managing data pipelines with Cloud Composer, Apache Airflow, or a comparable orchestration platform.
• Strong understanding of database architecture, distributed systems, and data integration patterns.
• Proficiency in Python, Java, or a similar programming language used in data engineering environments.
• Ability to independently lead a technical evaluation from initial design through implementation, testing, and final recommendations.
• Experience creating testing strategies and measurable database performance benchmarks.
• Strong technical documentation and presentation skills.
• Ability to clearly communicate findings and recommendations to engineering and architecture stakeholders.
• Experience conducting proof-of-concept projects or technical product evaluations is preferred.
• Experience migrating workloads between relational and NoSQL databases is preferred.
• Knowledge of cloud-native architectures and modern data platforms is preferred.
• Experience collaborating with architects, engineering leaders, data engineers, and data scientists in enterprise environments is preferred.

FAQ

1. What is the scope of a Senior Data Engineer working with MongoDB and BigQuery?

A Senior Data Engineer designs and maintains data pipelines that connect operational MongoDB data with analytical workloads in BigQuery. The role covers data ingestion, transformation, modeling, quality controls, performance optimization, and reliable delivery of data for analytics and reporting.

2. How is MongoDB used within a data engineering workflow?

MongoDB commonly serves as a source system for application or operational data stored in document-oriented collections. The engineer extracts and transforms this data, including nested or semi-structured fields, so it can be consumed effectively by downstream analytical systems.

3. How does BigQuery support the analytical side of the platform?

BigQuery provides a cloud-based analytical environment for querying and processing large datasets. Senior Data Engineers use it to build analytical tables, develop transformations, optimize query performance, and prepare data for business intelligence and advanced analytics.

4. What are the main considerations when moving MongoDB data into BigQuery?

Engineers need to account for schema differences, nested documents, data types, incremental updates, and changing source structures. The integration approach should also maintain data accuracy, appropriate refresh frequency, and traceability from the MongoDB source to the analytical dataset.

5. What level of SQL expertise is expected for BigQuery?

Strong SQL skills are important for complex transformations, aggregations, validation, and analytical data modeling. Candidates should also understand BigQuery-specific performance considerations such as partitioning, clustering, and efficient query design.

6. How is data quality managed across MongoDB and BigQuery pipelines?

Data quality can be maintained through schema validation, reconciliation checks, automated tests, duplicate detection, completeness checks, and pipeline monitoring. Engineers investigate discrepancies and establish controls to prevent unreliable data from reaching downstream consumers.