Data Engineer – Venn Department

About the role

The Data Engineer will join the VENN department, the data and technology backbone of Raising The Village. Reporting to the Senior Data Scientist, you will design, develop and maintain data pipelines, warehouse layers and quality systems that power program analytics, machine learning platforms and field evaluation tools.

Key responsibilities

• Design and deliver batch and streaming pipelines that ingest data from field collection platforms such as SurveyCTO, ArcGIS and custom mobile apps into the Databricks warehouse.
• Implement ELT/ETL workflows in PySpark and Databricks SQL, applying Medallion Architecture (Bronze, Silver, Gold) to produce clean, versioned data layers.
• Build and maintain Delta Lake tables with strong schema enforcement, ACID‑compliant writes and time‑travel capabilities.
• Participate in sprint‑based delivery cycles, owning pipeline work‑streams from design through deployment and monitoring.
• Contribute to data modeling decisions, including star‑schema design and integration of structured field data with AI model outputs.
• Develop observability frameworks to ensure pipeline reliability and data quality.

Required profile

• 1‑3 years of professional experience in data engineering or a related field.
• Ability to work in a fast‑moving, mission‑driven environment and deliver against an accelerated roadmap.
• Willingness to travel up to 20% for field data integration activities.

Required skills

• PySpark
• Databricks SQL
• Delta Lake
• Medallion Architecture
• ELT / ETL workflow development
• Batch and streaming data pipelines
• SurveyCTO, ArcGIS, custom mobile app data ingestion
• Data modeling (star schema)
• Observability and data quality frameworks
• Python
• SQL