Data Engineer

About the role

The Data Engineer is a core builder on RTV's expanding data infrastructure team, reporting directly to the Senior Data Scientist within the VENN department. You will design, develop, and maintain pipelines, warehouse layers, and data quality systems that power programmatic analytics, machine learning platforms, and field evaluation tools.

Key responsibilities

• Design and deliver batch and streaming pipelines that ingest data from field collection platforms (SurveyCTO, ArcGIS, custom mobile apps) into the Databricks warehouse.
• Implement ELT/ETL workflows in PySpark and Databricks SQL using Medallion Architecture (Bronze, Silver, Gold) to produce clean, versioned data layers.
• Build and maintain Delta Lake tables with schema enforcement, ACID‑compliant writes, and time‑travel capabilities.
• Develop data observability frameworks, including freshness, volume anomaly detection, schema drift monitoring, and validation checks with tools such as Great Expectations or dbt tests.
• Integrate household data, image classification outputs, and ML model predictions into unified warehouse layers for the WorkMate AI platform and support feature‑engineering pipelines for computer‑vision and scoring models.

Required profile

• Bachelor's degree in Computer Science, Data Engineering, Statistics or related field, or equivalent practical experience.
• 1‑3 years of experience building production‑grade data pipelines.
• Demonstrated ownership of pipeline design through deployment and monitoring.
• Ability to work in sprint‑based delivery cycles and collaborate with cross‑functional teams.

Required skills

• Python (including Pandas)
• SQL
• PySpark
• Databricks platform
• Delta Lake
• Unity Catalog
• Great Expectations or dbt testing
• AWS (cloud environment)
• BI tools