Senior Data Engineer

Senior Data Engineer job vacancy in C ape Town. Build serious data systems that actually matter. This is a high-impact senior role for a data engineer who knows how to get the best out of Spark, writes strong Python, and enjoys turning messy legacy pipelines into clean, scalable engineering. You will join a fast-moving team working on modern cloud data platforms, lakehouse architecture, and large-scale processing where performance, quality, and good engineering judgment count. The core tech includes Spark, PySpark, Python, Delta Lake, Parquet, Azure Synapse, SQL, Docker, and modern orchestration approaches. Salary: R80 000 per month CTC. Type: Remote. Key Responsibilities: Design, build, and optimise high-performance data pipelines using Python and PySpark Improve Spark workloads through better memory use, partitioning, shuffle tuning, and DAG optimisation Refactor legacy SQL-heavy ETL processes into modular, reusable Python libraries Build and maintain lakehouse data layers across Bronze, Silver, and Gold Work with Delta Lake and Parquet to improve versioning, schema management, and storage performance Help drive a code-first approach to orchestration and reduce reliance on cloud-specific tooling Support a cloud-agnostic engineering approach with portable, scalable solutions Contribute to code reviews, testing standards, and overall platform quality Partner with analysts, data scientists, and business teams to deliver practical data solutions Mentor junior engineers and help shape strong engineering standards across the team Design, build, and optimise high-performance data pipelines using Python and PySpark Improve Spark workloads through better memory use, partitioning, shuffle tuning, and DAG optimisation Refactor legacy SQL-heavy ETL processes into modular, reusable Python libraries Build and maintain lakehouse data layers across Bronze, Silver, and Gold Work with Delta Lake and Parquet to improve versioning, schema management, and storage performance Help drive a code-first approach to orchestration and reduce reliance on cloud-specific tooling Support a cloud-agnostic engineering approach with portable, scalable solutions Contribute to code reviews, testing standards, and overall platform quality Partner with analysts, data scientists, and business teams to deliver practical data solutions Mentor junior engineers and help shape strong engineering standards across the team

Profil recherché

Formation / Diplômes

Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field

Expérience

• 6+ years of experience working with Spark or PySpark in production environments

Compétences requises

Strong Python skills, with experience building maintainable, production-grade applications Proven ability to identify and fix Spark performance bottlenecks using the Spark UI Solid SQL skills, including the ability to interpret and migrate existing ETL logic Experience with Azure Synapse Analytics, Dedicated SQL Pools, and Data Factory Strong hands-on experience with Delta Lake and Parquet

ℹ Pour postuler :** Veuillez cliquer sur le bouton Postuler présent sur cette page pour découvrir les instructions de candidature.