We are looking for a
Data Engineer Confirmed
to join our growing Data team. This position involves the design, development and industrialization of data processing pipelines within a modern Data Mesh architecture, with a high demand for quality, performance and documentation.
Main duties
Design, develop and optimize data processing pipelines using Python, Spark and Scala to ensure efficient and reliable data ingestion.
Participate in the definition and implementation of the Data Mesh architecture, structured in Bronze, Silver and Gold layers, for modular and scalable data management.
Implement and optimize object data storage on MinIO, ensuring the scalability and security of stored data.
Model and partition data in PostgreSQL to ensure optimal performance and scalability for the volumes being processed.
Structure and store data in Parquet format, with a view to optimizing storage and speed of requests.
Expose and query data via federated SQL using Trino/Starburst for seamless integration with different data sources.
Participate in the operation and deployment of services on a containerized infrastructure, managed via Rancher, in a secure cloud environment.
Implement streaming and data change capture streams (Kafka, Debezium) and orchestrate processing with Airflow to ensure data responsiveness and consistency.
Job overview
Write technical documentation and presentations for each context or data area, making it easier to understand and maintain solutions.
Contribute to good data engineering practices, including quality, testing, code standards and code review, to ensure operational excellence.
Technical Stack:
Languages: Python, Scala.
Data processing: Apache Spark.
Storage object: MinIO.
Database: PostgreSQL (with table partitioning).
Data format: Parquet.
Architecture: Data Mesh (Bronze / Silver / Gold).
Federated SQL query: Trino.
Orchestration of workflows: Apache Airflow.
Streaming & CDC: Kafka, Debezium.
Containerization / Orchestration: Rancher (SUSE), Kubernetes.
Cloud: public cloud environment, S3 compatible storage, identity management and access (IAM).
CI/CD & Infrastructure as Code: GitLab CI, Terraform
Observability: Prometheus, Grafana, centralization of logs.
Data governance & cataloguing: data catalog tools.
Versioning: Git.
Documentation: writing presentation documents by context/field.
This post
Data Engineer confirmed
requires an experienced candidate who can effectively manage data processing and architecture projects while ensuring rigorous compliance with current standards.
Candidate profile
Experience
Level of experience required in the field of data engineering, with significant expertise in production environment.
Other information
Level of study required: Master or equivalent Bac+5 in computer science, engineering school or university.
Languages required: Bilingual French and English proficiency.
Education / Qualifications
Relevant business sectors for the position: Information Technology, Big Data, Cloud Computing, Software Engineering.
Qualifications and Skills:
Advanced Python and Scala for data pipeline development and analytics applications.
Good knowledge of Apache Spark for distributed processing of large data.
Concrete experience in the design and implementation of Data Mesh architectures, including Bronze, Silver and Gold environments, as well as managing lakes and data warehouses.
In-depth knowledge of Trino for federated SQL querying, allowing the querying of multiple data sources in real time.
Strong command of PostgreSQL, especially in table partitioning to optimize performance and scalability.
Ability to work in containerized and orchestrated environments, including Kubernetes and Rancher, ensuring effective deployment management.
Excellent writing skills for technical documentation and project monitoring.
Autonomy, rigor, team spirit, and ability to integrate quickly in the customer to ensure a quality service.
Sensitivity to DevOps best practices, including CI/CD to automate deployments and ensure stable environments.
Experience working in agile mode (Scrum, Kanban), with a strong ability to adapt to changes.
Excellent oral and written communication skills, with French-English bilingualism, facilitating international collaboration.
Job overview
Sense of innovation, curiosity for new data technologies, and ability to offer innovative solutions.
---
[Click the Apply button below to apply, and Create my CV to build a CV tailored to this offer, professionally]