Data Platform Engineer - 12 Month ContractKey ResponsibilitiesPlatform Engineering & DevelopmentDesign, implement, and maintain Big Data platforms (e.g., Hadoop, Spark, Kafka) used across the CIB environment.Build robust batch and real-time data ingestion pipelines using tools like Apache NiFi, Airflow, Spark, and Kafka Streams.Maintain and enhance enterprise data lakes and warehouse environments using technologies such as Hive, Delta Lake, and Azure Synapse.️ Cloud & Hybrid IntegrationArchitect and deploy data platform solutions on Microsoft Azure (Databricks, Azure Data Lake Storage, Synapse Analytics).Build hybrid cloud systems to integrate on-premises and cloud-based data infrastructure.Ensure optimal performance, scalability, and cost-efficiency across cloud workloads.️ Data Governance, Compliance & SecurityEnsure platform compliance with data governance and privacy regulations (POPIA, GDPR, BCBS239).Implement robust security controls across infrastructure including encryption, access controls, and audit logging.Work closely with data stewards and governance teams to integrate metadata management and data cataloging tools.Automation & DevOpsDevelop and maintain CI/CD pipelines for automated testing, deployment, and monitoring of data solutions.Automate infrastructure provisioning using tools like Terraform and Azure DevOps.Perform routine system administration, performance tuning, and issue resolution across data platforms.Monitoring & SupportImplement monitoring solutions (e.g., Prometheus, Grafana, ELK Stack) to ensure system availability and reliability.Provide L2/L3 support for production data environments, managing incidents and service requests effectively.Stakeholder EngagementCollaborate with cross-functional teams including data scientists, analysts, developers, and compliance teams.Translate business and analytical requirements into scalable platform solutions.Participate in Agile sprints and architecture design reviews.Qualifications & ExperienceMinimum RequirementsBachelors degree in Computer Science, Information Systems, Engineering, or related field.5+ years of experience in Big Data engineering or platform operations.Experience in enterprise-grade platforms in banking or financial services.Technical SkillsStrong proficiency with Hadoop ecosystem: HDFS, Hive, Spark, Kafka.Expertise in Azure cloud services: Azure Data Factory, Azure Databricks, Azure Data Lake, Synapse Analytics.Solid programming skills in Python, Scala, Java, and SQL.Familiarity with Terraform, Git, Jenkins, Docker, and Kubernetes.Experience with data governance tools such as Apache Atlas or Collibra is advantageous.Soft SkillsExcellent problem-solving and troubleshooting ability.Strong communication and collaboration skills.Ability to work in a fast-paced, high-stakes environment.Preferred CertificationsMicrosoft Certified: Azure Data Engineer AssociateCloudera Data Platform Certified DeveloperDatabricks Certified Data Engineer AssociateTOGAF or similar architecture frameworks (advantageous)
