Senior Data Engineer – Independent Contractor

• Build real-time ETL pipelines integrating Asana, SharePoint, AcceleratorApp, Fluxx, and Jibble.
• Design and implement scalable data lake and warehouse architectures supporting over 300,000 beneficiaries.
• Develop robust API connections with error handling and retry mechanisms.
• Optimize data synchronization to occur within 5 minutes, processing 100,000+ records per hour.

• 5+ years experience with Python, Java, or Scala for data engineering
• Proficiency with Apache Spark, Kafka, or similar streaming technologies
• Experience with cloud platforms (Azure preferred, AWS acceptable)
• Expertise in API integration and RESTful services
• Skilled in SQL and NoSQL databases
• Knowledge of data lake and warehouse architectures (e.g., Delta Lake, Databricks)
• ETL/ELT pipeline development and real-time data processing

• Microsoft Azure Data Factory
• Experience with CRM data models and impact measurement platforms
• SharePoint API / Microsoft Graph API experience
• Predictive analytics pipeline setup

• 5 to 8 years in data engineering roles
• Experience in non-profit or social impact sectors is advantageous
• Proven success with multi-system integrations and large data volumes

• Fully functional, integrated data pipeline
• Data lake architecture documentation
• Automated data quality monitoring
• Achievement of performance benchmarks (sync within 5 minutes)

Job Mail