The Senior Data Engineer plays a criticalrole in designing, developing, and maintaining scalable data infrastructure and pipelines. This position supports data-driven decision-making across the organization by ensuring data quality, accessibility, and performance.
Roles and Responsibilities
- Data Pipeline Development: Design, build, and maintain robust, scalable data pipelines using tools like Apache Airflow, Kafka, and Spark.
- Data Architecture: Develop and optimizedata models, data warehouses, and ETL processes to support analytics and machine learning initiatives.
- Collaboration: Work closelywith data scientists, analysts, and software engineers to understand data requirements and deliver tailored solutions.
- Data ǪualityG Governance: Ensure data integrity, implement validation checks, and enforce data governance policies.
- Performance Optimization: Monitor and improvedata processing speed,system uptime, and query performance.
- Leadership G Mentorship: Guide junior engineers, promote best practices, and foster a culture of continuous learning.
- Innovation G Research: Stay updated with emerging technologies in AI, automation, and data streaming; lead initiatives to integrate them into existing systems.
Profile
- Bachelor's or Master's in computer science, data science, information technology, or a related field, with 6+ years of experience .
- Proficiency in SǪL , Python , and ETL/ELT tools
- Experience in Microsoft solutions are as follows:
- Azure Data Factory
- Microsoft Fabric
- Azure Synapse Analytics
- Azure Databricks
- Power BI
- Azure SǪL Database/ SǪL Server
- Azure Purview
- Experience with big data technologies (Hadoop, Spark)
- Familiarity with cloud platforms (Azure, AWS, and GCP)
- Strong understanding of data modelling , data warehousing , and stream processing
- Knowledge of data security , compliance , and governance
- Knowledge of CI/CD pipelines using Azure DevOps
- Understanding of data security and compliance (e.g., GDPR, HIPAA)