Data Engineer - Databricks, Pyspark, Azure

Job Title:

Data Engineer – Databricks, PySpark & Azure

Location:

Toronto Hybrid

Job Summary

We are seeking skilled Data Engineers with strong expertise in Databricks, PySpark, and cloud-based data platforms. The ideal candidate will design and build scalable ETL pipelines, work with large datasets, and contribute to modern data platform development in a fast-paced, Agile environment.

Primary Skills

• Databricks

• Python (PySpark, Pandas)

• Java (ETL development)

• SQL

• Elasticsearch

• CI/CD (Jenkins, Git)

• Azure (preferred) / GCP

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines and data workflows

• Build and optimize data models for large-scale data processing

• Develop and maintain data applications with complex integrations

• Write, optimize, and execute SQL scripts for data processing and analysis

• Work with large datasets across distributed data platforms

• Collaborate with cross-functional teams on data architecture and solutions

• Implement CI/CD pipelines and DevOps best practices

• Develop and integrate APIs and web services

• Ensure high performance, reliability, and scalability of data systems

• Participate in Agile development processes and follow TDD practices

Required Qualifications

• Minimum 4+ years of experience in data engineering or related roles

• Strong programming skills in Python (PySpark, Pandas) or Java

• Hands-on experience designing ETL pipelines and data models

• Experience building and maintaining large-scale data applications

• Strong SQL skills

• Experience with data technologies and databases such as:

• PostgreSQL

• MS SQL Server

• Oracle

• Apache Spark

• Apache Kafka

• Elasticsearch

• Experience with data platforms:

• Databricks or Snowflake

• Experience with cloud platforms:

• Azure (preferred)

• GCP

• Experience with workflow orchestration tools:

• Apache Airflow or Azure Data Factory

• Experience with:

• Web Services & APIs

• CI/CD tools (Jenkins, Git)

• Experience working in Agile environments

• Understanding of Test-Driven Development (TDD)

• Bachelor’s degree in Computer Science, Engineering, or related field

Preferred Qualifications (Nice to Have)

• Understanding of networking protocols and security principles

• Knowledge of Capital Markets domain

• Experience with:

• Docker

• Kubernetes

• Experience with real-time, high availability, and low-latency systems

• Experience developing multi-threaded applications