Data Engineer with Spark

Job Title

Data Engineer with Spark

Job ID

101177

Responsibilities

• Write effective and scalable data processing code using Spark with Scala, Java, or Python.
• Design and implement robust data solutions.
• Debug applications to ensure low‑latency and high‑availability using Docker and Kubernetes.
• Interface with existing APIs and build REST APIs.
• Implement and refactor ETL processes with Airflow.
• Implement security and data protection using Vault.
• Support various data storage solutions such as COS API and Cassandra.

Required Education

• Bachelor's Degree

Preferred Education

• Bachelor's Degree

Required Technical and Professional Experience

• Proficiency in data processing with Apache Spark (Scala, Java, or Python).
• Experience with Docker and Kubernetes.
• Experience implementing and refactoring ETL processes with Airflow and Python.
• Experience implementing security and data protection with Vault.
• Previous experience with SQL, COS API, and Cassandra.

Preferred Technical Experience

• Knowledge of Kafka, GitLab, and Jenkins.

Equal Opportunity Statement

IBM is an equal‑opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, genetics, pregnancy, disability, neurodivergence, age, or other characteristics protected by the applicable law. IBM is also committed to compliance with all fair employment practices regarding citizenship and immigration status.