Data Engineer with Scala and Spark/ Pyspark

Capco
Bengaluru +5 more
On-siteCareer-pivot friendly

Who this role is best for

A natural match if you have experience in large-scale data processing with Scala and Spark.

Best fit for

  • Candidates with experience in building high-performance data pipelines for financial services clients
    — “enterprise-scale data environments within Financial Services, Payments, or FinTech domains”
  • Professionals adept at optimizing distributed systems and handling massive datasets
    — “optimize Spark jobs, data workflows, and distributed computing processes to ensure high throughput and low latency”
  • Individuals with a track record of delivering end-to-end data engineering solutions in hybrid work setups
    — “Hybrid (Capco Office)”

Things to consider

  • The role requires daily processing of up to 90 million records, emphasizing high-volume data handling
    — “processing 80–90 million records daily with a focus on scalability, reliability, and efficiency”

How to stand out

  • Highlight experience in optimizing Spark jobs and improving data pipeline efficiency
    — “optimize Spark jobs, data workflows, and distributed computing processes”
  • Demonstrate hands-on expertise with Scala and Spark in large-scale environments
    — “Strong hands-on experience in Scala, Apache Spark, and Java”
  • Showcase skills in troubleshooting performance bottlenecks in data pipelines
    — “Troubleshoot and resolve performance bottlenecks across data processing and ingestion pipelines”
  • Emphasize your experience with data governance and operational excellence practices
    — “Implement best practices for data quality, monitoring, governance, and operational excellence”
  • Demonstrate your ability to collaborate with cross-functional teams on complex data projects
    — “Collaborate with cross-functional teams to gather requirements and deliver robust data engineering solutions”
Pace · SteadyCollaboration · HighAutonomy · MediumDecision Impact · Team

Derived from job-description analysis by Serendipath's career intelligence engine.

What success looks like

  • Design, develop, and optimize large-scale data processing applications
  • Build and maintain high-performance data pipelines
  • Develop and integrate Native APIs and data services
Typical background
4+ years of experience in data engineeringExperience in enterprise-scale data environments

Skills & requirements

Required

ScalaApache SparkJavaData ModelingEtl/elt ProcessesDistributed SystemsConcurrencyHigh-volume Data ProcessingSQLRelational/non-relational DatabasesDebuggingAnalytical SkillsProblem-solving Skills

Preferred

AWSAzureGCPKafkaAirflowHadoop EcosystemCi/cd PipelinesDevops PracticesAgile Methodologies

Stack & domain

ScalaApache SparkJavaSQLRelational DatabasesNon-relational DatabasesKafkaAirflowHadoopCi/cd PipelinesDevops PracticesAgile MethodologiesProblem-solvingTeamworkData EngineeringFinancial ServicesEnergy Sector

About the role

Original posting from Capco via Greenhouse

About Us

“Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors. We are recognized for our deep transformation execution and delivery. 

WHY JOIN CAPCO?

You will work on engaging projects with the largest international and local banks, insurance companies, payment service providers and other key players in the industry. The projects that will transform the financial services industry.

MAKE AN IMPACT

Innovative thinking, delivery excellence and thought leadership to help our clients transform their business. Together with our clients and industry partners, we deliver disruptive work that is changing energy and financial services.

#BEYOURSELFATWORK

Capco has a tolerant, open culture that values diversity, inclusivity, and creativity.

CAREER ADVANCEMENT

With no forced hierarchy at Capco, everyone has the opportunity to grow as we grow, taking their career into their own hands.

DIVERSITY & INCLUSION

We believe that diversity of people and perspective gives us a competitive advantage.

MAKE AN IMPACT

Job Title: Data Engineer 

Position: Data Engineer

Experience: 4+ Years

Work Mode: Hybrid (Capco Office)

Key Responsibilities

Design, develop, and optimize large-scale data processing applications using Scala, Apache Spark, and Java.

Build and maintain high-performance data pipelines capable of processing 80–90 million records daily with a focus on scalability, reliability, and efficiency.

Develop and integrate Native APIs and data services to support business-critical applications and analytics platforms.

Collaborate with cross-functional teams to gather requirements and deliver robust data engineering solutions.

Optimize Spark jobs, data workflows, and distributed computing processes to ensure high throughput and low latency.

Implement best practices for data quality, monitoring, governance, and operational excellence.

Troubleshoot and resolve performance bottlenecks across data processing and ingestion pipelines.

Participate in code reviews and contribute to engineering standards, architecture decisions, and continuous improvement initiatives.

Required Skills & Experience

Strong hands-on experience in Scala, Apache Spark, and Java.

Proven experience building and supporting large-scale distributed data processing systems handling tens of millions of records daily.

Expertise in developing and consuming Native APIs and microservices.

Strong understanding of Spark architecture, performance tuning, partitioning, caching, and optimization techniques.

Experience with data modeling, ETL/ELT processes, and large-scale batch and streaming data pipelines.

Solid understanding of distributed systems, concurrency, and high-volume data processing.

Experience with SQL and relational/non-relational databases.

Strong debugging, analytical, and problem-solving skills.

Preferred Qualifications

Experience working in enterprise-scale data environments within Financial Services, Payments, or FinTech domains.

Exposure to cloud platforms (AWS, Azure, or GCP) and containerized deployments.

Familiarity with Kafka, Airflow, Hadoop ecosystem, or similar big data technologies.

Experience with CI/CD pipelines, DevOps practices, and Agile methodologies.

What We're Looking For

Engineers passionate about solving complex data challenges at scale.

Professionals who can design highly performant systems capable of processing and transforming massive datasets efficiently.

Team players who thrive in fast-paced, collaborative environments and take ownership of end-to-end delivery.

Source: Capco careers (Greenhouse)

Similar roles

More roles at Capco