Senior Data Engineer - Spark / Kafka / NiFi

Mastercard is hiring a Senior Data Engineer - Spark / Kafka / NiFi in Pune.

Our Purpose Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we’re helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential. Title and Summary Senior Data Engineer - Spark / Kafka / NiFi

Responsibilities

Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi. Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes. Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity. Translate business and technical requirements into scalable software solutions. Contribute to architecture discussions, technical designs, and engineering standards.

Streaming & Data Platform Engineering Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently. Validate and support Apache NiFi data ingestion, transformation, and routing workflows. Build reliable data pipelines supporting real-time and near-real-time processing requirements. Implement solutions for data replay, recovery, checkpoint management, and failure handling.

Performance & Scalability Analyze system bottlenecks and optimize application performance, throughput, and resource utilization. Improve scalability, reliability, and availability of distributed applications. Develop and test solutions leveraging modern storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms. Build deployment automation and operational tooling to improve platform reliability. Monitor production environments and proactively address operational concerns.

Requirements

Bachelor's degree in Computer Science, Engineering, or related technical field. 6+ years of hands-on software deployment, validation, testing, development experience on large-scale data-intensive applications. Strong expertise in Apache Spark (Batch and Structured Streaming). Strong experience with Apache Kafka and event-driven architectures. Experience developing, testing and supporting Apache NiFi data flows. Proficiency in Scala, pyspark or Python. Strong SQL and data modeling skills. Experience working with distributed storage systems such as Apache Ozone, Ceph, HDFS, or cloud-based object stores. Hands-on experience with Linux, Git, shell scripting, and CI/CD pipelines. Strong analytical, problem-solving, and communication skills. Experience operating production-grade distributed systems in cloud or hybrid-cloud environments.

Preferred qualifications

Experience building observability solutions using monitoring and logging platforms. Knowledge of data governance, metadata management, and data platform best practices. Experience with containerization and orchestration technologies (Docker, Kubernetes). Knowledge of performance engineering, resilience testing, and production readiness assessments. Experience building engineering automation frameworks and reusable validation platforms. Familiarity with observability tools, monitoring systems, and operational analytics. Experience in highly regulated, transaction-processing, or large-scale enterprise environments.

Corporate Security Responsibility

All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

• Abide by Mastercard’s security policies and practices;

• Ensure the confidentiality and integrity of the information being accessed;

• Report any suspected information security violation or breach, and

• Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.

About Mastercard

See the company's official careers page for full details, then apply using the button below.