About the position
PayPay is looking for a Data Engineer to work on our payment system to deliver the best payment experience for our Japanese customers.
Our ideal candidate is an problem solver and a learner at heart. You should have a very high degree of comfort in a programming language of your choice. We offer a flexible work schedule and you get your choice of tools (both hardware and software) and most importantly, the opportunity to work on tough problems with the brightest minds around you.
- Work directly with Data Analysts and Platform Engineering Team to create reusable experimental and production data pipelines
- Understand, tune, and master the processing engines (like Spark, Hive, Cascading, etc) used day-to-day
- Keep the data whole, safe, and flowing with expertise on high volume data ingest and streaming platforms (like Spark Streaming, Kafka, etc)
- Sheppard and shape the data by developing efficient structures and schema for the data in storage and transit
- Explore as many new technology options for data processing, storage, and share them with the team
- Develop tools and contribute to open source wherever possible
- You have previously worked on building data pipelines ingesting and transforming large number of events per minute and terabytes of data per day.
- You have worked with Spark and Kafka before and have experimented or heard about Flink/Druid/Ignite/Presto/Athena and understand when to use one over the other. Preferralbly, you have worked with cloud based big data processing platform such as AWS EMR, Google Cloud DataProc.
- You are passionate about producing clean, maintainable and testable code part of real-time data pipeline.
- You understand how microservices work.
- You can connect different services and processes together even if you have not worked with them before and follow the flow of data through various pipelines to debug data issues.
- You understand issues with ingesting data from applications in multiple data centres across geographies, on-premise and cloud and will find a way to solve them.
- Proficient in Java/Scala/Python/Spark
- Software development experience in one or more general purpose programming languages, preferably Scala.
- Experience with implementing platform components such as RESTful APIs, Pub/Sub Systems, Database Clients.
- Experience with AWS services.
- Contribution to open source projects.
- Knowledge about the Japanese payment industry