Senior Data Engineer (Spark) - Big Data Startup
These products will accompany the startup’s core product, a self-service big data governance and insights platform that is used by employees at enterprise clients. The platform is essentially an Excel spreadsheet on steroids for non-technical users to parse and manipulate large data sets.
The Senior and Lead Data Engineer will be building streaming ingestion pipelines to connect with data lakes at their clients. These pipelines will bring in new data which is then processed by Spark for analytics.
The company is approaching 1 PB of data. This would be entirely green field development. And you’d use all modern technologies with the language of your choice.
Required Skills & Experience
- Proficient in Scala, Java, or Python
- Strong experience with Spark
Desired Skills & Experience
- Understanding of AWS cloud environment
- Experience with Kafka, SQS, or similar
- Exposure to Flink or other streaming big data technologies
- Experience with Scala
- MS or PhD in CS or mathematics related discipline
What You Will Be Doing
- 100% new development in language of choice – some existing Spark jobs in Java
- 100% open source
- End-to-end project-based design and implementation of new ingestion pipelines
- 100% hands-on, flat hierarchy, first engineers hired for new product team will
- XX% Management Duties
- XX% Team Collaboration
- Competitive salary: up to $200K per year, depending on experience
- Equity package
- Well-funded startup with paying Fortune 100 clients
- New development of two new high-profile products