Senior Data Engineer

If you like the challenge of streamlining data collection processes, managing and improving Big Data related infrastructure, and wrangling hundreds of millions of data points, consider joining us as we build our next-generation data procurement platform to produce the foundational data that fuels Moz’s product, services, and research.

 Responsibilities:

• Build, maintain, and support large-scale collections and processing systems.
• Assist in the migration of our Hadoop/HBase infrastructure to AWS EMR managed services that serves our keywords relationship metrics plus keywords suggestions and corpus.
• Be part of a Scrum team that designs, swarms, and pairs together
• Implement and improve deployment, monitoring, testing, operational and foundational tools.
• Mentor junior developers
• Be an excellent problem solver and be willing to roll up sleeves to tackle any issue thrown your way

 Required Experience/Skills:

• 5+ years experience architecting and developing large-scale distributed systems
• B.S. or higher in Computer Science or equivalent training and experience
• Experience on Amazon EMR using EMRFS and S3 or HBase cluster using Hadoop’s HDFS and MapReduce infrastructure
• Familiarity with Python, Java, Spark, Ruby and/or Node.js
• Familiarity with source control. Github is preferred.
• Experience with Linux and cloud-based environments
• Experience with Infrastructure as Code such as Terraform or CloudFormation
• Ability to communicate with management, software architects, and developers about project status, APIs and overall system health.
• Work in an agile process – be able to break down work into bite-size chunks and communicate this to the team
• Working knowledge of modern web technologies including cloud based APIs and protocols (REST, JSON)

 Preferred Experience/Skills:

• Data Science background or experience
• Amazon Web Services (AWS) experience with EC2, ECS and S3
• Infrastructure as code provisioning tools such as Terraform/CloudFormation, etc.
• Participation in a technical community is a major plus: e.g., contributions to open-source projects, published works, presentations in academic conferences or industry circles
• AWS networking such as VPC and security groups
• AWS Analytics such as Athena, Glue, Sagemaker, Kinesis

Technologies you will be working with! 

• Python
• Node.js ES6
• Ruby
• Java, Scala
• HBase/EMR clusters, Riak, Redis/Elasticache, MySQL
• AWS Services
• 1000’s of servers