Connecting to LinkedIn...

Data Engineer - Spark, Hadoop, Python, AI, Machine Learning

Job Title: Data Engineer - Spark, Hadoop, Python, AI, Machine Learning
Contract Type: Permanent
Location: Sydney
Salary: 140000
REF: 2817044
Job Published: about 1 month ago

Job Description

Big Data Engineer | Data Engineer | Data Scientist (Scala, Spark, Hadoop, Python).

Fantastic opportunity for a talented Data Engineer to join a successful Tech House that provides data science services and complex software solutions to a range of industries helping them innovate and drive transformation.

As a Data Engineer, you will have a number of responsibilities including; documenting solution requirements, managing and transforming high volume data, performing advanced analytics to identify trends associated with adverse behaviour and automating analysis to proactively alert on high-risk activity. This is a challenging and interesting role where you will be using leading open source data science tools including Scala, Spark and Hadoop. 

Based in Sydney CBD, you will be working alongside a world-class team of friendly, motivated and talented people, who are passionate about their craft.

  • Experienced in Java, Scala and/or Python, Unix/Linux environment on-premises and in the cloud Java development and design using Java 1.7/1.8. Advanced understanding of core features of Java and when to use them
  • Experience with most of the following technologies (Apache Hadoop, Scala, Apache Spark, Spark streaming, YARN, Kafka, Hive, HBase, Presto, Python, ETL frameworks, MapReduce, SQL, RESTful services).
  • Hands-on experience building data pipelines using Hadoop components Sqoop, Hive, Pig, Spark, Spark SQL.
  • Must have experience with developing Hive QL, UDF's for analysing semi-structured/structured datasets
  • Experience with time-series/analytics db's such as Elasticsearch
  • Experience with industry standard version control tools (Git, GitHub), automated deployment tools (Ansible & Jenkins) and requirement management in JIRA
As a Data Engineer (Spark, Hadoop) you can expect to earn a competitive salary plus bonus and benefits. This is an immediate need offering a quick interview process.

Social Stream