Secrets On the Lake jobs

• Azure Databricks & Scala/Spark
• SQL, Python & ETL/ELT
• Sydney-based Data Engineer role

Company Description
Showtime Consulting is a leading provider of Shielded Cloud and Digital Solutions across Australia and New Zealand. We specialise in delivering secure, enterprise-scale technology solutions across cloud, infrastructure, cybersecurity, business applications, and digital transformation programs.

We partner with government and enterprise organisations to build high-performing technology teams that deliver scalable, efficient, and future-ready business solutions.

The Role
We are seeking an experienced Data Engineer to join a large-scale enterprise data environment in Sydney.

This is a hands-on engineering role focused on designing, building, and optimising high-throughput data pipelines using Azure Databricks, Spark (Scala), SQL and Python.

You will be responsible for delivering scalable ETL/ELT pipelines, supporting real-time and near-real-time integrations, and developing modern data solutions using Medallion Architecture and Delta Lake best practices.

Working closely with data, engineering and business stakeholders, you will also help operationalise workloads through CI/CD, scheduling, secrets management and observability.

Key Responsibilities

• Design, build and optimise high-throughput data pipelines using Azure Databricks.
• Develop scalable ETL/ELT pipelines using Spark (Scala), SQL and Python.
• Implement data solutions following Medallion Architecture across Bronze, Silver and Gold layers.
• Apply Delta Lake principles and best practices across data workloads.
• Design and develop real-time and near-real-time data integrations.
• Develop data processing solutions using Spark Streaming and Apache/Confluent Kafka.
• Build and manage Databricks jobs, workflows and scheduling processes.
• Support CI/CD pipelines using Git and tools such as Azure DevOps.
• Implement appropriate secrets management and observability practices.
• Work across cloud platforms including Azure and AWS.
• Support large-scale data processing using technologies including Spark, Hive, Impala and Sqoop.
• Collaborate with technical, business and delivery stakeholders to deliver effective data solutions.

What You'll Need

• Strong hands-on experience with Azure Databricks.
• Strong commercial experience with Spark using Scala.
• Strong SQL skills.
• Strong Python skills.
• Experience designing and building scalable ETL/ELT data pipelines.
• Strong understanding of modern data warehousing concepts.
• Experience with Medallion Architecture and Delta Lake.
• Proven experience designing real-time and near-real-time integrations.
• Experience with cloud platforms, particularly Azure and/or AWS.
• Experience with big data technologies such as Hive, Impala, Sqoop and Spark.
• Experience with real-time data processing frameworks such as Spark Streaming and Apache/Confluent Kafka.
• Understanding of CI/CD practices and associated tools.
• Excellent communication and stakeholder management skills.

Preferred Skills

• Experience working with both Azure and AWS environments.
• Experience with Master Data Management solutions.
• Exposure to large-scale enterprise data platforms.
• Experience with tools including GitHub, Jenkins, Jira and Confluence.
• Strong understanding of data warehousing concepts.
• Experience operationalising Databricks workloads.
• Understanding of automation, monitoring and observability within data environments.