Data Engineer (Kinesis & Databricks)

Data Engineer, AWS Kinesis & DatabricksPermanent | Canberra or Melbourne | Up to $140k incl. super | Baseline clearance | December startOne of Australia's most active Data & AI practices is growing its sovereign delivery team and needs a streaming specialist for secure, cleared work. Amazon Kinesis is the primary streaming backbone on this program, so this is a hands-on role moving high-volume, real-time data into a Databricks Lakehouse.The roleYou'll build and run the real-time layer: Kinesis Data Streams and Firehose feeding Databricks through Spark Structured Streaming into Delta tables. You'll own latency, reliability and cost, build monitoring in CloudWatch, and work alongside the batch engineers so streaming and batch pipelines fit together cleanly.Build and support real-time pipelines on Kinesis Data Streams and Kinesis Data FirehoseDevelop streaming solutions in Databricks using Spark Structured StreamingIngest and manage streaming data in Delta tablesTune pipelines for performance, latency, reliability and costBuild monitoring and alerting with AWS CloudWatchWrite clean, testable Python and SQL, and troubleshoot production issuesWhat you'll bringHands-on production experience with Kinesis Data Streams and/or Kinesis Data Firehose (essential)Spark Structured Streaming in productionHands-on Databricks, including Delta LakeStrong Python and SQLAWS fundamentals, particularly IAM, S3 and CloudWatchSound design instincts for scalable, reliable pipelines and strong troubleshooting skillsAustralian citizen with active Baseline clearance (NV1 also welcome)Good to haveUnity CatalogKafka or MSKWhat's on offerUp to $140k including super, Consultant levelPermanent role, starting 1 DecemberCanberra or MelbourneStructured career development, local and international career paths, and flexible work arrangements