Observability Engineer

What You'll Be Doing

• Designing and developing a next-generation, high-performance observability platform using modern tooling and best practices.
• Driving the adoption and implementation of OpenTelemetry (OTEL/OTLP) for metrics and log ingestion across the organisation.
• Building and maintaining Infrastructure as Code (IaC) and configuration management pipelines to automate platform operations.
• Contributing to CI/CD pipelines and deployment workflows to support continuous delivery of observability components.
• Owning and improving the reliability, scalability, and performance of the observability stack.
• Collaborating with engineering teams to define and track Service Level Indicators (SLIs) and establish reliability standards.
• Participating in code reviews, architectural discussions, and technical planning sessions.
• Supporting capacity planning, release processes, and change management practices.
• Championing an automation-first mindset and contributing to a culture of operational excellence.

What We're Looking For

• Infrastructure & Configuration Management

- Proven experience with Infrastructure as Code, specifically Terraform - Configuration management experience using Ansible and/or Chef

• Coding Skills

- Strong scripting and development capabilities in Bash, Python, and Go - Experience with Rust and/or Java is a nice to have

• CI/CD & Version Control

- Solid understanding and hands-on experience with Git and GitLab - Experience with ArgoCD and/or Jenkins for CI/CD pipeline management

• Monitoring & Observability

- Metrics platforms: Prometheus, Mimir, InfluxDB, VictoriaMetrics - Log aggregation: Splunk, Loki, Elasticsearch, VictoriaLogs - Visualisation: Grafana - Log forwarding: FluentBit, Vector.dev - OpenTelemetry (OTEL/OTLP): Metrics and log pipeline experience

• Containers & Orchestration

- Strong experience with Docker and/or Podman - Kubernetes (including workload management and cluster operations) - Helm for Kubernetes package management

• Cloud

- Hands-on experience with AWS

• Core Concepts & Principles

- You'll need a solid understanding and proven experience of the following: - Systems Architecture: ability to design and reason about complex distributed systems - Networking principles: TCP/IP, DNS, load balancing, proxies, and beyond - TSDB (Time Series Database) principles: understanding of storage, retention, cardinality, and query patterns - Logging principles: structured logging, log pipelines, and log management at scale - Messaging queues: pub/sub patterns, event-driven architectures - Clustering principles: high availability, fault tolerance, distributed coordination - Load balancing: strategies, implementations, and traffic management - API / JSON principles: RESTful APIs, data serialisation, and integration patterns - Caching mechanisms: In-memory caching, external caches, and cache invalidation strategies - Code reviews: Contributing to and leading thoughtful, constructive reviews - Testing and documentation: Writing quality tests and clear technical documentation - Agile mindset: Comfortable working in iterative, collaborative delivery environments - Automation mindset: Defaulting to automation and repeatability in everything you build

• Nice to Have

- Experience with AI concepts and tooling, including the creation of MCP's and agent Skills - Microservices architecture: Design patterns, service meshes, inter-service communication

• Operational Experience

- Change Management: Structured approaches to managing infrastructure and service changes - Service Level Indicators (SLIs): Defining, measuring, and acting on reliability metrics - Capacity Planning: Forecasting infrastructure needs based on usage trends and growth - Release Processes: Experience with progressive delivery, feature flags, and rollback strategies.