Senior Big Data Engineer (Data Platform)

Responsibilities

• Develop and enhance the best e-commerce data platform on the market.
• Design and develop ETL pipelines on Airflow for Spark (Batch, Streaming), Iceberg, Trino.
• Optimize existing pipelines and jobs on Spark - monitor cluster load and resource efficiency, and optimize as necessary.
• Collaborate with analysts and ML specialists to create/deploy tools and datasets that meet their use cases - maintain regular communication with data ops for platform development.
• Analyze and implement data engineering tools (Lineage, DQ).
• Design data warehouses based on k8s, configure performance, monitoring, plan cluster capacity and other infrastructure - think about what and how to load and in which partitions.
• Solve tasks related to cloud infrastructure implementation - develop new storage architecture.
• Conduct code reviews and participate in architectural decisions regarding the data platform within the team.

Requirements

• Understanding of Kubernetes resource model, configuration review, troubleshooting.
• Experience with Hadoop (HDFS, Hive Metastore).
• Experience with Spark (Spark SQL, streaming, understanding nuances of working with PySpark).
• Proficiency in Apache Iceberg - procedures, snapshots, maintenance.
• Deep understanding of Airflow (2, 3) - idempotency, assets, backfills.
• Strong knowledge of Git, CI/CD.
• Proficient in Python, Scala/Java - OOP, library design, integration/unit tests.
• Good understanding of databases, SQL.
• Bonus:

- Experience with Docker (proficient for setting up local project environments). - Experience with Kafka - understanding architecture and weaknesses. - Experience with cloud technologies: primarily Yandex Cloud. - Experience with Clickhouse. - AI-native development approach: active use of Claude Code / Codex and similar tools, experience building harness systems - agents with tools, automation through LLM, RAG.

Conditions

• Full-time employment.
• Work schedule: 5/2.
• Remote or hybrid work format.