Senior LLM Inference Backend Engineer

Responsibilities

• Infrastructure and high-load inference (Platform Engineering):

- Design and support GPU inference cluster (Nvidia) for thousands of users: scaling, load balancing, and request prioritization, user limits. - Performance optimization: caching, batching, etc. - Reliability and observability: ensure SLA and prevent regressions. - Evaluate and implement new LLM.

• Product development (LLM Application):

- Develop AI services: AI Code Assistant (Co-Pilot), AI Chat, AI Code Review. - Design automation pipelines: tool/function calling, error handling, context/state management. - Integrations: embedding AI services into corporate environment (API, DB, legacy systems).

• Research and prototyping (R&D):

- Conduct experiments, create MVPs, and build paths to production. - Research new technologies and approaches to improve products.

Requirements

• Expertise in Python & Backend:

- 5+ years of Python development experience, knowledge of standards: typing, asynchronous programming, design patterns. - Experience creating high-load APIs (FastAPI or similar), working with queues, workers, and background tasks.

• Architecture and reliability (Production):

- Experience building gateway/router, key management, routing, user limits. - Experience supporting high-load services in production: contract stability, error handling.

• Deep knowledge of LLM & Inference:

- Experience with inference tools (vLLM, SGLang, OpenAI API), understanding their internal workings. - Experience configuring inference for high load: latency/throughput, GPU resource management. - Experience building solutions with tool/function calling: MCP, guardrails, combating hallucinations.

Conditions

• Full-time employment.
• Work schedule: 5/2.
• Remote or hybrid work format.