Role overview
We are looking for an experienced backend engineer to improve the reliability, consistency, and statistical accuracy of our real-time data streams. Our systems currently take in 10,000+ events per second and we are looking to redouble our commitment to data quality by building new tooling around our core data feeds. As a backend engineer, you will have an extensive impact on our market data infrastructure and deal with highly complex data coming in from numerous sources at blazing speed.
This is a unique role where you will work full-time on our market data streams, writing both new features and statistical models to supervise application code. This means adding and managing real-time and historical data providers, implementing changes to our price and volume calculation methodologies, protecting our databases against invalid input, and more. This role is focused on our market data streams, which are our highest-throughput components, with potential to expand in scope over time.
Our data is both real time and historical, which each present unique challenges in how we store and provide access to our data. Our APIs in this area of our system are RESTful, and we use a lot of websockets. We like Go for its simplicity and maintainability, and we embrace Rust for the most resource-intensive components. We deploy using Kubernetes on Amazon Web Services. That being said, we're not dogmatic and always believe in using the best tool for the job.
What you'll work on
- Streamline the ingestion of high throughput trade events we receive from dozens of sources around the clock
- Design and build new features to supervise our ingestion system, detect outliers and ensure data quality
- Write and maintain probabilistic & statistical models to help solve ingestion problems: how do we know when a source of data is degraded or down? How should we measure the quality of incoming data in real time?
- Scale up our systems by an order of magnitude