Home > Technology

Real Time Data Technology

The core of our technology is a machine learning platform that trains and executes predictive models. Supporting this is a complex structure of real-time data-flow engineering whose job is to get the right data into the platform at the right time.

The platform introspects the structure of data and extracts salient features for the models. It also provides I/O connectors into various data flows allowing for easy integration into partner and client platforms.

Platform Differentiation

We use our platform to quickly and efficiently build and iterate data-driven  applications using a robust set of development tools.

  • Data sources and models are highly compressed and stay on a single machine, similar to financial trading platforms.  This allows for real-time requirements to be met and requires fewer machines to execute a given model.
  • The performance focused architecture of the core platform is designed and engineered for massive scalability across cores and machines.
  • Modeling is performed via simulations which can be run with perfect fidelity. This provides a rich and productive modeling environment and can be leveraged to provide what-if scenario testing and back-testing.
  • The platform is able to automatically take new data sources into account and to update models in real time.

Platform Architecture

We use a service-oriented architecture,with a heavily optimized core system in C++ glued together with Javascript and zeroMQ. We do everything we can to avoid going to disk or involving multiple machines in a single calculation. Our machine learning is built off the open-source JML library. We use Redis to hold operational data and a combination of nginx and node.js to route and process incoming requests and data events. Data is stored via a combination of proprietary purpose-built technology and SQL. Our visualization framework is built on Protovis, and we use Graphite and Nagios for operational monitoring. We’re big fans of open source of which we use copious amounts and contribute back wherever possible.