In September, we announced the launch of a new version of Resource Tracking in Q4 to increase stability, speed up performance, increase transparency and allow for continued scalability.
On October 29, 2018, we will commence the integration of Phase 1. Clients included in Phase 1 will experience a simpler layout, a powerful pivot and visualization tool, and increased performance. If you are not currently set to be included in the Phase 1 rollout, contact your CSM to discuss migration into the new infrastructure.
Through various working sessions with our clients, we synthesized 3 primary needs around resource tracking:
- Getting a high-level understanding of consumption
- Using a flexible tool to construct their own analyses
- Accessing a detailed grid of interactions for interaction-level investigation
To meet these needs, we:
- Simplified the number of tabs and graphs
- Introduced a powerful new pivot table
- Increased the performance of our data grid and Excel export
1. Simplified Navigation and Overview
To help our clients quickly find their important information, we removed the static profile pages (Broker, Team, User Profile) and left only 3 tabs: Overview, Data Pivot, and Data Grid. Additionally, we simplified the Overview tab to contain only the “Interaction by Brokers” and “Interactions by Teams” charts, which answer the two most common questions asked by users:
- Who is giving me the most value?
- Who is consuming the most value?
2. Data Pivot
We greatly increased the capabilities of the pivot table to give our users the ability to analyze their data through their preferred lens.
- Pull interaction data into pivot table - select from a wide range of fields for the rows and columns of the table and the ability to sort and filter
- Construct nested pivot tables – create rows and column with different levels of drill-downs across various categories. For example, you can group rows by Broker and Sector and columns by Interaction Type and Meeting Types
- Customize variables and displays – construct your own calculated values by combining the existing fields with a host of mathematical and logical operators. For example, you can create a custom variable that counts the number of interactions which have less than 2 companies and cost greater than $1,000.
- Select the type of chart you prefer - choose from variety of visualizations for your data pivot
3. Data Grid
While the appearance of the data grid may seem unchanged, we halved the load times for the page and are testing additional performance improvements.
Resource Tracking 2.0: Technical Overview
The following section provides a technical overview of the infrastructure driving the performance and stability gains behind Resource Tracking v2.0 (RTv2). Please contact your CSM if you want a more detailed walk-through with our engineers.
The existing Resource Tracking tech stack is Nginx, MSSQL, and Knockout.js with architecture built around a single large application and database. This decision was instrumental in the quick buildout of the product, but could not sustain the increasing volume and complexity of the data over time. The performance of the system became database (DB) bound and failure in one area of the application would affect multiple components.
To hit the performance, scalability, and stability targets for Resource Tracking v2, we massively simplified the data model for interactions and introduced a new architecture and tech stack. To minimize disruption, RTv2 is loosely coupled with the existing Resource Tracking system. This allows us to run the two systems in parallel and transition clients in phases.
The end goal is completely transition away from the existing Resource Tracking architecture.
Performance and Scalability
RTv2 is built around a micro-services, CPU-bound architecture utilizing a tech stack of Python, Gunicorn, Nginx, MySQL, and Knockout.js. All services run behind Nginx and are horizontally scalable, making the addition of instances to handle load spikes a straightforward task. Further improvements to the performance of the system include implementing a modern task queuing and distribution software (RabbitMQ and Celery), separating the Audit Database from the Interactions Database, and simplifying the data model to reduce the number of tables used by interactions from 200 to 50.
We enhanced and tightly integrated the infrastructure for testing, deployment, and monitoring to increase the stability of Resource Tracking. (1) Testing harnesses utilizing py.test for Back-end and stress-testing and Selenium for Front-end. (2) Deployment down-time is minimized to minutes as services run on Docker. Additionally, by containerizing applications in Docker, service failures are isolated and do not start cascading failure through system. (3) We implemented robust monitoring to help Visible Alpha in both being proactive in identifying issues and to aid in debugging. The new monitoring uses a tech stack comprised of Sentry, Nagios, New Relic, Metabase, Slack, and Pagerduty.
Services and Database
- Nginx: Web service load balancer
- MSSQL: Database for Resource Tracking Version 1 data
- Gunicorn: Python Web Server Gateway Interface, used by largest Python-powered web applications (like Instagram)
- Celery: Python task queuing service focused on real-time operation in background for tasks that would reduce performance of HTTP call
- Rabbit MQ: Message broker software that distributes incoming messages to Celery Workers
- Celery Flower: Monitoring tool for Celery and Rabbit MQ services
- New Relic: Application monitoring centered around software and API calls, web response times
- Nagios: Hardware monitoring of CPU utilization, available disk pace
- Sentry: Monitoring, notifications, and analysis of application exceptions. Alerts are routed to internal Slack channels.
- Metabase: Data record count analysis and alerts, used to track number of interactions loaded and any unexpected changes in record count (large increase/decrease)
- Cacti: Visualization of CPU load and network bandwidth utilization
- MySQL Audit: Database for storing compute audit used in interactions audit tool of RTv2
- MySQL Interactions: Database for storing interactions information, including calculated numbers, for RTv2
- Elk: Support tool used for quick and efficient searching of application logs
Interactions are entered by the Sell-side or Buy-side and stored in existing Resource Tracking MSSQL Database (DB). As interactions are entered into the Resource Tracking MSSQL DB, a copy of the interaction data is sent, via round-robin task switching, to the primary parallel servers of RTv2.
Interactions data is processed for grouping and valuation. Metadata is checked against RTv2 Interactions DB to ensure proper groups are created against existing interactions. Due to the heavy-processing nature of these tasks, they are split into 3 separate services:
- Precompute: responsible for data aggregation and runs in parallel across 4 workers
- Compute: responsible for running the analytics on the aggregated data and runs in parallel across 8 workers.
- Postcompute: responsible for generating fast views on aggregated data and runs in parallel across 4 workers. This service sends data back to the user front-end.
An audit service tracks every step of the compute action, distributed across 4 workers, and stores information in the Audit DB.
RTv2 uses two databases: Interactions DB and Audit DB. The Interactions DB stores all the interaction information and valuation data. The Audit DB stores all calculation audits. Due to the size of audit data, Audit DB is maintained separately from the Interactions DB to prevent performance impacts.
The entire system is monitored and logged for proactive and quick support.