From Batch to Stream

Prior to the system change, SA Taxi’s existing solution involved feeding data from vehicle tracking devices to an on-premises, Hadoop-based data warehouse, and presenting aggregated data to users through a simple interface. However, this system was limited in terms of the batch processing method that was used in order to receive and present the data to users. With batch processing, data is collected and processed on a schedule. This results in lengthy delays from the time that the data is generated, to the stage at which users are able to garner insights from that data. Unlike stream processing, batch processing does not feed data immediately into the user interface. As a result, data is not available in real time.

To efficiently manage its growing fleet of 22,000 minibus taxis across South Africa, financing, insurance and mechanical assistance firm SA Taxi relies on data. In particular, vehicle tracking data helps SA Taxi monitor driver adherence to obligations within their financing and insurance contracts. The more accurate and timely the data, the better SA Taxi can help 15 million daily commuter trips run safely and smoothly.

Data dams and delays

In 2019, SA Taxi realised that its existing data and analytics infrastructure could not deliver the reliable real-time intelligence required to manage its expanding fleet of taxis. Batch processing of vehicle tracking data delayed access for analysts, slowing down the process of identifying drivers operating outside the terms of their financing contracts. Infractions such as crossing international borders or using illegal mechanics risked invalidating insurance policies and increasing the risk of unsafe vehicles for passengers and other road users. Tracking data delays also limited SA Taxi’s ability to respond swiftly if a vehicle was stolen. Accessing driver account information was just as slow, with account managers required to manually extract data from multiple sources.

Furthermore, even if tracking and account data could be accessed in real-time, the existing user interface could not be relied on to respond at the same speed. Neither could it break down aggregate data to a more granular level.

Held back by these data issues, SA Taxi looked to Google Premier Partner DotModus and Google Cloud to create a data and analytics solution that would future-proof vehicle and account intelligence for years to come.

From data puddle to firehose

Opening the sluice gates on the data pipeline was the first step towards real-time centralised data and insight. Using Google Cloud Dataflow and Google Cloud Pub/Sub, SA Taxi replaced batch processing of vehicle tracking data to an on-premise Hadoop-based data warehouse with real-time stream processing to a BigQuery data warehouse. Adding Google Operations (formerly Stackdriver) ensured the data pipeline was fast, reliable and fully monitored.

With the data pipeline able to operate at full flow, attention turned to user access. Google App Engine and Google Compute Engine were selected to host a bespoke analytics engine, built using tools including Apache Solr, Django REST framework and Vue.js. The engine powered a user dashboard centred on a real-time search and filter capability, along with three new views analysts could use to interrogate data by location, vehicle or route. Google App Engine’s automatic scaling ensured the system was ready to respond quickly to changing user (or taxi) traffic.

'We no longer have to sip from a data puddle. DotModus and Google Cloud Platform have allowed us to drink from the firehose.'

A free-flowing system

With the new system in place, data now flows freely from SA Taxi’s 22,000 vehicles to analysts able to monitor and manage the fleet in real-time. Instead of delayed tracking data and slow, fragmented account data extraction, SA Taxi can view and access everything important to their operations quickly and easily in one place. Near-instant data filtering at individual and aggregate levels transforms how analysts identify and investigate issues by vehicle, route or location. And as the solution is now web-hosted, they can access the data they need anywhere at any time.

For the business, live tracking and insights enable a step change in operational performance and risk management. Fraud and theft can be identified and managed more efficiently to reduce costs and risk to passenger safety. Vehicle and passenger safety are further improved by more timely monitoring of driver compliance to contracted routes and vehicle maintenance terms.

Ready for future opportunities

“It's been a really powerful partnership. Historically, our tech limited us from doing certain things. Where we stand now, we’re limited only by our imaginations.” Stuart Wilson, Business Intelligence Manager, SA Taxi