From Batch to Stream

Prior to the system change, SA Taxi’s existing solution involved feeding data from vehicle tracking devices to an on-premises, Hadoop-based data warehouse and presenting aggregated data to users through a simple interface. However, this system was limited in terms of the batch processing method that was used in order to receive and present the data to users. With batch processing, data is collected and processed on a schedule. This results in lengthy delays from the time that the data is generated, to the stage at which users are able to garner insights from that data. Unlike stream processing, batch processing does not feed data immediately into the user interface. As a result, data is not available in real time.

Google Cloud Dataflow and Google Cloud Pub/Sub were selected and set up to enable the stream processing of data from SA Taxi’s on-board vehicle trackers into a BigQuery data warehouse. The addition of Google Operations (formerly Stackdriver) enabled DotModus’s data engineers to develop a fully monitored and managed data pipeline that was as reliable as it was fast.

''With our data, we are no longer sipping from a puddle. DotModus and Google Cloud Platform have allowed us to drink from the firehose.''

A User Interface to Match

With the data-streaming problem resolved, DotModus couldn’t risk the user interface becoming a bottleneck. So, a user interface providing search and filter functions with the same degree of speed and reliability was needed – any delay in response would make the interface difficult, as well as frustrating, to use.

A combination of Google App Engine and Google Compute Engine provided the perfect platforms to host the bespoke analytics engine, which was built using a combination of tools, such as Apache Solr, Django REST framework and Vue.js. Google App Engine’s automatic scaling meant that DotModus need not worry about over- or under-provisioning. Rather, the user interface would be ready to handle any amount of user (or taxi) traffic thrown at it.

The resulting user interface that was created provides the user with three different views:

1. The Vehicle Dashboard view allows the user to enter a taxi operator's account number and view the current location, aggregated activity metrics and account details specific to the operator’s vehicle.
2. The Current Location view provides a map that displays the current location and aggregated activity metrics of all 22 000 vehicles. Near-instant data filtering allows the user to filter by given route, vehicle year, operating region, credit provider, taxi association etc. and to chisel away at the provided data in order to find vehicles specific to their investigation or reporting needs.
3. The final view is a Route page that allows the user to filter by a specific route and view the taxis operating on that route, with a given DateTime range.

Plans for the Future

In order for bespoke software solutions to remain efficient and relevant, they need to be updated in response to new operational challenges and opportunities.

DotModus’s service-level agreements (or SLAs) provide the company’s customers with access to a diverse team of Google Cloud certified engineers – for a fraction of the cost of creating an in-house team. This agreement ensures that the app and system are monitored and maintained, guaranteeing that such a system will continue to meet the needs of the users and the company for which it was built.

With its DotModus SLA, SA Taxi has been able to add a number of new features and functionalities to its software:

• Immediate accident alerts for vehicles involved in collisions
• No-go-area alerts in order to help identify vehicles using illegal services, such as unlicensed panel beaters or mechanics.
• Stats integrated into an LED ticker or noticeboard in the company’s front office for better visibility.
• Expanded data analysis capabilities and new integrations with third-party systems.

''It's been a really powerful partnership. Historically, our tech limited us from doing certain things. Where we stand now, we’re limited only by our imaginations.''

In partnership, DotModus looks forward to growing the capabilities of the solution and further supporting the SA Taxi team in garnering increased value and insight from its wealth of data and data sources.