Indeed, one side of the equation of data collection in Indonesia has been going very well. Over the next five years, data centers are expected to grow by 11 percent across Indonesia. This is spurred by the fact that Indonesia saw an increase in 20 million new social media users in 2018 and 2019, with 10 million of those users being active. This substantial increase in social usage is guaranteed to create an enormous data pool in Indonesia.

On the other hand, big organizations are often data rich but insight poor. Organizations struggle to unlock the value of the massive datasets they are collecting; a common and growing problem in the Indonesian big data landscape.

Another thing that local organizations often misunderstand is that big data does not consist of just one monolithic database. Increasingly, organizations around the globe are combining their datasets, most frequently public data, to gain greater context of a particular issue. Sometimes, two medium-sized datasets can also generate some of the largest, most complex challenges.

Yet, Indonesia’s big data use landscape is not necessarily all discouraging. For instance, within the context of the COVID-19 pandemic, governments have started to recognize the importance of big data in monitoring the spread of COVID-19 through airplane flight and weather fluctuation patterns to achieve far richer and more comprehensive analysis.

More encouraging news is state-owned telecommunications company Telkom’s recent investment in Indonesian super-app, Gojek, seeking to mine the latter’s data pool. Gojek is an excellent example of how big data can boost business efficiency and enhance customer experience. This is done through gathering data from various apps to manage driver distribution, following the traffic as well as calculating the shortest route for a ride. Telkom’s investment in Gojek has demonstrated the importance of data analytics as the new business zeitgeist.

Along these lines, at a national level President Joko “Jokowi” Widodo has recently called on various business stakeholders to accelerate their digitalization process in order to also accelerate Indonesia’s post-pandemic economic rebound.

On a more positive note, he said that the Indonesian digital economy market had been quite promising, with rapid annual internet market penetration growth rate.

“To accelerate the growth of digital economy, we need more digital talents. We also need more software developers. We need more product designers and we also need content creators to support us,” said Jokowi in his speech during a Google Indonesia event on Nov. 18.

Needless to say, ample use of big data will also be part of this grand scheme.

However, to truly seize on this rich opportunity of inter-sectoral big data collaborations in order to gain actionable insights from the abundance of social media data generated by Indonesians, the country still needs to overcome common challenges in big data including scale, geospatial data types and visualization analysis siloization.

Let us take a look at these challenges, one by one:

Scale

Today’s data is simply too big. Often consisting of billions of rows and slow query times, data querying can take hours or days to get even a simple analysis result. Therefore, how can you accelerate the analysis?

The solution: organizations need full data fidelity to make the best decisions. Using artificial intelligence (AI) and machine learning, organizations will be required to expand their data pool, with accurate modeling types.

Geospatial

Geospatial data types, such as points, lines and polygons on a map, are becoming standard in data today. It seems like everything gets tagged with a location nowadays, think TweetMap.

Geospatial data types used to be the exclusive domain of data analysts, using separate tools that produce their own reports. Today, however, geospatial data is considered to be a critical part of the overall dataset.

Visualization

Because data visualization is done in the browser, the use of visualization tools in big data is often limited to simple charts and rendering of data. However, as hundreds to tens of thousands of records have to be transferred from the database to browser, this situation oftentimes results in wasted hours, painfully waiting for tools to transfer data to create visualizations or reduce the granularity of the results, rendering the entire process quite meaningless.

These challenges need a breakthrough solution, designed and purpose-built to handle these large datasets. OmniSci has the breakthrough technology to help governments and enterprises to do exactly that.

OmniSci’s accelerated analytics platform harnesses the massive parallel computing power of graphic processing units (GPUs). The system is built to take advantage of modern architecture: this is not a retrofit of another database or technology, but entirely new and purpose-built for this problem.

For instance, OmniSci offers 100x faster performance on data queries (look at the independent benchmarks). It can take 10B rows and query, filter, and render the results on a map in hundreds of milliseconds. Something that would take traditional technology hours (see a public demo with 12B rows of shipping data).

The rendering is another smart piece of the system. Because OmniSci runs on  GPUs, it can pass the results of a query back to the GPU to do a lightning fast rendering, sending a small image of the analysis result to the client. So a 10B point scatter plot contains all of the points, and still returns in hundreds of milliseconds.

Having an interactive experience with your largest datasets completely changes the way people think about and use data. Customers say it is the only tool that lets them interact with their data at the speed of their curiosity.

Another advantage: AI does not have to be a separate experience. AI can be put into practice by data scientists to use different tools from analysts. The analysts can also use programmatic interfaces like Python or R, where analysts can expect a visual dashboard experience.

Data science is often an experimental exercise, involving training and model building iterating hundreds of times before it becomes stable and predictive. The more speed they have, the more experiments they can run, the better their results will be.

OmniSci provides the interfaces that data scientists expect, giving fast, and familiar access to the largest datasets through the full data science workflow: ETL, feature engineering, training, modeling and predictive results analysis.

All things considered, 2021 will be a pivotal year, with some companies taking advantage of innovations like OmniSci, and others falling behind. Indicators for this prediction: not only the scale of data is growing, the complexity of data (including geospatial data) is also growing, and so does the opportunity with AI.

Without the right tools, organizations are not falling behind linearly anymore.Instead, they will fall behind exponentially, as competitors take leaps ahead. This fallout will be more evident in 2021, which will be the year big data goes from collection to insight.

Popular

Kadin urges stability after Constitutional Court ruling

Jokowi says transition to new government to begi

BCA loan disbursement surges 17.1 percent yoy in first quarter