1. Comment
  2. Comment
March 25, 2020updated 29 Nov 2021 12:31pm

Big Data: Technology Trends

By GlobalData Thematic Research


Big data combines traditional data management technologies with new forms of data processing that are better suited to modern data formats. Too much data is scattered across too many formats. This data is only valuable to businesses and consumers if it can be analysed accurately and efficiently. The trick is to ensure the accuracy and reliability of this data.

Listed below are the leading technology trends in big data, as identified by GlobalData.

Edge computing

Specific use cases of edge computing include the maintenance of data processing and analytics close to points of collection. The growth of edge computing is therefore closely associated with the Internet of Things (IoT). The proliferation of enterprise IoT initiatives and consumer IoT offerings will drive demand for edge computing solutions. The deployment of 5G cellular technologies will be a major stimulus for both IoT and edge computing.

Quantum computing

The race to reach quantum supremacy is well underway. AI, and particularly machine learning, stands to benefit, as quantum computers should be able to complete extremely complex calculations, involving very large data sets, in a fraction of the time it would take today’s machines.

AI chips

Central processing units (CPUs) have powered data centres for decades. Graphics processing units (GPUs) – of which Nvidia is probably the best-known exponent – that were once primarily used for gaming can process many threads in parallel, making them ideal for select data centres operations specific to big data, such as the training and modelling of large predictive data models. As the criteria for data centres moves from calculation speed to search speed, GPUs are moving into data centres.

Data centre interconnect (DCI)

As more data centres come on line around the world, so the need to transfer data between them at increasingly high speeds also grows. These data transfer speeds are especially important at the DCI level. The primary use case of DCI remains connecting data centres, but has recently expanded into capacity boosting. Given how interconnected data centres are, this optical upgrade is likely to happen globally in a relatively short time. The main beneficiaries include Applied Optoelectronics, Ciena, Corning, Finisar, Lumentum, Infinera, Oclaro, NeoPhotonics, and Viavi Solutions.

Silicon photonics

Big data technology trends include emerging technologies. Silicon photonics is an emerging technology that combines laser and silicon technology on the same chip. It allows data to be transferred between computer chips by optical rays, which can carry far more data in less time than the electrical conductors traditionally associated with semiconductors. This supports faster interconnects between data centres. Cisco, Intel, and Inphi are prominent vendors in this market.

Serverless computing

Companies that have outsourced their servers to cloud infrastructure as a service (IaaS) companies like Amazon Web Services (AWS) typically pay in advance for the amount of server capacity they require for process execution. But this kind of IaaS is fast being replaced by serverless computing. Pricing is based on the actual amount of resources consumed by an application. AWS Lambda is one of the leading serverless computing offerings. Leading cloud IaaS providers that could benefit from this trend are Amazon, Microsoft, Google, and IBM.

High performance computing (HPC)

HPC is one of the fastest-growing big data technology trends of an otherwise declining computing hardware market. China is going for world domination in HPC. The unveiling of the Sunway TaihuLight System in 2016 was a genuine breakthrough. Meanwhile, Huawei, Lenovo, and Inspur are aggressively competing against IBM and HPE in the enterprise and cloud transaction processing space.

3D sensors

3D sensors are likely to see phenomenal growth because of themes such as augmented reality (AR), AI, and autonomous vehicles. In some AR systems, advanced 3D-sensing cameras use vertical-cavity surface-emitting laser (VCSEL) systems for tracking objects and sensing 3D depth. This means the leading suppliers of VCSEL laser systems could get a boost over the next year or so. Many suppliers like Finisar and Lumentum come from the optical networking equipment sector, while players associated with such 3D sensing systems include Infineon, AMS, STMicroelectronics, Largan, and LG Innotek.

Software defined networking (SDN)

SDN is an architecture for data networks that allows software, rather than hardware, to control the network path along which data packets flow. It is becoming hugely disruptive. The rapid growth of cloud computing drove the early development of SDN. As cloud computing evolved, providers began offering additional services bringing about platform as a service (PaaS) and software defined everything (SDE). SDN is certainly disruptive in the networking market, but does not necessarily herald the demise of the traditional network equipment vendors. Cisco, Ciena, and Huawei all are transitioning towards SDE.

Container software

Containers package a software application together with its dependencies and can be run on most Linux or Windows servers. They enable applications to be easily moved between different IT infrastructures. Open-sourced in 2013, Docker has become an important standard in container technology. Lightweight Linux OS container software might transform the speed at which apps can be created, tested, and deployed. Leaders in container management software include Docker’s Swarm, Google’s open-sourced Kubernetes, Red Hat’s OpenShift, and Amazon’s Blox.

Real-time analytics

The combination of streaming data and analytics has the potential to generate value for companies that rely on making prompt decisions. A 2018 study by Harvard Business Review Analytic Services revealed that 70% of enterprises have increased spending on real-time customer analytics solutions. Amazon, Microsoft, and Google are all pushing their real-time analytics offerings to industries such as telecoms, transportation, and healthcare.


The full-scale mainstream adoption of 5G has the potential to increase data consumption globally. Within big data technology trends, 5G is expected to enable faster speeds (up to 20 gigabits per second (Gbps) per user). GlobalData estimates more than one-quarter of all data traffic to be carried over 5G by 2024. 5G is also likely to have a significant impact on applications.

This is an edited extract from the Big Data in Utilities – Thematic Research report produced by GlobalData Thematic Research.