Digital Infrastructure – the Foundation for AI



Over the last few weeks, I have been fortunate to listen to some interesting conversations on digital Infrastructure – cloud, hyperscaler data centres, data cables, satellites and power. I listened to Brad Smith, vice chair and president of @Microsoft, Joseph Dominguez, the President and Chief Executive Officer the CEO of Constellation, and then later heard an interesting panel organized by Simons & Simons LLP - Priyanka Nagpal @RasScolley @YuvrajDuggal and Jas Shoker.
If you are saving your data on the “cloud”, your data is "moving" via the internet that you access through physical infrastructure like WiFi routers. These routers connect to internet cables (2G, 4G or 3G) via radio waves. The data in these cables is encrypted and then transferred to data warehouses and stored in one of the racks of servers (called cloud).
In some cases, you are sending data across countries. This is done through a large network of cables. In 2024, there were 559 cable systems and 1,636 landings. The growth of content providers like Amazon, Google, Facebook, and Microsoft led to greater usage of cables from 10% prior to 2012 to 66% in 2020, resulting in their investment in thus area. The focus has been speed of data transfer o reduce latency – the delay in the transmission of data (measured in millisecond). Cables have lower latency (or less delay) than satellites.
As you can see in the map below, there are strong hubs – like Egypt via Telecom Egypt which enables over 90% of the international Eurasian traffic (200Tbps). Did you know that over 95% of all the world’s data goes through these undersea cables?

Data is increasing because of the time we spend online and the sensors that connect us to the internet. Because AI exploits data – by using data as a repository of knowledge, to increase speed of processes, create personal profiles for nudging, and for better customer service, it needs to be stored. This storage is in data warehouses or cloud storage or hyperscale data centres. A typical data warehouse can house anything from about 1000 servers which are made of CPUs, GPUs or DPUs that have a compute power of 100 to 1MW. Of the over 11,000 data centers worldwide, about 50% are located in the US.
Hyperscalers contain 50% of all installed data center servers and typically average abou 1 million square feet of space. There are about 25 companies that qualify as hyperscalers, operating 500+ data centers. Of all the hypersaclers, Meta alone does not share its servers – and its 24 hyperscale data center (53 million square feet) are still not enough for its demands and hence it leases more space. The density of servers in these type of data warehouses is huge – having tens of thousands to even millions of servers in the warehouse with a compute of 5-100 MW.
The average cost to build a hypersacler is USD 1 Billion and they need higher power than traditional data ceteres (10-14 kW per rack versus 5 kW per rack). One metric used is power usage effectiveness (PUE). I worry about this metric as PUE does not reflect the water, power, GHG, and e-waste figures correctly. Offset is also a problem if you are not looking at how that offset is happening.
Though these data warehouses offer Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS), they also need dedicated power, especially if they are doing training (as opposed to inference). Hence there has been a resurgence in other types of power: like nuclear (Microsoft and Constellation Energy (Long Island in USA) and Google recently with Kairos Power in USA) and geothermal (Microsoft and G42 with geothermal in Kenya). The challenge with renewables is that it is not constant and we all know how upset we get when services go offline - worse we have got rid of the processes that can manually help us which may be a challenge!
We have not even touched the complexity of where the hardware comes from. The chip wars illustrate a very complex supply chain. Hence, all these data servers will not work unless they have a fresh supply of chips. Considering that servers need to be replaced every 3-8 years - this requires some amount of foresight. What will be the change in technology (they are experimenting with biochips) and future availability of these materials for the manufacture?
We often do not talk about digital infrastructure and AI together but it is critical to understand that AI is a combination of hardware, software, data and human ingenuity and knowldege!
If you are curious about AI and other things like this. More www.melodena.com
Comments