Welcome to
the Forum

Open Compute Project’s Steve Helvie chats data center innovation from #OCPLisbon24

TechArena host Allyson Klein chats with Open Compute Project VP of Emerging Markets, Steve Helvie, about the proceedings in Lisbon this week and how OCP is helping to shape the cutting edge of infrastructure innovation.

ZeroPoint Technologies’ Nilesh Shah shares insights at #OCPLisbon24

TechArena host Allyson Klein chats with ZeroPoint Technologies’ VP of Business Development Nilesh Shah about the AI era demands for memory innovation, how advanced chiplet architectures will assist semiconductor teams in advancing memory access for balanced system delivery, and how ZeroPoint Technologies plans to play a strategic role in this major market transition.

HPE’s Jean-Marie Verdun discusses network delivered firmware via OpenBMC at #OCPLisbon24

TechArena host Allyson Klein chats with HPE’s Jean-Marie Verdun about his organization’s groundbreaking work to redefine firmware management using OpenBMC technology and how this breakthrough addresses data center customer demands.

Highlights from CircleB’s Matty Bakkeren at #OCPLisbon24

TechArena host Allyson Klein chats with CircleB’s Matty Bakkeren about how hisorganization is leveraging OCP specifications to deliver innovative and sustainable solutions to data center customers, how AI is re-shaping operator requirements, and how he sees the market shaping in 2024.

Portugal is an interesting place from a historic perspective. You could say that they peaked early, delivering the fastest ships in the Age of Discovery and carving a passage to India opening up the spice trade. This brought wealth and a colonial empire, and world influence to this small country, but as society advanced, other powers leapfrogged Portugal with maritime innovations of their own. In the 20th century, Portugal was a footnote on the history books becoming a military dictatorship until a democratic revolution freed the country 50 years ago this week. And its in this landscape that the Open Compute Project, an organization knowing a lot about the value of innovation, lands on Portuguese shores for its Regional Summit. How befitting of OCP to choose this location for arguably the most disruptive period in the history of data center computing, as we see AI place exponential performance demands on infrastructure and as infrastructure vendors feel the pressure to innovate or risk being disrupted to the footnotes of computing history.

The TechArena is delighted to once again be a media sponsor of OCP’s 2024 Summits, and we’ll be reporting this week on the latest innovations in open hardware configurations. I’m excited to engage with industry executives and leading operators about how OCP designs are influencing everything from balanced system performance across compute, memory and I/O, advances in cooling technologies to help operators deploy dense AI training clusters, and how a broader array of organizations beyond the largest cloud providers are leveraging OCP designs to advance their infrastructure. I’m also keen to see how European operators are navigating challenges of high energy costs and data regulatory restrictions and if we see any unique insights on the shaping of infrastructure and solution requirements flowing from European customers.

Watch this space as we report from the Summit, and please reach out if you’ve got specific questions you’d like answered from the experts assembled in Lisbon.

I recently attended NVIDIA GTC, called by some as the Woodstock moment of the AI Era, and I’m still unpacking what we learned there about industry innovation to fuel AI workloads. While the TechArena packed as many conversations possible with industry innovators at the event, one conversation that stood above the rest was our interview with CoreWeave’s Jacob Yundt. He leads infrastructure buildout for CoreWeave as they chart a trajectory for delivering unparalleled scale for AI training in the cloud.

How did they do it? As we have seen at many inflection points, CoreWeave took advantage of not being encumbered by legacy to deliver a cloud stack that was specially built for AI training clusters from initial provisioning to health checks, orchestration and scheduling. This enables the company to bring up a staggering amount of GPUs to a particular training task at warp speed while providing reliable compute throughout the training period. CoreWeave provides proactive oversight of its instances to ensure that precious training cycles are not disrupted based on potential hardware failures, I/O issues, or other maladies that confront data center infrastructure.

CoreWeave has developed a cult-like following amongst AI startups looking to train algorithms where speed to train often is the difference for market opportunity. Jacob clarified their market focus on any customer looking to do “ground-breaking work at incredible scale”, and this speaks to the type of underlying infrastructure requirements they have across compute, storage, and network. And the demand for this infrastructure is stark. CoreWeave has been on record stating that power demand alone from its training clusters may stress local power grids in the communities where it operates, and the demand for CoreWeave is also growing exponentially. Valued at $7B last December, the latest discussion of valuation of the company four months later has surged to $16B underscoring the growth potential for AI training.

So what infrastructure is CoreWeave tapping to deliver their AI service? It’s no secret that their training relies on NVIDIA GPUs, and CoreWeave will be integrating next generation Blackwell GPUs into clusters utilizing liquid cooling technologies. But Jacob stressed that there’s more than GPUs that goes into the groundbreaking scale they’ve been able to achieve. That scale starts with re-imagining the data pipeline, and CoreWeave has leaned into a strategic partnership with VAST Data to deliver innovative data management and control that scales with GPU performance needs. VAST Data’s platform has driven new capabilities for managing data sets to bring data more efficiently and quickly to the processing complex eliminating much of the overhead associated with traditional tiered storage solutions.

Jacob stated that the collaboration with VAST Data begins with his team’s love of QLC storage and the careful balance between performance, capacity and efficiency that QLC delivers. To say that Jacob is a fan of QLC is an understatement, and it’s no surprise given QLC’s advantages over TLC technology in delivering increased data density per cell. Jacob stated that his long-standing collaboration with Solidigm has ensured QLC deployment in his data centers with a partnership that extends beyond procurement to account and engineering support. When you consider the size of LLMs being trained at CoreWeave, it’s easy to guess that that’s a lot of QLC NAND being deployed.

So what’s next for CoreWeave? Watch this space to learn more about their continued infrastructure buildout as a harbinger of broader AI market adoption. I’m also interested to see if CoreWeave can make a dent in the cloud service provider landscape with their built for AI training stack. I’ll also be reporting on advances of the data pipeline infrastructure industry including in my Data Insights series with Solidigm.

TechArena host Allyson Klein chats with Eridan Communications CEO Doug Kirkpatrick about how a DARPA award innovation propelled an innovation in wireless radio technology that will help full broad proliferation of 5G.

The world of technology is in a constant state of flux, and with the rise of generative AI, the demand for more powerful and efficient processing is skyrocketing. I recently had a chance to chat with Mohit Gupta, Senior Vice President and General Manager of Custom Silicon and IP at Alphawave Semi, about how his company is leveraging their IP and engineering prowess to address customer demand for custom silicon innovation. Our conversation sheds light on why companies are increasingly turning to custom solutions and how Alphawave Semi is leading the charge in this exciting field.

Why the Rise of Custom Silicon?

Gone are the days of one-size-fits-all solutions. The era of generative AI, with its diverse workloads and specialized requirements, demands a more tailored approach. As Mohit explained in our interview, "The need for custom silicon arises because companies are facing a vast array of workload challenges. They need solutions that can address these challenges head-on and extract the maximum efficiency from their systems." This requires a focus on optimizing for specific parameters where generic, off-the-shelf solutions might not always be able to deliver. Custom silicon allows companies to optimize for the specific parameters that are critical to their specific AI infrastructure requirements. Mohit highlighted some key areas that he’s hearing from customers: "Imagine a scenario where memory bandwidth is the bottleneck in your system. With custom silicon, you can design a chip that prioritizes increased bandwidth, ultimately leading to a significant performance boost. Similarly, network latency or even cost reduction can be targeted for optimization based on your specific needs."

Alphawave Semi's Approach to Custom Solutions

Alphawave Semi isn't just another player in the custom silicon game. They've developed a distinct approach that focuses on four key pillars:

1. Connectivity: High-performance interconnects are the backbone of any complex chip design. Alphawave Semi prioritizes building a robust connectivity foundation of IP and chiplets within custom silicon designs, ensuring seamless communication between different processing elements on the chip, and between chiplets in an advanced 2.5D/3D package.

2. Compute: The heart of AI processing lies in the compute engines. Here, Alphawave Semi uses cutting-edge design techniques to deliver the optimal balance between power and performance for specific workload requirements, and their modular design delivers compute scalability. "As a founding member of the Arm Total Design Partnership, Alphawave Semi provides an accelerated path for specialized SoC solutions based on Arm Neoverse Compute Subsystems (CSS).

3. Complex Silicon Building Blocks: Building a custom chip isn't just about slapping together existing components. Alphawave Semi leverages its expertise in designing complex silicon building blocks that cater to the unique needs of each project. Their modular design enables speed in delivery of chiplet-enabled custom solutions as well as dialed-in engineering that leverages known silicon building block configs vs. expensive one and done design.

4. Advanced Packaging: As chip designs become more intricate, advanced packaging techniques are crucial for efficient integration. Alphawave Semi offers state-of-the-art packaging solutions that ensure optimal functionality and performance of the chiplet-enable custom silicon designs.

The TechArena Take

Alphawave Semi's dedication to custom silicon solutions has seen significant market success. Mohit noted that their engineering and IP progress has unlocked customer traction: "We've had successful designs on both 5-nanometer and 3-nanometer processes, showcasing our ability to deliver cutting-edge solutions at the forefront of technological advancement." With the ever-evolving landscape of AI, this progress on leading-edge process node and advanced packaging, bodes well for continued success in market. As companies push boundaries and explore new frontiers in generative AI, the demand for tailored solutions will only increase. With its focus on specific needs, cutting-edge technologies, and successful track record, Alphawave Semi is well-positioned to be a leader in this exciting future. I, for one, will be following their journey closely as the world of chiplet-based custom design takes off.

TechArena host Allyson Klein chats with Muriel Medard from MIT about the trends in network innovation, how AI is infusing into telco, and the shaping of 6G.

TechArena hosts Allyson Klein and Jeniece Wnorowski chat with CoreWeave’s Jacob Yundt about how his organization is delivering a scalable data pipeline to AI customers utilizing breakthrough VAST Data solutions featuring Solidigm QLC SSDs.

TechArena’s host Allyson Klein reprises conversation at MWC24 with a talk with GlobalLogic’s SVP of Global Communication Services Business, Sameer Tikoo, about the future of AI in the network and how his firm is building solutions that meet swiftly evolving customer demands.

This week was a whirlwind at GTC 2024 in San Jose, and it was a conference where I felt myself absorbing new insights about the tech industry, AI and where we’re collectively headed at turbo speed. With so many messages and so many companies aligning themselves to the green lantern that is NVIDIA, what are the key takeaways from the event?

#1 NVIDIA IS DISTANCING THEMSELVES FROM THE FIELD

2023 taught us that NVIDIA had unquestioned leadership in foundational definition of the AI era infrastructure landscape. Last year’s H11 introduction, shortages of GPUs in market, and meteoric rise in LLM training cluster deployments underscored their importance to the industry. What we saw this week was a company operating on all cylinders to keep and extend their lead. First, we got the unveiling of Blackwell with performance deltas equivalent to greater that we saw from the A100 to H100. Next, we saw the announcement of NIM showcasing that NVIDIA is not satisfied with AI training, they want to own the inference landscape as well with powerful software tools to aid deployment. Jensen also unveiled sweeping collaborations with industry leaders led by a massive collaboration with Microsoft to bring GB200 Grace Blackwell computing into Microsoft Azure. Finally, and notably covered previously on the the TechArena, NVIDIA unveiled their strategy to extend their dominance to the network with a 6G strategy that centers squarely on AI.

#2 THE INDUSTRY IS ALIGNING THEMSELVES DESPERATELY TO NVIDIA’S STAR

The energy at the San Jose Convention Center was palpable including on the show floor where infrastructure vendors and service providers hawked NVIDIA centric gear to position themselves as part of this disruptive force. They worked to get selfies with Jensen and were keen to highlight the depth of collaboration they had with the company. I haven’t seen this kind of engagement since the earlier days of the Intel Developer Forum in terms of a conference that set the pace for the industry. Those who execute in alignment of this strategy are poised to benefit greatly, and they know it.

#3 THE DATA PIPELINE IS LEGITIMATELY CRITICAL AND BEING RE-DEFINED

One of the most interesting elements in AI infrastructure today is re-definition of the data pipeline as broad enterprise begin training LLMs and tapping their data. This data is located all over the map – in the cloud, on prem, and at the edge, and getting a handle on how to aggregate it for training is, well, really difficult. Disruption in this space is massive, and many companies, VAST Data and WEKA come to mind, have interesting solutions to aid companies in this realm. For the large scale of the large scale, Voltron Data just delivered some new insight about Theseus that needs unpacking as well. While we have covered these firms on the TechArena, we’ll be going even deeper in our new Data Insights series with Solidigm to learn more.

#4 THERE IS AN INTERCONNECT WAR BREWING

I sat in many GTC sessions describing architectural models for deployment of GPU clusters, and as important as the GPU performance is to these workloads, the ability to connect systems together with high bandwidth switching is critical. NVIDIA’s answer to this is InfiniBand, but there was open discussions from others in the industry that Ethernet was in play as well. We covered the Ultra Ethernet consortium last year at the OCP Global Summit, and it’s apparent that service providers and infrastructure leaders alike want Ethernet to compete here. Put a pin in this topic as we’ll be exploring it next month again at OCP Summit Lisbon.

#5 DPUs ARE FIGHTING FOR SUPREMACY

NVIDIA’s Bluefield Network solutions are a leading force in delivering DPU capability to network offload, but here they are not the only game in town. AMD’s Pensando technology has raised some eyebrows with pure capability, and this is a diffuse field with entrants from everyone including network leaders Broadcom and Marvell to cloud service providers like Microsoft and AWS. What’s interesting to me is that this arena seems ripe for fierce competition, and I expect to hear a lot more about DPU innovation in the coming months.

So are we ready to declare a GPU victory and place CPUs as legacy gear? Do we want to throw the towel in as well for the AI startup silicon arena? The answer is…no. AI is moving at a pace that requires incredible amounts of silicon, and this opens the door for a heterogeneous array of viable solutions. AI is not one workload – it’s a broad array of training and inference across LLMs, image and voice recognition, recommendation engines and more – each with their unique computing requirements. It’s also a force that will be delivered across the data center, into the edge, and at the device, each requiring its own platform optimizations and performance characteristics. Finally, it’s a power-hungry monster, and we will meet a moment in the not so distant future where pure efficiency will become as critical as liquid cooling tradeoffs. We are still in the infancy of the AI era, and the room is open for broad innovation starting with silicon. I can’t wait to see what happens next.

TechArena kicks off a Data Insights Series in collaboration with Solidigm, and TechArena host welcomes co-host Jeniece Wronowski and Solidigm data center marketing director Ace Stryker to the program to talk about data in the AI era, the series objectives, and how SSD innovation sits at the foundation of a new data pipeline.

TechArena host Allyson Klein chats with EY’s Global Innovation AI Officer, Rodrigo Madanes, about what he’s seeing from clients in their advancement with AI and what this means for the industry requirements for innovation.

TechArena host Allyson Klein chats with Intel’s Lisa Spelman about how compute requirements are changing for the AI era, where we are with broad enterprise adoption of AI, and how software, tools and standards are required to help implement solutions at scale.

With new industry, collaborations VAST Data accelerates their ambition as the AI data platform.

VAST Data has been on a tear. If you haven't been following this company, you've missed a great story about transformation from storage platform to insightful data control for the AI era. Today vast unveiled multiple industry collaborations that fill in details of their strategy and showcase VAST’s bold direction to re-define data storage and oversight from cloud to edge.

Let’s unpack today’s news. We’ll start with an announcement on the new VAST Data AI platform featuring NVIDIA DPU's. This is a fundamentally different take on data and redefines platform architecture into a powerful GPU platform complemented with Bluefield DPUs each running a containerized version of the VAST OS embedding storage and database processing directly into the GPU platform.

What you’ll notice is missing is any mention of x86 architecture eliminating the concept of the CPU head node. VAST points to the increased efficiency of this design claiming a power utilization and footprint reduction of up to 70% and a “net energy consumption” savings of up to 5% vs previous platforms featuring VAST distributed data services and NVIDIA silicon.

VAST is also leaning into the core capabilities delivered with this platform with increased QoS through direct reads and writes to shared name spaces across the cluster, enhanced zero-trust security through use of industry standard NFS, SMB, S3 and Apache Arrow service delivery, and the addition of native block storage to compliment historic file and object storage.

So where do you get this platform? Enter collaboration announcement number two, a comprehensive collaboration with Supermicro to fuel system delivery for the AI factory. Here, the two companies point to delivering platforms that scale to exabyte level clusters. VAST and Supermicro promise support for the entire data pipeline from data prep to data training with VAST’s DataBase and DataStore solutions. Supermicro is known for swiftly getting innovative platforms to market, so I’m excited to see what they actually deliver to support this bold move for both companies.

What’s TechArena’s take? The AI training game is moving from GPU centric to GPU native with new architectural frameworks fueling these massive clusters. VAST Data’s history in high performance computing and their DASE (disaggregated share everything) architecture places them as a central player for organizations looking to integrate distributed data into AI training. AI is the killer app of the era forcing re-definition of the fundamental constructs that have fueled computing for the last thirty years, and this magnitude of disruption will reshape the infrastructure industry. Based on recent history and today’s announcements, VAST Data is positioning itself well as a disruptive force. I can’t wait to see more from the company in 2024.

TechArena host Allyson Klein interviews Netflix’s Tejas Chopra about how Netflix’s recommendation engines require memory innovation across performance and efficiency in advance of his keynote at MemCon 2024 later this month.

TechArena host Allyson Klein chats with Avidthink Principal Roy Chua about advancement of the network across 5G, edge, AI integration and more as the two share insights from MWC24.

TechArena host Allyson Klein chats with Hideyuki Tsugane, Vice President of Business Development at CTOne, about his organization’s vision of private wireless security and the staus of private 5G deployments as his company readies for a hockey stick of growth.

When you consider the massive investment in industry innovation based on AI, it doesn’t take too long to realize that cutting edge AI models are being constrained by their underlying infrastructure. While processor and accelerator advancements garner the lion’s share of the headlines, a key bottleneck to consider is the memory hierarchy. The faster data can be delivered for AI training, the faster a new large language model can be delivering inference to fuel a new workload. While servers continue to scale the amount of standard DRAM capacity - notably AMD’s industry leading 6TB of memory capacity across a dozen channels - the leading edge is seeking lower latency and more alternatives than standard configurations. In fact, cloud service providers have pointed to memory as one of the key infrastructure gaps facing AI training moving forward.

This industry challenge is why MemCon is a must attend event on my 2024 roadmap and one that the TechArena is delighted to sponsor. Pulling together some of the leading data center operators and the leading edge of memory innovation, MemCon features two days of discussions on the requirements for memory, the latest memory innovations, and how the industry should work together to bridge the gap for the insatiable demand represented by AI workloads. Highlights of this year’s events includes an opening keynote from Microsoft’s Zaid Kahn who also spoke at last year’s event. Since this time Zaid has become the chair of the Open Compute Project and continued his vocal evangelism for infrastructure innovation including in the memory arena. Joining Zaid from the operator perspective include sessions from Netflix, EY, Oracle, Shell, Roche, Berkeley Research Lab and Los Alamos National Labs. They’ll be joined by speakers from across the industry and from the industry consortiums shaping the standards that will fuel future memory innovation and collectively highlight the intersect between high performance computing and AI cluster development as well as broader scale opportunities with increased memory capacity, tiered memory with CXL, and more.

Whether you’re in the memory arena, run a data center and feel the pain of memory bound workloads, or are delivering platforms to the market and need to keep pace with the latest silicon advancements, prioritize your schedule to attend MemCon this year. As a media sponsor, we’re happy to deliver a registration discount code TECHARENA15 for 15% off registration. And if you’re going to be at the show, please reach out to meet up or even be on the TechArena podcast.

TechArena host Allyson Klein chats with Physia about their generative AI based patient care platform and how they aim to create a new AI + doctor model to improve patient care and transform the medical industry.

TechArena host Allyson Klein chats with Ansys Chief Technologist Christophe Bianchi about his company’s mission to deliver design simulation across industries, how the communications arena represents an opportunity from silicon to systems of systems, and how AI is accelerating capabilities in Ansys solutions.

I kicked off my Mobile World Congress reporting today with a fascinating interview with AMD’s Kumaran Siva on his company’s strategy for network, the intelligent edge, and AI. Kumaran leads AMD market development across the strategic industry segments where EPYC processors shine, and I was keen to get his views on 5G deployment progress and the key use cases and technology developments that will be featured at MWC this week.

At this year’s congress, all eyes are on the edge and specifically progress in adoption of VRAN solutions. These solutions require high performance and energy efficiency, especially in Europe where operators have been hit hard with spiked energy costs. At last year’s MWC, initial implementations of VRAN representing the final frontier of network virtualization was one of the hottest topics.

In our interview, Kumaran was quick to highlight AMD’s progress in 5G and increasing customer interest in EYPC CPUs for deployment success. Those who have followed the TechArena know that I’m somewhat obsessed with the efficiency and flexibility of chiplet architectures, so it may come as no surprise that AMD’s chiplet design enabled them to speed Siena processors to market, delivering the performance, efficiency, and security dialed in for this market. Kumaran confirmed that the Siena series has garnered operator attention delivering low latency, high bandwidth connectivity at the edge. Kumaran went further stating that he sees more uptick in edge deployments due to, in part, the need for high speed connectivity to fuel AI inference at the edge.

What is AMD’s approach in network and telecommunications? It starts with deep collaboration with both partners and operators. Kumaran called out AMD’s partner-centric approach as something uniquely prioritized here at AMD vs other stops on his career journey, and a central driver of the company’s continued gains in market share with their EPYC product line. As 5G continues to proliferate, I’d expect to see AMD continue to make inroads especially where single core performance and performance efficiency is required for workload delivery.

As for Kumaran, he’s interested to see the industry conversation on AI in Barcelona and the transformative juggernaut that continues to drive change in the comms arena. While AI will influence 5G workload evolution, its true force will be felt in 6G standards which AMD plans to be deeply engaged with, in regards to standards finalization. 

To learn more about AMD’s engagement in the telecommunications arena, check out our interview, and to learn more about AMD’s 8004 series processors and the entire AMD EPYC processor lineup, visit https://www.amd.com/en/processors/epyc-server-cpu-family

TechArena host Allyson Klein chats with Carsten Brinkshulte, CEO of Dryad Networks, about his company’s mesh network solution aimed at alerting forest fires before ignition, how his team tapped IOT and AI technology to develop their solution, and his aim to deploy hundreds of thousands of sensors in forests across the globe in 2024.

Whenever I talk to Rajesh Gadiyar, VP of Engineering for Telco and Edge at NVIDIA, I walk way with better insights about the state of the network arena and where GPUs fit into the future of network deployments. Rajesh is the type of technologist who beams when talking about new industry innovation or when sharing how a particular challenge has been overcome by engineering advancements, and you can’t help but catch the wave of excitement when talking to him.

I was lucky enough to catch up with him in advance of MWC to hear about the latest advancements NVIDIA has made in the network. It’s been about a half dozen years since NVIDIA made it’s ambitions known in this arena coming to MWC with claims of GPU superiority for the RAN. At the time, many dismissed the move as mere TAM expansion aspirations for the GPU vendor. Of course, the world’s changed a lot in that half decade, and NVIDIA’s Ariel platform has emerged as a serious contender for virtualized RAN infrastructure. Late last year, NVIDIA unveiled the world’s first GPU-Accelerated 5G Open RAN solution with NTT DOCOMO pointing to TCO improvements of 30%, network design utilization improvement of 50%, and reduction of power in half as compared to NTT’s legacy solution.

While these advancments are impressive and certainly give NVIDIA a claim to legitimacy in the space, the elephant in the room is on everyone’s lips in Barcelona this week…the infusion of AI into the network. AI is being discussed to help deliver improved service agility, drive deeper service and network automation, and just overall transform how networks operate. Operators are clamoring to integrate LLMs into everything from customer support to billing functions while seeking AI solutions to drive more efficiency to the network and help unlock the ROI from incredible investments in 5G network infrastructure. Looking to the future, those who steer 6G standards efforts are seeing seamless AI integration as a core pursuit within many standards workgroups. And with this the path for NVIDIA’s deeper engagement into the network becomes clearer as AI transformed workloads are immediately at least considerations for GPU acceleration. In our interview

Rajesh assured me that NVIDIA plans to be at the heart of 6G standardization efforts as they work to support their growing ecosystem with Ariel SDK and cloud support to fuel broad innovation. Watch this space for more information about advancements of NVIDIA in the network, and if you want to learn more about Ariel and what NVIDIA’s intentions are for the future of the network attend GTC 2024 next month where Rajesh and his team will be on hand to share the latest updates.