Welcome to
the Forum

The TechArena are fans of what WEKA is delivering in the market, and we’ve covered their data platform since last year as the company has unveiled innovations to help speed enterprise adoption of AI in market. It was, therefore, no surprise to us to see Contextual AI select WEKA as a strategic partner for delivery of enterprise AI services on Google Cloud this week. Contextual AI has made a name for itself with RAG 2.0, delivering enterprises fine-tuned models that provide foundational tools for enterprises wanting to build and customize specialized AI applications.

When you consider the application of contextual language models, a step beyond traditional RAG with integration across pre-training, fine tuning, and alignment with human feedback. Traditional RAG models uses an off-the-shelf model embedding, vector database for retrieval, and a distinct language model for generation, stitched together through an orchestration framework and therefore limiting both the final value of the application and difficulty and efficiency in delivery of the model.

RAG 2.0 was delivered, you guessed it, by the same leadership team that first delivered RAG at Facebook AI Research, so they know a bit about the RAG approach and how to make it better. RAG 2.0 has delivered proof of benefit across various industry benchmarks showcasing improved accuracy vs. RAG and Vanilla RAG models based on GPT-4 and Mixtral). That’s cool stuff. 

When you consider how much enterprises are focusing on RAG implementations to bring AI to the mainstream, you can understand why Contextual AI is garnering significant attention for delivery of its solution to customers. And when you consider the scale of data required for RAG 2.0, WEKA emerges as a perfect partner to support data pipeline requirements.

WEKA has delivered a fantastic data management platform to optimize GPU utilization delivering peak efficiency in model performance as well as value to the customer funding the Google Cloud instance. Contextual AI has deployed a total of 100TB thus far of WEKA data platform capacity to fuel data requirements and has seen increased developer productivity and faster model training times with the new solution. The delta in performance is eye-opening with a stated 3X performance gain across key AI use cases and 4X faster AI model checkpointing. They’ve done all this while reducing cost 38% per terabyte.

What’s the TechArena take? We love stories about taking technology innovation and actually implementing to customer value. What Contextual AI and WEKA are delivering here are useful tools that enterprises can tap today for real adoption of AI within their environments. We can’t wait to see more advancement on the contextual language models as the industry at large continues its bandwagon of support of RAG type frameworks for AI fine tuning and deployment. And we’re delighted to see WEKA’s data management solutions continue to garner momentum in market for this once-in-a-lifetime moment of AI-era computing advancement.

While Level 3 ADAS (conditional automation) requires the driver to be present and engaged, the resultant heterogeneous workload, a mix of compute and AI processing, is driving the need for new system-level architectures and state-of-the-art system-on-chips (SoCs) based upon leading-edge semiconductor processes and packaging technologies.  More specifically, solutions for Level 3 ADAS and above are driving the automotive industry to embrace chiplets to most efficiently and effectively address the demanding workloads by allocating the right task to the right compute engine optimized for a given task in a footprint that is most efficient from both a power and area perspective.  It is also the case that chiplet-based solutions can be more cost-effective when compared to an equivalent solution based on monolithic technologies.

Chiplet technology, still in its relative infancy, enables disparate technologies and semiconductor dice to be combined in a single package through the use of die-to-die interconnect which is physically connected via a package substrate while enabling performance that is equivalent to that of a monolithic device.     

In March 2022, a universal interface for the interconnection of chiplets was released. Dubbed UCIe 1.0 (Universal Chiplet Interconnect Express), the introduction of this standard, in part, reflects the broader industry awareness that the traditional benefits of scaling through process technology are being challenged. To be clear, the major semiconductor foundries continue to invest heavily in developing advanced process nodes which continue to offer improved power, performance, and area benefits. However, the economics of these advanced nodes present significant barriers to adoption for a large percentage of the ASIC / ASSP design community. Additionally, the benefits of scaling associated with migrating to the most advanced semiconductor process node do not apply uniformly across all circuit types - most notably analog circuits.

As such, while scaling through Moore’s law is still a very important vector for the semiconductor industry, the move to chiplets and the use of advanced packaging technologies are rapidly becoming an important vector for the industry - and in fact, hold the promise of spawning a new industry where chiplets from different vendors can be readily interconnected via UCIe to rapidly build new products that combine best in class technologies into a single package delivering fast time to market, low risk, and low development cost. The benefits of chiplets are quite meaningful enabling the following benefits:

• Ability to build SoCs that are larger than the reticle of the semiconductor manufacturing equipment
• Building SoCs with higher yield and correspondingly lower cost vs. monolithic solutions
• Reduced R&D cost vs. employing advanced nodes
• Accelerated time to market
• Smaller footprint - vs. discrete two-chip solution
• Higher performance vs. discrete two-chip solution
• Lower power vs. discrete two-chip solution 

UCIe is an open specification that defines the interconnect between chiplets within a package, enabling the formation of a chiplet ecosystem with a common interconnect footprint at the package level.  In effect, through industry standardization, the equivalent of a common footprint, similar to Lego®, is being established.  The UCIe standard is endorsed by 72 contributing members, and 26 adopting members at the time of this writing.  

The initial focus of the UCIe 1.0 specification was in the following areas   

• Physical Layer: Die-to-Die I/O with industry-leading Key Performance Indicators (KPIs)
• Protocol: CXL/PCIe for near-term volume attach
• Well-defined specification: ensure interoperability & evolution

The UCIe physical layer supports I/Os that can provide up to 32 Giga Transfers per second (GTs) with 16 to 64 lanes and uses a 256-byte Flow Control Unit for data, similar to PCIe 6.0.  The protocol layer is based on Compute Express Link (CXL).  In short, UCIe is leveraging tried and tested technologies that have a strong legacy of adoption across many markets.

As a testament to the importance of UCIe to the automotive industry, In August of 2023, the UCIe consortia introduced a 1.1 version of the specification with a focus on the following areas:

• Enhancement for automotive - including runtime health monitoring and repair for high-reliability applications
• New usages for streaming protocols with full UCIe stack
• Protocol support with end-to-end link layer functionality
• Cost optimization for advanced packaging resulting from new bump maps
• Enhancements for compliance testing

While all of these components of the 1.1 specification reflect a keen focus on the critical next-level details associated with chip-to-chip interconnectivity and device-to-device communication, the specific considerations for automotive applications in the UCIe 1.1 specification underscore the strong anticipated adoption of chiplets in safety-critical automotive applications. 

Addressing those safety concerns, the UCIe 1.1 version of the specification includes preventive monitoring to ensure that the die-to-die signaling  “eye” height and width are optimal and can be re-trained as needed. (As a note, the “eye” diagram is a way to measure the signal integrity of the link.  The more open the “eye” is, the greater the signal integrity).  There is also the addition of run-time testability of link health which includes the periodic parity Flit injection checking for the health monitoring of each lane with the ability to repair as required. 

The adoption of state-of-the-art technologies marks a sea change in the automotive industry where historically, automotive electronics employed mature products that were typically based on mature semiconductor process technologies.  With the advent of ADAS and AD, this has now changed. AI performance requirements that can reach PetaOPs levels for Level 5 ADAS are best addressed via heterogeneous computing solutions employing high energy-efficient, high-TOPs AI accelerators in a chiplet form factor. 

At the time of this writing, the UCIe consortia is preparing to release a 2.0 version of the specification.  When the results are made publicly available, we will take a look and see what additional features and considerations have been made to the specification and place a focus on those that specifically address the automotive market.

Learning to learn in a structured manner is the only way to maintain a long career in technology. I tell those I mentor they should settle into the reality they will likely have 5+ careers in this industry.

“Don’t be scared, just stay curious,” I say. “Do everything in your power to be the positive influence on every collaboration you engage in.”

Technology domains are small communities, and being the individual who consistently demonstrates high EIQ, dignifies vs. degrades others (regardless of the role they play) is a superpower for the long haul.

Many of us with a longer tenure in technology see patterns in industry transformations, both for good and for ill. Example: years ago, I was informed by a long tenured mainframe/UNIX expert that virtualization was a mainframe feature for decades. If a server solution was considered to be equivalent to “…a toy you find at the bottom of a cereal box”, why bother with virtualization technology that limits performance? (Extra credit: Name the CEO that gave us the soundbite comparing servers to Fruit Loops without doing a web search!)

To kick off my blog series with Tech Arena, I decided to share three patterns I have consistently seen across ~30 yrs of technology inflections.

The “best” solution in the eyes of a customer rarely offers the best raw performance.

As a hardcore engineer fresh out of college, I got my hands dirty coding some of the most complex SOCs with firmware and compiled languages. I was convinced RISC was the ‘best’ architecture. I was so excited to promote the first >100MHz System on Chip (SOC) for the ARM ISA, thinking it would be easy to sell ‘better technology.’ I instead learned about legacy code and ‘good enough’ x86/68K/PowerPC incumbency. “The best technology” may cause such an operational hassle, customers will admire, but never purchase “the best.” A company can’t maintain a profitable business on glowing reviews from technology analysts or journalists.

Corollary Lesson: Being boring, reliable, easy to debug, offering ways to sample operational data without impacting run time performance is surprisingly underrated by technology providers. To service providers, this list is REALLY important to deliver a good customer experience. Using a car analogy, a Lamborghini may be the “best” auto on a racetrack as measured by its top end speed, but when driving grade school carpools, a Lamborghini is limited to the same 25 MPH zones that a Toyota Corolla or Minivan comfortably achieves.

“What compute performance giveth, the network can taketh away.”

Maintaining optimal compute/throughput/storage is a never-ending battle. Equilibrium with enough compute power that processes data, balanced by enough storage in the right tiers, plus the right network capacity to keep the data moving – it just never lasts for long.

There are so many examples, I’ll just point you to two deeper articles to explore. Both The Register on high core count CPUs and GPUs, and Broadcomm’s article with Meta’s data on AI instance efficiency offer depth on the consequences of imbalanced systems. Now, imagine the ‘inside voice’ conversations within infrastructure operators who pay ~$1B for AI processing hardware and only see ~$500m of work done on it.

“Immutable ‘laws’ vs. Ideology and narrative”

A favorite comic in my family included something related to exercise in every routine. A classic quote is “I don’t do ups, I do DOWNS, because gravity is the law, and I obey the law!” In commercial airplanes, applied physics temporarily suspend the effects of the law of gravity, but eventually gravity wins, and the plane must land. Similarly, in every technology era, exuberance for a hot technology seems to temporarily suspend the laws of economics and the principles of supply and demand. Countless breakthroughs have promised a new utopia, to replace or better humanity, and to keep humanity from…well, being itself! The internet evangelists in 2001 claimed ‘borders didn’t matter’ and ‘information wants to be free!” Here in 2024, how many paywalls have you run into this week? Economics clapped back on utopia, because it turns out that you need to make more money than you spend to stay in business. There are a few hard lessons in the fact that customers/consumers will not spend at the level of the difficult engineering that built the product. Do consumers save more money? Do enterprises see benefits well beyond what they must spend to afford the product?

Concluding thoughts

Many of these ‘rules of thumb’ don’t need to be stated when business and technology together offer a solution that legitimately solves more problems than it causes. That often happens years after the discovery of a new technology innovation – so stay curious, keep learning and share your thoughts in the comments!

Lynn A. Comp Bio

Lynn A. Comp is a Vice President and Director of the Microsoft Global Account Team in the Intel Sales and Marketing Group. Lynn’s mission is to align the unique benefits that Intel and Microsoft offer their customers through access to unparalleled technology at scale, enabling the broadest ecosystems to innovate solutions to the most challenging business problems. Lynn returns to Intel from AMD where she served as the VP and GM of the EPYC CPU Cloud Business and the VP of EPYC Product Marketing.

Lynn has a wide range of experiences spanning her ~30 years in the tech industry, from strategic planning and go to market of RISC SOCs for both communications infrastructure and mobile phones, to software offers laying the groundwork for rapid video-based services innovation, to pioneering the foundational libraries that paved the way for ‘software defined’ networking with telecommunications operators.

Lynn has extensive experience in marketing, product management, product planning, and strategy development across software, hardware, cloud, and communications service providers (CoSPs).

Lynn has a Bachelor of Science in electrical engineering from Virginia Technology and an MBA from University of Phoenix. 

TechArena's Allyson Klein and Jeniece Wnorowski from Solidigm sit down with Kelley Mullick, Vice President Technology Advancement and Alliances, from Iceotope to discuss the latest in data center cooling technology. They dive into the role of liquid cooling in supporting AI workloads, the sustainability benefits of advanced cooling solutions, and the future of edge computing.

While no day in tech is technically slow, we’ve hit mid-summer, which means that conferences have slowed, Europeans are jetting off to holiday, and everything else is feeling a bit like wading through molasses in terms of the regular maelstrom of advancement that we’ve come accustomed to in the AI era. A day like today makes me look forward to what is on the horizon, and there is plenty to look forward to in the months ahead. Today’s focus is the OCP Summit coming up October 15-17 in San Jose, and if you have peered into the roster of industry sponsors like I have you’ve noticed that OCP has…well…super-sized.

I remember the days that this Summit would feature about three top tier sponsors vying for the spotlight, and this year features nineteen. NINETEEN. This is just the top tier players, and as you scroll through the list of companies spending major money to feature their tech at the show you realize that OCP has become the nexus of what is happening in AI infrastructure. Silicon providers, infrastructure builders, power and cooling entities, oil companies – all showing up to talk about the future of the data center. All likely engaged in OCP configuration delivery and participating in conference proceedings in a major way.

The TechArena is excited to be part of the OCP Summit this year again as a media sponsor, and we are especially excited to dip into conversations regarding the advancement of sustainable infrastructure, how the fabric of the future will be shaped including advancement of Sonic (which we covered extensively in Lisbon this April), what’s next in memory advancement, and of course power and cooling innovations required to fuel these ridiculously hungry racks. We’ll be podcasting daily, engaging with vendors in video discussions from the show floor, and writing insight stories for those unable to make it to San Jose. We’re keen to know what is on the top of your mind in regards to hyperscale infrastructure priorities as well! Please take part in our daily polls on my LinkedIn feed this week to share your views on what will grab the world’s attention during the summit in October.

“We need NASA, because we need a tower of flame 60 miles high burning at temperatures hotter than the surface of the sun.” - Nova, Sharknado III

Technology: it’s all about the sublime new features. That intricate blend of performance and programming, design and development, brilliant insights leading to giant leaps of innovation. Right? We’re talking about the best and the brightest innovators, those ultra-cool kids who opted to take all those math classes, knowing that the Psychology majors were already hitting quarter beer night while they wrote up their fourth page of derivatives. But it was all worth it, because someday…some wonderful day…the world would recognize that Linear Algebra would make things safe for, um, whatever they’d get paid a lot to algebrate.

As cool as technology is, the reality of any good high-tech solution involves more boring logistics than elegant engineering. For all the results, the hours of sweat are more about jamming six new ideas in a space that holds three and hoping that something doesn’t break once the test suite is finished. Innovation is built on the shoulders of a lot of swearing in small rooms with smudged whiteboards. And maybe the archaeology of those smudges reminds some about how it was done before.

The modern data center began sometime in the early ‘00s, when commodity servers began to fill the racks, displacing RISC machines. (Sometime later, we’ll talk about how it was a stock market bust that revolutionized the datacenter.) Virtualization hadn’t hit volume yet, so most servers were single-task machines. The really innovative IT organizations were just starting to think at a rack level with stateless transactions.

The move from computer rooms to data centers was only twenty years old or so. The common architecture was a raised floor that delivered power and pulled air downward (it was fun to watch the airflow using the cigarette smoke back in the day). We were moving on from a legacy of big servers that filled a tile or two and pulled in the range of 6-8kW of power, and now facing a world where evolving commodity servers in a rack were - gasp - demanding 10kW, maybe 12kW, per tile. This was also when the data center was generally located in prime real-estate on the buffer floor in between IT and management, so forgive us for the infrastructure costs back then. We thought local, not global.

The conversations back then went to two simple vectors. One: make servers with less power. It would take a bit for us to figure out how to turn off portions of silicon that weren’t doing anything, and performance dictated that we’d probably claim that power back anyway. Two, and usually prevalent: brute-force the data center to adapt. In some cases, that meant that a rack or two of servers on a tile would be isolated by a few blank tiles to keep up with the limited cooling capacity in the room. It was not entirely uncommon to find a box fan or two on top of a rack to shove air over to another air intake.

The next innovations focused on logistics, mechanics, and air flow over server engineering. The data center began to evolve as the cloud companies led the charge to dedicated buildings with specific architecture to optimize power delivery and air movement. Hot/cold aisles started to appear, as did plenums and upward air flow. That alone bought a few kW of breathing room, even as the racks began to evolve to support a mix of compute, network, and storage to handle virtualized workloads. These days, it’s not uncommon to see 20kW racks in a vanilla data center, standing on those shoulders of innovation.

Of course, the demands never end. The commodity server isn’t as cool (pun intended) as it used to be, we processor folks blame the memory. And the rise of dedicated GPU workloads is back to driving power of individual rack units to new heights. It’s not uncommon for a full rack of AI/ML servers to demand up to 60kW of power. As before, the brute-force solutions appear to be back in vogue. Depopulating space to support the boutique solutions is back, minus the box fans. We don’t allow humans in the data center anymore, they’re just wasted heat. It’s likely that commodity will start to pull in AI innovation, and a bit of re-architecture will save some power, but the end result is likely the same. It’s likely time for another innovation in the actual buildings to accommodate the new rack reality.

It’s not hard to imagine another twenty years when we’re all chuckling about those old days where 50kW created a week of meetings in the conference room, and maybe the AI can peer back through the smudges to realize the old solutions might be useful again. So for now, here’s to the next generation of geeks skipping out on the vape party to do homework. We’ll need you then like we always did.

The world is rapidly transforming with new applications of AI across numerous industries. I’ve been fascinated by the advancement of AI in the world of pharma research in particular, given the complex process to identify an early drug candidate, conduct preclinical research and run phases of clinical trials before reaching mass market introduction.

We recently discussed pharma within our paper on semiconductor requirements for AI advancement. I am reminded of this quote from Oii’s CEO and founder Bob Rogers:

“For each application area in drug development, the speedups reported by AI vendors are 3x to 10x. These accelerations by themselves are significant, but the real magic lies in the fact that every step of the drug development process is currently built from interlocking, inefficient human tasks. Replacement with AI tooling will result in wholesale reductions in the time it takes to propose, test, and report on new drugs in the market.” 

So where is the low-hanging fruit? I’d argue that one is the world of microscopy, where early stage research is performed under electron microscopes to seek information about the chemical structures of compounds for potential efficacy to target disease. This stage of the process is incredibly human intensive with work that – frankly – can be done faster and with more precision by AI visualization techniques.

Enter Health Technology Innovations, Inc. (HTI), a National Science Foundation-funded AI operation that is targeting the space with collaborations with the leading microscope vendors and research institutions. Led by industry veteran Tuan Phamdo, HTI has built a comprehensive Cryo-FAST software platform that helps accelerate pharma research. According to the company’s claims, HTI has driven average screening time from six hours to 5 minutes, halved scope collection time, and cut annual lab costs for this function by 50%. Today, the folks at HTI announced the fantastic news that their software is ready for mass market adoption as a 1.0 offering.

This advancement has gotten the attention of large players in the sector. Gatan, JEOL, and Thermo Fischer – all major players in electron microscopy – have collaborated with HTI on the delivery of this solution. The Northwest Cryo-EM center at Oregon Health Sciences University have also deeply engaged, and I’d expect to start seeing major research labs deploying HTI’s software in tight integration with their scope of choice. 

What’s the TechArena take? Well, we’ll go where this blog started. AI is a massive disruptive force, and solutions like HTI’s will help advance scientific discovery at record pace. We’re excited to see this as a proof point of the larger transformation happening in pharma and can’t wait to see societal benefits from solution adoption. We also expect HTI to become much better known quickly in the world of pharma research as a player that is delivering real technology to customers with the help and support of industry leaders. We can’t wait to hear more about solution deployments in the months ahead.

TechArena host Allyson Klein chats with Eric Dahlen from Intel and Alex Rakow from Schneider Electric about their roles in the Compute Sustainability group of the Open Compute Project. They discuss AI's impact on data center sustainability, power and cooling innovations, and the upcoming OCP Summit.

The pace of innovation in AI is moving at light speed, so it was only a small surprise recently when Antti Karjalainen shared his company’s vision beyond large language models (LLMs). 

“So what is an agent? It's a piece of software that, contrasting to traditional software that we've been used to, does not just make you more efficient at your work. Agents will actually complete the work for you. So think about them as sort of a knowledge worker or software that can reason, collaborate and act with humans.”

Antti went on to explain that what we’ve seen with generative AI is just the beginning, and it’s in the application of these models, in his view with agents, that the real transformation of our industries and society will start taking shape.

What Sema4 is delivering is fascinating in terms of its broad reach across industries and the disruptive productivity it places in the hands of human collaborators. In Sema4’s vision, each individual can be unleashed with unlimited AI workers that can research, reason, and deliver task work, freeing the human for more value-added work. At first blush, this vision looks similar to Microsoft co-Pilot. But upon further analysis, it’s more optimized for unique industries or business processes. Antti explained the difference, being that “the AI agent is something that you can actually interact with to get a full work product done.” 

“Instead of just assisting on the side, an AI agent is kind of the main thing. An AI agent is going to be a new way to interact with enterprise applications, whereas a co-pilot is more of an additive layer on top,” he said.

When you consider the autonomy in this statement, you can start understanding the power that agents represent to autonomous control of digital functions holistically and start imagining the complexity of delivering this level of control to the enterprise. The team certainly has the technology chops and enterprise awareness to deliver the goods with far-ranging backgrounds from tech leaders including AWS, Cloudera, Docker, HortonWorks and more. Today, the Sema4 team is working with a range of business functions including customer support and finance operations to deploy initial agents into enterprise environments. Tomorrow, Antti described additional functions including HR and software development as part of a near-term vision for deployment.

So what about guardrails? Sema4 is tackling security as well as data integrity flowing into training agents as keys to success. Layer on top of that the organizational compliance and enterprise-class resilience of model adoption, and you can understand that the human trust in bringing AI agents into the enterprise is likely the main time investment in this powerful technology transition. Antti expects to see a wealth of use case deployment examples emerge in the second half of 2024 as the first signal of broad agent adoption.

What’s the TechArena take? We aren’t surprised anymore by the speed of innovation being driven by AI. IT organizations are being pushed to their limits to be agents of change, no pun intended, within the enterprise, and we expect business functions to directly adopt agents with or without internal IT support – much as they did with the initial activation of cloud services to fuel business agility. 2024 and 2025 will be rife with stories of success in this arena, and we can’t wait to hear them. As to worker response to agent deployment…we’ll leave that topic for an upcoming blog as it’s a lot to unpack and urgently needs to be discussed in order to get full value out of these powerful tools.

Advanced Driver Assistance Systems (ADAS) are all the rage. Increasingly, consumer car purchasing decisions are based on ADAS features as compared to vehicle style, engine size, or branding, which historically has been the case.

A quick internet search will provide one with an obligatory understanding of the capabilities associated with each ADAS level. A high-level summary of the different ADAS levels is as follows:

Level 0: No Driving Automation

Level 1 - Driver Assistance

Level 2 - Partial Driving Automation

Level 3 - Conditional Driving Automation

Level 4 - High Driving Automation

Level 5 - Full Driving Automation

What is profound, at least in my mind, is that there is no consistency regarding what specific ADAS capabilities are associated with a given ADAS level. Furthermore, the industry has introduced yet a new level called L3+. Not quite High Driving Automation but approaching autonomy. Apparently, good enough isn’t good enough.

The general goal of ADAS is to improve the overall safety of the driving experience. To keep this tone upbeat, I won’t focus on the mortality rate associated with driving, however, one key point to note is that it is estimated that 97% of all auto accidents are due to driver negligence. The key takeaway is that most accidents can be easily avoided through the use of technology, which is what ADAS brings to the table.

At the lower ADAS levels, the goal is to provide the driver with safety assistance features so the driver is more aware of their surroundings and can respond to the environment appropriately. Features like blind spot detection or backup cameras, typically considered Level 1/ Level 2 don’t have any physical impact on the operation of the vehicle itself but provide valuable insights to the driver to avoid what may have otherwise been a significant incident.

The progression from Level 2 to Level 3 and beyond is significant in terms of the level of electronic content that is employed to actively prevent accidents leading to the point where, at levels 4 and 5, the vehicle is driving autonomously without some level or any level of human intervention.

The impact in the reduction of accidents associated with each progressive ADAS level along with the benefits of freedom that comes from emancipating the need for a driver are profound. These benefits range from enabling a “greener” relationship between our cars and the environment to providing mobility to those who otherwise might be immobile.

The technology that is required to pull off these increasing levels of ADAS is pushing the state-of-the-art in every category. The overused phrase “data center on wheels” is an understatement regarding the level of complexity and technology that is under the hood of the modern car. Achieving level 3 ADAS typically requires 18 or more high-resolution cameras in addition to a dozen radar sensors and typically a LIDAR (light detection and range) sensor. The underlying computing engine that processes these massive amounts of sensor data typically employs AI computing that delivers 100s of Trillion - Tera Operations Per Second (TOPs).

To date, few passenger vehicles have been certified to be Level 3 compliant. And yet, Level 3 still requires a driver to be present and ready to take over control of the vehicle in the case where the capabilities of the electronics are exceeded in their ability to control the vehicle – a far cry from the vision of the self-driving car. Achieving Level 3 ADAS and beyond is pushing semiconductor process technology limits – TSMC has announced an automotive-qualified 3 nm process technology expressly to address the demands of this market as the number of transistors and their associated thermal footprint have become meaningful. One of the hottest technologies – chiplets – is also being readily embraced by the automotive market to most effectively address the mismatch in technologies required to achieve the vision of the self-driving car.

The emerging industry standard UCIe™ (Universal Chiplet Interconnect Express™) announced a 1.1 version of the specification expressly to address the automotive application space focusing on areas such as data integrity and reliability. More about chiplets and the UCIe specification will be discussed in future blogs. In short, outside of quantum computing, it seems like the automotive market is taking a leading role in defining the future of many of the emerging technologies and is a good reason to follow the exciting innovations that the automotive market is now “driving.”

Cornelis Networks shot a huge salvo today into the fabric community with the announcement of Intel veteran Lisa Spelman assuming the CEO seat for the startup. Rumblings started on the Internet yesterday that a major Intel executive was leaving the firm for a new home. Spelman was an Intel veteran, having served long-standing leadership roles in the data center and AI group as well as leading the company’s Xeon processor strategy. She’s been widely hailed as a face to watch for higher levels of corporate leadership, making this departure yet another strategic talent exit for the chip giant.

What does this mean for Cornelis? The company has a history of leadership in the HPC arena, having delivered Omni-Path fabrics to data centers and having delivered a robust portfolio of high speed fabric solutions. For those who may remember, Omni-Path was originally designed within the walls of Intel before being re-born as an independent company four years ago. And while HPC fabrics are interesting, the strategic value of Cornelis, and likely the reason why Spelman was interested in driving this tech further into the market, is AI’s insatiable fabric demand. We’ve written about the need for a fabric alternative to NVIDIA’s InfiniBand solutions on the TechArena at length before, and the Ultra Ethernet consortium is certainly building a groundswell of momentum for alternative solutions for AI clusters.

If you’d asked me yesterday who was going to lead new fabric innovation for AI, I likely wouldn’t have Cornelis on the top of my list. But the addition of Spelman, and her deep knowledge of the AI landscape, incredible relationships with ecosystem and customers built from her time driving Xeon processors, and frankly business savvy in terms of what it takes to create categories and grow tech leadership with customers, Cornelis has placed themselves squarely in the mix. I’m interested to learn more about how the Omni-Path based IP mixes with AI customer requirements, how Lisa will place a different focus on delivery of tech from previous leadership, and how strategic collaborations will be brought to bear to scale Cornelis’s ascent in the marketplace. Watch this space for more information in the weeks ahead as Cornelis begins to unveil it’s expected evolution.

TechArena host Allyson Klein chats with Sema4.ai co-founder Antti Karjalainen about his vision for AI agents and how he sees these powerful tools surpassing even what current AI models deliver today.

In our latest TechArena Data Insights interview, Jeniece Wnorowski and I had the pleasure of chatting with Doug Emby, Vice President of Cheetah RAID. We delved into the fascinating world of cutting-edge storage solutions tailored for edge environments, which are crucial for industries such as entertainment, defense, and autonomous vehicles.

Doug shared insights about the remarkable Cheetah RAID Raptor and Prowler servers. These servers are designed to handle the rigorous demands of media and entertainment as well as military applications. The Raptor 2U server, in particular, is a powerhouse with its high storage capacity, robust performance, and rugged design. It boasts up to 737.28TB (using Solidigm D5-P5336 61.44 SSD) of storage capacity, hot-swappable NVMe canisters, and support for PCIe Gen4, ensuring rapid data transfer and reliable performance even in extreme conditions. You can imagine how these hot-swappable canisters could be used to quickly capture data in rugged environments and transport it swiftly at the end of a shoot day or in a migrating environment. Given the reliability of SSD technology, this use case works effortlessly without significant risk of data loss or drive degradation.

How much data can these solutions handle? A key part of our discussion was about the importance of scalable storage solutions for managing vast amounts of data at the edge. The Solidigm D5-P5336 SSD, which is integrated into Cheetah RAID’s systems, stood out for its high capacity and performance. This SSD is optimized for data-intensive workloads, including AI-driven data lakes, big data analytics, and scale-out NAS, providing efficient and rapid storage and retrieval of extensive datasets. Jeniece shared some insights on how Solidigm’s SSD technology is integral to these advancements. She highlighted features such as enhanced power loss data protection, hardware encryption, and temperature monitoring, all of which are essential for maintaining data integrity and performance in various edge applications.

One of the most compelling parts of our conversation was understanding the synergistic innovation driven between the two companies. Their collaboration has resulted in powerful and efficient storage solutions that are being relied on by IT leaders across industries. These servers ensure that critical data is stored securely and accessed quickly when needed, making a significant impact in real-world scenarios. Cheetah RAID’s high-performance server/storage, featuring hot-swap drive canisters and Solidigm’s 61.44TB drives, makes a powerful statement unmatched by others in the market.

For those interested in the technical details and practical applications of these innovations, the full interview provides a wealth of information on the future of scalable storage solutions at the edge. To dive deeper into our discussion, you can visit the TechArena interview here and learn more about Cheetah RAID's innovative products on their official page here.

The Road to the AI Era is Paved in Semiconductor Manufacturing Innovation

This report provides insight to the force and speed of innovation required to propel artificial intelligence (AI), new requirements from across the computing landscape, and why foundational principles of semiconductor manufacturing are requiring re-invention to deliver the performance and scale of this new age. We cover the impact of generative AI and large language models (LLMs) across industries, current challenges in delivering performance to meet LLM requirements, how High Bandwidth Memory (HBM) has emerged as a foundational element of AI compute platforms, and a foreshadowing use case of upcoming chiplet-based processing solutions. We also look at how advanced 2.5D and 3D packaging, delivered in collaboration with market leader Lam Research, ensures the future of AI and continued semiconductor innovation.

Introduction

We’re seeing our world transform at warp speed, and the opportunities AI will unleash are just beginning to surface. What once seemed science fiction is actually closer than we may realize. OpenAI blew the doors off this arena with its recent demonstration of its Figure One robot performing complicated tasks and demonstrating complex decision processes. NVIDIA CEO Jensen Huang underscored this innovation, stating that robots will integrate across industries providing support for manual tasks and more.1 While robotics captures incredible inspiration for many of us, it is the tip of the iceberg for potential societal advancement in the years ahead. One of the most compelling areas for near-term benefit is pharmaceutical discovery. Here we’ve seen major tech players collaborating with traditional pharma and biotech startups to fuel a new generation of drug research that’s estimated to improve profitability in the sector by up to 25% according to McKinsey.2 What’s driving this investment? Bob Rogers, co-founder and Chief Scientific Advisor at leading healthcare AI startup BeeKeeperAI, explains,

“For each application area in drug development, the speedups reported by AI vendors are 3x to 10x. These accelerations by themselves are significant, but the real magic lies in the fact that every step of the drug development process is currently built from interlocking, inefficient human tasks. Replacement with AI tooling will result in wholesale reductions in the time it takes to propose, test, and report on new drugs in the market.”

Pharma transformation is being echoed across industries, and while each industry carries different near and far-term potential and will move at different speeds, it’s easy to conclude that entire industries will be transformed and our definition of work reshaped. In fact, we’re seeing the rapid evolution of a symbiotic relationship between humans and machines where AI can take on routine or tedious tasks, freeing humans to focus on innovation. This symbiosis even extends into the most complex invention undertaken by humans — the continued advancement of semiconductor manufacturing.

‍

I’m excited to kick off a blog series on All Things Automotive. This topic is close to my heart. It is pretty much falling off a log to garner broad interest from a broad audience as we are now entering an era where the decades of fascination around cars that could drive themselves are now becoming a reality. I have been fortunate enough to have held many different roles throughout my career, both technical and business, that have been centered around the automotive market such that I have had a front-row seat to witness the transformation of this industry.

Cars are cool.

Why do I say that? Well, everyone who has ever owned a car has their own story about their car. It’s personal. For us boomers in the crowd, we knew exactly how to feather the accelerator when trying to start the car. It was somewhat of an art. If you didn’t understand what it took to get the car started, you would end up flooding the engine or running down the battery. Ultimately, over time you developed a personal relationship with your car and “understood” it.

Cars are personal.

When you ask someone about the first car they owned, you typically will hear a story that reflects some kind of unique relationship between them and their car. Many people have given their cars names, and everyone seems to have a mix of good and bad memories to share. If you’re ever looking for a good icebreaker– ask them about their first car. You probably won’t be disappointed.

My first car was a ’71 Ford LTD. Canary yellow with an olive-green interior. – What a color combo. This car was enormous. The trunk was big enough to allow me to pack all my possessions to go back and forth to college. With gasoline at 60 cents a gallon, fuel economy wasn’t even a thought. Complete with rotor and distributor cap, on a rainy day, if water got into the distributor cap the power steering, power brakes, power everything would fail. The first time this happened to me, it was quite a traumatic experience. In fact, every time it happened it was a traumatic experience. Today you probably need to go to a museum to see a distributor cap and most certainly, if you want to buy a fleer gauge to adjust the points. This has been long since replaced by electronic ignition and such.

For the first 150-plus years since the car was first invented, the adoption of electronics and technology in general in the car was slow. Seat belts were first introduced in 1885 and weren’t mandatory until somewhere around 1968. In California, it wasn’t until 1986 that you would receive a ticket if you weren’t wearing a seat belt. Growing up in the 60’s I used to enjoy watching Batman on TV. It was a light-hearted tongue-in-cheek adaptation of the comic book series. Not only was it one of the early shows to be filmed in color, something that was lost on me because we didn’t own a color TV, but every time Batman and Robin got into the Batmobile, they paused to show Batman and Robin both putting on their seatbelts. Kind of a public safety message at the time.

Fast forward to 2024, the advances in automobiles through the adoption of the most advanced semiconductor technologies, are, in my mind, even more profound than the introduction of the automobile itself. Self-driving cars, which have been the stuff of science fiction, today are a reality, albeit with some restrictions. These advances, which have only happened in the past 10 – 15 years, have led to significant improvements in safety over the seat belt, anti-lock brakes, or airbags, the mainstay for several decades.

Historically, the electronics in the vehicle were based upon mature, low-technology semiconductors at the level of 8-bit microcontrollers fabricated on semiconductor nodes that were at least a decade old. For many of the major automotive OEMs, electronics was considered context, not core to their business. This led to the introduction of Tier 1’s including Visteon and Delphi which were the spin-out of the electronics groups within Ford and GM respectively. Ford’s spin-out of their electronics group, named Visteon happened in 2000.

Today, auto OEMs have come to realize that the use of technology to enable a safer and more enjoyable driving experience is driving consumer purchasing decisions, not brand loyalty. This is a dramatic change from the past when brand, styling, engine, and transmission were the traditional factors that drove consumer purchase decisions. The technologies adopted in today’s vehicles are not for the faint of heart; they represent some of the most leading-edge technologies across multiple disciplines including, semiconductor technologies, packaging, artificial intelligence, and computing architectures, with many still on the drawing table as they are still being defined by some of the best minds across many different industries.

Automotive OEMs grappling with the fact that electronics has quickly moved from being context to core are now starting to “spin in” electronics organizations – a term, I have been told is referred to as “the double helix”. In 2017 – just 17 years after Ford spun out Visteon, they hired 400 engineers from Blackberry to accelerate the development of vehicle electronics. This is just one very small example of the expansive disruption that has been and continues to occur across the entire automotive value chain. The disruptions are profound and make for rich stories for a blog series. I will most definitely talk about these in future blogs.

As an introductory blog, I thought that this would serve as a good backdrop to understand the motivations for the adoption and development of leading technologies across multiple disciplines. Future blogs will be more technical, covering topics including chiplets, cybersecurity, functional safety, artificial intelligence, sensors, the evolution of automotive architectures, and the very popular software-defined vehicle, amongst many others.

TechArena host Allyson Klein and Solidigm’s Jeniece Wnorowski chat with Cheetah RAID VP Doug Emby about the innovative solutions his company is delivering to edge environments across a wide swath of applications from the entertainment industry to defense, and how innovative SSD designs from Solidigm help provide a foundation for storage performance and efficiency.

TechArena host Allyson Klein and Solidigm’s Jeniece Wnorowski chat with Taboola Vice President of Information Technology and Cyber, Ariel Pisetzky, about how his company is reshaping the marketing landscape with AI infused customer engagement tools.

TechArena spoke to over a dozen industry experts from  Circle B, Credo, London South Bank University, the Open Compute Project, Palo Alto Electron, PLVision, Qarnot, the Research Institutes of Sweden, and ZeroPoint Technologies,  and  to publish this comprehensive report on the state of open compute infrastructure innovation and how organizations should align data center planning and oversight with sustainability and performance objectives. If you manage an IT organization or oversee data center infrastructure, software, or sustainability initiatives, this report offers practical value for your organization.

I sat in on Andrew Dieckmann and Nidhi Chappell’s session at MS Build today to learn more about how Microsoft is delivering new AI capability leveraging the MI300X accelerator. Andrew leads Instinct accelerator development at AMD, and Nidhi oversees Azure AI and HPC infrastructure at Microsoft. While today’s MI300X instance delivery is a tremendous milestone for the companies, this has been a multi-year journey in the making starting in 2020 with the MI50 accelerator instance which was focused on a small scale cluster implementation.

Andrew called out that generative AI is the most demanding data center workload requiring incredible performance and capability from infrastructure and specifically silicon. The MI300X has been designed to integrate AMD technologies and manufacturing prowess to deliver a compelling choice of solutions to the marketplace. Nidhi furthered this concept, stating that until today’s launch, Microsoft did not have a choice of solutions to offer customers being limited to NVIDIA instances. For those customers who are seeking higher memory capacity and better price performance, the AMD based instances provide notable value.

She extended this thought by stating that this is not just about a silicon optimization but a holistic view across data center, AI accelerator, CPU, IO and network optimization, delivering the infrastructure environment that allows Microsoft to keep pace with broader corporate objectives on scaling both performance capability and energy efficiency within the Azure environment. Nidhi’s team is leveraging MI300X for Microsoft’s own Azure AI, a stunning group of workloads that collectively delivers 7.5 trillion characters translated per month, 54 million meeting hours transcribed in Teams per month, and 100 million monthly active users of AI text predictions per month. While she didn’t go into detail on how much of this work is delivered using MI300X today, we recommend watching this space for growth of platform usage given the value the platform represents to generative AI.

One notable observation from the discussion was the centrality of low-latency access to data for generative AI. Nidhi and Andrew both discussed the capabilities offered with the MI300X platform in HBM memory support as well as platform memory capacity scale. Another attribute of note was the central focus on Hugging Face and their use of MI300X services giving callouts to the software optimizations as well as core platform capability as differentiating factors for their use.

Congrats to Microsoft and AMD for this great milestone. We at the TechArena can’t wait to see more collaborative innovation.

MS Build has always been a fantastic conference for developer innovation within the Microsoft environment. In 2024 it has transformed into a must attend event to track AI innovation. Today, Satya Nadella and team did not disappoint as they delivered a maelstrom on new announcements to Azure AI, Co-Pilot, and more. The speed of announcements in the keynote was reflective of the speed of Microsoft innovation, and it starts with the foundational innovation of Azure infrastructure.

Satya shared a massive buildout of Azure data centers across the world from Thailand and Malaysia to Spain and Wisconsin. Microsoft announced the world’s largest supercomputing cluster in the world last fall, and Satya shared that they’ve grown this supercomputing capability by 30X in the last six months, an incredible pace of deployment reflective of the customer demand for Azure AI services.

Silicon Collaborations Fuel Azure AI Growth

They’re delivering this with tight partnerships with industry leaders along with home grown innovation starting with Microsoft silicon. This starts with their deep partnership with NVIDIA. The collaboration was discussed earlier this year at GTC and covered on the TechArena. Microsoft’s Nidhi Chappell discussed the nature of this collaboration as true co-invention in her interview on TechArena last week, and this was reflective of plans for delivery of H200 based instances later this year and expectations for Blackwell platforms among the first available cloud instances on Azure. These will be available to fuel MS365 and Co-Pilot Acceleration.

NVIDIA, however, is not the only game in town for Azure, and Satya stressed a commitment to the broadest choice of acceleration. Today, Satya announced expansion of the strategic collaboration with AMD with delivery of the industry’s first NDMI300 instances for customers. This is an enormous milestone for the two companies offering the best price performance instances for GPT 4o instances. I expect to hear more about this collaboration at the conference reflective of AI providers desire to support competitors to NVIDIA’s dominance of the AI acceleration arena.

Microsoft extends their investment in this space with their own silicon, and Satya did give a shout out to Microsoft Maia acceleration. However, more attention for home grown silicon was given to Microsoft Cobalt processors. Satya announced the public preview of Colbalt based VMs for cloud native computing. These ARM based solutions are being delivered to customers including Elastic, MongoDB, Snowflake and more and put the silicon industry on notice that while Microsoft was comparatively late to indigenous silicon development, they are not slowly exploring this space but integrating rapidly into customer services.

With this rapid development of compute capacity and capability, we need to consider Microsoft’s utility bill to power this infrastructure. Satya gave an update on his team’s goals in energy efficiency stating that Microsoft is on track to meet 100% renewable energy use across global Azure data centers by next year. He pointed to specific innovations in advanced power and cooling technologies helping Azure to meet these commitments. While this is a fantastic achievement especially given the challenge of renewable energy availability across the diverse geographical landscape that Microsoft is operating, I'd like to learn more about advancements on embedded carbon investment and true circularity given the speed of innovation investment.

Infrastructure Innovation Fuels AI Integration and Societal Transformation

So what does this buildout and innovation deliver? Satya spoke to the performance and efficiency advancements that Microsoft is delivering to customers giving an example of Chat GPT4 achieving 12X cost savings and 3X performance improvements since its launch in Q4 2022. That’s 1.5X performance gains vs. Moore’s Law in case you’re tracking.

But Chat GPT 4.0 is not the only LLM being delivered in Azure AI. Satya spoke to broad model support being tapped by over 50K organizations around the world, all grounded on the foundational partnership with OpenAI. GPT-4o the industry’s top performing model announced just last week, has already been integrated with MS Co-Pilot and in Azure AI.

Microsoft has also delivered Model as a Service (MaaS) capabilities with a handful of partners including NTT Data and expanded their ongoing opensource collaboration with Hugging Face with new capabilities for developers. Satya also claimed leadership on small language models including expansion of Phi-3. Microsoft is delivering Phi-3 vision as well as Phi-3 small, medium, and mini models– all with sizes to fit developer needs from ~ 3 billion to 12 billion parameters.

All of this capability fuels opportunity for integration across industries, and Satya briefly covered examples of customers taking advantage of this technology. A notable example of society changing integration of AI into our world is a new collaboration with the Khan Academy propelling AI’s power directly into US classrooms. Khanmigo, a Khan Academy AI tool will help support US educators to offload some of the crushing operational work for managing the classroom freeing time  to for educator engagement with students. And while the capability of AI will transform industries, deliver new revenue streams and create eye opening efficiency to work, this example provides a glimpse of how transformational a time we live in. We’re excited to see more and are thrilled to see what Microsoft is delivering to help usher in this new AI Era.

TechArena host Allyson Klein chats with Microsoft’s Vice President of Azure AI and HPC Infrastructure, Nidhi Chappell, in advance of Microsoft Build 2024. Nidhi shares how her organization is accelerating deployments of critical technology to fuel the insatiable demand for AI around the world and how Microsoft’s AI tools including co-pilot, Open AI and more have been met with overwhelming engagement from developers. She also talks about Microsoft’s silicon plans and strategic collaborations with NVIDIA and AMD.

Supermicro has been a player in the tech industry for over 30 years, focusing on building breakthrough solutions for data center compute requirements. Their history as a nimble infrastructure supplier has driven them ahead as a leader in AI era compute delivery. This is why I was so excited to invite Supermicro’s Paul McLeod to the TechArena Data Insights podcast sponsored by Solidigm. My co-host Jeniece Wnorowski and I put Paul through his paces to discuss Supermicro’s perspective on AI era computing, what customers are demanding of infrastructure, and how the data pipeline is a central innovation focus for today’s deployment targets.

The changing landscape of data management in AI workloads

Paul started by discussing the history of data management across data center environments stating that traditionally, IT has involved infrastructure silos for specific storage needs, with limited data accessibility across storage solutions. Paul added that with AI this is changing. AI demands all these data types and pipeline workloads to function simultaneously. Supermicro utilizes this evolving requirement to deliver value to customers. Paul pointed out that Supermicro’s heritage includes early use of NVMe technology, giving them valuable experience in storage solutions for AI.

This has shaped by a flattening of the traditional tiered storage model. Previously, cold tiers existed for data that rarely needed to be accessed. However, with AI, fast access to almost all data has become critical meaning that the cold tier is heating up into warm tier storage where flash alternatives shine. For this transition, Supermicro's solutions have featured Solidigm’s D5P5430 SSDs. These SSDs were designed to solve the unique challenges of data center enviroments, including delivery of high-density storage high performance storage drives needed for AI training. The P5430, their premier QLC-based offering, is available in various form factors to accommodate different server designs and thermal requirements and delivering impressive capacity, reaching up to 30 terabytes. Paul noted that the technology was dialed in for Supermicro’s requirements highlighting that bottlenecks have shifted from storage to compute and network. This is made even better with key collaborations with storage partners taking advantage of underlying infrastructure to fuel even the most grueling customer requirements.

Looking ahead: The future of data storage for AI workloads

Where does the market take platform innovation next? Paul pointed out the need for continued innovation of the data pipeline to reach additional scale in performance, compute density and efficiency. As large language models scale and customers demand more compute to train algorithms, keeping the data pipeline in balance and fed with rely on continued industry collaborations with partners like VAST Data and Solidigm. Be sure to visit Supermicro and Solidigm’s websites for more information about storage and compute solutions for the AI era, and continue following the TechArena as we explore data insights.

TechArena host Allyson Klein chats with Research Institute of Sweden’s Jon Summers about the latest research his team has conducted on efficient infrastructure and data center buildout in the wake of massive data center growth for the AI era.

TechArena host Allyson Klein chats with Palo Alto Electron CEO Jawad Nasrullah about his vision for an open chiplet economy, the semiconductor manufacturing hurdles standing in the way of broad chiplet market delivery, and how he plans to play a role in shaping this next evolution of the semiconductor landscape.

TechArena host Allyson Klein chats with OCP’s Raul Alvarez on his new charter accelerating growth of the data center market in Europe as well as his ongoing work in immersion cooling technologies from OCP Lisbon 2024.