Blog

It’s a mad, mad, mad, AI-driven world out there – and semiconductor design, from system-on-chip (SoC), to chiplets, to network-on-chip (NoC), is no exception.

AI is ‘definitely’ changing the approach to design, as Paul Martin, global director of SoC architecture at SoC and ASIC design provider Aion Silicon, explains. “Traditionally, SoCs are a form of compute platform – essentially we’re providing processors to run software – and the way we’ve dealt with increasing complexity is either scaling up or scaling out CPUs, either driving them harder or adding more cores,” says Martin. “And that’s not going to cut it, basically, on its own for AI workloads.

“For AI workloads, often the limiting factor isn’t the processing – it’s how you can move the data within the system,” adds Martin. “So, that really changes our focus on how we approach things.”

Take an ADAS system with a camera as an example. As a data source, give or take, you could have around 80 MB/s that needs to be fed into the AI model, at a hypothetical frame of 720x1280 pixels. But how will that AI model – perhaps a basic CNN (convolutional neural network) – process that? “A lot of engineers may look at that and say, ‘that’s the data bandwidth I need to deal with, that’s where my system bottleneck is’ – but it really isn’t the whole story,” says Martin. 

The CNN would take the initial frame data, represented by a 3 dimensional matrix of 720x1280x3 values, Red, Green and Blue for each pixel, into the model, which would comprise of multiple layers, with each layer computing a new three-dimensional matrix. If the first layer has 64 filters in it, the matrix you’re now looking at is 720x1280x64 values, and you’ll see another matrix computed for each layer – clearly a much greater amount than the original input data. “The important thing to understand here is that we need to be moving that data in and out of the processing element – in this case, it might be a Neural Processing unit, or NPU,” says Martin. “So, it’s going back and forth with the memory. That’s where you’ve got quite a complex data movement problem that you need to solve.”

Constantly making those trips with your primary memory source is out of the question – even more so if you’re using LLMs, because you’re likely moving between CPUs and NPUs, as they handle different processes. For Martin, whose role involves working closely with customers on complex semiconductor design projects, an architecture which deals with that data movement efficiently, therefore, is key.

“At Aion, the way we would deal with that is to essentially build a system model and to analyse how the workload is utilising the resources in the chip,” explains Martin. “How does the I/O behave? How does the processing element access memory, in terms of loads and stores; understanding the traffic profile from the workload? We do that by analysing the workload graph, building a traffic model, and then modelling the system, the bus fabrics, and the memory and the I/O, and that really then helps us to start to tune the system in terms of how it deals with that data.

“It’s a much more complex problem than simply putting a bigger processor in the way.”

Is AI moving the needle in terms of making those big chip design decisions? Martin describes it as a ‘shift-left in architecture’. “We’re moving into the system architecture space,” he says. “We’re not dealing with things in isolation. You are having to deal with both the software and hardware concurrently.

“Very often, what we’re doing with that system model is finding ways to optimise the microarchitecture, the processing elements, how we configure the on-chip fabrics, the NoCs as they commonly are, and the memory configuration,” Martin adds. “You really have to look at the whole thing holistically early on, because that’s really where the major decisions are being made – the kind of first-order decisions that we have to go through.”

Likewise, NoCs should no longer be seen as a late-stage interconnect choice, but a first-order design decision. That was the verdict of a whitepaper from Aion Silicon and partner Arteris, published earlier this year exploring best practices for implementing NoCs in AI-driven SoCs. It is a verdict that Martin also asserts, noting the importance of paying a lot of attention to the ‘traffic system within the chip’ early on in the design cycle.

“In the chiplet world, you’re also having to deal with this idea of having to connect NoCs across the die-to-die interface, so it really adds a lot of extra complexity because, in itself, that interface has to be designed so that we can move things in an efficient way,” adds Martin. “We may also be dealing with things like cache coherency, complexities like that across those die-to-die boundaries.

“So that’s in our perspective that a poor NoC architecture really can starve the best compute engines you have. You really have to get that part of the design right.”

Another partnership, announced this year, was Aion Silicon teaming up with Baya Systems for next-generation SoC and chiplet designs. Martin says he’s observed many chiplet deployments tackling the reticule and yield limitation problem, but there is a caveat. Ultimately, it’s about being able to move the data around seamlessly – which is where these partnerships come into their own.

“The issue with these things is they only really work if you get the die-to-die interconnect and the packaging correct – so there’s a whole new bunch of issues here and you really need to understand the trade-offs,” says Martin. “I think what we need to be cognisant of is that splitting the system up into chiplets gives us those new problems, and you may well be dealing with chiplets that are essentially sourced from different vendors, maybe on different process nodes, and then the underlying architecture may be different to the one you have in the die you’re producing, or at least across the multiple dies that you might be developing.

“Again, you really need to be thinking about the full system architecture, not just designing individual chiplets and then trying to bolt them all together at the end,” adds Martin. “That’s just not going to work.”

Martin is speaking at Microelectronics UK, on September 29-30, on AI transformation across the microelectronics value chain, and he concludes that, beyond shifting from chip design to system, AI is pervasive across the whole design process. “We’re increasingly seeing development of AI-enabled EDA tools to support the complexity of implementation, but also, because the design scope has grown so significantly, we’re looking at ways to boost our own productivity,” says Martin. “And that’s really where AI is starting to play a major role.

“I’m really looking forward to going to Microelectronics UK,” Martin adds. “It’s interesting to go to an event like this in the UK. Most of the events I go to are overseas, so actually, being able to engage with the UK community, how to get from concept to production silicon and really work and engage with the UK design community, I’m really looking forward to it.”