Imaginability

“But the piece of paper on my desk doesn't have discrete methods. If I decide for example, to burn it for fuel, or fold it into a paper airplane, does that mean that there is a ‘burn’ or ‘fly’ operation that's somehow built into the paper, and that it inherits these operations from a superclass of ‘flat things’? Nonsense. There are an almost infinite number of things I can do with a simple piece of paper, none of which may have been anticipated by the creator of that paper” Talin

Lack of imagination is a big problem in verification.  If you could imagine all of the ways a design could fail, then you could make sure that you checked for each one of them.  On the other hand, if you couldn’t imagine any ways in which it could fail, then your verification is going to be pretty hit and miss.  You’d hopefully put the design through its paces to see what else fell out, but to be honest, your heart probably won’t be in it.  Your chances of hitting many of the bugs will be slim.

To get round this, we try and involve multiple people with various specialities in the verification planning.  More brains mean more things that can be imagined, and the more that can be imagined, the more we can attempt to verify.

Of course, once you've thought of all of the things that you need to verify, you need to verify them, and directed testing just isn't up to the job any more.  You might have imagined a million interesting ways for your chip to die, but there's no way you'll type in enough code in time to verify them.  So the EDA vendors gave us coverage driven verification as a solution.  Functional coverage lets us specify what we want to see hit, and the constrained random stimuli hits them, and maybe some things you didn't even think of at all.

Isn't that cool?  By switching to a new technology, verification teams can extend their verification imaginability (even if they don't know it).  It’s not perfect though.  We still need to use our imaginations to guess where some of the more esoteric bugs might be.  If we don’t, then we probably won’t write the necessary functional coverage, stimuli generators and checkers, so it’s not all plain sailing. 

In a further attempt to fix the imaginability problem, the EDA vendors are developing to new technology in the form of "intelligent testbenches" and formal verification to automatically extend our imaginability without asking us to think any harder.

That’s nice of them.  There’s a limit to what these technologies can achieve (despite what their marketting will tell you), but they are certainly taking a lot of the imaginability burden off our fried brains.

So, we have an imaginability problem, we recognised it, the EDA vendors came to our aid, and we bought their solutions.

Now on to the point...

Imaginability is also a problem for testbench and verification IP design (let's call it verification design for narative convinience), but bizarrely, the verification industry is trying to move backwards rather than forwards, and even more bizarrely, they're clapping themselves on the back about it.

To see the problem with imaginability in verification design, you have to consider their users.  The users might be customers who buy your verification IP and put it in their own testbench, or they might be test writers who are using the testbench you have written for them. Either way, you might be under the impression that you have thought of everything you need your verification design to do, but your test writers and customers will probably disagree.  They will almost certainly try and do something with it that you didn't imagine. Why? Because there are more of them than you, so they can apply more brain power to it than you can.

To give you some examples of where imaginability has failed in verification design:

• One company I know has created several versions of the APB bus for themselves, which means no APB verification IP is suitable unless it’s very flexible in very strange ways
• Another breaks the AHB standard in four different ways that I know of, which means no AHB verification IP is suitable unless it’s flexible in even stranger ways
• My very first testbench fell over the day before tapeout because a last minute feature was added to the design and my testbench struggled to collate the data from various testbench components and pass them to the new checkers.

Imaginability problems don’t just come around because your customer is breaking the protocol, or because you weren’t good enough to out-think your users.  An important point to bear in mind is that the more you write your verification design, the more you can start to imagine because of a growing familiarity with the problem at hand.    Unfortunately, the longer you write your verification design, the more inflexible it becomes.  You might start to imagine what the user might want, but simply not be able to retrofit it to your design

So imaginability is a big problem in this arena as well.  Are we handling it as well as we are in verification planning and execution?

Are we hell.  The industry seems to be making a concerted effort to make things worse for itself.  Two and a half of the three big EDA vendors did it for political and financial reasons (which I can understand), but I’m not quite sure why the users started firing at their own feet.

The imaginability problem means that verification designs have to be flexible, and they have to be flexible by default, not just by design. Anyone needs to be able to modify a verification design's operation without changing the code directly.  That means we need very flexible verification languages. 

This is an area where e simply blows SystemVerilog away. The AOP features of e mean flexibility is built into your verification designs by default. No matter how much imagination you lack, your users will have a fighting chance of making your verification design useful to them. OOP on the other hand is only flexible if you make it flexible. You need to imagine all of the things the user would want to do with it (which is where we came in really - you won't imagine all of those things, because of the imaginability problem), and add hooks, callbacks, publicly accessible types, virtual methods, etc.

The only inbuilt flexibility of OOP involves inheritance. This sounds good in theory, but the reality is a bit more depressing. Non-virtual methods will rain on your parade, as will private and protected member variables that you want to replace with your new class type. So even the default inbuilt flexibility of OOP relies on the writer having the imagination to make their stuff virtual and accessible.

So e is flexible by default whereas SystemVerilog can only be flexible by design. SystemVerilog is a step backwards in solving the imaginability problem for verification design, and that seems to have been missed amongst all of the SystemVerilog marketing noise. In at least one respect, verification just got a little bit worse.

Luckily, it wasn’t lost on VerisityCadence when they worked on the OVM, and they appear to have done a very good job of putting a class library together that lets the end user decide what’s really going to be in the testbench.  It’s a shame that to get back to almost what we had for years with the eRM, they’ve had to resort to configuring the testbench using strings.  Nasty.

Of course, it’s only the OVM code that’s flexible by default.  You still have to imagine all possible uses of your custom code.  But you can’t.  Bummer.  How long until the next version of SV is due, and will we get over the whole “AOP is evil” marketing FUD in time?

Cheers
David

<shameless plug>

ps. If you want to know more about exploiting the flexibility of AOP, grab yourself a copy of Aspect-Oriented Programming with the e Verification Language: A Pragmatic Guide for Testbench Developers. It will teach you all you need to know about getting the most from AOP.

</shameless plug>

Comments

[Scott Hughes]

I'm relatively new to the world of OVM. Currently switching a project which used my own homebrewed Ruby scripting version of constrained random stimulus generation over to SystemVerilog/OVM. I have no experience at all with e, but I'm very familiar with OOP and AOP and, specifically, used AspectJ to inject a little AOP in my Java. Couldn't a similar approach to AspectJ be applied to SystemVerilog? It's a little easier in the Java world, since the Java bytecode can be directly manipulated, but I assume there is no (cross-simulator) method for manipulating compiled SystemVerilog. But a pre-processor on the source (which would be annotated with a little meta-data) could achieve the same effect, couldn't it?

[David Robinson]

Hi Scott,
writing this preprocessor (AKA weaver) should be possible, and you might not even need any meta-data (you just need to find class definitions, method definitions, coverage definitions, etc) although you’ll need to bear in mind that AOP in e is different from AOP in Java (it helps to consider AOP as a general term hijacked by marketting).

In particular, the active extensions (advice) in e can be selected based on the current values of determinant variables (simply enumerated or boolean variables), which makes deciding the final method that gets called more complex than in the Java case. It also makes AOP far more useful in e, as we can selectively apply the advice to specific instances of class FOO, not just all instances of FOO. I think some Java folk were moving towards “Dynamic AOP” which is probably the same thing.

A problem that will have to be solved is that any compile error messages will come from the generated code, and not the original code, which is a royal pain to debug (C++ templates used to be horrible because of this).

The 800 lb gorilla in the room is the fact that a lot of verification IP is encrypted, which means you can’t preprocess the source code. Still, something would be better than nothing. I’m pinning my hopes on someone at Accellera getting the AOP from e added to SV.

Cheers
David