Oh, one more thing

Requirements Based Verification

"There is nothing so useless as doing efficiently that which should not be done at all"

Peter Drucker

If you went round the good folks in Verilab and asked them what the biggest problem in verification was, you'd get a whole bunch of different answers.  After all, everyone has an itch to scratch.  Tommy would tell you that everything would be better if only teams had access to the best people.  Jason would probably argue that a whole bunch of problems would go away if only people really understood Liskov's substitution principle.  JL would argue that the biggest things we could do to improve verification would be to make everything an open source standard and to adopt continuous integration.  Kevin would get us all to refactor ugly code, and Mark would possibly suggest that asynchronous event handling in the various verification languages is the major thing to solve.  I'm not sure what Gordon, Robert & Gordon (the other one) would propose as our biggest challenge, but I bet it would involve Make. 

Me?  I'd tell you it would all be strawberries and cream if we just used verification processes. 

Don't run away.  Please.  Come back!  Process isn't a four letter word, and it isn't a meaningless word either as someone recently suggested.  A process is simply a name of a task and some steps to follow to best execute that task.  They don't have to be scary.  Your "Verification" process might just be:

1. Work out what to verify
2. Verify it

That's not too bad, now, is it?  It's arguably not the greatest process in the world, but even as it stands some projects could have benefited from it (it's amazing how many teams "forget" about step 1, and I have encountered one verification engineer who forgot about both steps(!)).

So why do I think it would all be sunshine and blue sky if we used processes?  It's simply because that in every troubled project I've ever encountered, the major root cause was lack of process.  Sure, there were issues with poor use of AOP and OOP, and some teams had guys who were "worse than a man down", and yes, good use of Make would have helped improve things, but even if all these things had been perfect, the fundamental problems were people either didn't know what they were meant to do, what they needed to do, or how to best execute on what they did know.  Or all three. 

By not following defined processes, essential tasks were missed completely or done badly.  Good practice wasn't captured and refined, and all the good people, tools, languages and standards in the world won't help if they're not used effectively.  I've decided on processes because I hate seeing all that project effort wasted. I hate seeing people work weekends firefighting on failing projects because they didn't follow some simple steps at the beginning.  

"It's not that we don't believe execution is critical. It certainly is. But we liken it to the Charge of the Light Brigade. If you have a plan that's fatally flawed, perfect execution can get you into more trouble because you dig yourself in deeper and faster."

Chunka Mui

Actually, let me clarify this a bit, because not all processes are equally important.  Here are the ones I think are worth adopting first:

1. Requirements extraction: How to discover what we could verify in this design
2. Requirement prioritisation: How to decide what we will verify in this design (and I don't mean have "must have", "really must have" and "essential" either).
3. Reporting: How to make sure the appropriate people get the information they need in order to make decisions

In fact, I think these processes are so important that I've formed a methodology (or is it a process?) called Requirements Based Verification around them and written a training workshop covering the first two processes [1].  I gave a short overview of the approach at DAC this year, and I'm giving a slightly different (better of course ;-) ) version at ClubT in Bristol next week, so if you want to hear more about it, come along to that.  JL's been ramping up on a slightly more detailed overview (1.5 hours instead of 45 minutes) which he's giving as part of a round the world tour with Certess, Denali, and SpringSoft.  If you're in the area at the right times, I'd suggest going along for what looks like a very interesting day. 

I'm planning to dedicate some blog space to some aspects of Requirements Based Verification over the coming months, so if you're interested, stay tuned. 

----------------

[1] Training for process 3 (Reporting) will follow at some point.  There are a whole bunch of other important processes that I'd like to produce training for at some stage.

Sometimes I Repeat Myself

"She was mostly immensely relieved to think that virtually everything that anybody had ever told her was wrong"

Douglas Adams, "So Long and Thanks for all the Fish"

I'm in the process of teaching myself Ruby on Rails at the moment.  There's no great reason for this, other than the fact that I kept hearing people talk about it and curiosity got the better of me.  That's not immediately relevant though.  What is relevant is that in parallel, I'm learning Javascript, and one of the cool new things I learned was this - white space, commenting, and descriptive variable names are bad.  Think about it.  All your Javascript, including your comments, white space and big variable names, has to move from the server to the user's browser, consuming bandwidth (think time and money) along the way.  Wow.  Ponder the implications of that for a moment.  Some of that indisputably good software advice you were given, such as GOTO's being evil, is just plain wrong. 

In some contexts. 

That's bad news for people who just accept what they're told, turn their brains off, and treat guidelines as unbreakable rules.  Actually, it's probably bad news for those who follow behind, dealing with the results.  But anyway...

The reason I'm writing anything here is that one of the big "rules" that's mentioned all the time in Ruby on Rails is "DRY" - Don't Repeat Yourself.  Don't duplicate code or information, because that's always bad.  Right?  Actually, no.  It's wrong. 

In some contexts.

Which is all very fortuitous for me, because I get to rehash a blog post I wrote internally in Sept 2005 ("Colouring outside the lines" for any Verilaber who want's to check how much reuse I managed to achieve here).  One of the many "rules" I looked at was "You should never duplicate code" because this bugs the hell out of me.  In testbench design, there are sometimes very good reasons for duplicating code, yet I've seen engineers mindlessly removing all duplication from a working testbench.  By unthinkingly applying rules they didn't really understand, they wasted time swapping probable advantages for improbably advantages, and risked injecting bugs into working code.  Like we don't have enough to do already in verification!

So, why is duplicating code bad?  Well let's be clear.  It's not bad.  It's only bad in some contexts, and to understand which ones, it's worth understanding why not duplicating code is good. 

You might think that an advantage of not duplicating code is that it's faster to just write the code once, but that's not always true.  Making specific code generic takes time and effort, so what commonly happens is that you find that you are repeating yourself, so you do a refactoring session to replace the duplicated code with a shared version.  This means that you have already spent time writing the code multiple times, and on top of that, you then have to write a version that can be shared, remove the original code, and then fix any issues.  It's not going to be faster than just duplicating, that's for sure. 

"Always program as if the person who will be maintaining your program is a violent psychopath that knows where you live"

Martin Golding

The advantage really comes during maintenance when you have to change the code.  Rather than change it in 100 places, you only have to change it in 1 place,  That's a great thing to have.  But it's only a great thing to have if the cost of removing the duplication is smaller than the cost of updating the code in P places.  When P = 100, it's a no-brainer.  When P = 2, it's more difficult to call.  Now, it depends on how often you'll have to change the code.  If you have to change it N times, and if N is large, then removing duplication is probably good.  So basically, if N*P is large, then removing duplication is probably a good thing.

Probably.  It's time to consider context now.  We write testbenches, and a lot of the time, these don't need to be maintained.  We verify the RTL, the RTL ships, and we move on to new designs.  Testbench maintenance only really occurs when we need multiple releases (respins or phased FPGA releases) of the design, or if we want one testbench to work with multiple derivative designs.  For many testbenches, N is only large if the design is unstable, so we're constantly modifying the testbench to keep up.  That brings something else to consider though.  We remove duplicated code because the code is doing the same thing in P places.  However, what if that becomes false after you've removed the duplication?  What if you were doing FOO in two places, but now because of a last minute, badly thought out design change, you have to do FOO in one of those places, and BAR in another.  In that case, you'd have been far better off just keeping the duplicated code, because now you have one block of code that needs two different behaviours.  Ouch.

So if N*P is high and D (the amount or potential amount of divergence) is low, then removing duplication is good.  Otherwise, you might be better off just allowing code to be duplicated (while keeping a close eye on what N, P and D do during the project). 

Time for a real example.  I have one DUT that can be targeted at an ASIC or an FPGA, and in either case, it can be in RTL or gate version.  How many testbenches should I have?  Someone blindly applying the DRY rule might say one.  You should instantiate the DUT once in just one testbench, and use `defines (or similar), to deal with any differences that come up.  It would just be pure evil to have "DUT dma(.clock(clk) ..." appear in different places. 

Someone who thinks about it a bit deeper might say...

• P = 4 (e.g. we connect the clock and reset in four places)
• N might be around 10.  We have four FPGA releases planned, and we'll probably get six gate level releases
• D will be pretty large because of signal name changes.  That is, the clock connection might remain constant across all releases, but the port map is going to change like crazy to deal with FPGA targeting and gate level renaming

...and go with four testbenches.  Sure, we're probably going to have to tinker slightly every time we release a new FPGA release, or generate a new gate level design (port changes), but the growing differences between the four design types will mean that a single testbench will become a massive headache of special case handling dealing with differences between the nominally identical versions of the design.  Any common code that needs to get changed will only need changed in four places, and as it's not expected to change much anyway, it's not a major headache.  Someone going through this process might decide that the flexibility offered by maintaining separate testbenches is more useful than the benefits offered by removing duplication.

"Part of the problem with brittle design is due to overgeneralization. Good programmers tend to like to factor out the common aspects of their code, incorporating widely-used functionality into a single subroutine or class. [...] These kinds of mechanisms tend to break when a platypus is encountered"

Talin

And that's really the tradeoff we're making here.  Being DRY means reducing your flexibility to deal with divergences in the functionality, but it means that maintenance will be easier if it doesn't diverge.  You have to think about that before declaring that duplication is good or evil.  Things are never that black or white.  My experience has been that flexibility has always been more useful to me than maintenance when doing testbench design.  Flexibility means I can deal with a change on the day of code freeze.  That's more important to me than saving a couple of hours during a more leisurely and unlikely maintenance phase.  So anytime I see duplicated code, and I feel my fingers start to itch to "fix" it, I take a moment to think about the context.  It might save some headaches later to just leave it as it is.

Are *you* confused?

While sipping on my revive-me-from-the-dead coffee this morning, I had a scan of JL's latest post on the ongoing VMM/OVM spat.  This pointed me at an article by Richard Goering in SCDSource which had the following:

Bartleson said, however, that Synopsys has no intent to support OVM. "Our goal is to support the Accellera [VIP] standard, so we get some unification. For us to support OVM would mean more confusion in the industry."

<splutter>Oh crap, I’m choking on hot coffee now. 

“Confusion in the industry”.  What?  Are we really going to get confused because we have two methodologies for writing testbenches?  I hope not, because anyone who is confused is going to be a real public danger driving home from work tonight. 

My experience of the IC industry is that it’s populated by a large number of very intelligent people, and as such, a prerequisite of being involved in ASIC design (or the human race for that matter) is surely the ability to understand and differentiate between two similar things? 

I seriously wonder how Synopsys view us all if they think two things will confuse us.  Perhaps two things confuse them?

Anyway, the whole “we must only have one methodology” cat fight reminded me of this blog which I was reading a few nights ago: 

“Ruby on Rails has pretty much nuked the field of Web development in Ruby, and I wonder if it's such a good thing.  For all the flak that Java receives because you can count at least a dozen different Web frameworks, there is something to be said about plurality and the constant chase for something better and different.  Each framework that comes out builds on the strengths of its ancestors while discarding the errors (and committing a few mistakes of its own, of course). The field advances a little bit every time while bowing down to the timeless laws of natural selection.

I am worried that Ruby on Rails will do to the Ruby world what JUnit did to Java:  a great tool when it came out but which condemned its community to an ice age where no innovation or competition appeared for years.  Whatever the fate of Ruby, I hope its fans will keep an open mind and will constantly challenge the Rails way, for the simple reason that it's always healthy to question what's in place, no matter how good it looks.”

I couldn't agree more.  Anyway, I’m going to finish my coffee, and wait for the next installment of “VMM vs. OVM – Handbags at Dawn”.

Imaginability

“But the piece of paper on my desk doesn't have discrete methods. If I decide for example, to burn it for fuel, or fold it into a paper airplane, does that mean that there is a ‘burn’ or ‘fly’ operation that's somehow built into the paper, and that it inherits these operations from a superclass of ‘flat things’? Nonsense. There are an almost infinite number of things I can do with a simple piece of paper, none of which may have been anticipated by the creator of that paper” Talin

Lack of imagination is a big problem in verification.  If you could imagine all of the ways a design could fail, then you could make sure that you checked for each one of them.  On the other hand, if you couldn’t imagine any ways in which it could fail, then your verification is going to be pretty hit and miss.  You’d hopefully put the design through its paces to see what else fell out, but to be honest, your heart probably won’t be in it.  Your chances of hitting many of the bugs will be slim.

To get round this, we try and involve multiple people with various specialities in the verification planning.  More brains mean more things that can be imagined, and the more that can be imagined, the more we can attempt to verify.

Of course, once you've thought of all of the things that you need to verify, you need to verify them, and directed testing just isn't up to the job any more.  You might have imagined a million interesting ways for your chip to die, but there's no way you'll type in enough code in time to verify them.  So the EDA vendors gave us coverage driven verification as a solution.  Functional coverage lets us specify what we want to see hit, and the constrained random stimuli hits them, and maybe some things you didn't even think of at all.

Isn't that cool?  By switching to a new technology, verification teams can extend their verification imaginability (even if they don't know it).  It’s not perfect though.  We still need to use our imaginations to guess where some of the more esoteric bugs might be.  If we don’t, then we probably won’t write the necessary functional coverage, stimuli generators and checkers, so it’s not all plain sailing. 

In a further attempt to fix the imaginability problem, the EDA vendors are developing to new technology in the form of "intelligent testbenches" and formal verification to automatically extend our imaginability without asking us to think any harder.

That’s nice of them.  There’s a limit to what these technologies can achieve (despite what their marketting will tell you), but they are certainly taking a lot of the imaginability burden off our fried brains.

So, we have an imaginability problem, we recognised it, the EDA vendors came to our aid, and we bought their solutions.

Now on to the point...

Wanted - Excellent verification engineers. Must be 6' 4"

Most people will (hopefully) agree that selecting a verification engineer by height would be a bizzare thing to do.  In fact, if we saw an advert that said "Wanted - Excellent verification engineers.  Must be 6' 4"", then we'd probably think that the advertising company were pretty strange, and didn't really know what they were looking for. 

After all, what's height got to do with verification?  It's a physical thing that you have no control over, and you are the height you are whether or not you did anything useful while attaining that height.  In fact, I'd pass the advert by and move onto a more sensible prospective employer. Any team selected by height probably isn't going to be much good, and I can imaging daily life on a project would be pretty dire.

Years of experience (YoE) is not much more useful than height in telling us how suitable the candidate is for the job, so why then, do companies still use it as a major selection, or rate-setting, criteria?

The tyranny of the, err, hem, actually, I'm not really sure

There was an interesting post on the verification guild this morning called "Testbench's Tyranny".  It's interesting to me because I have no idea what it's about.  I'm baffled, although I think I have it narrowed down to 2 possibilities:

1. It's going to be a discussion on why constrained random testbenches are evil, why OOP is evil, and why RTL based directed tests are all we need
2. It's a precursor to an announcement of yet another wonder product that's going to make verification push button, and remove the need for verification engineers and their brains

Based on the poster's name "xxx" and the fact this is his only post, I suspect the latter (unless we're encountering an interesting crossover from an entirely different industry!).  If it turns out to be possibility 1, then I'm looking forward to the follow up posts about why we should never have abandoned schematics in the first place, and a counter reply along the lines of "schematics!  It's been a disaster since we stopped using Rubylith".

Either way, I think it's worth watching.

But what prompted me to launch myself out of my writer's block and type was the following line:

"What on earth is the good of [sic] ``OOP, Layered'' TB to a DUT, which is written in plain Verilog modules/VHDL processes ready for synthesis"

Dissing this

Damn damn damn damn.  I promised myself that I wouldn't get involved in any more conversations about coding styles - ever, and then Janick has to go and write "Do they diss this “this”?"  Use of "this." is a pet hate of mine as it's nothing but a source of annoying bugs.  Janick has done an excellent job of explaining what "this." is for, and why you might want to use it, so I'm just going to be lazy and focus on why you might not want to use it. 

The "this." keyword has only one purpose, and that's to let you differentiate between class member variables and local variables (local as in scope, not local as in access rights).  I hope Janick doesn't mind me stealing part of his example here:

class packet;
   bit [47:0] da;

   function new(bit [47:0] da);
      this.da = da;
   endfunction
endclass

What we have here is a member variable called da (global scope within the class), and a parameter called da (local scope to the new() method).  When you're in new(), they're both visible and now you have to do something special to untangle them.  You could compulsively use "this" to solve the problem I suppose, but I'd suggest that avoiding the problem in the first place might be an all round better solution.

The problem with "this" is that if you forget to use it, or if it gets edited out of the code, then your code will continue to compile, but will now have a subtle error where the incorrect variable is used.  Here's what Gordon Allan, a colleague of mine (and I should add, one of the best coders I've ever met), said about this on his internal blog a while back:

"After much heated debate (religious wars even) I've decided to stop using "this." after chasing down a simple typo-induced bug for an hour, that was masked by "this." usage. Hungarian (or any similar) notation would have avoided it, and that wasted time is a good enough reason for me to switch. To what, though?

In our earlier debates we talked about 'this.' or 'me.' qualifiers being nothing more than unnecessary decoration in most cases. (I was one of those who sometimes liked that decoration, especially as it stood out nicely in my Xemacs window).

We mentioned that the only 'justifiable' time to use 'this.' was in an accessor method which took parameters named the same as the properties. But I now assert that this is dangerous and should not be used to justify 'this.'. For example:

    function void setFoo( int channel );
       ...   ...
       this.chan = chan;
       ...   ...
    endfunction

Can you spot the mistake above? I had one of those. I mis-spelt the parameter name in the method header, and so my 'this.chan = chan' line compiled OK with both lvalue and rvalue referring to the same thing and never involving the parameter. D'oh! A hungarian naming convention would look like:

    function void setFoo( int channel );
       m_chan = chan;
    endfunction

Which would of course fail to compile, alerting me to the typo."

Now, there's a whole new holy war here about what to use instead of "this".  Perhaps we should put "m_" in front of a class member variable because a member variable is "special". It's different from the other ones lying around you. It's global (to the class), so writing to it has side effects elsewhere in the code, and as a result it should be treated with respect. "m_" marks it as special. So does "__" in C. Incorporating "usage warnings" into variables isn't new. Consider them like the yellow bands on a wasp. Maybe not beautiful, but damn useful.

However, you can validly argue that this pollutes the public interface (which will be the topic of another blog soon), so it would be better to use "p_" for a method parameter or "l_" for something local to the method, or even (god forbid), use a different name.  Or as Gordon finished up:

"So I think I'm going back to my native Java/C# coding styles: PascalCase for names of public things (like classes, properties, methods), and camelCase for names of private things (like local variables, or parameters). That's the only distinction I need to make, so long as I name methods as verbs (and call them after what they do) and classes and properties as nouns (and call them after what they are)."

What you use is entirely up to you.  All I can say is that I've seem "this" do far more harm than good, and while my other suggestions might hurt the aesthetics of beautiful code, they help prevent bugs. As Joel said, "I can live with the shame if it makes me more productive". I have enough stupid bugs as it is without introducing another source in the name of "art".

Cheers
David

Yay - _another_ verification blog

That's it - I've finally caved in and got myself a public blog.  I'm not sure if it was because of the success of the Blogging Birds of a Feather session at DAC, the fact that a few of my colleagues (JL, Tommy & Gordon) are having fun with their blogs, or the fact that my flight home from DAC isn't for another 11 hours, and I'm at a loose end.

The hardest part so far has been choosing a name for this thing.  I'm sure content will be harder, but one step at a time, and all that.  I'll reckoned that I'll probably end up using this blog to write down the things I wish I had said during discussions [1], so "Oh, one more thing" seemed apt.

Have fun

[1] A few years back I had enough things to say after a particularly annoying conversation was over that I ended up writing a book. I'm hoping that the blog will prevent that kind of madness from happening again