Requirements Based Verification

"There is nothing so useless as doing efficiently that which should not be done at all"

Peter Drucker

If you went round the good folks in Verilab and asked them what the biggest problem in verification was, you'd get a whole bunch of different answers.  After all, everyone has an itch to scratch.  Tommy would tell you that everything would be better if only teams had access to the best people.  Jason would probably argue that a whole bunch of problems would go away if only people really understood Liskov's substitution principle.  JL would argue that the biggest things we could do to improve verification would be to make everything an open source standard and to adopt continuous integration.  Kevin would get us all to refactor ugly code, and Mark would possibly suggest that asynchronous event handling in the various verification languages is the major thing to solve.  I'm not sure what Gordon, Robert & Gordon (the other one) would propose as our biggest challenge, but I bet it would involve Make. 

Me?  I'd tell you it would all be strawberries and cream if we just used verification processes. 

Don't run away.  Please.  Come back!  Process isn't a four letter word, and it isn't a meaningless word either as someone recently suggested.  A process is simply a name of a task and some steps to follow to best execute that task.  They don't have to be scary.  Your "Verification" process might just be:

1. Work out what to verify
2. Verify it

That's not too bad, now, is it?  It's arguably not the greatest process in the world, but even as it stands some projects could have benefited from it (it's amazing how many teams "forget" about step 1, and I have encountered one verification engineer who forgot about both steps(!)).

So why do I think it would all be sunshine and blue sky if we used processes?  It's simply because that in every troubled project I've ever encountered, the major root cause was lack of process.  Sure, there were issues with poor use of AOP and OOP, and some teams had guys who were "worse than a man down", and yes, good use of Make would have helped improve things, but even if all these things had been perfect, the fundamental problems were people either didn't know what they were meant to do, what they needed to do, or how to best execute on what they did know.  Or all three. 

By not following defined processes, essential tasks were missed completely or done badly.  Good practice wasn't captured and refined, and all the good people, tools, languages and standards in the world won't help if they're not used effectively.  I've decided on processes because I hate seeing all that project effort wasted. I hate seeing people work weekends firefighting on failing projects because they didn't follow some simple steps at the beginning.  

"It's not that we don't believe execution is critical. It certainly is. But we liken it to the Charge of the Light Brigade. If you have a plan that's fatally flawed, perfect execution can get you into more trouble because you dig yourself in deeper and faster."

Chunka Mui

Actually, let me clarify this a bit, because not all processes are equally important.  Here are the ones I think are worth adopting first:

1. Requirements extraction: How to discover what we could verify in this design
2. Requirement prioritisation: How to decide what we will verify in this design (and I don't mean have "must have", "really must have" and "essential" either).
3. Reporting: How to make sure the appropriate people get the information they need in order to make decisions

In fact, I think these processes are so important that I've formed a methodology (or is it a process?) called Requirements Based Verification around them and written a training workshop covering the first two processes [1].  I gave a short overview of the approach at DAC this year, and I'm giving a slightly different (better of course ;-) ) version at ClubT in Bristol next week, so if you want to hear more about it, come along to that.  JL's been ramping up on a slightly more detailed overview (1.5 hours instead of 45 minutes) which he's giving as part of a round the world tour with Certess, Denali, and SpringSoft.  If you're in the area at the right times, I'd suggest going along for what looks like a very interesting day. 

I'm planning to dedicate some blog space to some aspects of Requirements Based Verification over the coming months, so if you're interested, stay tuned. 

----------------

[1] Training for process 3 (Reporting) will follow at some point.  There are a whole bunch of other important processes that I'd like to produce training for at some stage.