Agile is for losers…

…real men embrace agility.

The development world seems to be full of ‘Agile’ pretenders. If you were to
object to my saying that ‘there is no such thing as Agile’ then I’m sorry to say
that there’s a reasonable chance that you too fall into this category.

There is no ‘Agile manifesto’. There is a ‘Manifesto for agile software
development’, which lays out some general principles which one might like to follow in order to help them develop software with agility.
Of course it seems entirely reasonable to shorten ‘agile software development’ to simply ‘agile’, but herein lies the same problem we see in almost every other aspect of our day to day lives – words are allowed to be abused and end up losing any sense of meaning. Worse still they end up meaning something entirely different than they did to start with. We see it all the time in the world of politics, with words such as ‘extremist’ or ‘terrorist’. We talk about ‘morality’ as if it’s a shorthand for ‘whatever I subjectively feel is best’ – even if ‘what I subjectively think is best’ is internally logically inconsistent. We see people talking about ‘doing’ ‘Agile’, as if ‘Agile’ is a name for some prescriptive process (invariably involving moving cards around a board) which, if followed obediently, will yield some kind of magical gain in productivity.

Are you an Agile pretender?

* Firstly I’d ask you what ‘Agile’ actually is? It sounds like a simple question but I would bet that a very many people would struggle to give an answer that didn’t sound far from waffle.

* Secondly I’d ask you why do we want it? Again this should be a simple question to answer, and an adequate answer could be given in 3 short sentences.

* Lastly I’d ask you why & how ‘doing Agile’ is beneficial to your particular project?

<br/>
‘Agile’ is meaningless. Unfortunately the word has been entirely destroyed and I fear will never be recovered (although reluctantly I still have to use it when talking to agents, etc). Dave Thomas (one of the original signatories of the manifesto) wrote a blog post recently on the topic here: http://pragdave.me/blog/2014/03/04/time-to-kill-agile/
Having felt enormously frustrated for the last couple of years, seeing Dave’s post has made it clear to me once and for all that I’m not imagining the whole thing. The software development community is unfortunately easy prey for a whole cottage industry of snake-oil salesman that has sprung up around ‘Agile’, and the hidden – even if unintended – payload of destruction is that a whole generation of developers (the guys at the fake agile coalface) is emerging comprised of men & women who feel jaded towards the whole idea of ‘Agile’.
The reality is that the ideas and mechanisms to develop software with agility are very simple. They are simple because they make logical sense, and don’t come in the form of complicated & prescriptive tasks, but can be summarised in a general sense by saying ‘Only do what adds value. Don’t do what costs more than the value it adds. Agility is valuable, so a present cost may be worth future agility. Embrace collaboration in order to leverage the agility you’ve built into your engineering processes.’. For a thoughtful developer, working on a real agile project will fill them with joy – the joy a professional experiences in the efficient delivery of quality software, and the joy they experience in seeing their own productivity and professionalism grow from beginning to end.
What many men & women face today is a mysterious, almost cult-like, ritualism resembling some kind of black magic. The thoughtful ones never feel quite comfortable with it because they are unable to rationalise any sense out of it – they never experience the rewards that offset the costs of those mysterious ceremonies. it all feels like a fraud. Those people tell their friends that ‘Agile’ is a fraud, in the same way that people who don’t know better tell their children that capitalism is all about the rich taking advantage of the poor.

Am I an agile pretender?

* “‘What is ‘Agile’?”
As I’ve already suggested, ‘Agile’ is the process by which software development is done with agility in mind. The agility to change direction as quickly as possible, and as cheaply as possible. The agility to fail fast and fail cheap. The agility to continuously adapt and improve our own processes and techniques in order to deliver maximum value over a given period of time.

* “Why do we want it?”
We want agility because it increases our chances of success. It often reduces costs. It increases the chances of the customer getting the product they wanted, even if it’s not necessarily the product they asked for. In a dynamic global market things change fast and we need the agility to change the product during development according to market forces.
Human beings aren’t always great at knowing or articulating what it is that they actually need. We want the agility to draw those developing requirements out during the course of development, and we want the agility such that our product’s design can be emergent according to the feedback that we get from both the customer and our own processes.
We want the agility to fail fast, because to fail fast is to fail cheap.

* “why & how is ‘doing Agile’ beneficial to your particular project?”
This one can often be the slam-dunk! As it happens the project I’m currently working on, while labelled ‘Agile’, experiences absolutely no benefit from any agility (other than those associated with the good engineering practises which are a prerequisite for any successful agile project – which we don’t really do!). My current project is a fake ‘Agile’ one. We follow a Scrum paradigm which supposedly makes us agile, yet none of the reasons to adopt Scrum are of any benefit to us. Our project consists of what is essentially a very well defined list of discrete alterations/fixes which are required for regulatory reasons. The implementation of one of those fixes very rarely has any bearing on the implementation of any of the others, and what’s more we have a fixed and very specific scope (and a fixed deadline too :D). With this being the case there is absolutely no benefit to having the notional 2 week sprints that we do. There is very little benefit to the sprint retrospectives. Sprint reviews are pretty pointless when a task is so well defined, trivial & discrete from all others. In the daily scrum/standup we essentially try not to bore each other too much as we explain what we did yesterday, and intend to do today; no one cares because what I do today has no bearing on what anyone else will do tomorrow – there is no sprint goal.

For the last 6 months, or so, I’ve been compelled to be an agile pretender. In an environment where every man and his dog is sent on a ScrumMaster certification, and the new vocabulary that comes with it is thought to imbue the team with magical powers of increased delivery potential through wilful thoughtlessness (a perversion if ever I saw one), I have little choice.

Agile is for losers, and these posts are my attempt, for the time being, to be a thinking, ‘real’ man, who instead chooses to seek agility in software development.

See: Baby-steps to developing with agility.

Posted in Uncategorized | Tagged , , | Leave a comment

Story Points, Bugs & asking yourself “To What End?”

Some people like to invent complication. I’ve noticed this a lot when it comes to agile software development among teams that are following the fashion without really understanding what they are doing, or why they are doing it; and it doesn’t help that since seventeen or so guys met up in a ski resort in Utah over ten years ago, a whole cottage industry has sprung up around agile coaching & training – of which I’m certain a significant number lean more towards being snake-oil salesmen selling ‘Agile’ the noun (as Dave Thomas – an original signatory of the agile manifesto – laments here).

I’ve got a few ‘Agile’ items in my backlog to moan about later, but for now I wanted to talk about a particular irritation – story points & bugs. I keep finding myself in discussions with people asking what is the proper way to estimate bugs? Often this apparent difficulty stems from the idea that story points represent business value deliverable to the client – and as such we shouldn’t assign story points to bug fixes because they are mistakes, rather than value to the client and that our velocity should only reflect our ability to deliver value to the client, rather than our ability to clean up after ourselves when we mess up. Here are a few quotes for clarity:

“Our theory is that points, and our resulting velocity, should demonstrate business value to our customers. Customers shouldn’t “pay” for bug fixes. The product should be of a valuable quality.”

“Hone skills for collaborating with the customer. Story points represent what the customer values. Better to hone the skill of writing and estimating stories, as the vehicle for delivering that value, instead of trying to figure out the convoluted ways to write a story about fixing a memory leak.”

“I don’t assign story points to defects since I think story points are a measure of the team’s ability to deliver value in an iteration. Fixing defects, while important and valuable, is NOT delivering value…it’s a drag on the team’s ability to deliver value. The more the quality is built in –> the less defects arise –> the more bandwidth the team has to deliver new stories –> the velocity should be seen to increase. Assigning story points to defects, I think, pollutes this.”

“…fixing defects, while important and valuable, is NOT delivering value…insomuch as value is measured as quality product in the hands of customers. If we’re using story points to measure our effort to make ANY changes to the product, be it new functionality or fixing defects, then have at it and assign story points to defects.

If we’re using it as a measure of a team’s ability to deliver high quality product into the hands of the customer (my preference), then no, don’t assign story points to defect work.”

 

I think that the common theme here that underlies the misunderstanding of story points is the apparent misunderstanding of the term ‘business value’. I think that this is often the case because people have failed to ask themselves the most important question anyone should always ask when developing software – to what end? Perhaps the dark side of the previously mentioned cottage-industry preys on those that are quicker to buy a productised – step by step – solution than they are to wonder: ‘to what end?’ and in the process grok the rationale behind agile development methodologies and principles. Give a man to fish and he’ll eat that evening; Send him on an Certified ScrumMaster course and he’ll progenate a team of jaded and cynical developers who feel no joy in their supposed agility – thanks to the nagging dissonance associated with constantly trying to resolve contradictions which needn’t exist if they only knew ‘to what end?’ they were adopting these methods & techniques, and with that understanding were able to choose and apply them appropriately.

 

Hold on – ‘Bug’ is a pretty ambiguous term…

Yes it is. So to put this post in context, we’re talking about generally well defined defects that have managed to be introduced in a sprint and have found their way outside despite the associated stories being ‘done’. We’re talking about bugs that should have been caught, as opposed to the kind that only manifest themselves when a butterfly flaps its wings a certain way.

Other kinds of bugs include those of similar origin, but which aren’t well understood or defined – these would likely require a spike as it would make no sense to relatively size a poorly understood problem. Although there’s no reason that spike couldn’t spawn point-estimated task if it was worth doing so.

One last scenario worth mentioning is bugs not introduced during a sprint, but the kind that might exist in a backlog to address defects in an existing legacy system. Lets assume that these are all well understood by now so that we don’t need spikes. Should we assign story points to these? No – these probably shouldn’t even be part of the Scrum (i’m assuming Scrum as it’s pretty ubiquitous these days) paradigm – particularly if they are all individual and discrete issues (as they typically are). Why on earth would we think that Scrum or even ‘Agile’ (the noun) is particularly appropriate? To what end would we introduce such a framework?

That’s the subject of another rant I’ll get around to sometime, but to cut a long story short we use agile methods and techniques to achieve a state of agility in the business:

  • Agility to respond to change in a dynamic global market.
  • Agility to fail fast and fail cheap.
  • Agility to minimise time to market in order to maximise return on investment.

 

Those kinds of bug fixes, as discrete individual items, don’t participate in the tight feedback loops that facilitate such agility in the first place. How will Scrum, when fixing these defects, help us to achieve those attributes of agility?

 

To What End?

Story points exist to decouple our estimation process from units of time. It’s as simple as that. Why?

We live our lives in hours and days, and our sprints (if we’re using something like Scrum) are measured in aggregations of those days. Our releases are probably planned in aggregations of those sprints, etc. These are all fixed measurements and their relations, in a quantitative sense, always remain fixed relative to each other (except for leap years perhaps!); The problem that this poses is that it makes it incredibly difficult to estimate accurately (in hours for example), over an extended period of time with all kinds of changing factors which influence our ability to deliver value to the client, just how much value we can deliver in any given timeframe.

If we can decouple our estimations from units of time we can decouple our ability to give useful estimates from the factors that effect the time it takes to deliver – team alterations, changing product complexity, increasing productivity with familiarity, physical environmental factors, or anything else you might imagine.

By sizing stories relative to one another, using points, we can measure our velocity – our ability to deliver value to the client – in a way which is self-correcting without ever having to estimate anything other than how large one story is compared to another.

Different people in different teams can all easily estimate, with reasonable accuracy, the size of one story compared to another – regardless of how long it would actually take each of them to implement those stories compared to their colleagues. With a little effort and discipline we can have discrete teams with different velocities, all normalised according to a common understanding of story point allocation; This grants a product owner otherwise unimaginable flexibility when it comes to planning and allocating stories to multiple teams.

 

So this is the ‘end’ I alluded to when we wondered what story points were used for. So to what end would we conclude that story points shouldn’t be used to estimate bug fixes?

 

Business Value

Simply put, business value is something that the client wants. Something that the client cares about. Something they value.

The client doesn’t care whether you need to spend time upgrading your development machines to the latest OS version, or whether you need to spend 15 minutes every day rebooting your flakey git repo, or whether you need to spend more time testing because quality is slipping (a new screen roughly similar to a previously built screen is worth roughly the same to him whether you spend twice the time or half the time).

Your client DOES care about getting stories delivered, and your client DOES care about emergent bugs being fixed (hence bug fixes do provide business value – otherwise your client wouldn’t care whether you fixed it or not).

With this clearer understanding of business value, is there any end to which you can imagine that not assigning story points to estimate bugs will be beneficial? Lots of people like to assign a spike instead. The trouble with this is that (in the context which I set out earlier) the bug definitely needs to be fixed, and spikes are by definition time boxed efforts (which implies that bug resolution is optional). Secondly, which ever word you use to describe the allocation of resource involves an implicit conversion from time to story points – otherwise how do we know how many points fewer to commit to this sprint? The point of time-boxing spikes is that it makes no sense to try to size them relative to the usual stories; If it does make sense to compare them to the size of stories, then it almost certainly shouldn’t be a spike. The only reason to do so would be to accommodate this faulty ‘bug fixes aren’t business value’ logic.

 

How I would account for Bugs

In the context outlined earlier, I like to keep it as simple as possible unless there is a compelling reason not to. I would size the bug relative to other stories as usual – lets say we think that it will be roughly 2 points. It’s important that we assign these two points because they represent work that needs to be done, which means that there is 2 points worth of other work that won’t fit into the sprint.

We also need to remember that the original story from which this bug leaked (if we can pin it down like that) was estimated with a definition of done which, presumably, implied that the deliverable was bug-free. The fact that we’re now addressing that bug means that the original story isn’t in fact complete, and if we kept our velocity as it is we would go into our next sprint estimating how much work we can ‘nearly finish’, rather than actually finish (especially if this bug leakage is becoming habitual – otherwise in a one off situation this may not be worth the bother). So I’d reduce our velocity by 2 points, for example, too. That means that this sprint we’ll actually plan to deliver 4 points-worth fewer stories, and then going forward our velocity will be 2 points less until such time that we stop leaking bugs and manage to gain some velocity back.

 

Posted in software | Tagged , , | Leave a comment

Java lacks multiple inheritance of implementation because it’s not necessary?

Well… you could argue that Java is unnecessary when we have C. You could argue that C is unnecessary when we have assembly code.  So the question should be – is it useful? Lots of people suggest no – it’s dangerous and we can achieve the same thing through interfaces and composition… Deadly Diamond of Death… Think of the children!

My personal view is that yes, it can be useful. And yes it can be confusing/dangerous. I’d also argue that a spoon can be dangerous in the hands of a moron, yet we don’t abolish the use of spoons, because they have value; And the decision to use them in a dangerous manner is entirely voluntary. I want multiple inheritance of implementation, and I don’t like it when other people say that I should’t have it because they can’t trust themselves not to break stuff.

Here’s one line of reasoning I’d like to address. Someone recently stated in a sweeping and absolute manner:

“Because there is a better way without multiple inheritance.”

So lets look at this example of the observer pattern which I’ve roughly taken from an old Bob Martin document…
We have a Clock which is a self contained class which understands the way in which time is measured and represented. It ticks every second. We’d like to keep it self-contained and with a single responsibility.

We also want to implement an Observer pattern, whereby objects can register themselves as Observers of the clock so that they receive a notification whenever the clock updates it’s internal representation of the time (we don’t want to continuously poll the clock, wasting cpu cycles, when the time will only ever be updated once every second).

Given that Java doesn’t support multiple inheritance of implementations (ignoring some blurred lines introduced by Java 8 just recently), an implementation of the Observer pattern would have to look roughly like this:

public class Clock {

   public void tick() {
      // Update time each tick.
   }
}

interface Observer {
   public void update();
}

interface Subject {
   public void notifyObservers();
   public void registerObserver(Observer observer);
}

class SubjectImpl implements Subject {

   List<Observer> observers = new ArrayList<>();

   @Override
   public void notifyObservers() {
      for(Observer observer : observers) {
         observer.update();
      }
   }

   @Override
   public void registerObserver(Observer observer) {
      this.observers.add(observer);
   }
}

class ObservedClock extends Clock implements Subject {

   private Subject subjectImpl = new SubjectImpl();

   @Override
   public void tick() {
      super.tick();
      notifyObservers();
   }

   @Override
   public void notifyObservers() {
      subjectImpl.notifyObservers();
   }

   @Override
   public void registerObserver(Observer observer) {
      subjectImpl.registerObserver(observer);
   }
}

 

Now compare the above code with the implementation using a hypothetical Java which supported multiple inheritance of implementations:


class MIObservedClock extends Clock, Subject {

   @Override
   public void tick() {
      super.tick();
      notifyObservers();
   }
}

class Clock {

   public void tick() {
      // Update time each tick.
   }
}

interface Observer {
   public void update();
}

class Subject {

   List<Observer> observers = new ArrayList<>();

   public void notifyObservers() {
      for(Observer observer : observers) {
         observer.update();
      }
   }

   public void registerObserver(Observer observer) {
      this.observers.add(observer);
   }
}

 

Now… did writing that code unleash any zombie apocalypse? Which version is cleaner? Which is more elegant? Which is more decoupled? Which is ‘better’ ?

I’d be very interested to hear if anyone can demonstrate a better solution using standard Java.

I’ve heard ‘Uncle’ Bob refer to Java interfaces as ‘a hack’, made for the sake of simplicity – but not for us. For the sake of the JVM implementation. I have no way of knowing whether this is true – the road to hell is paved with good intentions, as they say; perhaps James Gosling and the gang were genuinely trying to look out for us. But what is certain in my mind is that Java is inferior for it’s lack of multiple inheritance of implementation.

Would Java have become such a successful language with multiple inheritance of implementation? Well thats a whole other can of worms.

Posted in coding, java, software, Uncategorized | Tagged , , | Leave a comment

Java is always pass by value

Ok, so this is explained ad infinitum elsewhere on the net. It is however a question asked over and over again by newbies in forums such as LinkedIn discussions & StackOverflow, etc; so with that being the case I’ll take the opportunity to provide a link to this ‘article’, thereby directing some traffic my way and avoiding the need to post a snarky lmgtfy – everyone’s a winner!

The reason Java appears at first glance to be pass by reference (at least insofar as Objects are concerned) is because when we create a new bar.Foo instance

 Foo someFoo = new bar.Foo();

and pass it into the method:

 public void handleFoo(bar.Foo theFoo) {
   //Do some stuff.
}

we don’t get a copy (i.e. the value) of the original Foo instance – we get a copy of the reference.

Hold on! the invoked handleFoo method doesn’t get the value – it get’s a reference to Foo? That’s pass by reference then, right?
No.

To elucidate the subtle difference I’ll try to remember back to my C++ programming days.
When programming using C++ we might have a reference to an object, which can be considered roughly similar to an object reference in Java (i.e. someFoo & theFoo). The value of this reference is a memory address which corresponds with the location of the object in memory – we can pass this memory address into a C++ function just as we can pass the someFoo reference into a Java method, and then from inside the invoked function/method the referenced Object can be manipulated or actioned upon.

Java:


JFoo someFoo = new bar.JFoo();
someMethod(someFoo);

...

public void someMethod(JFoo aFoo) {

//doStuff with aFoo

}

C++:


Bar bar;
someFunction(bar);
...

void someFunction(Bar &aBar) {

// do stuff with aBar

}

Here’s the key difference: With our C++ reference we’ve chosen to pass the reference itself (by using the ampersand with the function parameter). With Java we don’t have that option – we can only pass a copy of the value of the reference (a copy of the memory address – conceptually at least – which the someFoo reference ‘points’ to).
Here if we choose the first option (passing the reference itself) we are passing ‘by reference’. If we choose the second option we are making a copy of the memory address which the reference points to (i.e. the reference’s value) to be used in a duplicate reference and are therefore ‘passing by value’ a copy of that original reference’s value.

Ok. So passing by reference or by a copy of the memory address that a reference ‘points’ to is really the same thing – they’re both still references, right? When people argue that Java is pass by value they’re just being pedantic, right?
No.

I’ll use an example of cars & car keys as a kind of metaphorical explanation (or perhaps that should be an analogy; I said the word a few times in my head now and it doesn’t make any sense – like ladybird. Say ladybird 20 times and see if it still makes you think of a spotty red & black bug)

Beebop owns a Ford Escort (the Object) and as such also possesses a car key (the reference) for that Ford Escort.
Rocksteady owns a Porsche 911 (again, an object) and a key (again, a reference) for that car.

In a Java world where only ‘pass by value’ is possible, Beebop can enjoy Rocksteady’s shiny Porsche at night when it’s not being used, even though he doesn’t possess the key, if Rocksteady is kind enough to have a copy of the key made for Beebop. With a copy of the key (i.e. a copy of the ‘value’ encoded in that key) Beebop can take the 911 for a joyride when Rocksteady is asleep at night.

In a C++ world, where it is also possible to ‘pass by reference’, Beebop can be naughty. In this case Beebop sneaks into Rocksteady’s bedroom at night and ‘borrows’ the key for the Porsche, before making a copy for himself. Now there are two keys for the Porsche, just like there ended up being in the Java world; But here’s where Beebop gets nasty – he alters Rocksteady’s car key so that it no longer opens the porsche, and puts it back on Rocksteady’s bedside table. Now Beebop doesn’t even have to wait until night time to take the Porsche for a spin because Rocksteady’s key doesn’t open it anymore – the Porsche will be sat waiting on the driveway anytime Beebop feels like going for a ride. Just for fun, Beebop altered Rocksteady’s key so that it now starts the microwave when he presses the unlock button.

In the Java world Beebop could only copy the Porsche key. In the C++ world he could also change the original Porsche key. As you can see, that subtle difference when it comes to passing the actual reference, or a copy of the reference, has very important consequences.

This is why it’s important to understand the difference between passing by reference & by value. Or to make it explicit, the difference between passing by THE reference, or by a copy of the VALUE OF that reference.

Posted in coding, java, software | Tagged , , , | Leave a comment

Simple Java DSL with fluent interface

To date, I’ve not put an awful lot of time into considering when it is and isn’t appropriate to introduce a Domain Specific Language into a Java application. It’s something I’ve just started to pay some attention to, having thus far acted on gut feeling that it could be useful in some particular scenario.

Here I’ll present a very small one which I created recently to perform password validation (that is to validate the proposed password meets some minimum strength criteria/standard) in place of an existing static utility method which needed duplicating, with additional requirements, in order to validate a particular flavour of password. I’m taking some time to decide whether it’s actually an appropriate pattern of usage or not.

Here’s the client code making use of it’s fluent interface:

private static final PasswordValidator SECRET_STRENGTH_VALIDATOR = new PasswordValidator()
  .mustContain().atLeast(SECRET_WORD_MIN).and().noMoreThan(SECRET_WORD_MAX).charactersInTotal()
  .mustContain().atLeast(1).numbers()
  .mustContain().atLeast(1).ofTheseCharacters(ALPHA_CHARS)
  .mustContain().atLeast(1).ofTheseCharacters(SPECIAL_CHARACTERS_STRING)
  .mustOnlyContain().anyOfTheseCharacters(ALPHA_CHARS + SPECIAL_CHARACTERS_STRING).or().numbers().andNothingElse()
  .mustBeMixedCase();



Lets for now ignore the fact that this isn’t injected into the dependant client components – there’s only so much refactoring that one can or should attempt in one sitting.

I like this because it reads pretty much like an English sentence (not that ‘English’ or ‘French’, etc, is what the ‘Language’ stands for in DSL, as an old colleague of mine points out here – but in this particular case I think it’s a nice read-ability bonus), rather than a series of methods which must be read through in their entirety, or worse – a big old uber-method which does all of the validation in one hit, before it’s possible to know exactly what the particular validation rules are.

Of course all of that validation code still exists, but now it’s hidden away in validation rules which are constructed according to however the client code makes use of the possible combinations provided by the DSL, and executed when the validate() method is invoked on the validator. In this way many combinations of validation rules can be applied to different validator instances in a self describing way, using minimal extra code to do so.

So lets see where the complexity has been hidden:

public class PasswordValidator {

  private List rules = new ArrayList();

  public boolean validate(String passwordString) {
    char[] password = passwordString == null ? new char[0] : passwordString.toCharArray();
    boolean result = true;
    for(PasswordValidationRule rule : rules) {
      result = result && rule.execute(password);
    }
    return result;
  }

  public MustContainRule mustContain(){
    MustContainRule rule = new MustContainRule(this);
    rules.add(rule);
    return rule;
  }

  public MustOnlyContainRule mustOnlyContain() {
    MustOnlyContainRule rule = new MustOnlyContainRule(this);
    rules.add(rule);
    return rule;
  }

  public PasswordValidator mustBeMixedCase() {
    rules.add(new PasswordValidationRule() {
      @Override
      public boolean execute(char[] password) {
        String passwordString = new String(password);
        String upper = passwordString.toUpperCase(Locale.ENGLISH);
        String lower = passwordString.toLowerCase(Locale.ENGLISH);
        return (passwordString.equals(lower) | passwordString.equals(upper)) == false;
      }
    });
    return this;
  }

}



Ok. So there isn’t anything too complicated here; just a validator with 3 kinds of validation configurable – mixed case, must have something and must ONLY have something. The mixed case requirement is handled here directly as it is a complete requirement in it’s own right. The other two kinds of requirement are meaningless without some further input (what exactly is it that we must have? Must we have a certain number of that thing? etc). You can see that the rules are constructed to a PasswordValidationRule interface (which specifies the execute() method).

Here’s where those extra details are constructed (I won’t show the must ONLY have option – it’s the same but slightly more simple):

public class MustContainRule implements PasswordValidationRule {
  private String chars = null;
  private boolean numbers = false;
  private int from = 0;
  private int to = 0;
  private MustContainRuleDetailAppender detailAppender;

  MustContainRule(PasswordValidator passwordValidator) {
    this.detailAppender = new MustContainRuleDetailAppender(this, passwordValidator);
  }

  public MustContainRuleDetailAppender atLeast(int num) {
    from = num;
    return detailAppender;
  }

  public MustContainRuleDetailAppender noMoreThan(int num) {
    to = num;
    return detailAppender;
  }

  @Override
  public boolean execute(char[] password) {
    boolean isValid =
         validateParticularCharacterRequirements(password)
      && validateOverallLengthRequirements(password)
      && validateNumberRequirements(password);

    return isValid;
  }

  private boolean validateParticularCharacterRequirements(char[] password) {
    if(chars == null) {
      return true;
    }

    int count = 0;
    for (char passLetter : password) {
      count += StringUtils.countMatches(chars, String.valueOf(passLetter));
    }

    return validateCount(count);
  }

  private boolean validateOverallLengthRequirements(char[] password) {
    if(chars != null) {
      return true;
    }
    return  validateCount(password.length);
  }

  private boolean validateNumberRequirements(char[] password) {
    if(numbers == false) {
      return true;
    }
    int count = 0;
    for (char passLetter : password) {
      if(Character.isDigit(passLetter)) {
        count++;
      }
    }
    return validateCount(count);
  }

  private boolean validateCount(int num) {
    boolean pass = from > 0 ? num >= from : true;
    pass = pass && (to > 0 ? num <= to: true);
    return pass;
  }

  public class MustContainRuleDetailAppender {

    private final MustContainRule mustContainRule;
    private final PasswordValidator passwordValidator;

    private MustContainRuleDetailAppender(MustContainRule mustContainRule, PasswordValidator passwordValidator) {
      this.mustContainRule = mustContainRule;
      this.passwordValidator = passwordValidator;
    }

    public MustContainRule and() {
      return mustContainRule;
    }

    public PasswordValidator ofTheseCharacters(String str) {
      mustContainRule.chars = str;
      return passwordValidator;
    }

    public PasswordValidator charactersInTotal() {
      return passwordValidator;
    }

    public PasswordValidator numbers() {
      mustContainRule.numbers = true;
      return passwordValidator;
    }

  }

}

 

And that’s it.
Here we can see that this more complicated rule makes use of a public inner class – this is required in order to limit the client code to make only valid combinations of validation rule detail, such that the client code is only able to construct a single validation rule, in it’s entirety, at a time. This keeps the validation rule construction code simple (and therefore better tested) and the client code is forced to be written in an easy to understand way. This is achieved by controlling which methods are available depending on which object type is returned from each method, and therefore the methods which are then available in the chain; as such being forced to use the api in a specific way is not a bad thing as it actually limits the client code to invoking only a few sensible options at a time. For example, after

  .mustContain().atLeast(SOME_NUMBER)

the client code only has the option of invoking and() in order to add a numerical limit to that same validation rule currently being constructed, ofTheseCharacters() to specify the character set of which SOME_NUMBER must be used, numbers() to specify that SOME_NUMBER of numbers must be present, or finally charactersInTotal() to indicate that the there must be at least SOME_NUMBER of characters total in the password. Each of these methods, apart from and(), returns the original validator object which means that any further validation criteria must exist as another ValidationRule object. Because the MustContain and MustOnlyContain rules are so tightly coupled with their ….DetailAppender classes they have been implemented as inner classes; they’re effectively part of one larger class, but using this technique it’s possible to simulate a kind of context dependant method accessibility (unless the client code decides to deliberately break the mechanism by not chaining their invocations).

DSL_Methods


Why am I concerned about using this pattern rather than simply implementing the rules as private methods in the Utility class, or even as separate rules still, but manually added to a list and executed in sequence? To be honest the more I think about it the better I feel; in this case I feel like I can justify the marginal one off increase in complexity for the reusable simplicity and readability.

The DSL apparatus itself is really pretty simple, and as such it can easily be unit tested with confidence; and with that being the case it feels as though it is actually safer to now implement various validation routines using the DSL’s guiding hand, like this:

private static final PasswordValidator STRENGTH_VALIDATOR_1 = new PasswordValidator()
    .mustContain().atLeast(8).and().noMoreThan(20).charactersInTotal()
    .mustContain().atLeast(2).numbers()
    .mustContain().atLeast(3).ofTheseCharacters(ALPHA_CHARS)
    .mustContain().atLeast(1).ofTheseCharacters(SPECIAL_CHARACTERS)
    .mustOnlyContain().anyOfTheseCharacters(ALPHA_CHARS + SPECIAL_CHARACTERS).or().numbers().andNothingElse()
    .mustBeMixedCase();

private static final PasswordValidator STRENGTH_VALIDATOR_2 = new PasswordValidator()
    .mustContain().atLeast(8).and().noMoreThan(20).charactersInTotal()
    .mustBeMixedCase();

private static final PasswordValidator STRENGTH_VALIDATOR_3 = new PasswordValidator()
    .mustContain().atLeast(10).charactersInTotal()
    .mustContain().noMoreThan(3).numbers()
    .mustOnlyContain().anyOfTheseCharacters(ALPHA_CHARS).or().numbers().andNothingElse();



…than it would be to code them up in the typical, and less structured way. It makes it easier to compare the difference between various validation routines, avoid bugs or assumed behaviour which doesn’t actually exist (which I discovered in the old code during this process, although decent tests should have highlighted that in the first place), and helps to keep that utility class closer to 1 thousand lines than 2 thousand o.O

I’d be interested to hear any criticisms or pointers. Since doing this I’ve been able to cut/paste the code which builds up the validator into emails to send to other non/less-technical people that have needed some information regarding what password-type validation is performed in various places, and it’s worked pretty well. Although it’s a bit of a leap to label this code an internal DSL, it has been easy to share with those less technical people (and is extra convenient for the techies too) because like something such as SQL it has an enforced structure, uses a limited syntax, and is relatively expressive in a self-describing way.

I’m going to go on a hunt now for other scenarios where this kind of mini DSL approach also adds value from an internal perspective, and I’ll add to this if I come up with anything compelling.

Image | Posted on by | Tagged , , | Leave a comment

Boy-scout refactoring

I recently had a puzzling discussion with another  developer regarding refactoring; more specifically the use of private methods – he didn’t like them, favouring giant public methods which presumably require a whole load of inline comments to explain what it’s doing at various (probably deeply nested) stages.

Anyway, I thought I’d post a trivial example of method refactoring as these kinds of discussions tend to pop up occasionally, and this way I don’t have to repeat myself. Moreover the all-seeing eye of Google might benefit a novice to the craft looking for some guidance.

The following before & after snippets are production code (albeit slightly disguised) whereby a user is attempting to set/update a secret passcode which is used elsewhere in a hashing algorithm in order to verify the authenticity of any transmitted messages for that user. Ultimately the routine invokes the static PasswordUtil.validatePassword() (which validates that the new password supplied meets the minimum strength requirements – e.g. special characters, mixed case, etc) and returns the result as well as doing any cleanup required. My task wasn’t to alter this code, but to modify the implementation of PasswordUtil.validatePassword() which this method invokes.

Here’s the original method:

protected boolean validate() {
  boolean ok = true;

  if (JaasUserUtil.hasPermission("Use New Profile Pages")
      && JaasUserUtil.hasPermission("Change redirect SECRET")) {

    EnvironmentPropertiesDataBean data = (EnvironmentPropertiesDataBean)retrieveBean();

    // verify if a newly entered SECRET is secure enough
    if (!data.getDisableRedirectSecret() && data.getRedirectSecret()!=null) {
      ok = data.getRedirectSecret().length() <= 20
          && PasswordUtil.validatePassword(data.getRedirectSecret());
      if (!ok) {
        data.addFeedbackLine(new FeedbackItem(FeedbackItem.Type.WARNING,
        new BundleKey("account_profile.environment.account_secret_invalid",
        new String[]{data.getAccountCode()})));
        // SECRET value entered by the user is invalid -- now, if the SECRET *WAS* empty (disabled), then make sure
        // that it remains that way. Otherwise, the user would be told 'SECRET is set' while that's not actually true!
        String originalValue = AccountCache.get().getAccount(data.getAccountKey()).getRedirectSecret();
        if (originalValue==null) {      // It was disabled;
          data.setDisableaRedirectSecret(true);    // so, make sure it remains disabled.
        }
        data.setResult(FAILURE);
      }
    }
  }
  return ok;
}

Here you can see that the method requires some study in order to determine exactly what it does, and I think genuinely requires the provided inline comments in order to elucidate that behaviour. To start with the very first comment on line 9 would be entirely redundant if only the method had a self-describing name (such as ‘validatePasswordStrength’ rather than plain old ‘validatePassword’). In this particular instance I didn’t change the method name in order to minimise my refactoring footprint, but you will see in the refactored example that the static method on PasswordUtil now has a different name which makes it’s purpose clearer, and in turn the purpose of it’s invoking code is clearer.

You should also see that the latter comments are definitely necessary in this snippet if one wants to get a quick overview of the functionality without having to think about the code. The problem with code comments is that there is no mechanism to force them to stay up to date (i.e. we often find comments in code which say that it does something entirely different that it in actual fact does), and apart from looking untidy a reasonable assumption to make is that the code which they describe is almost certainly more complex than it needs to be (otherwise why write an explanation of what it does?). So comments in code tend to indicate a ‘code smell’ – they hint that something probably isn’t as good as it could be, or typically, much worse.

Below is a refactored version of the same method. It’s not perfect, and it’s not meant to be – it is however significantly better than it was when I found it. The purpose of taking a boy-scout approach is not to make the codebase perfect (perfection costs more money than it saves), but to encourage it’s maintainability.

protected boolean validate() {
  EnvironmentPropertiesDataBean data = (EnvironmentPropertiesDataBean) retrieveBean();
  return validationNotApplicable(data) || validateSecretStrength(data);
}

private boolean validationNotApplicable(EnvironmentPropertiesDataBean data ) {
return (data.getRedirectSecret() == null || !userHasPermissionToChangeSecret() || data.getDisableRedirectSecret());
}

private boolean userHasPermissionToChangeSecret() {
  return JaasUserUtil.hasPermission("Use New Profile Pages") && JaasUserUtil.hasPermission("Change redirect SECRET");
}

private boolean validateSecretStrength(EnvironmentPropertiesDataBean data) {
  boolean result = PasswordUtil.validateSecretStrength(data.getRedirectSecret());
  if(!result) {
    handleInvalidSecret(data);
  }
  return result;
}

private void handleInvalidSecret(EnvironmentPropertiesDataBean data) {
  BundleKey bundleKey =  new BundleKey("account_profile.environment.account_secret_invalid", new String[]{data.getAccountCode()});
  data.addFeedbackLine(new FeedbackItem(FeedbackItem.Type.WARNING, bundleKey));
  resetDisabledRedirectSecretToOriginalState(data);
  data.setResult(FAILURE);
}

private void resetDisabledRedirectSecretToOriginalState(EnvironmentPropertiesDataBean data) {
  String originalValue = AccountCache.get().getAccount(data.getAccountKey()).getRedirectSecret();
  if (originalValue == null) {
    data.setDisableRedirectSecret(true);
  }
}

Ok… Hopefully the ‘after’ version of the snippet should be much easier to read. The idea is that it reads more like plain English – like a story. We’ve minimised having to remember that we’re already nested 3 conditional statements deep – we’ve separated all of the different responsibilities out into single private methods which handle only one or two responsibilities each. We’ve removed the need to assign and later refer to boolean flags. The descriptive method names mean that comments are no longer necessary, and that we should be able to read the code from top to bottom, left to right, and understand the sequence of checks and tasks with the minimum of effort – the new methods are written in the order that they are invoked, and contextual highlighting in modern IDEs make following them even easier.

As a result of this refactoring, which didn’t involve altering any program logic, I am now able to understand clearly the logic contained within this method which means that I can be more certain that I’m doing the correct integration level testing, as well as the unit tests which will run for my altered code which this unmodified logic invokes. In fact during the same piece of work, elsewhere, following this process identified a few bugs, and a swathe of dead code which was otherwise hidden under unnecessary complexity.

As I’ve hopefully demonstrated, proactively refactoring as we interact with existing code not only prevents the code-rot from setting in, but also can help us to do a better job in the moment; it’s not just a good idea – it’s your responsibility as a professional engineer (as I’m sure you’ve heard before).

I’m still dealing, seven years later,  with the aftermath of my house’s previous owner’s DIY electrics. The difference between someone who can do the work, and a professional who can do a proper job really shows in both contexts.

Posted in coding, java, software | Leave a comment

Java 6 IKM test

I had to do a Java 6 IKM test the other day. I thought i’d post the result sheet up because it has lots of green bars :D

Screen Shot 2014-02-04 at 22.27.40

I’m led to believe that the percentage score (99) is not particularly useful, but rather the percentile score (97 on the right) is what’s looked at. I don’t know how good scoring in the 97th percentile is, as that’s not the 97th percentile of competent test takers, but (presumably) 97 percentile of ALL test takers, including the clueless ones.

It was enough, however, to get me an interview with a large investment bank in the city when I had no prior experience. According to the agent I “smashed it” :D but my man on the inside had informed me already that 97 (i think) was the score that I would need.

Anyway, the purpose of the post is to talk a little about the test and how it works:

Firstly – it’s an adaptive test which means that the difficulty varies dynamically according to how well you answer. I did one previously with very little preparation and scored in the 95th percentile, and it was very noticeable how the test is reactive doing it for the second time; I’d prepared properly this time and as a consequence the questions became quite a lot more obscure.

As a consequence of the test being adaptive there is another important thing to remember, which is potentially very off-putting if you weren’t expecting it – you cannot skip questions and come back to them later. This is a big deal to me, because now one has to decide on the fly how much time to give to a particular question should it be a tricky one; because the latter question difficulty depends on how you answer the earlier ones, you have to answer it right now or not at all!

Which leads into the third significant point. There can be up to four (if I remember correctly) correct answers out of 5-ish options, and the points you get for your answer will depend on how close your chosen answer/s are to the correct answer. This means that, for example, you may score partial credit for selecting a reasonable, even though incorrect answer. In the same way you will score negatively for selecting an entirely nonsense answer. I like this mechanism as it accounts for random guesses, while rewarding partial understanding with partial credit.

Timings… there was a time limit of about 2 hours. You can see from the results report above that the test taker is marked on their speed & accuracy combined. I have no idea how much the time taken affects the score as the first time around I used all of the time available, while the second time around I had a lot of time left over AND the questions were harder (because I was doing better), yet the score differed by only 2 percentage points. If we imagine a bell curve of results then it makes sense that the scoring differences at the higher ends especially will correspond with performance in a non-linear fashion. I would *guess* though, as an experienced professional, that speed is less important than accuracy.

What should you study to prepare? Look at the scoring sheet above. It’s worth digesting the lava.lang package especially I think as you can easily spend a lot of time figuring out how some class that you haven’t used in 10 years is supposed to work. If you haven’t done much multithreading before then definitely study that.

Apart from that there isn’t much advise I can give other than to understand that this exam is HARD. Much harder than the SCJP, or whatever Oracle is calling it these days.

Posted in coding, java, software, Uncategorized | Leave a comment