Qualification Based Software Procurement

A recent article on the New Zealand Institute of IT Professionals Techblog describes a different approach to software procurement that has been adopted by the Ministry of Business, Innovation and Employment (MBIE) and the NZ Police, for the development of an Emergency Response System.  This approach is a variant of Qualifications-based Selection (QBS - https://en.wikipedia.org/wiki/Qualifications-Based_Selection).  In QBS price is specifically excluded as a consideration during the 'request for proposal' phase of a procurement initiative, which instead focuses on identifying the best collaborative, solution partner and the most creative solution.  This apparent shift in the approach to software procurement by a major NZ government department, if perceived as successful, could be followed by other NZ government departments. If this is the case it would have significant and wide ranging implications for suppliers, and hopefully the country as a whole if it were to result in more successful, better value digital systems, avoiding some of the debacles that have occurred over the years (e.g.. INCIS, Novapay).

The QBS process looks at qualifications and experience of tendering suppliers, then works closely with a short list to identify the firm that exhibits the most creativity, innovation and deep thinking about the entire lifecycle of a system.  Only after that firm has been nominally selected is price negotiated.  If there is a failure to reach agreement the purchaser can fall back to the next best firm.

Although initially it might appear that the QBS process costs more, purchasers actually end up paying less in the end.  When a priced based selection process is operated, suppliers are motivated to be creative in pairing down design and construction costs to the absolute minimum in order to be perceived to tick all of the requested delivery boxes. There is little incentive to be concerned with post project operation and maintenance costs of a system or product.  That can lead to very much greater costs over all.

QBS is being used increasingly for complex Engineering projects in different parts of the world, including for building architecture, roading and other civil construction projects, particularly for 'intellectual' roles and phases of projects where design competency and innovative ability are critical.  In software engineering projects, all phases demand significant design and intellectual capability down to the lowest level of code development and testing. QBS means that:

  • Software development companies can hire the best and brightest minds rather than the least costly resources.
  • Software designers can focus on the most innovative, robust, and in the end most cost efficient design solution rather than lowest price. 

Although QBS is touted as something relatively 'new', I have over the years helped organisations to progress a similar process for software procurement, and can attest that in my own experiences at least, project success rates have been very much higher and overall costs lower than when the procurement process has focused on delivery price.  My observation has also been that quite often, newer, smaller companies have been able to deliver a better, more innovative, cheaper system than larger, 'brand name' type organisations.  An implication is that if the aim is to obtain the most creative, best value system, 'qualification' in QBS needs to be something other than just long-standing reputation.

From a practical, procurement process perspective this can be addressed by:

  • Looking carefully at 'potential to deliver' not just 'has delivered', based on similar classes of applications, possibly in different industries, rather than necessarily the exact same application, particularly as similar solution patterns are often applicable across industries and contexts.
  • Looking to contrast and compare a range of supplier types, large, small, more established, newer, whose strengths and weaknesses are assessed during later intensive, joint, conceptual design workshops.

An approach that I've found works quite well is to take a view on the optimum shortlist mix then look to qualify firms accordingly, the aim being to ensure that there is if possible variety in contender types.  For example if the entire short list comprises relatively large companies with many years of experience delivering relevant solutions, it is possible to miss the kind of innovative ideas and approaches that often come from smaller, newer firms.  Part of the QBS approach should be to ensure that an optimum solution is pursued rather than just a firm that can deliver a solution.

Intensive, joint, conceptual design workshops are where proposed design staff from short listed suppliers get together with the purchaser's assessment team to jointly develop a high level conceptual design of the target system.  In these workshops the purchaser focuses largely on describing the business and technical requirements, whilst suppliers focus on presenting enabling, high level design options.  Workshop outcomes include:

  • A clear view on whether potential suppliers have grasped the key implications of the business objectives.
  • An assessment of whether suppliers have been able to translate these into innovative, effective, economic design options.
  • Whether design options have accounted for the entire lifespan of the system, particularly future enhancements.
  • An understanding of the key risk areas and system dependencies, potential candidates for proof of concept.
  • An assessment of suppliers' software engineering structure and processes.
  • A view on the cultural fit of the suppliers' proposed staff.
  • An agreed, overall design approach that will be carried forward.

Such workshops can last hours to days, depending on the size and complexity of the target requirements.

This process is much more demanding of the purchaser than predominantly price based selection, requiring an in-depth understanding of:

  • Good software engineering practice, together with an understanding of related processes, organisational culture, and also a solid appreciation of the potential impact of individual competencies.
  • Design innovation, e.g.. key characteristics of practical, creative software design, such as simplicity and reusability.
  • Factors underlying project success and failure.

To achieve a successful procurement outcome, ironically perhaps, purchasers need to set to one side any notion of a firms obvious qualifications and of course 'brand name', 'prestige marketing' and persuasive sales pitches.  The selection team should embody a high level of general software engineering and related management competence that at least matches if not exceeds that of potential suppliers, although not necessarily in the same specific areas. The team should embark on joint, conceptual design workshops with a 'strawman' design. The process of developing a strawman is essential to gaining insight into potentially difficult areas. Consequently the team will be well prepared to ask relevant, searching questions and immediately understand the implications of design options being proposed by shortlisted suppliers.

A QBS process can work equally well for large or smaller procurement projects, with only the timescales being adjusted. For example for a very small system, joint conceptual design workshops may be limited to an hour each, whereas for a very large system they may take multiple days.

The MBIE/NZ Police, Emergency Response System RFP, indicates that each short listed supplier (at least two) will need to develop a competing, 'proof of concept' prototype, for which they will receive a grant to cover costs.  I am not entirely convinced that a prototype of itself is necessarily an optimum method of selection. It is certainly an expensive one (grants are around $75,000).  It really depends on how it is progressed, whether for example prototypes will be allowed to focus on relatively superficial user interface aspects, or will be required to address more critical, under the hood engineering aspects, providing confidence that the more creative, riskier aspects of each design vision can be achieved.

The point is it shouldn't be necessary to build anything in order to figure out whether the ship will float or the bridge will stay up, unless the construction hinges on new and possibly risky innovations, in which case it those that should be proven up-front. So hopefully that will be the focus in this case.  In any case, prototype development will certainly enable the purchaser and would-be suppliers to experience working together over an extensive period, in similar conditions to the project proper, which will no doubt provide an invaluable opportunity to gauge working compatibility, supplier methods and approach in reality, not just on paper.

My own experiences have indicated that intensive, joint, 'conceptual design' workshops can be a very adequate method for comparing innovative proficiency and 'cultural fit' of short listed suppliers. 

So, hopefully this QBS 'experiment' will prove successful, becoming the norm for NZ government and beyond, with price based selection a thing of the past.  It does however require a very different cultural approach (actually on both sides) and I wonder who is leading that in this case.