COTS procurement, 20 years after the Perry Memo

Story

January 15, 2015

COTS CONFIDENTIAL. Every month the McHale Report will host an online roundtable with experts from the defense electronics industry ? from major prime contractors to defense component suppliers. Each roundtable will explore topics important to the military embedded electronics market. This month we discuss the effect of the memo former U.S. Secretary of Defense William Perry issued in 1994 essentially directing all DoD to use commercial off-the-shelf (COTS) products wherever and whenever possible.

Today’s panelists are: Rob Smith, VP of C4ISR for Lockheed Martin’s Information Systems & Global Solutions business area; Rodger Hosking, VP and Co-Founder of Pentek; David Jedynak, CTO, Curtiss-Wright Defense Solutions; and Doug Patterson, VP of Military Aerospace sector for Aitech Defense Systems.

MCHALE REPORT: It’s been just over 20 years since the famous – some say infamous – memo from Defense Secretary Bill Perry directed the DoD to use COTS technology wherever and whenever possible. Looking back, what impact did that have on the defense electronics industry?

SMITH: I think that there used to be many more government-focused foundries that produced defense electronics. As we shifted more toward COTS they became a lot less relevant. It is the right thing to do to challenge industry to use COTS wherever they can. However, we have to think about ensuring the COTS supply chain is trusted and can deliver robust military systems. You also can’t create the system you need solely with COTS. We still have to design hardware and software that is unique for government and DoD missions.

HOSKING: It has boosted small company participation in defense programs, saved the government money, promoted open standards, and allowed easier insertion of new technology.

JEDYNAK: Overall, the impact has been very positive. It’s allowed the defense industry to leverage the quality, rapid pace, and innovation of adjacent technology markets (e.g. consumer, telecom, industrial, medical, automotive). At its core, electronics acquisition is a make vs. buy decision, and the Perry Memo shifted the real burden of proof on the make decision, which is good. Because COTS technology is typically derived from adjacent markets, it comes with entire supporting ecosystems – including tools, people, and experience – which could never survive, much less thrive, while enduring the ups and downs of the defense market.

On the other hand, the transition to COTS is not complete – we’re not quite there yet. There continues to be a big emphasis on older acquisition models, older verification and validation methodologies, and a lot of waterfall design. While COTS technologies grow and are refined in an agile world, the defense industry is still very focused on long-term waterfall development models that strike and adhere to technical design baselines. That makes technology refresh difficult. It’s a culture issue, but we’re trying to get there, and the good news is that we are starting to see the change occur.

PATTERSON: The worldwide, near universal, adoption of COTS has fundamentally changed the landscape of embedded computing for defense and aerospace. Electronics-based systems and their host programs are being fielded in a handful of years, not a decade as it had in the past. This saves everyone – including the taxpayer – literally millions of dollars and thousands of hours in development costs, allowing technologically-advanced, sophisticated systems to be deployed much sooner.

Standardization, and the ability to mix and match multiple vendor’s products within these systems, brought with it open competition and new companies. Innovative products sprouted up, filling the voids where nothing like this had ever existed before. This further fueled the spirit of innovation and a sense of urgency, coupled with an extended level of patriotism not widely seen in the “old-crows” somewhat staid and stodgy defense electronics industry.

MCHALE REPORT: Back then many scoffed at the commercial part of the acronym, claiming it justified the use of “radio shack technology,” and many definitions of what COTS means sprouted up over the last two decades. How do you define the role of COTS in defense procurement today?

PATTERSON: COTS is mainstream, prime time – the mocking has ceased almost completely. The top 10 defense and aerospace primes now use the word COTS to freely describe their own products and services in trade publications and it’s literally up in lights and banners at trade shows and industry conferences. COTS is the mantra, the words ‘custom’ and ‘customized’ being eschewed and rarely referenced in today’s defense and aerospace electronics marketing campaigns. For several years now, these top 10 primes have been sending their engineers and management personnel – sometimes even en masse – to the standards organizations’ meetings to help develop (and influence) the standards as they are generated by the embedded computers vendors.

SMITH: I will just say we are integrating as much COTS as we can. In a lot of ways the COTS movement is excellent as it allows us to leverage commercial technology. Working on commercial production scales is also an advantage as the cost per unit goes down tremendously when production lines pump out thousands and thousands of units. It comes down to enhanced capability at a lower cost in many cases.

Clearly transitions are hard to do and when you are pushed hard there are always growing pains but I’m pretty pleased with where Lockheed Martin and the industry as a whole are when it comes to COTS procurement today.

HOSKING: COTS vendors who understand and consistently meet the operational and support requirements of defense programs earn recognition by the primes, integrators, and end customers as worthy suppliers and partners. The best of these vendors offer the latest technology, competitive pricing, software tools, and applications assistance to help their customers win programs, integrate systems meeting specification, and then deliver them on time.

JEDYNAK: COTS isn’t consumer-off-the-shelf; it’s commercially developed. The two are very different things. The intention was never to go down to the local consumer electronics store to buy a computer or radio for a military vehicle. The intention is to utilize commercially developed (as opposed to government developed) technologies, such as processors, network interfaces, memories, and other support circuitry. COTS means that vendors choose the appropriate parts (e.g. industrial temperature), out of a massive market of commercially developed parts, to create industry specific designs. Defense isn’t the only industry that takes this approach. The same processors, memory, power management chips, and other various elements are used throughout many industries, including medical, industrial, and automotive – as well as defense.

The challenge is to make sure that our industry stays focused on the intent of COTS in the Perry Memo. For example, don’t spend the time and money to develop the same technologies that have already been developed for commercial (non-government) reasons. Really, the Perry Memo is an outsourcing directive. The DoD and primes should stay focused on the front-office work of designing and integrating defense systems, while reaping the advantages of outsourcing the back-office work of designing memory chips, processors, logic chips, and hardware/software tools.

MCHALE REPORT: Are prime contractors/system integrators leveraging more COTS technology today due to the budget-constrained environment or is COTS use more a result of a push toward commonality within the defense industry?

JEDYNAK: In short, yes. Commonality is one of the holy grails in the defense market and getting there is a difficult challenge. The issue of commonality is really one of scope. [So] what are we referring to when we talk about commonality? Chips? Boards? Common to the National Stock Number (NSN)? Ultimately, NSN commonality is the goal, since that’s how the government procures in the logistics chain, but the issue is how to scope that. If we are talking about a box that is specific to a particular vehicle, then commonality is hard. If the boxes may be opened, and the boards managed separately for a more modular view, then commonality moves into the realm of the possible. While a vehicle-specific chassis and backplane might not constitute commonality, the standard Line Replaceable Modules (LRMs) within the box do. The counter-argument is that that the architecture forces additional cost over a monolithic packaged design, which is a valid and fair assessment.

There’s a breakpoint on commonality and cost. Use of common parts results in better purchasing volumes, which then drives lower recurring costs. It also delivers various training and supply chain benefits that also drive lower non-recurring and recurring costs. What’s really behind the drive within the defense industry for using COTS is cost. Commonality is viewed as a way to get there.

PATTERSON: The real power of COTS remains as true today as it was in 1985, when a couple companies introduced the concept, as Wayne Fisher of Force Computers coined the COTS phrase. COTS saves the system integrators time, resources, and equally important (to the accountants and their investors), money. Today, the end users (i.e. military services) see hardware deployed in 1/4 to 1/3 the time, as compared to the past. Those primes who were the early adopters of COTS were awarded more defense programs due to faster delivery and lower costs.

In some cases, primes were able to deliver working prototypes or even pre-production hardware with their proposals. Those companies who resisted the COTS wave were slowly eroded away like the sands on a windswept beach.

SMITH: Commonality, much like anything else, comes down to mission requirements. Missions that encompass multiple agencies and supporting departments will require more than others. What the budget-constrained environment has encouraged is increased use of open architectures which add more design agility to upgrades and enable more COTS use as obsolete components can more easily be swapped out.

HOSKING: The most important factors to prime contractors/system integrators are cost, performance, delivery, support, and risk reduction.

MCHALE REPORT: Regardless of your place in the supply chain, obsolescence has been the dark side of COTS use. Has the defense industry gotten better at managing this or are they still hopelessly at the mercy of commercial life cycles?

HOSKING: Responsible COTS vendors offer their customers life cycle management programs to mitigate this growing problem. We can secure the required quantity of end-of-life components to sustain production for program lifetime requirements. Of course, continual monitoring for additional components becoming obsolete is essential.

JEDYNAK: It’s a fallacy to say that obsolescence issues are limited to COTS. In a 100 percent government-focused and government-driven supply chain, production of parts is highly inefficient and costly. Consider running off 1,000 chips for a 10-15 year system procurement: that means that the development, tooling, training, process, and storage of those parts is all undertaken for a single program. If any requirements are changed, the entirety of those parts may potentially become worthless. Preventing commercial obsolescence issues is very expensive and obsolescence, like commonality, is really about cost.

The real issue with obsolescence and COTS is the lack of focus on technology refresh cycles. Rather than striving to understand the agile COTS market and the tempo of technology refresh, defense acquisition programs still try to drive the early leading-edge of a technology during the technology development phase (to be as far from obsolescence as possible). Next, a baseline is usually struck on that technology generation. In the “bathtub curve” of cost and maturity this baseline is located at the high cost/immature edge. By the time a program is in production that baseline technology has already become obsolete, and now sits on the high cost/obsolete edge of the bathtub. This approach is incredibly costly, and results in little obsolescence management benefit over a purely government-focused and government-driven supply chain. Instead, the focus should be on understanding how an agile technology culture (where we are always designing to produce to the low cost/mature center of the bathtub) can enable us to keep pace with technology cycles. Then we can see that obsolescence isn’t the dark side of COTS use, it’s really the benefit.

PATTERSON: Component obsolescence as a result of COTS usage is a common misconception and misnomer. Regardless of whether or not a company designed their systems themselves from the ground up, or utilize COTS products, they all use the same memory devices, processors and FPGA chips, passive component resistors, inductors and capacitors as the COTS suppliers. “Parts is parts” as they say, it matters not if it’s COTS or proprietary designs. The shift in the semiconductor industry to determine the viability of a product line by the projected yields and ROI is a flawed concept. If a critical component in an ESM system isn’t selling millions of devices a month into other commercial or industrial markets, it’s obsoleted without a thought as to the impact that decision has on our troops.

Yet the troops are the people who are constantly in harm’s way, providing those very companies with the protection they need to even exist, let alone grow. The greater good of providing our servicemen and women with the latest and best technology – and to be able to maintain it throughout that system’s lifecycle – is not part of a balance sheet. The concept of nationalism or patriotism never enters the semiconductor board rooms, where business school grads and accountants only look for gain to provide their investors the dividends they believe their stocks must return to remain “economically viable.”

SMITH: We always think about obsolescence when using COTS products. Today the defense industry has a lot more experience integrating new products when previous COTS devices go obsolete. This is accomplished through enhanced relationships with COTS vendors, which enables us to be thoughtful with what we expect to be future obsolescence challenges.

MCHALE REPORT: Going forward the DoD is likely to want to share more R & D costs with industry, which many feel means more COTS procurement. Agree or disagree with that statement? Please share why.

SMITH: I think the DoD will continue to share R & D costs with industry, but I’m not convinced that will be analogous to increased COTS procurement. If something developed in the commercial world meets our government customer needs, we’ll leverage it.

However, it is important to note that our R & D is mission-focused, and we will use whatever technologies or products are necessary to meet the mission needs of our customers. Many times mission requirements cannot be satisfied by COTS and require specialization that is beyond what COTS solutions can deliver.

HOSKING: This is a smart move, and it would mean more COTS procurement. We would like to see DoD engineers becoming more directly involved in system design and product selection, reducing the tendency to out-source those decisions.

JEDYNAK: That is absolutely true. If you consider Peter Drucker’s “front-office/back-office” model, the DoD’s front office is increasingly centered around bleeding-edge technology, sophisticated software applications, and unique operation challenges. At the same time, doctrine and tactics are continually evolving. Money, time, and sheer brainpower need to be spent at the front office to maintain superiority and capability over-match which necessitates more COTS procurement. The DoD absolutely needs to stop expending energy and resources in their front-office activities on those solutions that are already been addressed in the COTS market, e.g., rugged computers, displays, network, storage, operating systems, middleware, etc. It’s more important for the DoD to focus on innovation for the required capability sets, the user community, and the actual end-applications. Conversely, on the COTS side of the equation, the defense supplier industry needs to increase innovation in its own front office, bringing continual reduction in SWaP-C, higher security, and those industry-focused software toolsets that enable the DoD customer to innovate. Sharing R & D is really no different than the procurement process. Ultimately, the DoD relies on a healthy and stable industrial-base. Greater involvement in the investment must come with additional opportunities, and those opportunities will ensure health and stability.

PATTERSON: Agree, and the reason is self-evident as I shared above. COTS reduces development, deployment, and overall program life cycle costs – and it’s always about the money in the end. That which maximizes the ROI gets the lion’s share of the business.