All material on this web site is copyrighted by Honourcode, Inc. 1999-2007.


Systems Engineering Return on Investment (SE-ROI)

A Practical Research Project

  • Quantify the relationship between SE practices and project success.
  • Discover leading indicators of a project’s future success and risks based on SE practices used.
  • Identify good SE practices and tools based on their contribution to success

Systems engineering is widely viewed as a discipline that has high value. Yet there is a uniform lack of proof. This multi-year project is proving the relationships. This web page is used to report public information as it becomes available.

Your company or organization can contribute and gain the benefits before they are public! A project advisory group includes individuals from:

The Aerospace Corporation
AFIT
Alchemists Int'l
Applied Physics Lab
ARDEC
ASC Shipbuilding
BAE Systems
Boeing
Canadian Space Agency
CEDISC
Codarra Adv Systems
Corning
Defense Acquisition University
DRS
George Mason University

 Harris
IBM
Israel Aircraft Industries
Johns Hopkins University
L-3 Communications
Loughborough University
Loyola Marymount University
Massachusetts Institute of Technology
The Mitre Corporation
NAVAIR
US Navy Chief Engineer
New York City Transit
Nokia
Northrop Grumman
US Office of the Secretary of Defense (AT&L)

Parsons
QinetiQ
Rafael
Rand Corporation
Raytheon
Saab Sytems
SAIC
Swedish Defence Material Administration
Sysnovation
Systems and Software Consortium
Technion University
University of Alabama Huntsville
University of Minnesota
University of South Australia
Washington Savannah River Co.

Contact Eric Honour for information at ehonour@hcode.com or +1 (850) 479-1985.

What Did We Know Before?

Results from the preliminary Value of Systems Engineering work:

  • Better technical leadership correlates to program success. [Ancona 1990, Miller 2000]
  • Better/more systems engineering correlates to shorter schedules by 40% or more, even in the face of greater complexity. [Franz 1995, Honour 2004]
  • Better/more systems engineering correlates to lower development costs, by 30% or more. [Gruhl 1992, Barker 2003, Kludze 2004, Honour 2004]
  • Optimum level of systems engineering is about 15% of a total development program. [Gruhl 1992, Honour 2004]
  • Programs typically operate at about 6% systems engineering. [Kludze 2004, Honour 2004]
  • Parametric cost estimation of systems engineering is possible. [Valerdi 2004]
  • SE practices correlate to program success. [Gamgee 2006]

Download data from the prior work:

Ontology of Systems Engineering

A review of the current systems engineering standards reveals a categorization of activities that appears to represent the industry ontology ("shared conceptualization"). The table below shows seven categories of activities. These are the basis for the SE-ROI project data to be gathered.

Download a usable version of this table

Approach

The project is now doing interviews to obtain data from in-work and recently completed projects. The data required includes:

  • Project characterization data such as project size, project type, development phases, bounding parameters, risk levels.
  • Project success data such as cost/schedule compliance and technical quality measures.
  • Systems engineering data such as hours expended on systems engineering tasks, quality of those tasks, specific nature of the methods and tools used

The only effective method to obtain this data is through an interview process with the key individuals. Project databases store some equivalent data, but the data is usually organized in unusable structures and always misses key information. Interpretation of the data is needed to convert it into a common structure.

Your company can receive the benefits of early data by providing access to projects. Regular reports to participating companies will give you the knowledge edge.

Data access is protected by confidentiality agreements and by separation of the data from the project identifications. Only aggregated data is shared with others.

Where Are We Now?

Initial work has already:

  • Defined primary scientific hypotheses.
  • Surveyed the current SE standards.
  • Created an ontological classification based on the common data in the standards
  • Defined secondary scientific hypotheses.
  • Created first and second drafts of interview data sheets, and reviewed them widely with many people.
  • Run pilot interviews to prove the data sheets.
  • Settled on the final form of the interview data sheets.
  • Started interviews with real programs.

Further work continues to

  • Gather data in full interviews over several years
  • Provide benchmarking results to contributing companies
  • Publish public information as it becomes available

Interim Results

The following reports provide interim results from the project..

Paper Reference Summary
Honour, E, "Gathering Data to Measure Systems Engineering Return on Investment (SE-ROI)," Proceedings of the Systems Engineering Test & Evaluation Conference, Melbourne, VIC, 2006 [.pdf 79KB] Describes the interview data gathering methods to be used on the SEROI project.
Honour, E, "A Practical Program of Research to Measure Systems Engineering Return on Investment (SE-ROI)," Proceedings of the INCOSE International Symposium, Orlando, FL, 2006 [.pdf 90KB] Describes the SEROI project plan to provide public information and gather public interest, thereby enhancing the likelihood of obtaining access to programs.
Honour, E, & Valerdi, R “Advancing an Ontology for Systems Engineering to Allow Consistent Measurement,” Proceedings of the Conference on Systems Engineering Research, Los Angeles, CA, 2006 [.pdf 159KB] Continues and revises the categorization of systems engineering efforts to include additional standards and definitions of key terms such as "systems engineering effort," "amount of effort," "type of effort," "quality," "success," and "optimum."
Honour, E, "Design of Experiments as Applied to Systems Engineering Return on Investment," Proceedings of the Conference on Systems Engineering Research, Hoboken, NJ 2007 [.pdf 93KB] Describes the statistical basis for the SEROI project, using the Design of Experiments method to separate the effects of many parameters while combining the data collection.

Support

SE-ROI is primarily funded by Honourcode, Inc.and the Systems Engineering and Evaluation Centre (SEEC) of the University of South Australia. Further funding will be needed. If your organization is interested in guiding this research and obtaining early data, contact Eric Honour for information at ehonour@hcode.com or +1 (850) 479-1985

Page last modified: 11 Apr 07