The relationship between software stability and version is tenuous at best. Yet we software designers and engineers --people who should know better-- have accepted software versioning as the primary indicator and feedback mechanism about software's stability since people first started writing it.

Software designers know, as a matter of training, that it is unacceptable to assign multiple meanings to a single variable, no matter how close in concept those meanings may be. Yet, we happily go on overloading our software version numbers to also connote the stability of the underlying package. Again, because there is some --not quite quantifiable-- relationship there, and, well, that's the way its always been done.

We've also always known at some level that there was something wrong, somewhere, with the whole notion. We've filled many tombs with attempts to understand and document the illusive relationship between a software project's longevity and the resulting software's quality. We've approached it with mathematical precision, religious zeal, and edgy rebellion, but have we ever really been able to square it?

This paper proposes completely separating the concepts of software stability (or "quality") from version maturity.

There are obvious advantages to separating software version numbers from the software's stability rating:

1. A credible trailing indicator of software's real, experienced, stability is required before progress on good predictive indicators can be made. - Attempting to create acceptable leading indicators --such as development practices and methodologies used-- without first developing a consistent set of trailing indicators of experienced stability, is like trying to take world-class photographs from inside an isolation tank. Widely accepted trailing indicators are needed first, in order to validate (or invalidate) proposed methods for producing more stable software.

2. A non-ambiguous indication of a given piece of software's quality (stability) can be made available to the people responsible for using, supporting, upgrading, insuring, and valuing it (within any context).

3. With two separate values to denote version and stability, the subtle relationship between these two concepts can be better studied and understood simply by analyzing historical records of both data sets.

   At the same time, the relationship will become less important, because a value based on something more germane than a software product's maturity can be used to estimate how likely the product is to work or fail within a given context or system.

The overriding philosophy of this project is simplicity. If we have to tell people to defocus their eyes and concentrate on a point somewhere behind the page (or some similar-sounding instructions) in order to understand this system then we have failed.

It is important to the usefulness of this rating system that educated people in any field be able to grasp the connection between the rating value and the likelihood that the rated product will work or fail within their given situation. It must also be explainable in such a way that, if they wish, people can grasp the reasons and mechanisms that the system claims will insure it reflects and predicts real-world probabilities.

Design Goals:

1. The rating system shall be simple, unambiguous, and concise without being too simple to be useful
2. The rating system shall be based on a small, fixed set of self-evident criteria that all who use it can comprehend and validate to their own satisfaction.
3. The rating system shall be independent of the code, the distribution files, or the deliverables that make up the software. A rating shall never be embedded within software distribution files or code because it will be free to change without any changes to the underlying software or package.
4. Primarily, the first phase/task in the rating system shall be a trailing indicator with an eye to establishing future leading indicators. Initially, it will be based on the software's real-world, historically observable behavior with regard to its stability and generally not on other behaviors or attributes of the software.
5. Where little history is available to produce a trailing rating, that fact should be easily discernable through observation of the rating alone. This will be employed in the initial role of the rating system as a trailing indicator and in future where a leading indicator rating may be published.
6. For future leading ratings, when special processes or practices have been employed during the design and/or development of the software to insure safety and reliability, such special measures may be reflected within the rating so long as they do not interfere with the rating's initial purpose of relating the historically observed stability of the underlying software.

7. [?tbd? probably too soon to be applicable to the current scope] For minor determinations to a new project's rating (before sufficient historical data has accumulated) should the rating system consider the software stability record of individuals who perform the new development if a clear and quantifiable connection between the individuals' past, " first release" software's stability and the stability of the current software can be made and statistically verified?
8. [?tbd? probably too soon to be applicable to the current scope] To the extent, if any, that the rating system is linked to the people who design and develop the software being rated (as provided above), should the only credential that shall be considered regarding the contributing individuals is the complete documented stability history of first-release software they've produced in the past?
9. [?TBD?] Should the consideration put forward above for individuals be extended to include organizations?

• Function or Method
  For the sake of this paper, these two names are used interchangeably. A function is a callable block of code, which may return a value, produce side-effects, or both. Functions may include parameters to be referenced by the code that is within the block. Parameters may be altered as a side effect of calling the code in the function (such as when they are passed by reference). Functions may have identifiers or may be anonymous.

• Independence
  Because we've lived with version-stability as an overloaded variable for so long, this paper will often reinforce the concept of independence. Changes to the stability rating of a piece of software are not necessarily accompanied by changes to the code or the distribution files. While a stability rating can change based on software changes it will often change based on events that are completely external to the code and distribution files.

• Library Grade Function (LGF)
  LGFs are the embodiment of the notion of "re-usable code". An LGF will have no non-standard or non-LGF function, identifier, or storage dependencies. That means, it may call outside functions or reference identifiers that are part of a formally standardized library within its own language (e.g. the standard C library), or it may call other functions or reference identifiers packaged in the same library that it is itself a part of and linked from. All other dependencies are disallowed. Should any other function calls or identifier dependencies exist and be used within a function, directly, or as a result of calling another function, the function is NOT an LGF

• Libraries
  Functions ("methods") within a function or class library MUST each be rated separately.
  • For a function ("method") within a library to be rated, it must be an LGF. Otherwise it is un-rated which should be thought of as equivalent to a rating lower than Alpha (A).
  • The rating given for a function or class library as a whole MUST be the rating of the lowest rated function ("method") within the library.
  • If A library includes un-rated functions ("methods") within it, these are considered the lowest rating and so the entire library must be un-rated.
  • Functions ("methods") that are part of function or class libraries may have additional rating requirements, which include; unit testing (min, max, min-max exceeded, and typical inputs), thread behavior (unit tests with multiple calls on the stack simultaneously), external re-entrancy behavior (unit tests of multiple stack depths from within a re-entrant caller), self re-entrant behavior (if the function or method being rated is re-entrant, then test under a multitude of stack conditions), Statement lines per function (average, percentage above threshold, max).

• Release vs. Version
  Warning: These definitions for release and version are not currently compatible with IEEE definitions. If anyone has ideas for how to maintain compatibility while allowing for independence, please contact me.

  A single release of a given piece of software may involve multiple versions.

  • Version
    A version denotes that changes have been made to the software package from a prior version, however small, since the last time the package was published outside of the developer.
  • Release
    A release is a version of the software that has been functionally changed and/or functionally improved in a significant way from the preceding version (or from inception).

• SSR
  SSR is an abbreviation for Software Stability Ratings as they are defined in this paper.

• User, Developer, & Tester
  These are the three entities who may have a direct effect on a software stability rating at various levels.
  • Developer
    The entity that is actually developing the software. In the case of multiple independent developers being sub-contracted, this is the top-level developer responsible for the overall unit which a given rating applies to.
  • Tester
    Testing agencies may also perform assurance and insurance functions as well, these are generally not brought in at levels below Hi-rel but may be.
  • User
    The most important effector. Experience and observations by this entity have power to downgrade (or upgrade) a rating at all levels.

Publication is a foundational purpose of any rating system. This specification prescribes a way in which to publish downgrades to a software version's stability rating.

When a reliability rating of a version of software is changed the new rating must be published to an easily accessible web-page where users of the software have been clearly instructed such ratings can be found.

A reliability rating that is downgraded to any rating other than "Unsecure" must be published in this way within ten business days from the day it is downgraded.

A reliability rating that is changed to "Unsecure" must be published in this way within three business days of the day it is changed. Also, in the case of a rating that is changed to "Unsecure", users who have purchased the downgraded version in the past year and have not purchased or received more recent versions must be sent email stating that the version was downgraded. If there is no email address, or an attempt to send mail to the email address on file has failed, no further emails need be sent to that user.

When a previously unreported, non-security related defect that can affect a software version's current reliability rating is reported, the software must be tested within 20 business days of the report to verify if the defect exists. If the defect is found to exist, or the test for verification is not performed the reliability rating must be changed to reflect the defect. The existence and non-existence of software defects MUST be verified in writing and signed by the individual making the determination.

When a previously un-reported defect is reported that can cause a software version's current reliability rating to be changed to "Unsecure", the software must be tested within 3 business days of the report to verify if the defect exists. If the defect is found to exist, or the test for verification is not performed, the reliability rating must be changed to "Unsecure" to reflect the defect.

The existence and non-existence of software defects MUST be verified in writing and the individual making the determination must sign a declaration attesting to the veracity of the reported result.

Generally, stability ratings SHOULD NOT be used in product advertisements. If a stability rating is used in a product advertisement in a dynamic media, it must adhere to the same time restrictions as have been documented here for publicizing rating downgrades along with a link to the webpage where the downgrade ratings for the product are published in accordance with the above recommendations. If the stability rating is used in a fixed media type, such as magazines, the date it was valid on MUST be displayed along with the URL of the webpage where downgrades are published.

Color is optional, but if rating values are portrayed in such a way that each rating value is assigned a different color or set of colors, the color scheme defined here MUST be used.

An optional-use color scheme is defined for the recommendation so that if colors are associated with rating values they will be consistent. This will eliminate color differences as a possible source of confusion when viewing stability ratings.

The rating values currently defined, along with the criteria requirements for software to be assigned these ratings are documented in this table.

To use the SSR helper functions, include the following in the <head> section of your web document, like this:.

---

    <script 
            type="text/javascript" 
            LANGUAGE="JAVASCRIPT"
            SRC="https://creativyst.com/Doc/Std/StableSW/SwSr.js">
    </script>

---

Right now, there is one function

        ssrLg(ssr,UseClick);

To call this function inside html you have to surround it with a <script type="text/javascript"> tag. Also, this returns a string with the html of the button, which you must hand to document.write() in order to display. Here's an example of how this will be called inside your d/x/html to display an SSR button ('C' in this case):

---

    <script type="text/javascript">
      document.write(ssrLg("C",1));
     </script>
 

---

Pass an SSR string to this in the "ssr" parameter (as defined above and here). The UseClick parameter says to make a button that pops up a small reference window when it is clicked. Keeping UseClick at 1 will allow users to get an explanation of the Software Stability Rating they're seeing.

Here are some examples of this function's output along with the calls that produced it. Click on these to see the behavior of the UseClick option.

<script type="text/javascript"> document.write(ssrLg("A",1));  </script>	<script type="text/javascript"> document.write(ssrLg("B",1));  </script>	<script type="text/javascript"> document.write(ssrLg("C",1));  </script>
<script type="text/javascript"> document.write(ssrLg("E",1));  </script>	<script type="text/javascript"> document.write(ssrLg("H",1));  </script>	<script type="text/javascript"> document.write(ssrLg("S",1));  </script>
<script type="text/javascript"> document.write(ssrLg("Ah",1));  </script>	<script type="text/javascript"> document.write(ssrLg("Cs",1));  </script>	<script type="text/javascript"> document.write(ssrLg("Eh",1));  </script>
<script type="text/javascript"> document.write(ssrLg("0",1));  </script>	<script type="text/javascript"> document.write(ssrLg("0b",1));  </script>	<script type="text/javascript"> document.write(ssrLg("0cs",1));  </script>
<script type="text/javascript"> document.write(ssrLg("/",1));  </script>	<script type="text/javascript"> document.write(ssrLg("/e",1));  </script>	<script type="text/javascript"> document.write(ssrLg("/cs",1));  </script>

This article is © Copyright, Creativyst, Inc. 2002 - 2005 ALL RIGHTS RESERVED.

Links to this article are always welcome.

However, you may not copy, modify, or distribute this work or any part of it without first obtaining express written permission from Creativyst, Inc. Production and distribution of derivative products, such as displaying this content along with directly related content in a common browser view are expressly forbidden!

Those wishing to obtain permission to distribute copies of this article or derivatives in any form should contact Creativyst.

Permissions printed over any code, DTD, or schema files are supported as our permission statement for those constructs. JavaScript code may be linked to but not copied (to ensure fresh content).

If you're interested in improving or promoting this project please contact me. If used, your code and contributions will be attributed to you with a link to your site.