Patches Required

So we get this call from a customer who has been using our identity software for sometime now with a huge complaint that there was a security hole in our software that left them exposed for several years.  Needless to say, they were not happy.

Why had we, the vendor, not made them aware of the security hole, sent them a patch, and insured it was fixed?

After all, are they not paying a not so insignificant sum in support costs?  The software they have deployed is an older version of the software, but it is still under active support.  They demanded we patch their version of the software, as they did not want to go through an upgrade in production, and we were bound by the support agreement to do so.

Well, we have been spending the last several weeks trying to show them the error of that approach.

There are several issues here, but first some background.  The software they deployed was a slightly older version (v11.1.1.3.0) delivered in mid 2010.  The current release of the software is 11.1.2.1.1.  The devil, as they say, is in the details.

To read these numbers, lets go left to right.  The “11” indicates the major version 11g, versus 10g, thus both are 11g.  The first “1” is the family of software, which rarely changes.  The third number (“1” in the first, “2” in the second) is the major release, thus the first is “11g R1” and “11g R2”.  This usually indicates significant changes between the versions, including new features, functions, etc.  They are not interchangeable and would require a formal software upgrade procedure to implement the new features.

The fourth number is the sub-release within that software version, with cumulative bug patches and improvements to the software.  The last number is the bug patch set (BP) for that version.

Our customer was running 11.1.1.3.0, which indicates this was the first unpatched release of that particular version of software. They spent many hours integrating, developing, and rolling out the software into production.  That was in mid 2010.  But there was a flaw that was discovered later that year, a shortcut left by a developer to aid testing, that would allow someone to circumvent the login process if you knew how.  A major hole, yes, for us and the ex-developer.  And quickly repaired in a patch and rolled into BP2 (11.1.1.3.2). This was in late 2010

Which is where the story gets interesting (I hope).  We did not broadly announce a security issue we found with one customer, as this would immediately put the rest of our customers at risk with a zero day flaw. We do quickly release a tested patch that corrects the issue and notify our support customers through support contacts to apply the patch as soon as possible without tipping the bad guys off. Then we roll the patch into the next bundle patch, in this case BP2.

Bundle patches are collections of patches (the goal is 20 to 60 or so) rolled together and tested to not break the current software. Most of the times the are cumulative.  However, our customer chose the path of least resistance (or resources required) and did not implement a patch process to their production environment, nor test any updates to the software released. Thus they ran the better part of three years with a major whole in their public website.

It was finally when a new project person looked at the last bundle patch release (BP4 or 11.1.1.3.4) notes did they see there was this flaw.  That is when things got screwy.  The customer wanted us to back port the one patch for the flaw to 11.1.1.3.0, as it was still under active support.  We recommended they apply the patches up to BP4 to at least benefit from all of the fixes we have implemented over the last three years.  They consider it an upgrade and say we are not supporting our product. We are.  We fixed the problem over two years ago.

Here is the flaw in their logic.  First, if we did do the one off fix, they would now have a unique production deployment. No other customer would have the 11.1.1.3.0 release with a solo patch on it, so it would complicate the support effort going forward. Second, the customer would still be flying in production with the initial release of the software.  Given an estimated 50 fixes per bundle patch, four bundle patches means roughly 200 things have been fixed and tested together.   The customer may fix the one issue they are concerned with in production, but will still run into problems as they run into other software glitches that have already been fixed. We would fix one issue, but they would still be 199 fixes behind.

One other quick tidbit: the  grace period.  As a vendor rolls out an update or patch, the grace period is the time a vendor allots for its customers to migrate to the newer version (not new release).  This  takes into consideration i would take up to a year to apply a patch, so the older version is still kept up on active support.  So if BP3 comes out, BP2 falls into the grace period (usually 1 year) before active support ceases and customers should use the newer version. Note, if BP 4 comes out within 9 months, BP2’s grace period continues for another 3 months or one year after BP3 was released. Note that the old grace period clock starts ticking on BP3 the day BP4 comes out.

Once out of the grace period, active support for that particular BP ceases.  In our customer’s case, BP0, was well past the grace period, so technically the lawyers would argue we were not obligated to actively support it.   We still actively support 11.1.1.3.x software, but only the latest release (BP4 in this case) and any BP3 software living out its grace period.  BP0, BP1, and BP2 would be considered deprecated versions of patch sets.

So here are the important points and learnings from all of this:

1. The vendor must supply fixes and patches to the software as bound by the support contract, but it is up to the customer to stay aware of the releases and apply patches in a timely fashion.
2. All projects must include resources to maintain patch levels.
3. Bundle patches are usually cumulative and only the latest one needs to be applied. Usually.
4. When notified of a vendor patch set release, someone on the customer side must invest the time to investigate the bugs and the patches and determine if any apply to the currently deployed stack. If so, it should be applied in a timely fashion.  If it does not address a particular combination of software currently being used, only then can the decision be made to forgo the update.
5. At a minimum, patch grace periods should be noted (see vendor support documentation).  If current software falls out of the grace period, support may not be able to help and the customer may have to apply released bundled patches first if they run into a problem in a deprecated version of the code..
6. There is a benefit to applying bundle patches, as the usually contain several dozen patches and they have been tested together, so one avoids running into the same problem someone else already has going forward.
7. Do not expect the vendor to shout from the roof tops any major security issues fixed. It gives the bad guys too much information on the rest of the customer install base.
8. Doing nothing year after year will only lead you into trouble.

Remember, this is security and identity software, so you need to make sure patches and updates are reviewed an applied in a timely manner.

Read Full Post »