Forum

Notifications
Clear all

Regression Verification and Validation

14 Posts
14 Users
0 Likes
1,635 Views
(@fgk4)
Posts: 51
Trusted Member
Topic starter
 

As we all have learned this week from Dr. Simon, If any of the Verification or Validation test cases have failed during the V&V processes, the specifications have to be either reviewed or changed and these tests have to be repeated. From my experience, sometimes a regression test is sufficient to test the changed specifications. In other cases, the whole verification and validation protocols had to be re-executed. The decision of conducting a regression test or a full execution of the protocol is not an easy one.

- re-executing full verification and validation protocols can be costly and time consuming.
- Conducting a regression test (even when testing all the changed specification) may not provide the full picture, since a changed specification may have impacted another specification in an indirect way.

What do you think is the best strategy to take when a Validation or Verification report have failed to achieve the expected results?

 
Posted : 03/02/2017 3:14 pm
(@lg236)
Posts: 51
Trusted Member
 

This is an interesting question. In my opinion, it may depend on the type of changed specifications and how that may affect the device. If it is a change in a minor detail of the aesthetics of the device, it can just require only the regression test on that one changed specification. But if the change involve a major part or function of the device as mentioned by Fady, that can require a further analysis. For instance, if the changed specification impacts a major part and a few other depending features of the device, it may need further investigation to see if it is truly required. Because with any change, you still have to confirm the main purpose of the device still meeting up to user’s needs, the safety and efficacy.

 
Posted : 04/02/2017 3:39 pm
 tme3
(@tme3)
Posts: 24
Eminent Member
 

In an failed V&V testing portion of a new product could point to many factors such as poor design, test method, test operators or as you suggested a product simply not able to meet a spec. If a root-cause analysis of these failures indicate that the cause of the problem is indeed the spec, then it is early enough in the life of the product to deviate from the intended specs, or as you mentioned a regression analysis can be performed from accelerated aging. This regression can indicate what the actual threshold of the specific product or design spec is. I feel though that even with the regression a spec deviation could be in order.

 
Posted : 05/02/2017 1:46 pm
 tn58
(@tn58)
Posts: 72
Trusted Member
 

It would depend on the client's requirements. In many cases, the client may specifically require you to re-execute the entire verification and validation protocals. This can be costly but if the client does not receive the expected results, that would mean loss of money and future deals. So the best way to handle this situation would be to see if the final result is what the client wants. If this can be fixed by regression test, than it would cost less and you are able to deliver the expected results. However, if you want to save the time and cost by taking a short cut of a regression test and the final result is not what was expected, you end up with more cost and time to your company. So in many cases, it is better to speak with the direct clients to avoid the loss of time and cost.

 
Posted : 05/02/2017 4:42 pm
 gh56
(@gh56)
Posts: 51
Trusted Member
 

i see a lot of you have gave their opinion on the matter of how to trying to get the best strategy to take when the validation and verification had fail. Now we know is really bad when this two fail, so we have to make sure before everything is done properly to "never" counter this matter, so we have to consider that instead of rushing to design verification, the opposite approach might be more beneficial. spend more time defining design inputs and that design verification becomes smoother, leaving 0 change to fail. implementing this approach will help who ever is working on this medical device to have a better design input and help you to have the ability to manage medical device product development.

 
Posted : 05/02/2017 5:38 pm
(@merzkrashed)
Posts: 123
Estimable Member
 

I think the best strategy to take when a Validation or Verification report have failed to achieve the expected results is to think about the Requirements you need for your design, because common problems in the verification “story” come from poor requirements.
Also, Plan It Before You Do It, always plan right before starting verification process, because testing is often very expensive, time consuming, and subjective. Use testing where it makes sense to do so. So, Know the requirements & Plan it the best strategy to use.

 
Posted : 30/01/2018 12:06 pm
 Sk90
(@sanam)
Posts: 109
Estimable Member
 

V&V demonstrates that design of a new product fulfills its requirement and meets user needs.A well defined requirement links to a customer need.Requirement should also be clear and measurable .Tracing back to Requirement is a significant challenge when test fails.Regression test allow a consistent ,repeatable validation of each new release of product.such testing ensures that product defects have been corrected for new release and no new quality program were introduced in maintenance process

 
Posted : 31/01/2018 7:14 pm
(@srg36)
Posts: 117
Estimable Member
 

I agree that it is important to have a good plan of how to handle testing failures when doing your V&V, but I like the points that @gh56, @merzkrashed, and @Sanam brought up, about spending more time on coming up with a reasonable and appropriate value for the acceptance criteria to reduce the risk of having failures during V&V. This definitely will not eliminate the possibility of having failures, but I think it will definitely reduce the possibility for failure. In projects I have worked on, I have seen it happen often where acceptance criteria for a test is taken from a previous project or some predicate product, the device then fails in testing, and an investigation into the failure reveals that the acceptance criteria was not really applicable or reasonable, which then means the team must agree on a new acceptance criteria. This is a long process that I believe can often be avoided if you put in a little more work defining your requirements, so I agree that spending a little more time defining your testing acceptance criteria can save you a lot of time in the long run.

 
Posted : 01/02/2018 10:08 am
(@hm243)
Posts: 85
Trusted Member
 

The Validation or Verification test report is very significant in determining the success of a product. Once the Validation or Verification fails, it is important to go through the protocol step by step to find the factor that caused the failure. Although this may seem to be time consuming and waste in money, this would be the best process in fixing the issue. The regression test would be a faster route but there is no guarantee that all the factors are considered. As mentioned previously, there could be a specification that is overlooked and then affected by another specification. Even though redoing the entire Validation or Verification test report will take more time, it is a more accurate way of finding the issue at hand. With the regression test, if something is missed, then there could cause a defect in the product, which could possibly then harm someone using the product. In all cases, it is better to repeat the Verification and Validation protocols, if the test fails the first time around.

 
Posted : 01/02/2018 3:44 pm
(@julienneviuya)
Posts: 68
Trusted Member
 

I agree with the above points that this debate would have to be on a case by case basis. If there is a failure in V&V testing, it is primarily important to make sure that your testing method itself is valid for testing this case. If the test is invalid, a failure or a pass would mean nothing to the protocol. Once your test method is validated, the failure must be evaluated if it is an isolated event or if it is indirectly affected by another factor of the device. If the change can be tested in isolation, then a regression test may be sufficient to save on time and cost. However, there are instances when the whole protocol must be carried out once again, while it may be time consuming, may be the only way to produce valid results, such as having many integrated parts that do not work in isolation.

 
Posted : 02/02/2018 12:14 pm
(@savery115)
Posts: 82
Trusted Member
 

In agreement that acceptance criteria needs to be re-evaluated. Before that, a deviation should be opened to address the failure of a validation and why the deviation occurred and how it will be addressed. I would re-evaluate what additional testing or changes need to be done, whether its a regression test or opening up tolerance and specifications. From there, I would review if the acceptance criteria needs to be changed due to the added test or changes in tolerances and specs. Lastly, you have to ensure to rerun the entire validation protocol that failed to ensure it passes with the intended results.

 
Posted : 04/02/2018 11:30 am
(@dipanpatel)
Posts: 71
Trusted Member
 

I agree with your method of using a regression test to save resources. But this might not always identify the issue. By using a less involved method first and then later a more involved re-executing full verification and validation protocols if not solved. However, the changes that are made can also involve the proper track to take. A minor change (in the opinion of the company) can implicit a regression test. However, a functional change will need a full re-evaluation. Each solution, even those not mentions are highly dependent on the situation.

 
Posted : 04/02/2018 11:43 am
(@bb254)
Posts: 113
Estimable Member
 

From prior experience, when a verification/validation had failed I was obligated to redo the verification and validation per the updated specification. Yes, this is time consuming and can be costly, but it is the cleanest way to reroute the protocols. By redoing the protocol and validation the acceptance criteria will change, but within the documentation one should state the reasoning for new acceptance criteria. A justification for the failure must be documented along with the new expected output. Therefore, it is critical to spend time brainstorming the test and failures that can occur well in advance to the execution of the protocol. Based on the type of failure, one should decide whether they want to complete a regression test or re-evaluation. It is also based on upper managements decision on whether they want to fund a re-evaluation or to just move on to a different project.

 
Posted : 04/02/2018 2:11 pm
(@monicagoncalves)
Posts: 59
Trusted Member
 

I believe that deciding between a re-executing full verification and validation protocols or conducting a regression test depends on the situation. In my company there base all decision on the severity of the situation and risk factor. This past week we had a failure occurs on one of our devices during validation and we decided to redo validation on an updated specification. We needed to write up a justification for why we could re-execute the full verification protocol on this specific failure. All failure need to be justified and documented. But like I stated, in the beginning, this decision should primarily be made based on the severity of the risk of the failure.

 
Posted : 04/02/2018 5:24 pm
Share: