PdM: What you need to know

Rude questions to ask an IIoT PdM vendor

The cross-disciplinary team tasked with evaluating Industrial IoT Predictive Maintenance solutions often lacks direct Machine Learning discipline expertise. Here are four questions that can shed light on a vendor’s technological capabilities and value add.

As industrial plants digitalize their production assets, IIoT Predictive Maintenance is now an integral part of the Smart Factory roadmap.  A common scenario is that the decision-making process to procure IIoT Predictive Maintenance technology can be led by representatives of plant management, Operations & Maintenance (O&M), Operational Technology and Information Technology.  In many cases, the industrial organization is unable to analyze vendors’ capabilities and must rely on third party consultants for guidance. 

This article provides four questions to ask any IIoT Predictive Maintenance vendors you are considering.   These questions can be used in a RFI/RFP or informally during solutions briefings. 

#1 Is the Machine Learning Algorithm selection process manual or automated?    

It is important not to gloss over the topic of Machine Learning algorithm selection.  If your vendor is providing Automated Machine Learning, then the selection of the algorithm to be used for a given situation is automatically determined by the solution itself without the need for human input. 

When the selection process is manual, a Big Data professional needs to make the decision about which algorithm to use from a library of hundreds of even thousands of algorithms. 

Industrial plants cannot hire enough Big Data scientists and engineers to deploy solutions locally.  Equally troubling for the industrial sector is that higher paying industries including Financial Services, Insurance, IT and Professional Services are competing for the same professionals. 

How is this relevant to Machine Learning algorithm selection?  Industrial plants cannot rely on recruiting sufficient numbers of Big Data professionals to manage onsite IIoT predictive maintenance.  At stake is the ability to scale a solution across an entire plant. If, for strategic reasons, management sets policy that only internal Big Data resources will take responsibility for tasks such as algorithm selection, it should recognize that there will be a high premium associated with this decision. 

Are you interested in learning more about this topic? Download our whitepaper on an Automated Machine Learning architecture for industrial IoT analytics.

#2 What level of technical expertise is required to use the Industrial IoT Predictive Maintenance solution? 

In the research we conducted with Emory University on the future of IIoT, Reliability and Maintenance personnel were asked to identify the largest blockers of Industrial IoT Predictive Maintenance.  Skills shortage (data scientists), was cited as the strongest factor negatively impacting the deployment of Industrial IoT Predictive Analytics.  Over 80 respondents indicated that skills shortage had a strong (44%) or moderate (37%) impact.  Only 8% suggested that it had no or limited impact on deployment. 

From interviews with O&M professionals, there is a concern that Maintenance and Reliability staff will be required to gain knowledge of discipline for which they are untrained.   Whether or not these concerns are warranted, they are not isolated. 

What does this mean for the industrial plant?  Selecting a solution that requires existing plant personnel to gain new skills is risky.  Technicians and engineers can be expected to access monitoring systems and dashboards with their existing level of skills.  In other words, solutions need to adapt to the current industrial plant environment, not the other way around. 

#3 Can you give me a customer reference from my vertical that I can call? 

This does not need explanation.  A vendor that cannot provide a customer reference from their vertical may not be ready for prime time.  Perhaps there is a good reason that that a company starting out does not yet have references, but consider the risks involved. 

#4 How does the vendor’s anomaly detection results compare with benchmark data?    

When evaluating a vendor, you want to compare their performance relative to others.  One effective way to do so is to compare vendor test results against published benchmarks. Anomaly detection is a domain that is researched and used in a variety of applications.  A partial list includes computer servers, cybersecurity, medical fields, consumer behavior and now, machine monitoring. It serves as the basis for all industrial failure analysis and prediction solutions. 

For instance, SKF Enlight AI has benchmarked its anomaly detection capabilities against three algorithms for anomaly detection developed by Yahoo, Netflix and Twitter, respectively. These companies released their anomaly detection algorithms to the public as open-sources. 

In the first test compared with Yahoo, Netflix, and Twitter, the results were decisive: SKF Enlight AI outperformed the other algorithms in all measures – the accuracy of the results, calculation time and memory consumption. Using a cost-function that inputs the above 3 measures and calculates a single measure, SKF Enlight AI’s score was 88% whereas the next best runner-up scored only 51%. 

In the second test, SKF Enlight AI was used to predict Remaining Useful Life (RUL) of jet engines using a data set that was released as part of a contest organized by NASA.  The criteria measured by the organizers were: Recall, Precision, F-measure, Mean Absolute Percentage Error (MAPE), and Root Mean Square Error (RMSE) 

SKF Enlight AI results of correct RUL predictions surpassed all those submitted by the winners of this contest, including Microsoft MAPE. Presenso outperformed by 7.7 times and RMSE by 5.6 times better than the next best competitor. Precision, recall, and F-measure were 6%-16.2% better. 

Recommendation and Conclusion 

This list we provide in this article is a starting point and questions should be developed by the team assigned to decision making process. 

There are two steps that are critical to the purchase decision: 

  • First, business and functional requirements should be formalized, and each vendor should be evaluated relative to objective criteria.   
  • The second step is to conduct Proof of Concept (PoC) before reaching a purchase decision.  For a step-by-step guide for conducting a PoC, please refer to this document.