How deep learning helps automate inspections for the automotive industry
Automotive manufacturers were among the first in industry to embrace the mechanization, automation, and productivity enhancements offered by machine vision. Today, nearly every component, system, and sub-assembly within an automobile is manufactured with machine vision and barcode reading technology. Thanks to great advancements in artificial intelligence, many automotive and parts manufacturers are now casting their gaze to deep learning-based image analysis software to automate remaining manual processes considered, until recently, simply too complex to automate.
Historically, these applications have involved unpredictable part location, cosmetic inspection on confusing backgrounds, and classification—a category of inspection which machine vision algorithms simply cannot automate. By automating these categories of applications with the use of deep learning software, manufacturers can increase their productivity by limited defects and false rejects. The other byproducts on deploying this technology include increasing overall quality and potentially minimizing labor.
Read on to see how automotive manufacturers are employing deep learning-based image analysis in factories right now to improve defect detection, optical character recognition (OCR), assembly verification, and classification.
Automotive manufacturers have a huge incentive to ensure the integrity of their components and sub-assembly, where the slightest defects can compromise a part’s function and safety. And yet not all cosmetic defects are cause for functional concern.
Take, for example, a piston in a reciprocating engine. Scratches on a piston’s welding seams adversely affect its performance. Others, like rust spots and even some superficial cracks and fissures, are merely cosmetic. These differences in defects can be hard for automated inspection systems to evaluate, however, because of their subtlety in presentation (a scratch can look similar to a crack) and imaging issues.
Metallic weld surfaces cast specular glare, which can confuse cameras. For this reason, many manufacturers continue to employ manual inspectors rather than try to automate inspection of textured metal welds because human inspectors, though slow and vulnerable to fatigue, are more well-suited to correctly identify and characterize cosmetic subtleties.
Fortunately, advancements in deep learning have made it possible to automatically detect and characterize unpredictable and variable defects on metal surfaces like welds with no manual inspection required. Powerful new software is able not only to identify but also to characterize these defects, much like the human eye and brain would. Programming this type of inspection would require complex algorithms with extensive defect libraries; and even then, the inspections would still likely generate errors.
By contrast, deep learning algorithms learn from example images to form their own models of weld defects. These systems can tolerate blurry, unfocused images and challenging backgrounds after an initial training period so that they are able to appreciate and recognize even the most minor variations and effectively categorize them. And of course, the added benefit is speed. These programs, which can run as software-only or directly onboard smart cameras, are fast and consistent. In this way, deep learning systems offer an unbeatable combination of speed and consistency with human-like intelligence.
Given the many possible flaws and lighting challenges, deep learning-based analysis offers a simple and robust alternative to traditional machine vision inspection. After the system has been trained during run-time, the deep learning-based software, having learned to recognize and ignore irrelevant variations, is able to characterize underpowered and overpowered weld images as defective.
Optical Character Recognition
Automated character reading can also confuse an inspection system when specular light, reflection, and paint colors are present. Typically, OCR and optical character verification (OCV) tools recognize characters in order to read them or verify their correctness, while giving the user the options to optimize their systems for speed and read rate.
The majority of OCR/OCV tools available today can quickly and reliably read black fonts printed on clean, white backgrounds. Unsurprisingly, these ideal conditions are not the norm in most industrial settings. Even though advanced algorithms can now learn and read most printed fonts, even with little contrast between type and background and significant variation in width and height, problems mount when letters or numbers are touching, skewed, or distorted.
It can also be difficult to distinguish between similar shapes (for example, the letter “O” and the number “0”) when the tool hasn’t been pre-trained on that specific font. What happens when specular light, reflection, and paint colors are present? Or when printed alphanumeric figures are somehow deformed and no longer immediately recognizable? These situations would normally require excessive labeling during the training process, and even then could fail.
In the case of a car’s VIN code, manufacturers must be able to quickly decode a string of letters and numbers, which can be printed on varying surfaces. Environmental conditions as well as print distortion can make it difficult for a machine vision system to locate and recognize characters, which may either be direct part-marked (DPM), etched or scribed onto a metal plate, or printed on a sticker. For this task, automotive manufacturers can finally turn towards deep learning-based OCR—a level up from existing machine vision-based OCR tools—when they experience both challenging fonts as well as image formation difficulties.
Uniquely, Cognex’s deep learning-based OCR tool uses a pre-trained, omni-font tool to recognize characters even if they are obscured by print deformation, high or low contrast, or reflection. In the event of a misread, the deep learning model only needs to be retrained with additional examples on the missed characters, saving time and decreasing failure rates.
Incomplete or faulty assemblies can be difficult to inspect for a number of reasons: Multiple parts require painstaking algorithm development, and these scenes can be visually confusing because of their patterning and background. In these cases, an automated system needs to be capable of learning the correct, normal appearance of a scene while accounting for slight variations which may or may not affect quality.
For large and complicated assemblies, an inspection system also needs to be able to segment specific regions as areas of interest for inspection or as containing defects. These factors make rule-based programming too cumbersome and prone to error. Instead, deep learning-based defect detection, location, and layout tools can develop AI-based models to identify targeted regions of interest in an image, simply by learning possible variations in appearance, from a representative sample image set and inspect them for completeness.
In this way, the deep learning-based system creates a reliable reference model of an assembly such as a car door and can quickly and reliably confirm that all components are present, correctly placed in their location, and completely assembled. Though human inspectors are skilled at this type of judgement, deep learning can do the task with the speed, accuracy, scale, and robustness of a computer.
Let us return to the welding seam defect detection example to further understand deep learning’s value proposition. As previously explained, welding inconsistencies may be anomalous but not functionally defective. This requires an automated inspection system which, in addition to detecting defects, can correctly classify them as “good” (effectively, passable) or “bad” and the cause for rejection. Manufacturers need data to understand if there are too many of one type of defect causing rejections on the line in order to change production methods or limit the type of defect from possibly occurring in the first place.
Machine vision’s inherent limitations prevent it from classifying images. Yet advancements in neural networks mean that AI-based programs can now perform image-based classification. AI can actually classify images of the same part into sub-categories by learning their key visual differentiators. After being trained on a set of labeled representative images of each image class, the deep learning model learns to distinguish between real-time images and categorize them into classes such as spark plug types.
Deep learning is just the latest tool that inspection application engineers have in order to solve complex automotive inspection challenges. What was once deemed only possibly by human inspection can now be done through a vision system that leverages deep learning technology.