Cogniac Best Practices Iron Fist

Best Practices for Improving Applications:

The “Iron Fist” Technique

Cogniac Confidential

• Once you have an application in the Cogniac system with 100+ labeled media, you should start to see models released with assessed accuracy in the 90+% range, depending on the difficulty of the visual task.

• If you have a large dataset, you can mine it for the hardest images to force the model to learn harder. We call the technique of Labeling and forcing the model to learn the hardest examples the “Iron Fist” technique.

• This technique relies on the notion that at any given time, the hardest images for a model are the images for which its predictions are “most wrong”. These are also the most valuable images to label, forcing the model to “learn harder”.

The technique is as follows. After a new model is released:

1. Replay ALL (or up to 1000 RANDOM) images from the input subject

You want to draw from the largest sample pool to find the rarest, hardest images. The system will automatically ignore media already labeled in your target application. The real purpose of this step is to update the model predictions in the output subject based on the current best model. This is necessary to find the ‘worst-of-the-worst’ predictions from the current best model against the largest possible dataset.

2. Wait for the replay to complete 

Check the output subject (with the probability filter set to full range 0 to 1) and refresh until no new output items appear.

3. Prepare to replay again the most difficult items by selecting a probability range for the most difficult samples in a single subject

For example, when finding an ROI in preparation for a subsequent inspection and most images are expected to have the ROI, select a probability range of 0 to ~0.5, which contains the ‘most wrong’ predictions.

In other cases where the ROI is not always present, the ‘most wrong’ range might be closer to [~0.4, ~0.6], for example. Look at the output subject of interest and adjust the probability range to visually check the results of selecting a particular probability range.

In the case of OCR applications, the images with the lowest probabilities will tend to be the most challenging samples.

4. Replay from your application from a single output subject with a probability filter selected to contain the ‘most wrong’ outputs.

Configure the replay for 25 to 50 images and force feedback.

5. Wait for the replay to complete. 

It may take some time if you have many items in the output subject.

6. Provide feedback on the “most wrong” images from the feedback queue.

Ruthlessly rule the model with your “Iron Fist”!

7. Wait for a new model to be released and repeat. 

Important Note:

When utilizing the Iron Fist technique (or any labeling strategy that emphasizes more challenging images), the application accuracy reported by the Cogniac system will be much lower than the actual real-world application accuracy. Because the application is assessed on a random subset of the labeled images, you are effectively making the application test much more difficult than ‘real-life’ by labeling mostly the hard images. This is good because the more difficult training and validation set forces the model to “train harder”. As a result, it will perform better in the “real world” than the application accuracy indicates.