Model interpretability for models trained with a custom algorithm

Machine learning models can make good predictions, but it becomes challenging to adopt predictions without understanding the logic behind them. Being able to interpret a model increases the trust in a machine learning model.

Extracting insights from the models and the predictions is valuable for identifying feature importance, informing feature engineering, directing future data collection, informing human decision making, and debugging, on top of building trust in the output.

The insights can provide answers to questions like:

• What features in the data did the model identify as the most important?
• For any single prediction from a model, how did each feature in the data affect that particular prediction?
• How does each feature affect the model's predictions in a big-picture sense?

In order to help answer these questions, there is a model interpretability option available for models trained with custom algorithms. The model insights file is entirely dependent on the custom algorithm code; you need to include the desired code within the custom algorithm you will apply in the training to generate the file and define its contents.