The distribution of correctly identified images of Carabidae at both the genus and tribe level showed similar patterns. Most of the baseline, other Coleoptera, and other insect specimens showed a high level of detection accuracy by CarabID. However, due to the high variability in the distribution of confidence scores within the new species group, the level of accuracy appeared to be much lower than the other groups (Figure 4). The distribution of correctly identified images of Carabidae at both the genus and tribe level showed similar patterns. Most of the baseline, other Coleoptera, and other insect specimens showed a high level of detection accuracy by CarabID. However, due to the high variability in the distribution of confidence scores within the new species group, the level of accuracy appeared to be much lower than the other groups (Figure 4).

When we compared the differences in the confidence score accuracy between the genus and tribe given by CarabID, our result indicated that grouping Carabidae species into tribes gave a higher relative proportion of images that were correctly identified than those at the genus level for both the baseline and new species group. There was no difference in the level of detection accuracy for other Coleoptera and other insect groups as the tribes for these insects did not belong to those that can be currently identified by CarabID.

Additionally, most of the detections for baseline, other Coleoptera, and other insects showed a relatively high detection accuracy at both the genus and the tribe level, whereas the level of detection accuracy for the new species group was lower, with a higher proportion of images being incorrectly identified (Figure 5).

Interestingly, when images were manipulated, the CarabID application provided a different score from the unmanipulated images for most of the images that were tested. Higher scores were generally more common if the image was edited (Figure 6). None of the 229 unmodified images from the new species (n = 191) and other Coleoptera (n = 38) categories that were tested were accurately identified to the genus level, and only three images from the new species group were accurately identified to the tribe level, although there was no correct identification to the genus level for this group (Figure 6).

Check out pages 6, 7, & 8 of Version 5 Edition 1 for relevant graphs.

A total of 371 wings were used from 5 groups of images: the baseline, which contained images used to train the model (n=41); same species containing images of the same taxa but different specimens (n=33); new species with different species within the same genera (n=43); other flies, which contained Diptera of completely different genera and families (n=192); and other insects with various moth and wasp taxa (n=58). The highest confidence score sat at 1.0 for Procecidochares utilis within the baseline group (Figures 7, 8, 9). Large numbers of outliers were seen for baseline, new species, and other flies in particular. All data appears to be negatively skewed, with median values all above 0.78 (species confidence for other insects). Greater variation was seen overall in the species confidence scores as opposed to genus confidence (Figure 7). No predicted scores above 0 were correct for the categories ‘new species’, ‘other flies’, and ‘other insects’. 19 out of 27 correct scores were greater than 0.9 for the baseline and 12 out of 26 were for the same species group. Overall, there was a much greater number of incorrect predictions (mostly false positives) than correct ones (true positives or negatives). The genus scores have less variation in general. For the baseline, 27 of 32 correct scores were greater than 0.9, while 15 of 21 were for same species and all 20 were for new species. Mean confidence scores are 0.85, 0.80, 0.79, 0.76, and 0.74 for species, and 0.93, 0.78, 0.87, 0.88, and 0.75, for the genus level.

Discussion

The experiments demonstrated the potential of TephritID and CarabID to predict its namesake taxa from images of their wings and body. However, the results imply the model isn’t as robust as is needed at its current stage of development. Several sources of limitation have been identified, providing a scaffold for future steps.

Accuracy at different taxonomic ranks
The TephritID model was able to output reasonable mean confidence values at species-level identification, these being 0.85 (baseline), 0.80 (same species), 0.79 (new species), 0.76 (other flies), and 0.74 (other insects). At the broader genus level, the accuracy was unsurprisingly higher with scores of 0.93, 0.78, 0.87, 0.88, and 0.75, respective to the aforementioned list, and less variable overall towards higher values. A similar trend was shown after testing the CarabID model, where accuracy was higher and less variable when images were grouped into a higher taxonomic range at the tribe level consisting of multiple genera. These findings are indicative of promising progress and a decent current model. At the species level (Figure 8), it is evident that incorrect predictions given by TephritID considerably outnumber correct ones (315:53). This is also true for CarabID at the genus level, where there was a more confined range of insects. These skewed inaccuracies are largely influenced by the ‘New species’, ‘Other flies/Other Coleoptera’, and ‘Other insects’ groups. Additionally, there was a decrease in the score from the ‘baseline’ to ‘other insects’ for both models, with baseline images used in model training being the highest and ‘other insect’s being the lowest.

However, the models returned a range of scores with highly varied accuracy and predictions for specimens in the ‘new’ and ‘other’ groups when it should have been outputting low to no scores for taxa it did not recognise. This is because TephritID and CarabID were only trained to identify certain fruit fly and ground beetle taxa to their species and genus levels, and therefore were more likely to provide a score of zero for any images that belonged to the ‘Other flies/other Coleoptera’ and ‘Other insects’ groups. It is very likely that images from ‘Other insects’ had obviously different features compared to the ‘New species’ and ‘Other flies/other Coleoptera’ groups, which would increase the proportion of insect specimens that were correctly identified, with high accuracy scores expected. This could also explain why there was a much lower proportion of images incorrectly identified from the ‘New species’ group, as the specimens tested belonged to the same order (Diptera and Coleoptera) as those on which the model was trained, possibly increasing the difficulty for accurate detection due to the insect having similar features.

Potential issues with TephritID and CarabID
Since TephritID and CarabID tend to identify taxa with relatively high confidence when testing baseline images, but not those within the ‘New species’ category, this suggests that there may be an issue of overconfident misclassification and a decline in the models’ ability to generalise as images accumulate variations that were absent in the training data. This is a common challenge when working with neural networks such as the CNN, and a key obstacle in further development of deep learning techniques [15]. Generalisation failure is universally cited as a continual challenge in machine learning, and is often a repercussion of limited data availability in the sample set [16]. According to Zohuri & Moghaddam [17], deep learning models using CNN are reliant on the performance of AI and require a large dataset including at least 10,000 images as reference examples when training the model. The dataset used to train TephritID only contained approximately 800 images, whereas CarabID contained 1,764 original beetle images, with a total of 4,143 images after augmentation was applied. This could indicate that a larger dataset may be required to increase model accuracy and to reduce inconsistency in the identification of ground beetles to their correct taxa.

Another limitation of low data availability for model training is an effect called overfitting [18], which is when the weights adopted by the model for certain features are optimised for the training data, but result in poor performance when faced with new data [19]. This occurs when the model lacks the ability to make generalisations on new data that it was not previously exposed to, resulting in the model performing poorly on testing datasets although it shows good prediction on the training datasets [20]. Overfitting especially affects smaller sample sizes used in model training as the level of noises tend to be higher, and it is believed that increasing image augmentations can help reduce model overfitting [20], [21]. This variability points toward potential errors somewhere along the line between model training and testing, or a model that is not yet rigorous enough to provide consistently reliable predictions. Images used for testing the CarabID model were taken at slightly different angles under different lighting conditions, with some images appearing to have a lower resolution than others. Theoretically, if the CNN model was performing as expected, image quality and position should not affect the results after the preprocessing process with standardisation [14]. However, large differences in the scores obtained from CarabID between modified images and pre-modified images in the ‘New species’ and ‘Other Coleoptera’ categories was evident, which further indicates possible model overfitting.

A couple straightforward options exist to reduce bias, spurious correlations, and misleading predictions within the model. As mentioned above, the likely culprits of the large amounts of variation and overconfident misclassifications are overfitting, an inability to generalise, and a shortage of training set data. Highly accurate classifications require sizable datasets [22], so a next step for TephritID and CarabID would be to undergo more digitisation of NZAC specimens, as well as of other taxa from all over the world for continued training. Secondly, it is interesting to note that modifications such as tilting and mirroring of existing images tended to output different predictions than pre-modified images. Data augmentations should have accounted for these, but since differences were observed, more iterations may be needed, especially considering the small training dataset. Augmentation techniques increase the number of training samples during model training, and it also tends to reduce overfitting [18]. This is especially true for the CarabID model, as only a very small proportion of specimens were correctly identified to both the genus and tribe level pre-modification, with no accurate detection scores given for images at the genus level, and only three images from the ‘New species’ group accurately identified to the tribe level. However, the above techniques have their own drawbacks, and there is work being done worldwide to compensate for them. All in all, TephritID and CarabID reaffirm the potential of using wing and body images and deep learning methods to identify taxa. Its potential applications in biosecurity surveillance and conservation make it particularly relevant — especially within Aotearoa, where the robustness of these systems can be the difference between ecological balance and devastation.

The development of AI models through deep learning tools for entomological studies in a New Zealand context has great potential that will benefit conservation and biodiversity. Although the web applications CarabID and TephritID may not currently be able to accurately describe and identify the correct Tephritidae and Carabidae genera for the ‘New species’ category that displays similar traits as the images used in model training, modifications of these web tools in the future could be effective in helping with the classification and identification of native and invasive insect species, reducing the amount of time required by entomologists and researchers. Digitisation provides opportunities that will be especially useful for biosecurity surveillance, conservation projects of threatened native and endemic species, and helping to maintain the biodiversity of Aotearoa. In the future, it would be beneficial to make appropriate changes to image-based deep learning models so they can provide greater accuracy in identifying fruit flies and New Zealand native beetles to more confined taxonomic ranges, which provide further opportunities for monitoring live fruit fly and ground beetle species for ecological research and pest eradication.