Figure 14: OMLET results for test samples from the GRUFF cup
database.
Figure 14 shows the plot of the average error per sample versus training set size for examples from the randomly generated cup category. As before, OMLET'S performance generally improves as the number of training samples is increased. A comparison of the error plots for the conventional chair data and the cup data reveals that the average error for the cups is higher for the same number of training samples, and the error rate decreases more erratically. The comparison of error rates between these two categories is valid since they are both at the same level in the learning hierarchy. As before, there are two performance factors that could be the cause of the different error rates. There are considerably more ranges that need to be learned for the cup category than for the GRUFF conventional chair category (17 versus 3). Also, from Figure 15 A, we can see that data set created by the cup generator program is of poor quality. Thus, due to the random nature of the synthetic cup generator program, the system was trained with shapes that, on average, are not very good examples of cups. Regardless of the poor training data, when more than 150 training samples are used, the actual evaluation measures for the cup test examples are within approximately 4% of the desired evaluation measures. In light of the ``bad" set of shapes used as training examples and the large number of ranges that must be learned, the higher average error for cups seems reasonable.
Figure 15: Histograms of desired evaluation measures of the synthetic
cup training sets.
As an additional test, we generated a set of 78 synthetic cups in the same manner as before (see Section 5.2). However, we required the distribution of the desired evaluation measures of the synthetic cups to have a similar distribution as the GRUFF conventional chair examples (shown in Figure 11 A). Figure 15 B shows the histogram of desired evaluation measures of the examples in this second synthetic cup data set. Since the number of training epochs, the number of training examples, and the quality of the training data are the same as for the first test using the GRUFF conventional chair examples, this experiment isolates the effect of the number of ranges that must be learned. Performing a leave-one-out test (77 training examples), the average error per sample was found to be approximately 0.08. In Figure 13, the leave-one-out results on the 78 GRUFF conventional chair examples show an average error of less than 0.01 per sample. Thus, it would seem that the number of ranges to be learned affects system performance considerably.
Finally, we created a set of 200 synthetic cups with a similar distribution as the GRUFF conventional chair examples. The histogram of desired evaluation measures of the examples in this third synthetic cup data set would look similar to the histograms in Figure 11 A, and Figure 15 B. Performing a leave-one-out test (199 training examples), the average error per sample was found to be approximately 0.023. Compared to the error rate of the original 200 synthetic cups (approximately 0.04), we again note that ``better" training data improved system performance considerably. Compared to the error rate of the 78 synthetic cup data set (approximately 0.08), which is similar in quality, we see the increased number of training samples significantly improved system performance. The error rate for this third synthetic cup data set with 200 examples is still higher than the error rate for the GRUFF data set of 78 conventional chair objects (less than 0.01), which has a similar quality distribution. Consider that for the GRUFF data set we used 77 training examples to learn the 3 ranges of the conventional chair category, and for the synthetic cup data set, we used 199 training examples to learn the 17 ranges of the cup category.