Analysis of Hidden Representations by Greedy Clustering

The hidden layer of backpropagation neural networks (NNs) holds the key to the networks' success in solving pattern classification problems. The units in the hidden layer encapsulate the network's internal representations of the outside world described by the input data. In this paper, the hidden representations of trained networks are investigated by means of a simple greedy clustering algorithm. This clustering algorithm is applied to networks that have been trained to solve well-known problems: the monks problems, the 5-bit parity problem and the contiguity problem. The results from applying the algorithm to problems with known concepts provide us with a better understanding of NN learning. These results also explain why NNs achieve higher predictive accuracy than that of decision-tree methods. The results of this study can be readily applied to rule extraction from NNs. Production rules are extracted for the parity and the monks problems, as well as for a benchmark data set: Pima Indian diabetes diagnosis. The extracted rules from the Pima Indian diabetes data set compare favorably with rules extracted from ARTMAP NNs in terms of predictive accuracy and simplicity.