In this paper, normal form (NF) theory and Hilbert-spectral analysis are used to characterize and quantify nonlinear modal interaction in stressed power systems. The method of NF is utilized to obtain a second-order approximation to system behavior following large disturbances. Based on this representation, nonlinear interaction measures are obtained to assess the extent and distribution of nonlinearity in the system. Hilbert spectral analysis is then applied to characterize and quantify the time evolution of the modulation process directly from the simulated power system data obtained from transient stability simulations as well as to validate normal form results. The use of the analysis methods is demonstrated on a 11-bus, 4-machine test system. Examples of application of the developed approaches are given to determine the strength of nonlinear interactions in power system behavior, and estimating the distribution of nonlinearity in the system states. The results of the simulations show that the combined application of these techniques leads to a more in depth analysis of system behavior.