The free vibration of bridges and patterns in bridge-vehicle dynamic interactions can help signify decaying components of bridges and predict structural risks, such as scouring. Traditional methods - including contact sensors, Laser vibrometers, and videogrammetric algorithms - often require a time-consuming process of manual interpretation to identify anomalous vibration modes that imply underlying defects. Engineers can hardly examine all possible correlations between vibration modes and various scouring possibilities, because the number of combinations of vibration modes and possible scouring conditions is exponentially large. Using a bridge as an example, this paper examines an approach that automatically correlates the vibrations of bridge components captured in videos with potential scouring problems through an algorithm that automatically updates a numerical simulation model of the bridge based on video analyses. An algorithm then simulates various scenarios of scouring on Finite Element Analysis Model of the bridge, thereby determining the most likely scouring condition as those that produce similar vibrations extracted from videos. The authors tested this algorithm on a real bridge and found the actual length of the scour of a column by correlating the frequency extracted from the video data to that of frequency determined from FE analysis.