Multi-fidelity data are common in almost every engineering and science discipline, which can be from simulation, experiments, and a hybrid form. High fidelity data usually associate with higher accuracies and expenses (e.g., high resolution experimental testing or finer scale simulation), while low-fidelity data are exactly opposite in terms of the accuracy and cost. Multi-fidelity data aggregation (MDA) in this study refers to the process of combining of two or multiple sources of different fidelity data to have a high accuracy and low computational cost. MDA has a wide range of application in engineering and science, such as multiscale simulation, multi-resolution imaging, and hybrid simulation-testing. This paper presents a novel framework named Multi-fidelity Data Aggregation using Convolutional Neural Networks (MDA-CNN) for multi-fidelity modeling. The MDA-CNN architecture has three components: multi-fidelity data compiling, multi-fidelity perceptive field and convolution, and deep neural network for mapping. This framework captures and utilizes implicit relationships between any high-fidelity datum and all available low-fidelity data using a defined local perceptive field and convolution. Most existing strategies relies on the collocation method and interpolation, which focuses on the single point relationship. The proposed method has several unique benefits. First, the proposed framework treats the multi-fidelity data as image data and processes them using CNN, which is very scalable to high dimensional data with more than two fidelities. Second, the flexibility of nonlinear mapping in neural network facilitates the multi-fidelity aggregation and does not need to assume specific relationships among multiple fidelities. Third, the proposed framework does not assume multi-fidelity data are at the same order or from the same physical mechanisms (e.g., assumption is needed for some error estimation-based multi-fidelity model). Thus, the proposed method can handle data aggregation from multiple sources across different scales, such as different order derivatives and other correlated phenomenon data in a single framework. The proposed MDA-CNN is validated using extensive numerical examples and experimental data with multi-source and multi-fidelity data. Conclusions and future work are presented based on the observations in the current study.