Laser-Based Additive Manufacturing (LBAM) provides unprecedented possibilities to produce complicated parts with multiple functions for lots of engineering applications. Melt pool is one of the most important signatures in LBAM and is indicative of process anomalies and part defects. High-speed temperature images of the melt pool captured during LBAM make it possible for in-situ melt pool monitoring and porosity prediction. This paper aims to broaden current knowledge of the underlying relationship between process and porosity in LBAM and provide new possibilities for efficient and accurate porosity prediction. We present a deep learning-based data fusion method to predict porosity in LBAM parts by leveraging the measured melt pool thermal history and deep learning. A PyroNet, based on Convolutional Neural Networks, is developed to correlate in-process pyrometer images with layer-wise porosity; an IRNet, based on Long-term Recurrent Convolutional Networks, is developed to correlate sequential thermal images from infrared camera with layer-wise porosity. Predictions from PyroNet and IRNet are fused at the decision level to obtain a more accurate prediction of layer-wise porosity. The model fidelity is validated with LBAM Ti-6Al-4V thin-wall structure. Our method can achieve 100% accuracy with high efficiency, allowing the method to be applicable for in-situ porosity detection in LBAM.