We present FEA-Net as an efficient data driven approach to learn Partial Differential Equation (PDE). Specially designed based on physics prior knowledge, FEA-Net needs less trainable parameters and training data while has certifiable convergence. Moreover, FEA-Net is fully interpretable and we can even infer the physics parameters from it. In this paper, inspired by the local support of Finite Element Analysis (FEA), we will first construct a convolution kernel that is suitable to model PDE. Secondly, inspired by the numerical solvers, we constructed the FEA-Net based on the proposed convolution kernel. Experiment results in predicting elasticity problems show that, FEA-Net is able to outperform purely data driven approaches like Fully Convolutional Networks (FCN) by a large margin on multiple tasks.