Photonic crystals have shown a great deal of promise for the realization of true integrated optics. Waveguides with small bends may be formed allowing compact integrated photonic circuits to be formed. Full threedimensional (3D) photonic simulations are required in order to realize very low loss, integrated photonic crystal circuits. Needless to say, the design and fabrication of such fully 3D structures is challenging, and thus efficient simulation tools are necessary to identify the optimum structures for different applications. Researchers at the Department of Defense (DoD) and Arizona State University (ASU) have independently developed parallel Finite Difference Time Domain (FDTD) codes, with the goal of scaling up each simulator for complicated structures such as 3D optical integrated circuits (OIC). As the name implies, FDTD is a popular time-domain method for solving Maxwell's equations for the electric and magnetic fields. These two curl equations are solved explicitly in time over half-step intervals, where the values of one set of field values (e.g., electric fields) are used at the successive interval to solve for the other field (e.g., magnetic field) in a time marching fashion. In previously reported work, a fully parallel FDTD code scalable to 107 FDTD grid points was presented and showed good scalability up to 200 processors. The goal of our current work has been to realize a more optimized and efficient implementation requiring significantly less memory and scalable to thousands of processors. This requires both a scalable parallel FDTD code, as well as one with improved absorbing boundary conditions and more efficient algorithms to reduce runtime. The work and results discussed herein address both the scalability and the efficiency of the time-domain algorithm.