Stochastic computing (SC) is an attractive computing paradigm because of its fault tolerance and hardware efficiency. Unfortunately, existing SC systems suffer from large latency and inaccuracy problems. In this paper, we propose a new parallel stochastic computing (PSC) system which has higher computing accuracy and faster processing speed compared to existing SC systems. It uses the nibble serial data organization to reduce the latency and a combination of memory based weighted binary generator and data shuffling to improve the accuracy. Simulations on finite impulse response (FIR) filters show that the proposed system with 4 nibbles achieves 35% improvement in accuracy with about 3.5 times increase in processing speed compared to traditional SC. Like SC systems, PSC is also more tolerant of soft errors compared to conventional 2's complement implementations. For a 4-tap FIR filter a 10% injected error causes the root-meansquare- error (RMSE) of PSC to be 0.053 compared to RMSE of 0.188 for 2's complement implementation.