Physical Unclonable Functions (PUF) have to be highly reliable especially when it is being used along with cryptographic hash modules for key generation. To achieve ultrahigh reliability, the conventional approach employs error correction codes (ECC) based on helper data input. Such an approach not only increases the hardware overhead of the PUF but also reduces the entropy of the system, resulting in both hardware and software security issues. In this paper we design a compact and highly reliable PUF architecture based on resistive random access memory (RRAM). We propose a new design where the sum of the read-out currents of multiple RRAM cells is used for generating one response bit. This method statistically minimizes any early-lifetime failure due to RRAM retention degradation at high temperature or under voltage stress. We employ a device model that is calibrated with IMEC HfOx RRAM experimental data and show that with 8 cells per bit, we can ensure 99.9999% reliability) for a lifetime >10 years at 125°C. We embed the RRAM PUF into SHA-256 and show that the hardware overhead of the proposed RRAM PUF based architecture is significantly lower than one that uses a traditional RRAM PUF with ECC.