Recent real-world attacks against Certification Authorities (CAs) and fraudulently issued certificates arouse the public to rethink the security of public key infrastructure for web-based connections. To distribute the trust of CAs, notaries, as an independent party, are introduced to record certificates, and a client can request an audit proof of certificates from notaries directly. However, there are two challenges. On one hand, existing works consider the security of notaries insufficiently. Due to lack of systematic mutual verification, notaries might bring safety bottlenecks. On the other hand, the service of these works is not sustainable, when any party leaks its private key or fails. In this paper, we propose a Tripartite Public Key Infrastructure (TriPKI), using Certificates Authorities, Integrity Log Servers, and Domain Name Servers, to provide a basis for establishing secure SSL/TLS connections. Specifically, we apply checks-and balances among those three parties in the structure to make them verify mutually, which avoids any single party compromise. Furthermore, we design a collaborative certificate management scheme to provide sustainable services. The security analysis and experiment results demonstrate that our scheme is suitable for practical usage with moderate overhead.