The VIMOS Public Extragalactic Redshift Survey (VIPERS): The distinct build-up of dense and normal massive passive galaxies