Effects of modal density in system identification using the Hilbert transform