Spectrally efficient modulation formats can be used to overcome the problems associated with limited channels and bandwidth of Dense Wavelength Division Multiplexing (DWDM) optical systems. Multilevel modulation formats are considered spectrally efficient and can double the transmission capacity by transmitting more information in the amplitude, phase, polarization or a combination of all. Here we have studied and evaluated different multilevel modulation formats in a practical optical transmission system. Such modulation formats are Differential Quadrature Phase Shift Keying (DQPSK), DQPSK...