Generalized Analytical Models of Filtering Penalties and Equalization in Coherent Optical Links

This paper presents a unified analytical framework for filtering-penalty assessment and receiver equalization in coherent optical links dominated by linear impairments. The formulation targets scalar amplitude filtering, under the assumption of negligible phase and polarization effects, and jointly accounts for cascaded optical filters, amplified spontaneous emission (ASE) noise, receiver-side transceiver impairments, anti-aliasing filtering, and digital equalization. Closed-form post-equalization signal-to-noise ratio (SNR) expressions are derived for the zero-forcing equalizer (ZFE), minimum mean-square error equalizer (MMSEE), and fractionally spaced equalizer (FSE), while semi-analytical matrix formulations are provided for the whitened finite-length equalizer (WFLE) and finite-length equalizer (FLE). The models include the colored noise induced by distributed ASE injection and optical filtering, and are generalized to arbitrary normalized transceiver-noise power spectral densities (PSDs). The resulting post-equalization SNR and Q-factor metrics are suitable for optical network digital twin (ONDT) applications and for measurement-based characterization of commercial transceivers. Time-domain simulations and experiments with two commercial transceivers confirm the accuracy of the proposed approach and support its applicability to filtering-aware performance estimation in coherent optical links.