Fluorescence imaging using voltage-sensitive dyes is an important tool for studying electrical propagation in the heart. Yet, the low amplitude of the voltage-sensitive component in the fluorescence signal and high acquisition rates dictated by the rapid propagation of the excitation wave front make it difficult to achieve recordings with high signal-to-noise ratios. While spatially and temporally filtering the acquired signals has become de-facto one of the key elements of optical mapping, there is no consensus regarding their use. Here we characterize the spatio-temporal spectra of optically recorded action potentials, and determine the distortion produced by conical filters of different sizes. Based on these findings we formulate the criteria for rational selection of filter characteristics. We studied the evolution of the spatial spectra of the propagating wave front after epicardial point stimulation of the isolated, perfused right ventricular free wall of the pig heart stained with di-4-ANEPPS. We found that short wavelength (<3 mm) spectral components represent primarily noise and surface features of the preparation (coronary vessels, fat, and connective tissue). High frequency components (> 100 Hz) also lack in the time domain of the optical action potential spectrum. Both findings are consistent with the reported effect of intrinsic blurring caused by light scattering inside the myocardial wall. The absence of high frequency spectral components allows the use of aggressive low-pass spatial and temporal filters without affecting the optical action potential morphology. We show examples where the signal-to-noise ratio increased up to 150 with less than 3% distortion. A generalization of our approach to the rational filter selection in various applications is discussed.