Principle

The concept of ZS-DeconvNet is based on the optical imaging forward model informed unsupervised inverse problem solver:

\begin{matrix} (1) & a r g m i n_{θ} | | y - (H \times f_{θ} (y))_{↓} | |_{2}^{2}, \end{matrix}

where $y$ denotes the noisy LR image, $H$ denotes the points spread function (PSF) matrix, and $f_{θ} (y)$ is the deep neural networks (DNNs) with trainable parameters $θ$ . × and $↓$ indicate matrix multiplication and down-sampling operators.

However, DNNs trained directly via the above objective function enhance both useful information of samples and useless noise induced from the acquisition process, e.g., shooting noise, thus their performance degrades rapidly as the SNR of input images declines. To equip ZS-DeconvNet with robustness to noises while maintaining its unsupervised characteristic, we adopted image de-noising schemes and classify ZS-DeconvNet cases as follows:

2D ZS-DeconvNet
3D ZS-DeconvNet
2D ZS-DeconvNet-SIM
3D ZS-DeconvNet-SIM

2D ZS-DeconvNet

The image pairs $(\hat{y}, \tilde{y})$ used for training 2D ZS-DeconvNet models were generated following a modified scheme from the original re-corrupted to re-corrupted strategy[1] under the assumption of mixed Poisson-Gaussian noise distributions, where three hyper parameters $β_{1}, β_{2}, α$ needed to be pre-characterized. The re-corruption procedure from a single noisy image y can be represented in matrix form as:

\begin{matrix} (2) & \hat{y} = y + D^{T} z, \tilde{y} = y - D^{- 1} z, \end{matrix}

where $D = α I$ is an invertible matrix defined as a magnified unit matrix by a factor of $α$ , which controls the overall magnitude of added noises, and $z$ is a random noise map sampled from a Gaussian distribution with zero means:

\begin{matrix} (3) & z \sim N (0, σ^{2} I), σ^{2} = β_{1} H (y - b) + β_{2}, \end{matrix}

where $β_{1}$ is the Poissonian factor affecting the variance of the signal-dependent shot noise, and $β_{2}$ is the Gaussian factor representing the variance of additive Gaussian noises. $b$ is the background, approximately regarded as a fixed value related to the camera, by subtracting which we extracted fluorescence signals from the sample. $H$ is a linear low-pass filter used to preliminarily smooth the image and reduce the noise, and we adopted an averaging filter with a size of 5 pixels in our experiments.

We have proven that the theorectical optimal value for $α$ , $β_{1}$ and $β_{2}$ is 1, 1 and variance of the additive Gaussian noises, respectively. The variance of the additive Gaussian noises is dependent to the camera and can be estimated from the sample-free region of the image itself or pre-calibrated following standard protocols. However, the estimation or calibration may not be accurate, so we generally chose $β_{2}$ with a small random deviation from the estimated value. Meanwhile, both the operations of approximating Poisson distribution with Gaussian distribution and applying an inaccurate intensity map introduce extra uncertainty to $β_{1}$ , therefore we typically adopted random value of $β_{1}$ within a small range, e.g., [0.5, 1.5]. For similar reasons, we randomly adopted $α$ within a small range, e.g., [1,2].

In practice, we use rotation, flipping and cropping to get patches of specified size from the raw data, and implement the above re-corruption process to each patch to generate re-corrupted pairs.

We designed a combined loss function consisting of a denoising term and a deconvolution term, which respectively corresponds to the denoising stage and the deconvolution stage:

\begin{matrix} (4) & L (\hat{y}, \tilde{y}) = μ L_{d e n} (\hat{y}, \tilde{y}) + (1 - μ) L_{d e c} (\hat{y}, \tilde{y}), \end{matrix}

where $(\hat{y}, \tilde{y})$ indicates the re-corrupted image pair mentioned above, and $μ$ is a scalar weighting factor to balance the two terms, which we empirically set as 0.5 in our experiments. The denoising loss $L_{d e n}$ and the deconvolution loss $L_{d e c}$ are defined as follows:

\begin{matrix} (5) & L_{d e n} (\hat{y}, \tilde{y}) = | | f_{θ^{'}} (\hat{y}) - \tilde{y} | |_{2}^{2}, \end{matrix}

\begin{matrix} (6) & L_{d e c} (\hat{y}, \tilde{y}) = | | (f_{θ} (\hat{y}) * P S F)_{↓} - \tilde{y} | |_{2}^{2} + λ R_{H e s s i a n} (f_{θ} (\hat{y})), \end{matrix}

where $f_{θ ‘} (\hat{y}), f_{θ} (\hat{y})$ are the output images of the denoising stage and the deconvolution stage, $R_{H e s s i a n}$ is the Hessian regularization term used to regulating the solution space, and $λ$ is the weighting scalar to balance the impact of the regularization.

Schematic of 2D ZS-DeconvNet

3D ZS-DeconvNet

Similar to the 2D case, we designed a combined loss function consisting of a denoising term and a deconvolution term:

\begin{matrix} (7) & L (z) = μ L_{d e n} (z) + (1 - μ) L_{d e c} (z), \end{matrix}

where $z$ is the 3D noisy image stack, and denoising loss $L_{d e n}$ and the deconvolution loss $L_{d e c}$ are defined as follows[2]:

\begin{matrix} (8) & L_{d e n} (z) = | | f_{θ^{'}} (S_{o d d} (z)) - S_{e v e n} (z) | |_{2} + γ | | f_{θ^{'}} (S_{o d d} (z)) - S_{e v e n} (z) - (S_{o d d} (f_{θ^{'}} (z)) - S_{e v e n} (f_{θ^{'}} (z))) | |_{2}, \end{matrix}

L_{d e c} (z) = | | f_{θ} (S_{o d d} (z)) * P S F - S_{e v e n} (z) | |_{2} + γ | | (f_{θ} (S_{o d d} (z)) * P S F)_{↓} - S_{e v e n} (z) - (S_{o d d} (f_{θ^{'}} (z)) - S_{e v e n} (f_{θ^{'}} (z))) | |_{2}

\begin{matrix} (9) & + λ R_{H e s s i a n} (f_{θ} (S_{o d d} (z))), \end{matrix}

where $S_{o d d}$ and $S_{e v e n}$ represent the axial sampling operators which takes an image stack and returns its odd slices or even slices, respectively, stacked in the same order as the original stack, $γ$ and $λ$ are weighting scalars of the Gap-Amending Regularization term and the Hessian regularization term.

Schematic of 3D ZS-DeconvNet

2D ZS-DeconvNet-SIM

We have proven that the SIM reconstruction noise is of zero mean. This zero-mean characteristics of reconstruction artifacts make it possible to perform denoising and deconvolution for SIM images in an unsupervised manner. In practical implementation of 2D ZS-DeconvNet-SIM, we first added additional noises for each raw SIM images of different orientations and phases, i.e., 3-orientation × 3-phase, via Eq. (2) to generate two sets of recorrupted raw SIM images, and then the generated images were reconstructed into two noisy SR-SIM images, denoted as $\hat{Y}$ and $\tilde{Y}$ , which were used as the input and GT in the training procedure.

For the dual-stage architecture of ZS-DeconvNet-SIM, we set its overall loss function of the same form with Eq. (4), and the denoising loss is calculated with the two recorrupted SIM images:

\begin{matrix} (10) & L_{d e n} (\hat{Y}, \tilde{Y}) = ‖ f_{θ^{'}} (\hat{Y}) - \tilde{Y} ‖_{2}^{2}, \end{matrix}

where $f_{θ^{'}}$ is the denoising stage of ZS-DeconvNet-SIM with corresponding trainable parameters $θ^{'}$ .

Similar to ZS-DeconvNet for acquired raw image processing, we next defined tha deconvolution loss for ZS-DeconvNet-SIM based on recorrupted SIM image pairs and the super-resolution PSF matrix $H_{S I M}$ as

\begin{matrix} (11) & L_{d e c} (\hat{Y}, \tilde{Y}) = ‖ H_{S I M} f_{θ} (\hat{Y}) - \tilde{Y} ‖_{2}^{2} + λ R_{H e s s i a n} (f_{θ} (\hat{Y})), \end{matrix}

where $f_{θ}$ is the entire dual-stage network with all trainable parameters $θ$ .

Schematic of 2D ZS-DeconvNet-SIM

3D ZS-DeconvNet-SIM

The applications of 3D ZS-DeconvNet-SIM for volumetric SIM modalities such as lattice light-sheet structured illumination microscopy (LLS-SIM) and three-dimensional structured illumination microscopy (3D-SIM) are similar to those of 3D ZS-DeconvNet described in Supplementary Note 1b with the primary difference being that 3D ZS-DeconvNet-SIM adopts spatially interleaved post-reconstructed SIM images rather than noisy raw images as inputs and GT in both training and inference phases. The objective function of 3D ZS-DeconvNet-SIM is devised as the combination of the denoising loss and the deconvolution loss, which is formulated as follows

\begin{matrix} (12) & L (Z) = μ L_{d e n} (Z) + (1 - μ) L_{d e c} (Z), \end{matrix}

\begin{matrix} (13) & L_{d e n} (Z) = ‖ f_{θ^{'}} (\hat{Z}) - \tilde{Z} ‖_{2}^{2} + γ ‖ f_{θ^{'}} (\hat{Z}) - \tilde{Z} - (S_{o d d} (f_{θ^{'}} (Z)) - S_{e v e n} (f_{θ^{'}} (Z))) ‖_{2}^{2}, \end{matrix}

\begin{matrix} (14) & L_{d e c} (Z) = ‖ H_{S I M} f_{θ} (\hat{Z}) - \tilde{Z} ‖_{2}^{2} + γ ‖ H_{S I M} f_{θ} (\hat{Z}) - \tilde{Z} - S_{o d d} (f_{θ^{'}} (Z)) + S_{e v e n} (f_{θ^{'}} (Z)) ‖_{2}^{2} + λ R_{H e s s i a n} (f_{θ} (\hat{Z})), \end{matrix}

where $Z$ , $\hat{Z}$ , and $\tilde{Z}$ are the entire stack, odd slices, and even slices of the noisy SIM image stack generated via the analytical SIM reconstruction algorithm, $H_{S I M}$ is the volumetric PSF of corresponding SIM systems.

Schematic of 3D ZS-DeconvNet-SIM

[1] Pang, T., Zheng, H., Quan, Y. & Ji, H. in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2043-2052 (2021).

[2] Qiao, C. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes. Nature Biotechnology (2022).