Tutorial

Our source code and Fiji plugin can be downloaded from https://github.com/TristaZeng/ZS-DeconvNet. We use MATLAB R2021b to generate training datasets from raw images, and TensorFlow to perform training and inference. We also develop a Fiji plugin for both data generation and model training and inference.

This page provides quick-start tutorials for MATLAB, Python and Fiji. For more detailed instructions see the ReadMe.md in our Github repository. An overlook of their respective functions are summarized in the table below:

Content

• Brief description of parameters
• MATLAB: How to generate training dataset and PSFs
• Python: How to perform training and inference
• Fiji: How to do all above with a click

Brief description of parameters

ZS-DeconvNet is a deep-learning method for denoising and super-resolution with no need for users to traverse through hyper-parameters. Basically you just make sure that ZS-DeconvNet models are trained using matched PSFs (otherwise the improper training with mismatched PSFs might result in lack of resolution improvement or periodically artifacts), then you can skip this part without inhibiting the using of ZS-DeconvNet. Apart from the typical hyper-parameters such as learning rate, total epoch, batch size and patch size, there are a few important inputs during training stage:

MATLAB: How to generate training dataset and PSFs

We use MATLAB R2021b but other versions might be compatible.

1.1 To generate training dataset, you can:

• Prepare a folder of raw data. Download our open-source raw data of various modalities or use your own raw data.

• Open ./Python_MATLAB_Codes/data_augment_recorrupt_matlab/GenData4ZS-DeconvNet/demo_augm.m and replace the parameter data_folder with your raw data directory.

• The default output path is ./Python_MATLAB_Codes/data_augment_recorrupt_matlab/your_augmented_datasets/.

1.2 To generate training dataset for ZS-DeconvNet-SIM, you can:

• Prepare a folder of raw data.

• Open ./Python_MATLAB_Codes/data_augment_recorrupt_matlab/GenData4ZS-DeconvNet-SIM/Create_corrupt_img_2D.m or Create_corrupt_img_3D.m and set the parameters.

• Perform SIM reconstructions on the corrupted dataset to generate the input and targets for ZS-DeconvNet-SIM.

2.1 To simulate PSF for (3D) ZS-DeconvNet, run ./Python_MATLAB_Codes/data_augment_recorrupt_matlab/GenData4ZS-DeconvNet/create_PSF.m and the simulated PSF will be saved to your designated file path.

2.2 To simulate PSF for ZS-DeconvNet-SIM, run ./Python_MATLAB_Codes/data_augment_recorrupt_matlab/GenData4ZS-DeconvNet-SIM/main_create_simu_beads.m and the generated raw 3D-SIM images of a simulated bead img_sim will be saved to your MATLAB workspace.

Python: How to perform training and inference

4.1 Building environment

Our environment is:

• Windows 10
• Python 3.9.7
• Tensorflow-gpu 2.5.0
• NVIDIA GPU (GeForce RTX 3090) + CUDA (11.4)

To use our code, you should create a virtual environment and install the required packages first.

$ conda create -n zs-deconvnet python=3.9.7
$ conda activate zs-deconvnet
$ pip install -r requirements.txt

After that, remember to install the right version of CUDA and cuDNN, if you want to use GPU. You can get the compatible version by searching

$ conda search cudatoolkit --info
$ conda search cudnn --info

then installing

$ conda install cudatoolkit==xx.x.x
$ conda install cudnn==x.x.x

4.2 Training Demo

Skip this part if you do not wish to train a new model. You can just test the demo test data using our provided pre-trained models.

To train a new model, you need to:

• Generate the training dataset following the instructions in the previous part.
• Choose a test image/volume and obtain the path to the corresponding PSF.
• Change otf_or_psf_path (or psf_path in the case of 3D), data_dir, folder and test_images_path in ./Python_MATLAB_Codes/train_inference_python/train_demo_2D.sh or train_demo_3D.sh.
• Run it in your terminal.
• The result wills be saved to ./Python_MATLAB_Codes/your_saved_models/.
• Run tensorboard --logdir [save_weights_dir]/[save_weights_name]/graph to monitor the training process via tensorboard if needed.
• Other detailed description of each input argument of the python codes can be found in the comments of ./Python_MATLAB_Codes/train_inference_python/train_demo_2D.sh or train_demo_3D.sh.

4.3 Inference Demo

We have provided a download link in the folder ./Python_MATLAB_Codes/saved_models/, which contains six different data samples, their corresponding well-trained ZS-DeconvNet models, and their ZS-DeconvNet outputs, organized in the format of:

• ./Python_MATLAB_Codes/saved_models/xx/test_data contains the test data of this sample type.
• ./Python_MATLAB_Codes/saved_models/xx/saved_model contains the pre-trained model (a .h5 file) of this sample type.
• ./Python_MATLAB_Codes/saved_models/xx/saved_model/Inference_demo contains the ZS-DeconvNet output of this sample type. You should be able to get the same result if you use the pre-trained model to process the test data in ./Python_MATLAB_Codes/saved_models/xx/test_data.

Now to test a well-trained ZS-DeconvNet model, you should:

• Change the weight paths in ./Python_MATLAB_Codes/train_inference_python/infer_demo_2D.sh or infer_demo_3D.sh accordingly, or just use the default options given by us.
• Run it in your terminal.
• The output will be saved to the folder where you load weights, e.g., if you load weights from ./Python_MATLAB_Codes/train_inference_python/saved_models/.../weights_40000.h5, then the output will be saved to ./Python_MATLAB_Codes/train_inference_python/saved_models/.../Inference/.

Fiji: How to do all above with a click

We provide a simple instruction for our plugin here. For latest updates, detailed parameter table and snapshots of usage, see the ReadMe.md in our Github repository.

Here is a demo video of screen captures of the training and inference procedures of the ZS-DeconvNet Fiji plugin on two different low SNR dataset with two types of operating systems (Windows 10 and Ubuntu 18.04.6 LTS):

5.1 Installation and Pre-trained Models

You can follow the instructions below to install the plugin:

• Download from here.

• Copy ./jars/* and ./plugins/* to your root path of Fiji /*/Fiji.app/.

• Restart Fiji.

• We provide pre-trained models in BioImage Model Zoo bundle and one corresponding test image or test stack for each model in the folder ./pre-trained_models. See the list below:

We mainly developed and tested the ZS-DeconvNet Fiji plugin on workstations of Linux and Windows operating system equipped with Nvidia graphics cards. Because TensorFlow-GPU package is currently incompatible with MacOS, we are sorry that MacBook users can only use the TensorFlow-CPU to run our ZS-DeconvNet Fiji plugin at present, which is relatively inefficient compared to Nvidia GPU-based computation. We’ll be looking for the solutions and trying to make our plugin compatible with MacBook for higher efficiency in the future.

5.2 About GPU and TensorFlow version

The ZS-DeconvNet Fiji plugin was developed based on TensorFlow-Java 1.15.0, which is compatible with CUDA version of 10.1 and cuDNN version of 7.5.1. If you would like to process models with a different TensorFlow version, or running with different GPU settings, please do the following:

• Open Edit > Options > Tensorflow, and choose the version matching your model or setting.
• Wait until a message pops up telling you that the library was installed.
• Restart Fiji.

5.3 Inference with ZS-DeconvNet Fiji plugin

Given a pre-trained ZS-DeconvNet model and an image or stack to be processed, the Fiji plugin is able to generate the corresponding denoised (optional) and super-resolved deconvolution image or stack. The workflow includes following steps:

• Open the image or stack in Fiji and start ZS-DeconvNet plugin by Clicking Plugins > ZS-DeconvNet > predict ZS-DeconvNet 2D / predict ZS-DeconvNet 3D.

• Select the network model file, i.e., .zip file in the format of BioImage Model Zoo bundle. Of note, the model file could be trained and saved either by Python codes (see this gist) or ZS-DeconvNet Fiji plugin, but has to be saved with TensorFlow environment <= 1.15.0.

• Adjust inference options if you like.

• After image processing with status bar shown in the message box (if select Show process dialog), the denoised (if select Show denoising result) and deconvolved output will pop out in separate Fiji windows automatically. Then the processed images or stacks could be viewed, manipulated, and saved via Fiji.

5.4 Training with ZS-DeconvNet Fiji plugin

For ZS-DeconvNet model training, we generally provide two commands: train on augmented data and train on opened img, which differ in the ways of data loading and augmentation. The former command loads input data and corresponding GT images which are augmented elsewhere, e.g., in MATLAB or Python, from two data folders file by file, and the latter command directly takes the image stack opened in the current Fiji window as the training data and automatically perform data augmentation.

The overall workflow of ZS-DeconvNet training with Fiji plugin includes following steps:

• Open the image or stack to be used for training in Fiji and start the ZS-DeconvNet plugin by clicking Plugins > ZS-DeconvNet > train on opened img; or directly start the plugin by the alternative command Plugins > ZS-DeconvNet > train on augmented data and select the folders containing input images and GT images.

• Select the network type, i.e., 2D ZS-DeconvNet or 3D ZS-DeconvNet and the PSF file.

• Adjust other options if you like, or do not.

• Click OK to start training. A message box containing training information will pop up, and three preview windows will be displayed after each epoch, showing the current input images, denoised output images and deconvolution output images.

• Three types of exit:
  (i) Press Cancel > Close to enforce an exit if you don’t want to train or save this model.
  (ii) Press Finish Training for an early stopping. A window will pop up and you can save the model by File actions > Save to...
  (iii) After the training is completed, a window will pop up and you can save the model by File actions > Save to...

  Of note, you can also press Export Model during training to export the lastest model without disposing the training progress.

5.5 PSF Generation

You can generate PSF via the Fiji plugin PSF Generator.