Schematics of the reconstruction approach. (A) Model training. We use
an adversarial training strategy adopted from Dosovitskiy and Brox (2016b), which
consists of 3 DNNs: a generator, a comparator, and a discriminator. The training images
are presented to a human subject, while brain activity is measured by fMRI. The fMRI
activity is used as an input to the generator. The generator is trained to reconstruct the
images from the fMRI activity to be as similar to the presented training images in both
pixel and feature space. The adversarial loss constrains the generator to generate
reconstructed images that fool the discriminator to classify them as the true training
images. The discriminator is trained to distinguish between the reconstructed image and
the true training image. The comparator is a pre-trained DNN, which was trained to
recognize the object in natural images. Both the reconstructed and true training images
are used as an input to the comparator, which compares the image similarity in feature
space. (B) Model test. In the test phase, the images are reconstructed by providing the
fMRI activity of the test image as the input to the generator. (Shen et al, 2018)