Dealing with data in Pytorch
torch.utils.data.Dataset: stores the samples and their corresponding labelstorch.utils.data.DataLoader: wraps an iterable around the Dataset to enable easy access to the samples.Source: click here
Source: click here
Image classification
Caltech101 Caltech 101 Dataset. Caltech256 Caltech 256 Dataset.
CelebA
Large-scale CelebFaces Attributes (CelebA) Dataset Dataset.
CIFAR10
CIFAR10 Dataset.
CIFAR100
CIFAR100 Dataset.
Country211( The Country211 Data Set from OpenAI.
DTD
Describable Textures Dataset (DTD).
EMNIST
EMNIST Dataset.
EuroSAT
RGB version of the EuroSAT Dataset.
FakeData
A fake dataset that returns randomly generated images and returns them as PIL images
FashionMNIST(root, train, transform, …)
Fashion-MNIST Dataset.
Image detection or segmentation
CocoDetection
MS Coco Detection Dataset.
CelebA
Large-scale CelebFaces Attributes (CelebA) Dataset Dataset.
Cityscapes
Cityscapes Dataset.
GTSRB
German Traffic Sign Recognition Benchmark (GTSRB) Dataset.
Kitti
KITTI Dataset.
OxfordIIITPet
Oxford-IIIT Pet Dataset.
SBDataset
Semantic Boundaries Dataset
Optical Flow
FlyingChairs
FlyingChairs Dataset for optical flow.
FlyingThings3D
FlyingThings3D dataset for optical flow.
HD1K
HD1K dataset for optical flow.
KittiFlow
KITTI dataset for optical flow (2015).
Sintel
Sintel Dataset for optical flow.
Image pairs
LFWPairs
LFW Dataset.
PhotoTour
Multi-view Stereo Correspondence Dataset.
Image captioning
CocoCaptions
MS Coco Captions Dataset.
Video classification
HMDB51
HMDB51 dataset.
Kinetics(root, frames_per_clip, num_classes, …)
Generic Kinetics dataset
Kinetics400(root, frames_per_clip, …)
Kinetics-400 dataset.
UCF101(root, annotation_path, …)
UCF101 dataset.
Source: click here
Source: click here
The torchvision.datasets module contains Dataset objects for many real-world vision data like CIFAR, COCO
In this tutorial, we use the FashionMNIST dataset.
Every TorchVision Dataset includes two arguments: transform and target_transform to modify the samples and labels respectively.
# # CIFAR10
# # # # Download training data from open datasets.
# training_data = torchvision.datasets.CIFAR10(
# root="data",
# train=True,
# download=True,
# transform=transform,
# )
# # # Download test data from open datasets.
# test_data = torchvision.datasets.CIFAR10(
# root="data",
# train=False,
# download=True,
# transform=transform,
# )# MNIST FASHION
# Download training data from open datasets.
training_data = torchvision.datasets.FashionMNIST(
root="data",
train=True,
download=True,
transform=transform,
)
# Download test data from open datasets.
test_data = torchvision.datasets.FashionMNIST(
root="data",
train=False,
download=True,
transform=transform,
)print("\n TYPE:\n",type(training_data))
print("\n SUMMARY:\n",training_data)
print("\n ATTRIBUTES:\n",training_data.__dict__.keys())
print("\n SHAPE:\n",training_data.data.shape)
print("\n FIRST DATA POINT:\n",training_data.data.shape)
img, label = training_data[0]
print(img.shape,label)
print(len(training_data))
TYPE:
<class 'torchvision.datasets.mnist.FashionMNIST'>
SUMMARY:
Dataset FashionMNIST
Number of datapoints: 60000
Root location: data
Split: Train
StandardTransform
Transform: Compose(
RandomRotation(degrees=[-20.0, 20.0], interpolation=nearest, expand=False, fill=0)
ToTensor()
)
ATTRIBUTES:
dict_keys(['root', 'transform', 'target_transform', 'transforms', 'train', 'data', 'targets'])
SHAPE:
torch.Size([60000, 28, 28])
FIRST DATA POINT:
torch.Size([60000, 28, 28])
torch.Size([1, 28, 28]) 9
60000
Feature batch shape: torch.Size([64, 1, 28, 28])
Labels batch shape: torch.Size([64])

Label: 9

Label: 0

Label: 0

Label: 3

Label: 0
You can also create your own datasets using the provided base classes.
Base Classes
Source: click here
# DATA AUGMENTATION
transform=torchvision.transforms.Compose([
# AUTO AUGMENT
torchvision.transforms.AutoAugment(fill=0),
torchvision.transforms.ToTensor(),
# MANUAL DATA-AUGMENT WITH ROTATION
# torchvision.transforms.RandomRotation((-20,20), expand=False,fill=0),
# torchvision.transforms.ToTensor(),
])
training_data=torchvision.datasets.ImageFolder("./butterflies/train/",transform=transform)
TYPE:
<class 'torchvision.datasets.folder.ImageFolder'>
SUMMARY:
Dataset ImageFolder
Number of datapoints: 338
Root location: ./butterflies/train/
StandardTransform
Transform: Compose(
AutoAugment(policy=AutoAugmentPolicy.IMAGENET, fill=0)
ToTensor()
)
ATTRIBUTES:
dict_keys(['root', 'transform', 'target_transform', 'transforms', 'loader', 'extensions', 'classes', 'class_to_idx', 'samples', 'targets', 'imgs'])
FIRST DATA POINT:
torch.Size([3, 224, 224]) 0
338
Feature batch shape: torch.Size([32, 3, 224, 224])

Label: 1

Label: 1

Label: 0

Label: 2

Label: 1