detail question - spli

[gilliM]

Is it normal that you reference the same file for train_data and val_data at line

earthPT/src/train.py

Line 142 in d8ceeb0

np.load(os.path.join(data_dir, f'TL{i:02d}_train_sar_float16.npy'), mmap_mode="r+")

?

and also a view lines later, train_data array is reused ?

    if split == 'train':
        data = >>>>train_data<<< [np.random.randint(len(train_data))]
        ii = torch.randint(0, data.shape[0] - 10000, (batch_size,))
    else:
        data = >>>train_data<<< [np.random.randint(len(val_data))]

[Smith42]

Here we take the final 10000 timeseries in a preprocessed data array as our validation set (i.e. torch.randint(data.shape[0] - 10000, data.shape[0], (batch_size,)) as below), and the rest are used as our training set (i.e. torch.randint(0, data.shape[0] - 10000, (batch_size,)) as shown below and in your snippet).

def get_batch(split):
        if split == 'train':
            data = train_data[np.random.randint(len(train_data))]
            ii = torch.randint(0, data.shape[0] - 10000, (batch_size,))
        else:
            data = train_data[np.random.randint(len(val_data))]
            ii = torch.randint(data.shape[0] - 10000, data.shape[0], (batch_size,))

Does that clear things up? We are certainly not mixing the training and validation sets here!