Sequential Data for Deep Learning

Deep learning models for sequences (RNNs, LSTMs, GRUs, 1D CNNs, Transformers) almost always expect a rank-3 tensor:

(samples, timesteps, features)

The whole job of data preparation is turning raw sequential data into that shape. This notebook does it twice:

Everything is a tiny toy example so the shapes stay readable.

Internal correlation

The importance of auto or cross-correlation in data

Temporal Discretization

Basic Univariate Time Series

Sequential Data vs. the IID Assumption

IID Data

  • Samples are independent and identically distributed
  • Rows are exchangeable: shuffle them and nothing is lost
  • Order of rows carries no information
  • Random train/test splits are valid
  • Example: 1,000 patients, one row each

Sequential Data

  • Samples have an intrinsic order (time, token position)
  • Rows are not exchangeable: shuffling destroys the signal
  • Position of a row is information
  • Splits must respect the arrow of time
  • Example: 1,000 days of one patient’s vitals

The key difference

  • Sequential data has correlations along the sample axis itself: \(x_t\) is predictive of \(x_{t+1}\)
  • Under IID, that axis is just a bookkeeping index; under sequence models, it is the axis being modeled
  • Consequences:
    • We reshape into (samples, timesteps, features) so the order axis becomes explicit
    • Overlapping windows make samples correlated with each other → random CV leaks
    • Architectures (RNN, CNN, Transformer) exist precisely to exploit this dependence

Recall: static correlation

Auto-correlation

Auto-correlation Internal correlation (self correlation)

Examples: Internal Auto-correlation (self correlation)

Internal Correlations → Free Labels

A magical idea: \(X \rightarrow X\) type learning

The observation

  • If \(x_t\) depends on \(x_{<t}\), then part of the data predicts the rest of the data
  • Yesterday's features can predict today's features

Autoregression

  • Model \(\hat{x}_t = f(x_{t-W:t-1})\): the window is the input, the next step is the target
  • The label was already sitting in the sequence: we just slid a boundary through it

Self-supervision

  • No human annotation needed: every position is a training example
  • \(N\) timesteps \(\Rightarrow \sim N\) supervised pairs, for free
  • Under IID this is impossible: \(x_i\) says nothing about \(x_j\), so there is nothing to predict from

Why it matters

  • Unlabeled text and unlabeled sensor logs are essentially unlimited
  • Pretraining = exploiting internal correlations at scale; the “label” is the data itself

Example: Auto-regression (Self Regression)

Self-Supervised Learning (SSL)

Self supervised learning s sort of fits between supervised and unsupervised learning because it doesn’t require an explicit human supervised

Honorable mention: The auto vanhilla encoder

While not a time series model. The auto encoder is a textbook example of self supervision where the input and target pair X and Y come directly from the inputs themselves. There’s no need for human supervision.

Honorable mention-2: The variational vanhilla encoder

  • The General idea is to build the latent space in a meaningful (structured) way. This is done by forcing latent space points to be sampled from a specified distribution

Time Series Data

Examples of time series data

Motivating example: Univariate time series

A univariate time series

A univariate series is one number per timestep. Here: 12 days of (fake) temperature readings.

  • Raw shape is (12,): a flat vector
  • There is 1 feature (temperature) measured at 12 timesteps
  • It is 1 long sequence (chunk of a series), not 12 samples
temp: [20 22 25 24 21 19 18 20 23 26 28 27]
shape: (12,)

“Looking back” two timesteps

Chunking with a sliding window (looking back three timesteps)

A model cannot learn from one long line. We cut it into many overlapping windows:

  • window size W: how many past timesteps the model sees (here W = 3)
  • horizon H: how far ahead we predict (here H = 1, the very next day)
  • Each window becomes one sample; the value right after it becomes the target

Slide the window forward one step at a time to generate every sample.

sample 0: X = [20 22 25]  ->  y = 24
sample 1: X = [22 25 24]  ->  y = 21
sample 2: X = [25 24 21]  ->  y = 19
sample 3: X = [24 21 19]  ->  y = 18
sample 4: X = [21 19 18]  ->  y = 20
sample 5: X = [19 18 20]  ->  y = 23
sample 6: X = [18 20 23]  ->  y = 26
sample 7: X = [20 23 26]  ->  y = 28
sample 8: X = [23 26 28]  ->  y = 27

X shape: (9, 3)  (samples, timesteps)
y shape: (9,)

Each row is one sample: blue cells are the window (inputs), orange is the target.

Adding the feature axis

X_uni is (9, 3): but the model wants (samples, timesteps, features). A univariate series has exactly one feature, so we add a trailing axis of size 1.

  • (9, 3) → (9, 3, 1)
  • No data changes; we are only telling the model “each timestep carries 1 number”
  • This is the single most common shape bug in sequence modeling
before: (9, 3) -> after: (9, 3, 1)

sample 0 as a (timesteps, features) matrix:
[[20]
 [22]
 [25]]

Adding more features: Two feature case

Example: A trivariate time series

Now each timestep carries three measurements: temperature, humidity, wind.

  • Raw shape (10, 3) = (timesteps, features)
  • The features axis is last: one column per variable
  • Windowing works exactly as before; only the feature axis grows from 1 to 3
series shape: (10, 3) (timesteps, features)
[[20 65  5]
 [22 60  7]
 [25 55  6]
 [24 58  4]
 [21 70  9]
 [19 75 12]
 [18 80 11]
 [20 72  8]
 [23 64  6]
 [26 58  5]]

Chunk in time and self-supervision

look-back three previous days

X shape: (7, 3, 3)  (samples, timesteps, features)
y shape: (7,)

sample 0 (rows = timesteps, cols = temp/humid/wind):
[[20 65  5]
 [22 60  7]
 [25 55  6]]

target 0: 24

Each sample is now a matrix, not a vector: 3 rows (time) by 3 columns (features). Stacking those matrices gives the 3D tensor.

Takeaway: the univariate and trivariate cases are the same code. Only the size of the last axis changes.

case raw data one sample tensor
univariate (12,) (3, 1) (9, 3, 1)
trivariate (10, 3) (3, 3) (7, 3, 3)

Terminology

Splitting Time Series: Holdout vs. Walk-Forward

The one rule

  • Train precedes validation precedes test. Never shuffle.
  • Random splits leak the future into the past: overlapping windows make it worse

Cross-validation for time series

Motivation

Standard k-fold cross-validation shuffles samples. That is invalid here:

  • Windows overlap, so shuffling leaks nearly identical rows into train and test
  • Training on the future to predict the past is not a test you can run in production

Temporal Holdout (fixed-origin)

  • One split point: three contiguous blocks (e.g. 50 / 25 / 25)
  • Model fit once → a single performance estimate
  • The deep learning default: refitting a network many times is expensive
  • Works well when the series is long and each block is representative

Cross-validation for time series

Note, similar to regular k-fold, we are training multiple models here. One per fold, and the aggregate scores a hyperparameter configuration rather than any individual model.

Walk-Forward CV (rolling-origin)

  • Many folds: train on the past, validate on the block immediately after, slide forward
  • Expanding window (grow the training set) or rolling window (keep it fixed-size)
  • Model refit per fold → a distribution of scores across time
  • Earns its cost when data is short or performance may drift across regimes

The simplest correct version is walk-forward (expanding window) validation:

  • Always train on an initial block, validate on the block that comes immediately after
  • Then grow the training block and slide forward
  • Every fold respects the arrow of time
fold 0: train = [np.int64(0), np.int64(1), np.int64(2), np.int64(3)]   val = [np.int64(4), np.int64(5)]
fold 1: train = [np.int64(0), np.int64(1), np.int64(2), np.int64(3), np.int64(4), np.int64(5)]   val = [np.int64(6), np.int64(7)]

Basic AR example-1

Basic AR example-2

Note that we could also use a more complicated curve fitting model here, such as a multi-layer perceptron.

Basic VAR example

This that we flattened the vector here.

Text and NLP data

NLP Data

Text is also a sequence: timesteps are token positions instead of clock ticks. The target shape is identical:

(samples, timesteps, features)
    |          |          |
 sentences   words    word representation

The only new problem: a word is not a number. We need to invent the feature axis.

Corpus, tokenization, vocabulary

  • Corpus: our tiny dataset: 4 sentences (4 samples)
  • Tokenization: split each sentence into tokens (here, lowercase words)
  • Vocabulary: the sorted set of unique tokens, each assigned an integer id

Two reserved ids come first by convention:

  • <pad> = 0: filler so all sequences have equal length
  • <unk> = 1: any word not seen during training
sentence 0: ['the', 'cat', 'sat', 'on', 'the', 'mat']
sentence 1: ['the', 'dog', 'sat', 'on', 'the', 'log']
sentence 2: ['a', 'cat', 'chased', 'the', 'dog']
sentence 3: ['the', 'dog', 'barked']

vocabulary size V = 12
{'<pad>': 0, '<unk>': 1, 'a': 2, 'barked': 3, 'cat': 4, 'chased': 5, 'dog': 6, 'log': 7, 'mat': 8, 'on': 9, 'sat': 10, 'the': 11}

Tokens

Integer encoding and padding

  • Map each token to its id → a list of ints per sentence
  • Sentences have different lengths, but tensors must be rectangular
  • Pad short sentences with 0 (and truncate long ones) to a fixed T

Result is (samples, timesteps): integers, not yet features.

seqs shape: (4, 6)  (samples, timesteps)
[[11  4 10  9 11  8]
 [11  6 10  9 11  7]
 [ 2  4  5 11  6  0]
 [11  6  3  0  0  0]]

These integers are ids, not magnitudes. cat = 4 and dog = 6 does not mean a dog is 1.5 cats. Feeding raw ids to a model implies an ordering that does not exist, so we replace each id with a vector. Two ways to do that follow.

Stemming and lemmatization

Shallow methods often simplify the vocabulary dramatically by truncating or simplifying words.

This is typically not done for deep models because they can handle the context, Do there increase model flexibility

Stemming and lemmatization

Classical NLP pipelines shrink the vocabulary by normalizing word forms before encoding:

  • Stemming: chop affixes with crude rules; the result need not be a real word (running → run, studies → studi)
  • Lemmatization: map to the dictionary form using morphology and part of speech (better → good, ran → run)

Why it used to matter:

  • Smaller V means smaller one-hot vectors and fewer embedding rows
  • With little data, run / running / ran each get too few examples to learn from separately
     word |    stem |   lemma
  running |     run |     run
     runs |     run |     run
      ran |     ran |     run
  studies |   studi |   study
     cats |     cat |    cats
   barked |    bark |  barked
   better |  better |    good

Bag of words: throwing the sequence away

Bag of words (BoW) counts the words in a sentence and forgets where they were.

  • Sum the one-hot vectors over the time axis
  • Shape collapses from (samples, timesteps, V) to (samples, V): rank 2, no time axis
  • Cheap, strong baseline for topic-style tasks (spam, sentiment)
  • But word order carries meaning, and BoW destroys it

The classic failure case: two sentences with opposite meanings produce the identical BoW vector.

Modern practice mostly drops these steps. Contemporary models rarely stem or lemmatize because:

  • Subword tokenization (BPE, WordPiece, SentencePiece) already splits running into pieces like run + ##ning, so related forms share representation without any hand-written rules
  • Capacity and data: large models see enough text to learn on their own that run, ran, and running are related, and they place those embeddings near each other
  • Information loss: stemming is lossy and error-prone (studies → studi), and tense/number/aspect often matter for the task
  • Modern architectures are complex enough to handle the surface variation directly, so normalizing it away throws away signal for no gain

One-hot encoding

  • Each word becomes a vector of length V with a single 1 in its own slot
  • Shape becomes (samples, timesteps, V): the feature axis is the whole vocabulary

Properties:

  • Very sparse: one nonzero out of V
  • Grows with vocabulary: a real 50,000-word vocab means 50,000 numbers per token
  • No similarity structure: every pair of distinct words is equally far apart
onehot shape: (4, 6, 12)  (samples, timesteps, features=V)

sentence 0, timestep 1 -> cat
[0 0 0 0 1 0 0 0 0 0 0 0]

nonzeros: 6 out of 72 cells

Example learning task: Binary text classification

Dot product

Cosine similarity between any two different one-hot words is exactly 0: “cat” is as unrelated to “dog” as it is to “on”.

cos(cat, dog) = 0.0
cos(cat, on ) = 0.0
cos(cat, cat) = 1.0

Dense vector representation (embeddings)

Replace the length-V sparse vector with a short dense vector of length d (d << V).

  • An embedding matrix E has shape (V, d): one row per vocabulary word
  • Lookup is just row indexing: E[seqs] → (samples, timesteps, d)
  • Values are learned during training, so similar words end up near each other

Below d = 2 and the vectors are handcrafted so the geometry is visible. Real models use d in the 100s and learn E by gradient descent.

E shape: (12, 2)  (vocab, embedding_dim)

embedded shape: (4, 6, 2)  (samples, timesteps, features=d)

sentence 0:
  t0     the  ->  [-0.9 -0.8]
  t1     cat  ->  [0.9 0.8]
  t2     sat  ->  [-0.9  0.7]
  t3      on  ->  [-0.8 -0.9]
  t4     the  ->  [-0.9 -0.8]
  t5     mat  ->  [ 0.9 -0.7]

One-hot and embedding are the same operation. A one-hot row times E selects a row of E, so an embedding layer is a matrix multiply that we implement as a lookup for speed.

onehot @ E == E[seqs]

identical: True

feature-axis size:  one-hot = 12  vs  dense = 2

Summary

representation shape time axis? notes
raw univariate series (12,) implicit one long sequence
windowed univariate (9, 3, 1) yes feature axis added by hand
raw trivariate series (10, 3) implicit features in last column
windowed trivariate (7, 3, 3) yes each sample is a matrix
integer sequences (4, 6) yes ids only, not usable features
one-hot text (4, 6, 12) yes sparse, no similarity
embedded text (4, 6, 2) yes dense, learned, similarity-aware
bag of words (4, 12) no order destroyed

The recurring recipe:

  • Decide what one sample is (a window of time, or a sentence)
  • Decide what one timestep is (a day, or a token)
  • Decide what features describe a timestep (a measurement, or a word vector)
  • Stack the samples and check X.shape before it reaches the model
  • Never shuffle time when splitting a time series