Recurrent Neural Networks for Sequential EHR Data

"The past informs the present — in medicine, a patient's history is the key to their future."

August 2026 · Giacomo Saccaggi

Why RNNs for Healthcare?

Patient medical records are inherently sequential. A patient doesn't arrive at the hospital with a single snapshot of conditions — they have a trajectory: a series of visits over months or years, each containing diagnoses, procedures, medications, and lab results. The order matters: developing hypertension before diabetes tells a different story than the reverse.

In Part 1, we learned to represent medical codes as dense embeddings. In Part 2, we built feedforward networks that take a patient's aggregated medical history and predict outcomes. But there's a fundamental limitation: feedforward networks ignore temporal structure.

Feedforward networks treat visits as a bag — they can't distinguish "diabetes → kidney disease" from "kidney disease → diabetes"

Consider predicting heart failure risk. A feedforward network sees:

• Patient has: hypertension, diabetes, obesity, coronary artery disease

But it doesn't see:

• Visit 1 (2019): Obesity diagnosed
• Visit 2 (2020): Hypertension developed
• Visit 3 (2021): Diabetes diagnosed
• Visit 4 (2022): Coronary artery disease
• Visit 5 (2023): ???

The progression pattern — metabolic syndrome evolving into cardiovascular disease — is invisible to a feedforward network. We need a model that can remember what happened at previous time steps and use that memory to inform predictions.

Enter Recurrent Neural Networks.

Basic RNN Architecture

The key idea of RNNs: maintain a hidden state that gets updated at each time step and carries information forward through the sequence.

RNN maintains hidden state h(t) that carries information across time steps

The Recurrence Relation

At each time step t, the RNN:

1. Takes the current input x(t) and the previous hidden state h(t-1)
2. Computes a new hidden state h(t)
3. Optionally produces an output o(t)

The critical insight: the same weights (U, W, V) are shared across all time steps. This is what makes it "recurrent" — the same transformation is applied repeatedly, with the hidden state carrying forward information from previous steps.

RNN Architectures: Many-to-What?

RNNs are flexible — different tasks require different input/output configurations:

For heart failure prediction, we'll use many-to-one: process all visits, then use the final hidden state for classification.

Forward Computation in Detail

Let's trace through the forward pass step by step. Consider a patient with 3 visits:

# Patient visits (each encoded as embedding vector)
# Visit 1: [Diabetes] → x₁
# Visit 2: [Diabetes, Hypertension] → x₂
# Visit 3: [Diabetes, Hypertension, CKD] → x₃

Step-by-Step Forward Pass

Loss Function

For sequence classification, we compute loss only at the final time step:

For sequence prediction (predict at every time step):

Backpropagation Through Time (BPTT)

Training RNNs requires computing gradients that flow backward through time. This is called Backpropagation Through Time (BPTT).

Gradients flow backward through the unrolled computation graph

The Gradient Flow

To update weights, we need ∂L/∂W, ∂L/∂U, and ∂L/∂V. The key challenge: W affects h(t) at every time step, so its gradient accumulates contributions from all steps.

The crucial term is ∂h(t+1)/∂h(t) — how much the hidden state at t+1 depends on the hidden state at t. This involves the recurrent weight W and the activation function derivative:

The Vanishing Gradient Problem

Here's where RNNs run into trouble. When backpropagating through many time steps, we multiply many gradient terms together:

If the eigenvalues of W are:

• < 1: Gradients shrink exponentially → Vanishing gradients
• > 1: Gradients grow exponentially → Exploding gradients

Exploding gradients can be mitigated with gradient clipping:

# Gradient clipping in PyTorch
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

But vanishing gradients require architectural changes. Enter: gated recurrent units.

LSTM: Long Short-Term Memory

LSTMs (Hochreiter & Schmidhuber, 1997) solve the vanishing gradient problem with a more sophisticated cell architecture. The key innovation: a cell state c(t) that acts as a "conveyor belt" — information can flow through unchanged, with gates controlling what gets added or removed.

LSTM cell with forget gate, input gate, and output gate controlling information flow

The Three Gates

LSTMs use three gates — each a sigmoid neural network layer that outputs values in [0, 1], controlling how much information passes through:

LSTM Forward Pass

Why LSTMs Solve Vanishing Gradients

The magic is in the cell state update equation:

c(t) = f(t) ⊙ c(t-1) + i(t) ⊙ c̃(t)

This is an additive update, not multiplicative! When backpropagating:

Unlike standard RNNs where gradients must pass through tanh derivatives at every step, LSTMs can maintain gradients across many time steps by keeping forget gates close to 1. This is the "constant error carousel" — error signals can propagate back hundreds of steps without vanishing.

GRU: Gated Recurrent Unit

GRUs (Cho et al., 2014) simplify the LSTM architecture while maintaining most of its benefits. Instead of three gates, GRUs use two: a reset gate and an update gate. There's no separate cell state — just the hidden state.

GRU with reset gate r(t) and update gate z(t)

GRU Equations

LSTM vs. GRU

Bidirectional RNNs

Standard RNNs process sequences in one direction — from past to future. But sometimes context from both directions is useful. A Bidirectional RNN runs two RNNs:

• Forward RNN: processes x₁ → x₂ → ... → x_T
• Backward RNN: processes x_T → x_T-1 → ... → x₁

The hidden states are concatenated:

Bidirectional RNN processes sequence in both directions

When is Bidirectional Useful?

In healthcare, it depends on the task:

For retrospective analysis with complete patient histories, bidirectional models typically outperform unidirectional ones by 2-5% AUROC.

Practical Implementation: Heart Failure Prediction

Let's build a complete bidirectional GRU model for predicting heart failure from sequences of patient visits. We'll walk through every component: data handling, model architecture, training, and evaluation.

The Task

Given a patient's sequence of hospital visits (each containing diagnosis codes), predict whether they will develop heart failure. This is a many-to-one sequence classification problem.

# Example patient data
# Patient 1: 4 visits → heart failure = YES
# Visit 1: [Obesity, Sleep Apnea]
# Visit 2: [Obesity, Hypertension]
# Visit 3: [Diabetes, Hypertension, Hyperlipidemia]
# Visit 4: [Diabetes, Coronary Artery Disease, Atrial Fibrillation]

# Patient 2: 2 visits → heart failure = NO
# Visit 1: [Acute Bronchitis]
# Visit 2: [Acute Sinusitis, Allergic Rhinitis]

Data Format and Challenges

Healthcare sequence data presents unique challenges:

• Variable sequence lengths: patients have different numbers of visits (2-100+)
• Variable-size visits: each visit has different numbers of diagnosis codes (1-50)
• Sparse codes: ~15,000 possible ICD codes, most patients have only 10-200 unique codes
• Imbalanced labels: heart failure affects ~5% of patients

Custom Dataset Class

We need a PyTorch Dataset that handles variable-length sequences. Each patient's visits are aggregated into a single vector per visit (sum of diagnosis embeddings), then padded to a fixed maximum length:

import torch
from torch.utils.data import Dataset, DataLoader
import numpy as np

class VisitSequenceDataset(Dataset):
    """
    Dataset for patient visit sequences.
    
    Each patient has a variable number of visits.
    Each visit contains multiple diagnosis codes (multi-hot encoded).
    """
    
    def __init__(self, patient_visits, labels, num_codes):
        """
        Args:
            patient_visits: List of patients, each is a list of visits,
                           each visit is a list of diagnosis code indices.
                           e.g., [[[10, 50], [10, 50, 200]], [[5, 100]], ...]
            labels: Binary labels (heart failure: 0 or 1)
            num_codes: Total number of unique diagnosis codes
        """
        self.patient_visits = patient_visits
        self.labels = labels
        self.num_codes = num_codes
    
    def __len__(self):
        return len(self.labels)
    
    def __getitem__(self, idx):
        visits = self.patient_visits[idx]
        label = self.labels[idx]
        
        # Convert each visit to multi-hot vector
        visit_vectors = []
        for visit_codes in visits:
            multi_hot = np.zeros(self.num_codes, dtype=np.float32)
            for code_idx in visit_codes:
                multi_hot[code_idx] = 1.0
            visit_vectors.append(multi_hot)
        
        # Stack into tensor: [num_visits, num_codes]
        visit_tensor = torch.tensor(np.stack(visit_vectors))
        seq_length = len(visits)
        
        return visit_tensor, seq_length, torch.tensor(label, dtype=torch.float32)


def collate_visits(batch):
    """
    Custom collate function to handle variable-length sequences.
    Pads sequences to the maximum length in the batch.
    """
    visits, lengths, labels = zip(*batch)
    
    # Find max sequence length in this batch
    max_len = max(lengths)
    num_codes = visits[0].shape[1]
    batch_size = len(batch)
    
    # Create padded tensor
    padded = torch.zeros(batch_size, max_len, num_codes)
    
    for i, (visit, length) in enumerate(zip(visits, lengths)):
        padded[i, :length, :] = visit
    
    lengths = torch.tensor(lengths)
    labels = torch.stack(labels)
    
    return padded, lengths, labels

Example Data Setup

# Example: create dataset
patient_visits = [
    # Patient 1: 4 visits
    [[10, 50], [10, 50, 200], [25, 200, 300], [25, 400, 450]],
    # Patient 2: 2 visits
    [[5], [5, 15]],
    # Patient 3: 3 visits
    [[100, 150], [100, 200], [100, 200, 500]],
    # ... more patients
]
labels = [1, 0, 1]  # Heart failure labels
num_codes = 1000

dataset = VisitSequenceDataset(patient_visits, labels, num_codes)
dataloader = DataLoader(
    dataset, 
    batch_size=32, 
    shuffle=True, 
    collate_fn=collate_visits
)

# Test batch
for padded_visits, lengths, batch_labels in dataloader:
    print(f"Batch shape: {padded_visits.shape}")   # [batch_size, max_seq_len, num_codes]
    print(f"Lengths: {lengths}")                   # [4, 2, 3]
    print(f"Labels: {batch_labels}")                # [1, 0, 1]
    break

Bidirectional GRU Model Architecture

Our model has three main components:

1. Embedding layer: transforms sparse multi-hot diagnosis vectors to dense representations
2. Bidirectional GRU: processes the visit sequence in both directions
3. Classifier: uses final hidden states to predict heart failure

import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

class BidirectionalGRU(nn.Module):
    """
    Bidirectional GRU for heart failure prediction from visit sequences.
    
    Architecture:
        Multi-hot diagnosis codes → Embedding → Bi-GRU → Concat final states → Classifier
    """
    
    def __init__(
        self,
        num_codes,
        embed_dim=128,
        hidden_dim=128,
        num_layers=1,
        dropout=0.3
    ):
        """
        Args:
            num_codes: Number of unique diagnosis codes (vocabulary size)
            embed_dim: Dimension of code embeddings
            hidden_dim: GRU hidden state dimension
            num_layers: Number of stacked GRU layers
            dropout: Dropout probability
        """
        super().__init__()
        
        self.num_codes = num_codes
        self.embed_dim = embed_dim
        self.hidden_dim = hidden_dim
        
        # Embedding layer: transforms multi-hot to dense
        # Using Linear instead of nn.Embedding because input is multi-hot, not indices
        self.embedding = nn.Linear(num_codes, embed_dim)
        
        # Bidirectional GRU
        self.gru = nn.GRU(
            input_size=embed_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            bidirectional=True,
            dropout=dropout if num_layers > 1 else 0
        )
        
        # Classifier: takes concatenated forward and backward hidden states
        # Hidden dim is doubled due to bidirectional
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Sequential(
            nn.Linear(hidden_dim * 2, hidden_dim),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_dim, 1)  # Binary classification
        )
    
    def forward(self, visits, lengths):
        """
        Forward pass.
        
        Args:
            visits: Padded visit sequences [batch_size, max_seq_len, num_codes]
            lengths: Actual sequence lengths [batch_size]
        
        Returns:
            logits: [batch_size, 1] - raw scores before sigmoid
        """
        batch_size = visits.size(0)
        
        # 1. Embed each visit's diagnosis codes
        # [batch, max_seq_len, num_codes] → [batch, max_seq_len, embed_dim]
        embedded = self.embedding(visits)
        embedded = torch.tanh(embedded)  # Activation after embedding
        
        # 2. Pack padded sequence for efficient GRU processing
        # This tells GRU to ignore padding positions
        packed = pack_padded_sequence(
            embedded, 
            lengths.cpu(),  # lengths must be on CPU
            batch_first=True,
            enforce_sorted=False  # Allow unsorted sequences
        )
        
        # 3. Pass through bidirectional GRU
        # packed_output contains all hidden states
        # hidden: [num_layers * 2, batch_size, hidden_dim]
        packed_output, hidden = self.gru(packed)
        
        # 4. Extract final hidden states from both directions
        # hidden shape: [num_layers * 2, batch, hidden_dim]
        # For single layer bidirectional: hidden[0] = forward, hidden[1] = backward
        
        # Get the last layer's hidden states
        # Forward direction: processes sequence left-to-right, final state at position -2
        # Backward direction: processes right-to-left, final state at position -1
        forward_final = hidden[-2]   # [batch, hidden_dim]
        backward_final = hidden[-1]  # [batch, hidden_dim]
        
        # 5. Concatenate forward and backward hidden states
        combined = torch.cat([forward_final, backward_final], dim=1)  # [batch, hidden_dim * 2]
        combined = self.dropout(combined)
        
        # 6. Classification
        logits = self.classifier(combined)  # [batch, 1]
        
        return logits


# Initialize model
model = BidirectionalGRU(
    num_codes=1000,
    embed_dim=128,
    hidden_dim=128,
    num_layers=1,
    dropout=0.3
)

print(model)

# Model architecture output:
# BidirectionalGRU(
#   (embedding): Linear(in_features=1000, out_features=128, bias=True)
#   (gru): GRU(128, 128, batch_first=True, bidirectional=True)
#   (dropout): Dropout(p=0.3, inplace=False)
#   (classifier): Sequential(
#     (0): Linear(in_features=256, out_features=128, bias=True)
#     (1): ReLU()
#     (2): Dropout(p=0.3, inplace=False)
#     (3): Linear(in_features=128, out_features=1, bias=True)
#   )
# )

Understanding pack_padded_sequence

Padding is necessary for batching variable-length sequences, but we don't want the GRU to process padding tokens. pack_padded_sequence creates an efficient representation that skips padding:

Original (padded):
Seq 1: [A, B, C, D]
Seq 2: [E, F, 0, 0]  ← padding
Seq 3: [G, H, I, 0]  ← padding

Packed (sorted by length, interleaved):
data: [A, E, G, B, F, H, C, I, D]
batch_sizes: [3, 3, 2, 1]  ← how many sequences at each time step

Training the Model

Now let's implement the complete training pipeline with proper evaluation metrics for imbalanced healthcare data.

import torch.optim as optim
from sklearn.metrics import roc_auc_score, precision_recall_curve, auc
import numpy as np

def train_epoch(model, dataloader, criterion, optimizer, device):
    """Train for one epoch."""
    model.train()
    total_loss = 0
    num_batches = 0
    
    for visits, lengths, labels in dataloader:
        # Move to device
        visits = visits.to(device)
        labels = labels.to(device)
        
        # Forward pass
        optimizer.zero_grad()
        logits = model(visits, lengths)
        
        # Compute loss (BCE with logits)
        loss = criterion(logits.squeeze(), labels)
        
        # Backward pass
        loss.backward()
        
        # Gradient clipping to prevent exploding gradients
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        
        optimizer.step()
        
        total_loss += loss.item()
        num_batches += 1
    
    return total_loss / num_batches


def evaluate(model, dataloader, device):
    """
    Evaluate model and compute metrics.
    
    Returns:
        Dictionary with loss, AUROC, and AUPRC
    """
    model.eval()
    all_probs = []
    all_labels = []
    total_loss = 0
    num_batches = 0
    criterion = nn.BCEWithLogitsLoss()
    
    with torch.no_grad():
        for visits, lengths, labels in dataloader:
            visits = visits.to(device)
            labels = labels.to(device)
            
            logits = model(visits, lengths)
            loss = criterion(logits.squeeze(), labels)
            
            # Convert logits to probabilities
            probs = torch.sigmoid(logits).squeeze()
            
            all_probs.extend(probs.cpu().numpy())
            all_labels.extend(labels.cpu().numpy())
            total_loss += loss.item()
            num_batches += 1
    
    all_probs = np.array(all_probs)
    all_labels = np.array(all_labels)
    
    # Compute metrics
    auroc = roc_auc_score(all_labels, all_probs)
    
    # AUPRC (Area Under Precision-Recall Curve) - better for imbalanced data
    precision, recall, _ = precision_recall_curve(all_labels, all_probs)
    auprc = auc(recall, precision)
    
    return {
        'loss': total_loss / num_batches,
        'auroc': auroc,
        'auprc': auprc
    }

Complete Training Script

def train_model(
    model,
    train_loader,
    val_loader,
    num_epochs=50,
    learning_rate=0.001,
    patience=10,
    device=None
):
    """
    Full training loop with early stopping and learning rate scheduling.
    """
    if device is None:
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    
    model = model.to(device)
    print(f"Training on {device}")
    
    # Loss function with class weights for imbalanced data
    # Assuming ~5% positive class, use pos_weight = 95/5 = 19
    pos_weight = torch.tensor([10.0]).to(device)  # Adjust based on your data
    criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
    
    # Optimizer
    optimizer = optim.Adam(model.parameters(), lr=learning_rate, weight_decay=1e-5)
    
    # Learning rate scheduler
    scheduler = optim.lr_scheduler.ReduceLROnPlateau(
        optimizer, mode='max', factor=0.5, patience=5, verbose=True
    )
    
    # Early stopping
    best_auroc = 0
    patience_counter = 0
    best_model_state = None
    
    history = {'train_loss': [], 'val_loss': [], 'val_auroc': [], 'val_auprc': []}
    
    for epoch in range(num_epochs):
        # Training
        train_loss = train_epoch(model, train_loader, criterion, optimizer, device)
        
        # Validation
        val_metrics = evaluate(model, val_loader, device)
        
        # Record history
        history['train_loss'].append(train_loss)
        history['val_loss'].append(val_metrics['loss'])
        history['val_auroc'].append(val_metrics['auroc'])
        history['val_auprc'].append(val_metrics['auprc'])
        
        # Print progress
        print(f"Epoch {epoch+1:3d} | "
              f"Train Loss: {train_loss:.4f} | "
              f"Val Loss: {val_metrics['loss']:.4f} | "
              f"AUROC: {val_metrics['auroc']:.4f} | "
              f"AUPRC: {val_metrics['auprc']:.4f}")
        
        # Learning rate scheduling
        scheduler.step(val_metrics['auroc'])
        
        # Early stopping check
        if val_metrics['auroc'] > best_auroc:
            best_auroc = val_metrics['auroc']
            patience_counter = 0
            best_model_state = model.state_dict().copy()
            print(f"  → New best model! AUROC: {best_auroc:.4f}")
        else:
            patience_counter += 1
            if patience_counter >= patience:
                print(f"\nEarly stopping at epoch {epoch+1}")
                break
    
    # Load best model
    if best_model_state is not None:
        model.load_state_dict(best_model_state)
    
    print(f"\nTraining complete. Best AUROC: {best_auroc:.4f}")
    return model, history

Running Training

# Example training run
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Create model
model = BidirectionalGRU(
    num_codes=1000,
    embed_dim=128,
    hidden_dim=128,
    num_layers=2,
    dropout=0.3
)

# Train
trained_model, history = train_model(
    model,
    train_loader,
    val_loader,
    num_epochs=50,
    learning_rate=0.001,
    patience=10,
    device=device
)

# Sample training output:
# Training on cuda
# Epoch   1 | Train Loss: 0.8234 | Val Loss: 0.7621 | AUROC: 0.6234 | AUPRC: 0.1523
#   → New best model! AUROC: 0.6234
# Epoch   2 | Train Loss: 0.6543 | Val Loss: 0.5987 | AUROC: 0.7012 | AUPRC: 0.2145
#   → New best model! AUROC: 0.7012
# Epoch   3 | Train Loss: 0.5234 | Val Loss: 0.4823 | AUROC: 0.7543 | AUPRC: 0.2876
#   → New best model! AUROC: 0.7543
# ...
# Epoch  25 | Train Loss: 0.2134 | Val Loss: 0.3245 | AUROC: 0.8456 | AUPRC: 0.4123
#   → New best model! AUROC: 0.8456
# ...
# Epoch  35 | Train Loss: 0.1823 | Val Loss: 0.3567 | AUROC: 0.8423 | AUPRC: 0.4056
# Early stopping at epoch 35
#
# Training complete. Best AUROC: 0.8456

Model Variants and Enhancements

Using Pre-trained Embeddings

Instead of learning embeddings from scratch, we can use pre-trained medical embeddings from Part 1:

class BidirectionalGRUWithPretrained(nn.Module):
    """Bidirectional GRU with pre-trained medical code embeddings."""
    
    def __init__(
        self,
        embedding_matrix,       # Pre-trained embeddings [num_codes, embed_dim]
        hidden_dim=128,
        num_layers=1,
        dropout=0.3,
        freeze_embeddings=True
    ):
        super().__init__()
        
        num_codes, embed_dim = embedding_matrix.shape
        
        # Initialize embedding layer with pre-trained weights
        self.embedding = nn.Linear(num_codes, embed_dim, bias=False)
        self.embedding.weight.data.copy_(torch.tensor(embedding_matrix.T))
        
        if freeze_embeddings:
            self.embedding.weight.requires_grad = False
        
        # Rest of architecture same as before
        self.gru = nn.GRU(
            input_size=embed_dim,
            hidden_size=hidden_dim,
            num_layers=num_layers,
            batch_first=True,
            bidirectional=True,
            dropout=dropout if num_layers > 1 else 0
        )
        
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Sequential(
            nn.Linear(hidden_dim * 2, hidden_dim),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_dim, 1)
        )
    
    def forward(self, visits, lengths):
        # Same forward pass as before
        embedded = self.embedding(visits)
        embedded = torch.tanh(embedded)
        
        packed = pack_padded_sequence(
            embedded, lengths.cpu(), batch_first=True, enforce_sorted=False
        )
        
        _, hidden = self.gru(packed)
        
        forward_final = hidden[-2]
        backward_final = hidden[-1]
        combined = torch.cat([forward_final, backward_final], dim=1)
        combined = self.dropout(combined)
        
        return self.classifier(combined)

Adding Attention Mechanism

Attention allows the model to focus on the most relevant visits for the prediction. This is particularly useful in healthcare where a single critical visit might be most predictive:

class AttentionBidirectionalGRU(nn.Module):
    """Bidirectional GRU with attention over visit sequence."""
    
    def __init__(self, num_codes, embed_dim=128, hidden_dim=128, dropout=0.3):
        super().__init__()
        
        self.embedding = nn.Linear(num_codes, embed_dim)
        self.gru = nn.GRU(embed_dim, hidden_dim, batch_first=True, bidirectional=True)
        
        # Attention mechanism
        self.attention = nn.Sequential(
            nn.Linear(hidden_dim * 2, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, 1)
        )
        
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Linear(hidden_dim * 2, 1)
    
    def forward(self, visits, lengths):
        batch_size = visits.size(0)
        max_len = visits.size(1)
        
        # Embed
        embedded = torch.tanh(self.embedding(visits))
        
        # GRU
        packed = pack_padded_sequence(embedded, lengths.cpu(), batch_first=True, enforce_sorted=False)
        packed_output, _ = self.gru(packed)
        output, _ = pad_packed_sequence(packed_output, batch_first=True, total_length=max_len)
        # output: [batch, max_len, hidden_dim * 2]
        
        # Attention scores
        attn_scores = self.attention(output).squeeze(-1)  # [batch, max_len]
        
        # Mask padding positions
        mask = torch.arange(max_len).unsqueeze(0).expand(batch_size, -1).to(visits.device)
        mask = mask >= lengths.unsqueeze(1).to(visits.device)
        attn_scores = attn_scores.masked_fill(mask, float('-inf'))
        
        # Softmax
        attn_weights = torch.softmax(attn_scores, dim=1).unsqueeze(2)  # [batch, max_len, 1]
        
        # Weighted sum of hidden states
        context = (output * attn_weights).sum(dim=1)  # [batch, hidden_dim * 2]
        context = self.dropout(context)
        
        return self.classifier(context)

Results Comparison

Typical results on heart failure prediction from EHR visit sequences:

Key observations:

• Sequential models beat bag-of-codes: GRU/LSTM outperform DNN by 4-7% AUROC
• Bidirectional helps: Looking at future context improves ~2% AUROC
• Pre-trained embeddings boost performance: Med2Vec embeddings add ~2% AUROC
• Attention provides marginal gains: +0.5% AUROC but adds interpretability
• GRU ≈ LSTM: Similar performance with fewer parameters

Summary

We've covered the theory and practice of RNNs for healthcare:

References

• Hochreiter & Schmidhuber (1997). "Long Short-Term Memory" — Original LSTM paper
• Cho et al. (2014). "Learning Phrase Representations using RNN Encoder-Decoder" — GRU introduction
• Choi et al. (2016). "RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism"
• Lipton et al. (2016). "Learning to Diagnose with LSTM Recurrent Neural Networks"
• Rajkomar et al. (2018). "Scalable and accurate deep learning with electronic health records"
• Shickel et al. (2018). "Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for EHR Analysis"