Pretraining objectives and encodings

Twelve papers behind a sequence model over account histories, the model under ACH return scores and credit underwriting, pretrained on a mix of CPC, replaced token detection (RTD) and a CoLES-style temporal contrast (TCL, where two periods of one user are positives). Three papers are the sources of those objectives and two benchmark their families on bank data. The rest cover forward-window targets, serving, long histories, numeric encodings and shrinking the per-event embedding from a transaction encoder.

CPC (van den Oord, Li, Vinyals, DeepMind, 2018). Self-supervised features need a target that rewards slow structure over local detail. CPC picks the true future embedding out of negatives (other candidates in the batch) from a summary of the past, so it keeps what predicts several steps ahead. A linear probe (a linear classifier on frozen features) reached 64.6% phone accuracy predicting 12 steps ahead against 28.5% at 2. In practice: predict a calendar horizon ahead with negatives from the same user, or CPC learns to recognize the user.

ELECTRA (Clark, Luong, Le, Manning, Stanford and Google, 2020). A masked language model learns only from the 15% of positions it masks. ELECTRA has a small generator fill them and trains the kept discriminator to label every token original or replaced, so every position carries a loss. ELECTRA-Small beat a same-compute BERT-Small on GLUE (79.9 against 75.1), and a generator that ignored context was clearly worse. In practice: other users' random events are that weak generator, so RTD needs same-user perturbations of amount, date or merchant.

CoLES (Babaev et al., Sber AI Lab, 2022). Bank histories have few labels. CoLES treats each user as a class (slices of one history are positives, other users' slices negatives), so the embedding learns what stays constant within a user. In the bank's own tables it trailed hand-crafted features alone (retail churn 0.65 against 0.74) and won mainly on identity-like tasks. In practice: TCL is CoLES across time, so it pulls a user's healthy and distressed periods together and blurs what a return model needs.

Local and global representations (Bazarova et al., Skoltech and Sber, 2024). Some labels belong to a client, others to a moment. Contrastive models capture the stable client, while generative ones (predicting the next or a masked event) keep the current state, so each wins its own kind of label. Per-transaction fraud ROC-AUC was 0.81 for a masked model against 0.38 for CoLES, though below-chance scores like that one suggest noisy local labels, and no model saw time. In practice: CPC sits near next-event models, RTD near masked ones and TCL near CoLES, so the loss weights pick which labels the joint model serves.

EBES (Osin et al., Skoltech, 2024). Event-sequence comparisons seldom control tuning effort or seed noise. Under 210 tuning runs, 20 seeds and significance tests, shuffling a trained GRU's test events cost 0.10% on banking data, so the models act as bags of events, and an MLP on aggregates came within about 5% of the best. Time entered as one min-max-scaled scalar, though, which erases calendar position. In practice: run the shuffle test on the production model and compare it against a tree on aggregated transaction embeddings.

PinnerFormer (Pancha, Zhai, Leskovec, Rosenberg, Pinterest, 2022). A user embedding computed once a day goes stale if it was trained to predict the very next action. The dense all-action loss has random positions each predict one engagement from the next 28 days, so the embedding holds what stays relevant between refreshes. Recall@10 rose to 0.229 from 0.186, and daily inference cost 8.3% against 13.9% for next-action training. In practice: predict typed events over the next 30 days from many positions.

TransAct (Xia et al., Pinterest, 2023). A real-time ranker has to relate the item being scored to the user's last few actions. Early fusion concatenates the candidate onto every row of the recent-action history, so attention compares each past action, typed and including hides, with that item. Removing TransAct cost 8.59% of repin hits, while mean-pooling the history gained only 0.21%. In practice: for ACH returns, fuse the pending debit's amount, originator and date onto every history row, with returns and NSF fees as event types.

Context Clues (Wornow et al., Stanford, 2025). Medical-record models usually read short windows of long histories. Longer context helps, but copied-forward diagnoses repeat events, and the most repetitive quarter of patients scored about 23% worse on Brier. Mamba (an attention-free long-sequence model) at 16k events reached mean AUROC 0.807 against 0.777 for the prior best model at 512 tokens, lifting every quarter about equally. In practice: collapse each recurring rent, payroll or subscription stream into a summary event and keep the deviations, which carry the signal.

MOTOR (Steinberg, Fries, Xu, Shah, Stanford, 2024). Risk tasks ask when an event happens, while next-code pretraining asks only what comes next. MOTOR predicts after every event the time until each of 8,192 codes next appears, under a censored survival likelihood, so it learns hazards over a horizon. It beat next-code pretraining on every task (celiac C statistic 0.802 against 0.774), assuming censoring is uninformative. In practice: pretrain on time to the next NSF fee, return or missed payroll, with account disconnection as a competing event.

FATA-Trans (Zhang et al., Visa Research, 2023). TabBERT, a Transformer doing masked prediction over transaction rows, copies static fields such as card type into every row, so a masked static field can be read off a neighbor. FATA-Trans encodes static fields once as their own token, removing the shortcut. On Amazon Electronics reviews AUC rose to 0.7206 from TabBERT's 0.7098, but the static split alone was worth 0.003, a small gain on 10-record windows. In practice: factor institution and account attributes into one conditioning token, or replaced events get caught by a copied field.

Numerical embeddings (Gorishniy, Rubachev, Babenko, Yandex, 2022). Deep tabular models take raw numeric scalars and lose to gradient-boosted trees. A learned embedding per feature, piecewise linear over quantile bins (PLE) or trainable sines and cosines (PLR), lets the network separate nearby values. PLR moved an MLP's average rank from 8.5 to 3.0 on eleven tree-friendly datasets though none had a heavy-tailed signed variable like an amount. In practice: encode signed log amount with PLE or PLR, since a log scalar barely separates rent of 1,400 from 1,550.

Matryoshka representations (Kusupati et al., University of Washington and Google, 2022). A model trained at one embedding size spreads information over every dimension, so truncation fails. MRL sums the loss over nested prefixes (8, 16, 32 dimensions and up), so the first $m$ dimensions work alone. On ImageNet the 64-dimensional prefix scored 75.82% against 76.80% for all 2048 and 48.10% for SVD truncation, though no test fed a prefix into a second model. In practice: train the transaction encoder with nested prefixes so the sequence model reads 64 dimensions.

What the papers say together. CPC resembles the next-event models that win time-local labels, RTD the masked model that best catches per-transaction fraud, and TCL is CoLES, which wins fixed per-user traits. Returns and default are future-event labels, where identity contrast does not help, and the forward-window target that beat next-event prediction in MOTOR and PinnerFormer is missing from the mix. No paper tests CPC with RTD, PinnerFormer's summed objectives never beat its best single one, and time and amounts are weakly encoded in most of these papers. Every gain should be reported against a tree on aggregates.

Experiments to run first on a CPC, RTD and TCL objective mix. (1) Ablate the objectives (each alone, each pair, all three, matched compute) on return dollars caught and default losses, with linear probes for institution, income band and distress onset to see whether the embedding is mostly identity. (2) Add a MOTOR-style time-to-event head with disconnection as a competing event, and a dense 30-day target, scored with embeddings 1, 7 and 30 days old. (3) Vary only the RTD proposal, from other users' events to same-user perturbations and encoder nearest neighbors. (4) Swap the amount encoding and run the EBES shuffle and random-timestamp tests. (5) Serve a 64-dimensional Matryoshka input plus a small real-time module, and report risk against latency.