Transaction foundation models in industry

Eight published transaction and banking foundation models, from Trustly, Nubank, Revolut, Featurespace, Visa (three papers) and Tencent's WeChat Pay, are the closest published counterparts of the reference stack. They test choices that production write-ups leave unstated: how amounts and time enter the model, which pretraining objective to use, and whether to freeze the backbone or fine-tune it.

Open Banking Foundational Model (Trustly, 2025). Open-banking accounts often hold only a few transactions. Each is written as a short sentence ([TYPE] DEBIT [AMT] <bucket> [NAME] <description>) and BERT is pretrained by masked language modeling (MLM, which hides tokens and predicts them), so a linear probe (logistic regression on the frozen embedding) can read an account from under 20 transactions. It led on demographics and location; aggregate features won three of six ACH-return tasks, on min-max scaled scores that hide gap sizes. In practice: recovered gender, age and location are fair-lending proxies that category-level reason codes would hide.

Trustly's frozen account embedding, built from fewer than 20 transactions, predicted gender, age and location at or near the top of the methods compared. If a production sequence-model embedding behaves the same way, what is the main concern for the default-risk model?

nuFormer (Nubank, 2025). Nubank wanted raw transactions in its credit model without losing 291 hand-built features. A GPT-style decoder pretrained by next-token prediction on compact transaction tokens is fine-tuned through LoRA (small trainable low-rank matrices on frozen weights) jointly with a network over those features, so its gradient carries what the features miss. This Joint Fusion gained +1.25% relative AUC over production LightGBM, about 0.01 AUC, against +0.97% for late fusion. In practice: fuse end to end, and train on as-of-score-date snapshots, since later edits to past records leaked future information.

Why can nuFormer's Joint Fusion beat late fusion, where the transformer is fine-tuned first and its embedding is then handed to a gradient-boosted tree with the tabular features?

PRAGMA (Revolut, 2026). Revolut wanted one model for every user event, from card payments to app screens. An event encoder reads each event alone and a history encoder attends across the resulting vectors, with MLM over tokens, whole events or whole fields, so event embeddings can be cached. With LoRA the 1B model raised credit PR-AUC 130.2% over undisclosed baselines, but a linear probe on it scored 47.1% below a network-aware model on anti-money-laundering (F0.5), since each user is read alone. In practice: cache per-transaction embeddings and keep the entity graph as a separate fraud input.

PRAGMA scored far below Revolut's production model on anti-money-laundering. What does the failure imply for a fraud stack that also has an entity graph?

NPPR (Featurespace, 2023). Featurespace asked which self-supervised objective best suits card transactions. A GRU (a recurrent network) predicts every field of the next transaction and rebuilds earlier ones from its state and a time offset, weighted by $e^{-\Delta t/\lambda}$ with $\lambda$ of 2 months, so the state carries longer-range behavior. It beat CoLES (contrastive learning on sub-sequences of one user) and RTD on four public tasks, default AUC 0.798 against 0.765, though averaged embeddings inflated its score on age. In practice: RTD, one of the stack's objectives, lost here, so test a next-event head.

TransactionGPT (Visa, 2025). Visa's anomaly model already had about 400 hand-built features per transaction, so the question was what history adds. Stacked transformers for the features, the transaction's own fields and the sequence are joined by 'virtual token layers' (a learned softmax mix plus MLP) that let each transaction enter the sequence as several tokens. The best model beat production by 22.5%, but a single-transaction model reached +7.7% and plainly concatenated history only +7.3%, on 8-transaction sequences. In practice: split a fraud model's gain into features, history and fused.

In TransactionGPT, a model that saw a single transaction with its hand features did about as well on anomaly detection as a sequence model with plainly concatenated features. How should a pretrained fraud model's headline improvement be read in light of this?

TREASURE (Visa, 2025). Predicting a card's next merchant out of over 100M rules out a full softmax. This causal transformer over a card's last 512 transactions scores the true merchant against 1,024 random negatives shared across all sequences and positions in the batch (InfoNCE, a softmax over the true item and sampled negatives), a large pool at little memory cost. Precision@1 rose from 0.060 with 5 independent negatives to 0.142. The headline +111% over production is mostly architecture and data, since abnormal-label training alone reached +96%. In practice: tune a contrastive encoder's negative count along with hardness.

LLM-initialized embeddings (Fan et al., Visa, 2025). TREASURE learns merchant, MCC (merchant category code) and location embeddings from scratch, so a rarely seen merchant gets a poor vector. Fan et al. prompt an open LLM such as Llama3-8B with enriched text for each ID and start its row at the last hidden state, which training then updates. Next-merchant accuracy rose from 0.076 to 0.099 while other fields moved about 1%, and the claimed prompt techniques were never tested against plain prompts. In practice: initializing ID tables from text is a cheap test for rare and new merchants.

Generative pretraining for fraud (Tencent WeChat Pay, 2023). WeChat Pay has billions of unlabeled payment events and few verified frauds. A GPT-style decoder pretrained to predict the next event's attributes feeds a head that runs a CNN over first differences of its hidden states, so the head reads behavior change. The top 1% of scores held 19.16% of about 800 verified frauds among 80M users, but there is no baseline, and pretraining included every fraud-reported user, leaking labels. In practice: differencing is a cheap, untested change-point head to try for account takeover.

What the papers say together. Encoding each event into a vector before the sequence model fits far longer histories than spelling transactions out as tokens, and Nubank found the token format changes cost while accuracy per transaction stays the same. Generative next-event prediction won the one controlled comparison of objectives, and no paper tests CPC or a mix like the reference stack's. Fine-tuning beat frozen use on risk tasks in most tests (a frozen model matched a fine-tuned one on held-out card issuers), aggregate features stayed strong baselines, and nobody ablates how amounts or time enter. Visa's gains include the target label in training, so they are only partly self-supervised.

Terms

MLM: hide random tokens (or fields, or whole events) and predict them from both sides
Next-token / next-event prediction: causal model predicts the next token or every field of the next transaction
Linear probe: logistic or linear regression on a frozen embedding; the model is not updated
LoRA: small trainable low-rank matrices added to frozen weights
CoLES: contrastive: two random sub-sequences of one user are positives, other users' are negatives
RTD / replaced-event detection: swap some events for other users' events; predict which were swapped
Past reconstruction (NPPR): rebuild an earlier event from the current state and a time offset, down-weighted with age
InfoNCE with shared negatives: softmax over the true item and one pool of negatives reused across the batch
Hard negative: an example that looks like the anchor but means something different
Joint Fusion (nuFormer): fine-tune the transformer end to end with a tabular network over existing features
Late fusion: fine-tune the transformer first, then give its embedding to a separate model
Virtual token layer (TGPT): softmax mix plus MLP that turns a set of vectors into k tokens of chosen size
Compositional embedding: hash each merchant ID to several rows of a small table and average them
As-of snapshot: serve each transaction as it existed on the score date, to avoid leakage from later edits

Quiz

Trustly reports each method's score min-max scaled across the four methods on each task, so the best is 1.00 and the worst is 0.00. What can you conclude when BERT scores 1.00 and feature engineering scores 0.00 on a task?
Nubank compared compact special tokens (about 14 per transaction) against plain-text serialization (about 33) at the same number of transactions, and the two scored the same. What does that imply about the compact encoding?
Adding a source of frequent, low-information transactions to nuFormer's input lowered AUC. What mechanism explains this, and what should be tested for a fixed-length history window?
Nubank found that Buy-Now-Pay-Later prepayments rewrote past transaction timestamps. What does the fix (serve each transaction as it existed on the score date) guard against?
PRAGMA's event encoder reads each event on its own before a history encoder looks across events. In a two-stage stack the per-transaction encoder takes most of the serving time. What does this design allow, and what limits it?
A team adapts its sequence model with small labeled sets. If 'lightweight adaptation' means a frozen backbone with a small head, what does PRAGMA's comparison of LoRA against linear probes suggest?
Featurespace's NPPR averaged event embeddings while its baselines used the last event's embedding. Averaging helped age prediction and hurt churn. Why?
TREASURE's switch from 5 independent negatives per positive to 1,024 negatives shared across the batch sharply improved next-merchant prediction and left country and category prediction almost unchanged. Why the difference?
Two papers raise questions about the data behind an embedding. Fan et al. initialize merchant embeddings from LLM text, and WeChat Pay chose its pretraining users partly from fraud reports. Which pairing of effect and risk is right?

Sources

Gustavo Polleti, Marlesson Santana, and Eduardo Fontes. Open Banking Foundational Model: Learning Language Representations from Few Financial Transactions. arXiv preprint, 2025.
D. T. Braithwaite et al. Your Spending Needs Attention: Modeling Financial Habits with Transformers. arXiv preprint, 2025.
Maxim Ostroukhov et al. PRAGMA: Revolut Foundation Model. arXiv preprint, 2026.
Piotr Skalski, David Sutton, Stuart Burrell, Iker Perez, and Jason Wong. Towards a Foundation Purchasing Model: Pretrained Generative Autoregression on Transaction Sequences. ICAIF, 2023.
Yingtong Dou et al. TransactionGPT. arXiv preprint, 2025.
Chin-Chia Michael Yeh et al. TREASURE: The Visa Payment Foundation Model for High-Volume Transaction Understanding. arXiv preprint, 2025.
Xiran Fan et al. Enhancing Foundation Models in Transaction Understanding with LLM-based Sentence Embeddings. EMNLP Industry Track, 2025.
Ze Yu Zhao, Zheng Zhu, Guilin Li, Wenhan Wang, and Bo Wang. Generative Pretraining at Scale: Transformer-Based Encoding of Transactional Behavior for Fraud Detection. arXiv preprint, 2023.