POPri: Private Federated Learning using Preference-Optimized Synthetic Data

Introduction: The Clash of Scale and Privacy in the LLM Era

Modern machine learning is caught in a structural squeeze. On one side, Large Language Models (LLMs) require vast, diverse datasets to generalize effectively. On the other, the most valuable personal data is trapped in silos across user devices—such as smartphones, laptops, and enterprise environments. Traditional Federated Learning (FL) with on-device training (like DP-FedAvg) was designed to solve this by training models locally on client data and transmitting noisy weight updates back to a central server. However, with modern foundation models scaling to billions of parameters, executing backpropagation on-device has become a computational and memory-related impossibility for consumer hardware.

To bypass this bottleneck, researchers have turned to a new paradigm: generating synthetic client data on the server and iteratively refining it using client feedback. At the forefront of this shift is POPri (Policy Optimization for Private Data), a novel framework developed by researchers at Meta. POPri leverages Reinforcement Learning from AI Feedback—specifically Direct Preference Optimization (DPO)—to fine-tune generative models under strict Differential Privacy (DP) guarantees, turning resource-constrained client devices into simple preference-voting engines.

The Limits of Private Evolution (PE) and In-Context Learning

To understand the breakthrough of POPri, we must first look at its predecessor: Private Evolution (PE). PE tackled the on-device constraint by keeping the massive generator LLM on the server side. The workflow operated as follows:

• The server-side LLM generates candidate synthetic data samples.
• These samples are sent to client devices.
• Clients use a small, lightweight text-embedding model to calculate the similarity (using cosine similarity or nearest neighbor voting) between the synthetic samples and their local, private data.
• The clients return a histogram of scores. To satisfy Differential Privacy, DP noise is added to these aggregated counts, and low-frequency samples are truncated to boost the signal-to-noise ratio.
• The server aggregates these noisy histograms to identify the highest-scoring synthetic samples.
• Finally, the server uses these top samples as in-context examples in the LLM’s prompt to guide the generation of the next batch of synthetic data.

While Private Evolution achieved impressive performance gains on image and text tasks with minimal communication overhead, it hit a critical bottleneck: In-Context Learning (ICL). Relying on prompts to steer generative behavior is inherently limited, inefficient, and fails to permanently align the model’s underlying weights to the target distribution.

POPri: Aligning Generators via Direct Preference Optimization

POPri replaces the fragile in-context learning loop of Private Evolution with robust, weight-level fine-tuning using Direct Preference Optimization (DPO). Instead of merely prompting the LLM with high-scoring samples, POPri treats the client-side similarity scores as preference indicators to perform reinforcement learning directly on the server.

The POPri Pipeline Step-by-Step

The core optimization loop of POPri is elegant and highly scalable:

1. Candidate Generation: The server-side generator LLM produces K synthetic candidate samples for each of P seed prompts (creating a pool of K x P samples).
2. Local Scoring: These candidate samples are distributed to client devices. Using their local private data and a small embedding model, clients calculate similarity metrics.
3. Differential Privacy Aggregation: The local preference rankings are aggregated globally, with DP noise added to satisfy formal differential privacy guarantees.
4. DPO Preference Triplet Construction: For each of the P prompts, the server constructs preference triplets. The highest-scoring synthetic sample is designated as the “chosen” response (y_w), while a lower-scoring sample is designated as the “rejected” response (y_l).
5. Policy Optimization: The server fine-tunes the generator model using the DPO loss, pushing the model to increase the probability of generating samples like y_w and decrease the probability of generating samples like y_l.

Why DPO Over Supervised Fine-Tuning (SFT)?

A natural baseline alternative would be to simply perform Supervised Fine-Tuning (SFT) on the top-ranked synthetic samples. However, POPri’s research reveals that SFT struggles in this setting because it treats top-rated synthetic samples as absolute ground-truth labels. Since these synthetic samples are highly noisy approximations of real client data, SFT forces the model to overfit to flawed outputs. DPO, by contrast, operates on relative preferences. It merely guides the generator to make better-than-average choices, requiring much weaker statistical assumptions and yielding highly robust, smooth policy alignment.

Empirical Breakthroughs and Living Benchmarks

Evaluating generative models in federated learning is notoriously difficult due to data contamination (i.e., LLMs having already memorized historical static benchmarks like StackOverflow). To counter this, the researchers curated a suite of “living datasets”—regularly scraped contemporaneous texts, such as impromptu UK, US, and Canadian parliamentary debates, which are free from training set contamination.

The team evaluated POPri against standard baselines on two primary downstream tasks:

• Next Token Prediction (bioRxiv Abstracts): Evaluating a downstream 72M-parameter DistilGPT2 model trained on synthetic data. POPri closed an impressive 58% of the gap between zero-privacy (direct training on private data) and the baseline of no fine-tuning.
• OpenReview Score Classification: Using RoBERTa-base as a downstream classifier in a centralized DP setting, POPri outperformed traditional DP-SGD and Private Evolution, closing 43% of the performance gap.

The Resource Trade-Off: Client Savings vs. Server Demand

Implementing POPri requires navigating a distinct set of system trade-offs:

• Client-Side Benefits: Compared to traditional Federated Averaging (DP-FedAvg), POPri dramatically reduces client burden. Clients do not need to perform expensive backpropagation passes or store massive gradients. They only run forward passes through a small local embedding model and return compact preference scores, keeping communication bandwidth minimal.
• Server-Side Cost: The primary tradeoff is server-side computation. While DP-FedAvg merely averages incoming model weights on the server, POPri requires running iterative, heavy RL/DPO training loops. For large technology enterprises with ample server compute but strict user-device constraints, this is a highly favorable trade-off.

Critical Operational Lessons for POPri

The researchers highlighted two pivotal hyperparameter strategies essential for successfully deploying POPri:

1. The Imperative of Staying On-Policy

In reinforcement learning, the data-generating policy should closely match the policy being optimized. POPri operates over multiple rounds of generation and feedback. If the server performs too many optimization steps per round, the model weights diverge too far from the policy that originally generated the candidate samples (becoming “off-policy”). While taking more optimization steps per round yields immediate short-term convergence gains, the researchers found that limiting optimization steps per round guarantees superior long-term performance.

2. The Goldilocks Zone of Rejection

When selecting the “rejected” sample for the DPO preference pair from the K candidates (where K=10), choosing the correct rank is vital. If you pair the top-ranked sample with the absolute worst-ranked sample (rank 10), the contrast is too stark, the task is too easy, and the model learns very little. Conversely, pairing rank 1 with rank 2 introduces too much noise. Empirically, the researchers discovered that pairing the top sample with rank 5 represented a “Goldilocks zone”—offering enough contrast to guide optimization without overwhelming the gradient with noise.

The Road Ahead for Private Federated GenAI

POPri represents a monumental shift in how we approach privacy-preserving machine learning. By transforming resource-constrained client devices from training engines into privacy-preserving preference evaluators, POPri opens the door to safely fine-tuning foundation models on highly sensitive real-world datasets. Future extensions of this work aim to explore multi-modal data generations and integrate cutting-edge policy optimization techniques like GRPO to further accelerate the transition of private generative AI from research to production.