←
Bogdan Kulynych1,705 posts
Bogdan Kulynych avatar

Bogdan Kulynych

@hiddenmarkov

privacy, security, and reliability of ML · Ex @EPFL, @hseas, @Google

◷ Joined September 2012▧ 159 Photos and videos
Gautam Kamath avatar
⟳ Bogdan Kulynych retweeted
Gautam Kamath@thegautamkamath·Jun 2

The NeurIPS position paper track was flooded with submissions that substantially used AI. More than other tracks, and despite a requirement otherwise. I appreciate the chairs' strong action. People are not entitled to reviewers' time, AI makes it exceptionally easy to waste it.

This year, the NeurIPS 2026 Position Paper Track made the decision to require that all papers be substantially human-written, with AI used for only copy-editing or similar peripheral changes to the main text! For more details, please check our blogpost: blog.neurips.cc/2026/06/02/a…

Bogdan Kulynych avatar
⟳ Bogdan Kulynych retweeted
Bogdan Kulynych@hiddenmarkov·Mar 23

A Gaussian mechanism with ε = 6 can be less private than one with ε = 8. This points to a problem with how we report privacy guarantees in machine learning. A thread 🧵

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

Here's a paper with all the details: arxiv.org/abs/2503.10945 @physics_Felipe presenting it at #SatML on Tuesday March 24. w/ @physics_Felipe, Borja Balle, Jamie Hayes, @FlavioCalmon, @ahonkela

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

But when it fits, it solves both problems: correct comparison via μ, and easy and precise mapping to full privacy profiles and attack success rates. We should use it to report guarantees in DP-SGD or other mechanisms that are Gaussian-like!

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

GDP isn't always the right choice. Some algorithms can't be precisely represented by it (e.g., Randomized Response) or we do not yet know how to (e.g., Report-Noisy-Max).

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

How precise? We can measure it with a metric called "regret." For the DP-SGD instance above, GDP has regret of 0.1%, and a single (ε, δ) pair has regret of 21%. Rule of thumb: with noise σ ≥ 2 and iterations T ≥ 400, GDP is very accurate for DP-SGD. arxiv.org/abs/2406.08918

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

Here's an example for a specific instantiation of DP-SGD in terms of f-DP trade-off curves (an equivalent operational version of privacy profiles). As we see, a non-asymptotic GDP trade-off curve fits the DP-SGD trade-off curve almost exactly.

Post attachment
Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

Many ML algorithms, especially those involving many compositions like DP-SGD, can be very precisely characterized with GDP. This is a *non-asymptotic* result, not just a central limit approximation!

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

GDP characterizes the entire privacy profile ε(δ) of a Gaussian mechanism exactly using a single number μ. Interpretation: if a mechanism satisfies μ-GDP, then running membership inference against it is as hard as distinguishing N(0,1) from N(μ,1) based on a single observation.

Post attachment
Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

Can we do better without reporting an entire privacy profile? Yes! With Gaussian differential privacy (GDP).

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

As the convention sets δ in a data-dependent way, this matters whenever you compare models across datasets or papers.

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

Issue 2: You can't properly compare two mechanisms by ε if their δ values differ. A Gaussian mechanism with ε = 6 at δ = 10⁻⁵ is less private than one with ε = 8 at δ = 10⁻⁹. This is because you cannot properly compare ε if δ is different.

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

No attacker in the universe can achieve that 98% rate: It's purely an artifact of compressing the entire privacy profile into one pair (ε, δ). My colleagues and I detailed on this problem in detail in this NeurIPS'24 paper: arxiv.org/abs/2407.02191

Bogdan Kulynych avatar
Bogdan Kulynych@hiddenmarkov·Mar 23

Issue 1: A single (ε, δ) pair can massively overstate privacy risk. Example: DP-SGD with ε = 8 at δ = 10⁻⁵ suggests a worst-case membership inference accuracy of ~98% using standard conversions. But using the full privacy profile, the actual maximum is only ~68%.