If we DP fine-tune a foundation model (trained without formal privacy guarantees), I agree there may be a disconnect between the actual privacy guarantees and what people expect. However, I think that the DP guarantee is still meaningful.
I often think of it like in a survey. I come to your door and ask to use your blog posts for my fine-tuning. You say "I am worried about an adversary using the fine-tuned model to learn my private information." I tell you "Don't worry, I am using differential privacy. The final model won't be significantly more attackable if you contribute."
Well yes, I think it *could* be still meaningful, but I am arguing that in reality how meaningful it is depends on the specific relationship of the finetuning dataset to the original dataset, and the size of the model.
If you DP finetune on blog posts that are already in the original dataset you've added no privacy guarantee to the model, because obviously your model is not becoming more private from training on data. And in that setting, the one thing you would care about is how large the model is (as larger models are more attackable).
Now, if you DP finetune on blog posts that are completely uncorrelated to the original dataset, then your DP is entirely meaningful. It gives you a meaningful bound about how attackable you are.
You need to bridge these two edge-cases, presumably continuously, and you end up with a trend that should look like figure 2.
In practice, your blog posts (finetuning dataset) are probably correlated to things that your original dataset is trained on, and so your DP guarantee is somewhat informative, but isn't directly corresponding to a (human) privacy guarantee; as it doesn't capture how attackable your data is at the end of the finetuning.
I _think_ it is fair to say that the DP guarantee only captures how much more attackable your model after finetuning, but I'd argue that's not what people care about. I think when a group says "this model has an epsilon DP guarantees", they want it to be interpreted as - no one could do a privacy-attack on your data with higher than e^epsilon probability. And I'm arguing that's not necessarily the case (and it's even not-likely).
If we DP fine-tune a foundation model (trained without formal privacy guarantees), I agree there may be a disconnect between the actual privacy guarantees and what people expect. However, I think that the DP guarantee is still meaningful.
I often think of it like in a survey. I come to your door and ask to use your blog posts for my fine-tuning. You say "I am worried about an adversary using the fine-tuned model to learn my private information." I tell you "Don't worry, I am using differential privacy. The final model won't be significantly more attackable if you contribute."
That guarantee is still preserved.
Well yes, I think it *could* be still meaningful, but I am arguing that in reality how meaningful it is depends on the specific relationship of the finetuning dataset to the original dataset, and the size of the model.
If you DP finetune on blog posts that are already in the original dataset you've added no privacy guarantee to the model, because obviously your model is not becoming more private from training on data. And in that setting, the one thing you would care about is how large the model is (as larger models are more attackable).
Now, if you DP finetune on blog posts that are completely uncorrelated to the original dataset, then your DP is entirely meaningful. It gives you a meaningful bound about how attackable you are.
You need to bridge these two edge-cases, presumably continuously, and you end up with a trend that should look like figure 2.
In practice, your blog posts (finetuning dataset) are probably correlated to things that your original dataset is trained on, and so your DP guarantee is somewhat informative, but isn't directly corresponding to a (human) privacy guarantee; as it doesn't capture how attackable your data is at the end of the finetuning.
I _think_ it is fair to say that the DP guarantee only captures how much more attackable your model after finetuning, but I'd argue that's not what people care about. I think when a group says "this model has an epsilon DP guarantees", they want it to be interpreted as - no one could do a privacy-attack on your data with higher than e^epsilon probability. And I'm arguing that's not necessarily the case (and it's even not-likely).