Privacy-Preserving Transformers: SwiftKey'S Differential Privacy Implementation

In this paper we train a transformer using differential privacy DP for language modeling in SwiftKey. We run multiple experiments to balance the trade-off between the model size, run-time speed and accuracy. We show that we get small and consistent gains in the next-word-prediction and accuracy...