I have to admit, I didn’t realise that text reszing behaved differently for user preferences compared to page zoom. For that reason alone, I’m going to avoid setting font sizes in pixels.
If 2 to 3% (or more!) of your users are relying on a custom font size, you should know that so you can either support that user preference or make a conscious decision to not support it. Doing anything less is frankly irresponsible, especially considering that users with larger font sizes may be using those sizes to compensate for visual disabilities.