If he were to dine with dashing actors like Jennifer Aniston (pictured with Stern in 2019), he'd have to stop going into the gutter with adult film actors