Scott grew a beard on family vacation in 2004