Yang-Mills origin of gravitational symmetries