We would still need to improve the protection so that Eli could throw a catchable ball. Though I don't think that's a rule anymore. All in all, I think that ref did well to keep the flag in his pocket most of the time.
Certainly what you say is a major factor in baseball. Managers make their lineups and call pitches with the ump in mind.