[haul down one's colors] or [strike one's colors] {v. phr.}
1. To pull down a flag, showing you are beaten and want to stop fighting.
* /After a long battle, the pirate captain hauled down his colors./
2. To admit you are beaten; say you want to quit.
* /After losing two sets of tennis, Tom hauled down his color./