From what I could see as a standard tourist, and I only checked out the Fort and Petrah areas, it’s best to minimise the time spent in Colombo! Unless, as always, you know some locals who can show you the real city and best places.
Sri Lanka has far more interesting and wow-factor experiences outside of the country’s capital.