The lifetimes of the 21+ and 41+ states of Po208 were measured in the α-transfer reaction Pb204(C12,Be8)Po208 by γ-ray spectroscopy utilizing the recoil distance Doppler shift method. The newly extracted transition strengths alongside ones of the decay of the 22+ state were compared to the results of large-scale shell-model calculations using an effective interaction derived from the realistic CD-Bonn nucleon-nucleon potential. The comparison indicates the importance of the quadrupole isovector excitations in the valence shell for a fine tuning of the two-body matrix elements of the shell-model interaction.