Serial intervals represent the time delay between illness onset in successive cases in chains of transmission. The serial interval distribution is often used as a proxy for the generation time distribution, representing the time delay between successive infections in transmission chains, since infections are generally unobservable while illness onset dates are observable. The serial interval distribution is a key input into common approaches to estimate the time-varying reproductive rate. In this study, we examined detailed contact tracing data on laboratory-confirmed cases of COVID-19 in Hong Kong between 1 January and 30 September 2020, and identified 860 pairs of cases with clear epidemiological links between infector and infectee, representing approximately 30% of all confirmed cases. Analysis of these 860 pairs identified a mean serial interval of 4.2 days and standard deviation of 5.0 days, with 102 (11%) observed serial intervals being negative. We found clear changes over time in serial intervals, with longer serial intervals of mean 6 days during the rising phase of a community epidemic, declining to a low mean of 2 days when incidence fell due to effective control measures. We were able to correlate the changes in serial intervals with more timely isolation of potential infectors, consistent with our hypothesis that this would reduce postsymptomatic transmission but not necessarily pre-symptomatic transmission. Methodological developments are now needed to account for changing serial interval distributions when estimating reproductive rates.