Montebello Road and Mount Hamilton Road are the two climbs we've done pretty much every year in the Low-Key Hillclimbs. They are thus the best source of data on speed trends in the series.
For men and women solo riders, I took the geometric mean of rider times for each of the climbs each time they were done. Hamilton was done twice in 1998, while Montebello was skipped that year, but every other year Montebello was week 1, Hamilton on Thanksgiving.
There's some interesting trends. In the 1995-1996-1997 as the series got more popular the average speed dropped for both climbs. 1998 was a slight down year for turn-out, but there's no Montebello data. There were two Hamiltons that year: the first was week 1 and it went off as normal, with faster times for men and slower for women. The second one, on Thanksgiving, was even quicker, but that one was broken into two portions due to a motorcycle crash, with the times added, so riders got additional recovery.
When the series started up again in 2006, times were faster than even in 1995. Again turn-out was small. Series turnout built through 2009, and as it did, average times increased. Starting in 2009, however, times have come down every year for men and the trend has been downward for women. 2013 was the fastest year yet on both climbs for men, and were relatively fast for women (there's many fewer women then men, so the result depends more heavily on who happens to come that year, yielding substantially more variation).
Running regressions, the rate of improvement is substantial: between 1.1% per year to 2.0% per year depending on the climb and whether you look at men or women.
So the end result is just because you score less than you may have in the past, you're not necessarily slower. The fields have been getting faster. In general, the more popular the Low-Key Hillclimbs, the growth comes preferentially from relatively slower riders, and average times increase. And, as has been the trend since 2009, as turn-out drops, the speeds increase.