No change in headlight beam power with increasing revs from idle indicates the R/R is not charging sufficiently.
Been there, have a few T shirts ! At idle the headlight is running directly off the battery, no charge from the Reg/Rec.

Seb mentions the Electrex unit - Sprock and I have their unit on our bikes.
Never had a problem since fitting.
Come with a year's guarantee.....(Honda spares have ZERO guarantee, so if it fails "out of the box" you have NO RECOURSE !) Honda spares are also about twice the price and not as well engineered.

 

I've disconnected my manual fan switch and will see if that has any effect.
I've also had another look at voltages: at 3,000 rpm I get 13.9V and at 4,000 rpm I get 14.5V at the battery. But when the headlamps are on it drops by about 1.5V. Normal?

 

Normal is about a 1V drop but 1.5V is probably within tolerances. Looks like the reg/rec is OK. Nice !
I'd check the fan and make sure it's not sticking or has some other malfunction. The solenoid for the fan could be a bit dicky, so make sure the fan comes in at the right temp.