Kinda what I was going to say too. I might mock this up this evening and share some scope outputs. The way I understand PWMs to work is a 50% duty cycle @ 5v, the load "sees" 2.5V. @ 100% it sees 5V, @ 25% it sees 1.25V. This is likely an over simplification and I'm sure the formula is way more complicated and utilizes symbols found scratched in to cave walls by ancient civilizations that may or may not have been influenced by extraterrestrial visits. For example:
I'm sure this means something to someone. That someone is not me.
Anyway using my, likely over simplified, understanding of PWMs, and looking at the datasheet for an LED, you'll notice that an LED's output curve is not linear. Your brightness goes from 0 - full bright in the course of 20mA and within ~1v. Look at these graphs.
current vs. voltage on the left, and intensity vs current on the right. Even if my over simplified understand of PWMs is 100% wrong, this is why I think you're seeing what you're seeing. LEDs current goes from 0 to blast-off in no time at all. Which then translates to the brightness output graph on the right. It's like, nothing, nothing, nothing OMg is bright in here!
SO! Superimpose my understanding of PWMs on top of that you get this. (assuming 5v PWM)
Duty - LED on?
0% - nope
5% - nope
10% - nope
25% - maybe
30% - maybe
50% - Yup.
75% - OMg it's bright!
100% - pretty much the same as 75% and a big shout-out to my current-limiting resistor for not letting me fry my $2 LED from AdaFruit. [Resistor: I got your back bruh]
You get the idea. So between your knob being turned 1/4 of the way and 3/4 of the way around, you've gone from no brightness to full brightness with most of the noticeable "dimming" being from 50-75% - 1/4 of a POT turn (see scaling comment below).
How do you overcome this? Dunno. I've mitigated some of this by using a micro (like Arduino or RaspberryPi) by scaling your output to the input voltage range and get true percentages which is nice. this makes the POT "feel" more directly connected to the output (1/4 turn = 25% duty cycle). You could use a 10 or 20 turn pot maybe? or preset your PWM to start closer to the "on" characteristics of the LED? Buy a prebuilt LED driver/controller (where's the fun in that?).
Another thing I've sort of, internalized is that LED's are really on/off kinda of devices. The "dimming" you get is more/less an illusion from turning them on/off really fast. I mean, between that "on" and "ON!" current range there is some actual dimming going on, but the bulk of what you are seeing is perceived brightness.
So add all of THIS to the fact that your driver transistor/MOSFET or whatever also has an "on" threshold you can see why there would be a VERY noticeable period of time where nothing is happening while you're turning your knob. So, as 1.21Gigawatts was saying, you'll first overcome the .6v for the diode/transistor, to get that to turn on, then you have to overcome the "on" current/voltage for the LED(s).
That's what I think...