There are literally millions of ex-Navy Seals, Green Berets, Delta Operators, Rangers, Force Recon out there than there ever was in the military since the beginning of our nation.
I have personally talked with one person who claims he was in Special Forces. He wasn't lying. What he didn't disclose is that he was attached to Special Forces as an Army Cook.
Yeah, grow a beard, put on a lumberjack jacket and jeans and voila, you are ex-something. You choose your fantasy, army cook.
Like Flap said, and there are also CST's who are attached to SF units who are not SF but they are on the front lines with them. Common would be women because Muslim women cannot or will not talk to non-family men (SF operators) often. So to have another female there to talk to them helps get intel. There's a woman who's actions are unknown for another few years until they're declassified. She died in combat and her story is waiting to get declassified. Basically, she was a badass in the SF community, and wasn't butch either.
Another example are the "Night Stalkers". They go thru much of the same training as operators do, but aren't operators. So it isn't uncommon for them to pick up some guys who they recognize from selection.
Then you have guys who claim they're "basically SF" cause they were attached for a few months to a SF unit. I know of a few guys who claimed this. They do not have a tab nor went thru selection. This was during the early stages of OIF and FO's were much needed. Knew a few 68W's and none of them made this statement, even when they were attached to SF units as well.