Can someone with a rifle treat it as an improvised weapon via rifle butt to threaten squares around him and provide flanking?
Yes, you can threaten with an improvised weapon such as a rifle butt, but you may need to change your grip on the weapon to do so
As the rules on Reach and Threatened Squares state:
If you are wielding a melee weapon or are otherwise capable of making a melee attack (e.g., if you have your own natural weapons), you threaten all squares into which you can make a melee attack, even when it is not your turn (the exception is unarmed strikes—if you’re making unarmed strikes, you don’t threaten other squares).
And when using Improvised Weapons:
If you’re using an object that wasn’t meant to be used as a weapon, treat it as a club.
Obviously, a rifle is meant to be used as a weapon, but it's not meant to be used as a melee weapon, and so it should count as a club if you want to hit people with it in melee. You might not be "wielding a melee weapon" but you are capable of making a melee attack which is not an unarmed strike, so you meet the criteria for threatening spaces around you.
Your GM might require you to change your grip on a rifle in order to use it in melee instead of as a ranged weapon. Changing grip is a swift action, so in this case you would have to have adjusted your grip by the end of your turn in order to be able to threaten with a melee attack on other creature's turns, and you wouldn't be able to rapidly switch between using it as a ranged weapon and a melee weapon on-and-off your turns since you can only make one swift action.
A rifle butt isn't a particularly dangerous weapon and you suffer the -4 penalty to attack rolls for using an improvised weapon, but that's irrelevant if your goal is just to help flank an enemy - you're there and you could conceivably hurt them with a melee attack, so they have to worry about you.
You definitly can threaten spaces when using a rifle (or a stick, or a Vesk thighbone, etc.) as a club.
The caviat I would place on it (at my table) is that you would not threaten during a round in which you used the rifle for a ranged attack unless you stated such and use a Swift or Move Action (see Change Grips) to reposition the item for use as a melee weapon. I would then require an additional Swift or Move action to use for a ranged attack again. If not used for ranged attack the next round, positioning for club use would be presumed.
I would not let someone both fire and threaten unless it was a specifically stated and deliberate thing.