Jeans and a t-shirt should be OK... it's an accessory IMO... so keep in mind some outfits weren't made to be accessorized.
If you're reaching for sneakers... should probably not wear the fedora... but if your belt matches your shoes then go for it!
I'd personally wear that sort of hat with some slick jeans and a solid T-shirt (or v-neck), or any of a million button down collared shirts.
Be grateful you have a good head for it... ha ha