A criminal who steals my cell is not going to be concerned about my whereabouts, present or past. He's going to sell it for crack $$.So you are OK with that criminal that just stole your phone knowing your every move? Where you tend to go and when?
If I was a millionaire, and the thief had something to gain about knowing my travel habits and timelines, yes...I'd be concerned.
The issue here is "big brother", not some thief....This whole "I don't want to be filmed when I go to get gas...It's an invasion of my privacy" is what we are talking about.