Safety
This is a big one, most big cities will suffer from some type of crime(s), and there are some noticeably worse than others (crimes and crime rates). As a guy, six foot plus, this doesn't concern me too much however I still wouldn't like to visit Ukraine right now or the slums of El Salvador. Yes, crime can happen in any country, even Switzerland, but some countries have much higher crime rates than others and that's just facts.
Use your street smarts and don't gamble with your life in questionable areas, especially if you are by yourself. Try to go in a group wherever possible.
Not finding adequate gyms
This one is a bit more of a personal rant. When I stay in a city one of the first things I look for is a decent gym, especially if I know I will be there for a while. I'll even try out a few of the gyms nearby and not just 'the best in the area'. Some gyms are just never going to cut it. Now, I love an old-school-steely-machines-pumping-music-type gym, I'm well accustomed to them and feel more at home in them. But some take old-school way too literally. A new set of dumbbells, floor panels, or an AC unit would not go amiss. However, if they are the only choice then for me it's better than no gym whatsoever.
Main cities are likely to have gyms everywhere these days but the smaller towns and remote areas are a little trickier in this case I tend to find the most active things I can to compensate. Hiking and bike riding are a good start.