Lots of us bananas have come and gone from HK and have had no problems at all with safety. I think HK is TOO safe sometimes; there are freaking rails along every sidewalk... so safe that it gets annoying simply crossing the street that you have to go underground to get to...
The main thing a banana would have to worry about is if he speaks poor or no Chinese... THEN you'll probably get weird looks ie: people think you're retarded.