You're right that it does make linking directly to a comment non-intuitive. (And without theming the expand collapse isn't clear in my module right now). It is, however, unlikely that they will come directly from Google to a comment as they don't index (as links anyway) the anchors in the page. This would really happen if you were using something like the comment page module that creates a separate page for each comment but is mutually exclusive of this sort of idea. Links from other sites, however, could land somebody looking at the title of the comment and having to expand it to see the body.
You're right that it does make linking directly to a comment non-intuitive. (And without theming the expand collapse isn't clear in my module right now). It is, however, unlikely that they will come directly from Google to a comment as they don't index (as links anyway) the anchors in the page. This would really happen if you were using something like the comment page module that creates a separate page for each comment but is mutually exclusive of this sort of idea. Links from other sites, however, could land somebody looking at the title of the comment and having to expand it to see the body.