Is it too nitpicky to get annoyed when someone suggests going to their neighborhood instead of a middle ground

I live in a city where most people use public transportation to get from place to place. If I lived in the suburbs where everyone drives it might matter less to me. But whenever a guy suggests his own neighborhood instead of somewhere middle ground for both people, I get kind of annoyed and usually say “what about meeting here (a neighborhood that should be easy for both people) instead?” But I was mentioning this to a guy friend at work and he was basically saying that that was rude of me. I think the rude part is making me commute somewhere that’s a 5 minute walk for you lol. Just curious if other people feel the same way