How wrong is it when i've liked a friend of my brothers ever since i was 14?

Okay i have nother but older brothers & of course they have guy friends, but there friends are losers to me because its either I've known them since i was 5 or there just dumb. & theres a friend of theres whos nothing like the other losers. He has a really good job, his own house, and hes single. Well since I can remember I would always flirt with him. & he likes it because he flirts back at me. I get really shy around him, but at the same time I'm myself around him. But the thing is hes my brothers friend and this p Friday after my brothers surprise b-day party I went back to his house & all we did was talk and chilled out and by the time i knew it it was 5am. He walked me out & i gave him a hug and told him to call me & he told me he had to think of what he was gettin him self in too. & just last night we hung out again after a softball game & we were talkin about b-day since mines around the corner & he remembered it & all i did was smile @ him. It made me fill good & i love it.