I have a horrible crush on someone who

1. I THINK might be attracted to me, but I am terrible at figuring these things out.
2. We are both married and have kids, and
3. He is a doctor. I'm...not. Medical ethics, etc. says he could get in a lot of trouble.
4. We have known each other for several years and nothing has happened. I definitely think he was hinting early on, but I was afraid I might be misreading so I dropped the ball, so to speak, and now I just want to kick myself for it.

Because I am a total idiot, I am obsessively thinking about ways to let him know that I am interested in him the next time I see him (like asking to grab coffee, which I have been afraid to do because I have a pathological fear of rejection). Guys, is this incredibly stupid? Would you ever risk this much because you really wanted to sleep with someone? I do NOT want a serious thing and am not going to ask him to leave his wife or anything like that. I just really want to kiss the guy...and some other things.