public sector worker

Hi I work in the public sector. I face clients every day. Even though I speak good English, time to time I tumble and find hard to speak to some body or explain things. Often i lost for words. My work place few people commented about that and one particular woman always make remarks about speaking skills with her friends around me. She never approached me and told me any thing so I found if hard to find a ground to retaliate or respond to her comments. When these people make comments around me I found it hard and harder to speak when they are around me. These days I feel very isolated and lots of people try to avoid me. I got a nice house hold and two kids so their company is not that important to me. So I did not try to go after them searching for a topic for a conversation. Then what happen is they attack me more. Some people who were nice to me before do not talk to me after talking to the people who avoid me. Some times they all get together and look at me as a group from distance and all of sudden they burst in to a laugh. I don't know the reason, but when I look at them it's like they are making fun of me. When they gather and talk I tried to go and join them then they dissolved into different directions. It is not easy at the present. but nobody told me whats going on. One male officer who work with me greets to me every moring several times. and what he did was tell people who are around him to watch and came to me and started me greeting for several times. First I responded to him. Then I told him not to do that as I found him making fun of me. I told him that I don't like what he is doing. When he continue to do that I stopped responding to him. Now he and his two male friends and one female friend all do not speak to me and some times do not give me a space to work. and They go around and talk to the others and laugh infront of me. I do not want to be in this shift but with my kids I can't go to another place to work.