I wouldn't worry too much, it's perfectly normal for social workers to explore family dynamics. I have a large family on my mother's side, biological father's side and stepfather's side. My stepfather (who I class as my dad) has been my father from age 8 and I have had no relationship with my biological father since then, I am now in my 30's. Additionally I have no relationship with any cousins, aunts, uncles, grandparents on any side. I also have a biological brother whom I have no relationship with. I see him maybe once a year at Christmas, but there is no connection there. My social worker explored this by discussing it in detail with me. However they were happy with my responses and understood, they did not need to get in contact with family members for further information as it wasn't necessary. Many family's have different dynamics and are not always close. I think the best thing for you to do is just be completely open and give detail as to your reasons behind your relationships with your family and then you shouldn't have any issues.
The main thing social workers want to establish is what support network you have around you and how strong it is. If you have strong relationships with your parents and they will be involved in supporting you and your adopted child then that is a support network. If you have a solid group of friends, even if it's only a small group, then this is a support network. My partner and I were approved and both of us have limited relationships with our family, however have very strong relationships with our parents, and we also only have a couple of close friends. None of this was a problem in becoming approved adopters, however we have had many situations arise from children's social workers where they will reject us as they feel we don't have a large enough support network, so just be prepared for things like that.
Many adopted children will come with a lot of challenges and ultimately there will be many situations where you will need respite from caring for the child, even if it's just a night out once or twice a month, they want to know that you have people in your life who will be able to step in and care for your child and provide you the support you will need. They want to know that you have people in your life that you can pick up the phone to and be able to offload any pent up feelings or frustrations, or if you need guidance and support.
Don't worry, your situation sound's perfectly normal to me. Good luck in the process.