The ideal team of reviewers:
-The "ideal reviewers": one person or two who are attuned to the writer’s train of thought, who can recognize exactly what he was trying to say, who like it (usually… and if they don’t it really means something) and know a way to make it better that will please you perfectly. A rare breed–they are something like your literary clone–but once found it is unwise to let them go.
-A few good average readers: they are people who don’t have a professional point of view on literature, who enjoy books and have enough insight to recognise what it is that made them tick. You won’t always agree with them and it is likely that at times they won’t even understand what you are trying to do. But they are precious, still. They are the people who can show you how people will react to your story, and why. They can give you tips on how to please more people. They are not here to improve your literary skills but to help you find an audience, and that’s just as precious.
-Random readers: they needn’t be the same people from one story to the next. Assuming that they read a bit in their spare time and enjoy the genre you write in, they provide good help as well. They are test readers, if you will. You probably won’t improve anything thanks to their help, they may even miss the mark entirely (anecdote: recently, I wrote a time-travel story that was set in Scotland. The time-traveller was just back from the fourteenth century, and remarked that "they all talk like us, not that silly English accent!". The reader pointed out that the American accent was maybe similar to 17th century English but certainly not 14th century, and later said that he assumed the story must be set in the US because the setting was not described. I did mention several times the fact that they were Scottish, though). Do not ignore them or turn them down. If they didn’t understand, or disliked your story, maybe other readers will and you will never be successful. Listen to them, even if they can give you no useful advice.
The readers you don’t need: people who don’t usually read unless out of curiosity, because their friend (ie. you) wrote it. They are unlikely to point out anything useful, and are not even representative as random readers, so why bother? Same goes for people who would not normally read the genre you wrote in. It is quite pointless to give a science fiction story to someone who will remark that it would be much better and more believable if you wrote about real stuff, or erotica to someone who assumes that all mentions of sex in a story are "gratuitous". Allow them to read your work if it would be awkward not to do so. But don’t listen to them.