Early childhood development plays a crucial roll in a person's education. Someone that comes from a broken home is far less likely to receive the attention they need to develop when they're young. They tend to struggle early in school and lag behind. This leads to lower test scores.
You can draw almost a direct correlation between higher single parent household rates and lower test scores. The one anomaly is that American Indians tend to score a little higher than Hispanics. This can probably be explained by the fact that some Hispanic children face a language barrier whenever they enter the school system.
Skin color doesn't have anything to do with someone's intelligence, but the environment they're raised in absolutely does.