Survey Frustration

Alright, so I've been working with these two data sets from two surveys for about a year and a half now, pulling many, often strange correlations out of them with fairly consistent success. They've been good to me. Recently though, I decided to return to the data for multivariate analysis (looking for relationships between three or more variables) to strengthen the research for submission to a journal and have just been frustrated by inconsistencies between the two surveys.

The Surveys: With the exception of a couple questions added and a few questions reworded, the surveys are textually the same. The big difference is that the first survey group was a sample of students at a community service event while the second was via the internet, advertised on my university's homepage (of sorts). Overall, there were some demographic differences. I got a larger proportion of male respondents at the service event than the general survey (not surprising since I'm studying why men don't volunteer). Of course, the service event survey had a higher number of hours volunteering in college and the previous semester as well. However, I'm not blind to the influence of gender and previous hours because I'm studying exactly those.

Key Differences: Among all of the analyses I'm doing, the most recent and most important are also the most inconsistent. I get one thing for one survey at a certain amount of hours and another thing for the other survey at a different amount of hours. Let me be more specific.

1. In the general survey, women are significantly more likely to do more service if they've already done a lot of service while men don't change. There's no such effect in the service event survey.

2. Organizational Influences: In both surveys, organizational affiliation significantly impacted the likelihood to volunteer in the future. But which organizations and how were completely different. In the Service Event Survey, clubs, greek organizations, and university organizations all had a significant impact on the likelihood to volunteer in the future (clubs and uni orgs positive, greeks negative). Organizational affiliation also significantly impacted women but not men. In the general survey, only community volunteering agencies had a significant general effect. By gender, women in CVAs were more likely to volunteer than those who were not, while they were also significantly more likely to volunteer if they had logged a lot of hours in the last semester and were associated with clubs or religious organizations.

3. Motivations: women with little volunteering experience cite resume building/networking motivations in the service event survey. Men cite the same with mid-range volunteer experience and little recent volunteer experience in the general survey.

These are just a few of the inconsistencies that I'm not sure how to explain. There's something about the groups; whether their environment or some unknown selectivity. I can generalize some explanations, but it's just not clear.