By Method

Quality Assessment

While reviewing the breadth of the methods used for nonprofit fundraising is useful, more important is assessing the rigor, credibility, and relevancy of the predictions in these published works. Wen et al. (2012) used a 10-question framework to assess the quality of each work. I used a similar method. I answered questions given in Table @ref(tab:qaquestions) for each published work; the possible answers were Yes, No, or Somewhat with weights of 1, 0, and 0.5 respectively. All questions, except for Q4 and Q6, are from Wen et al. (2012). Of course, these questions are suitable only for those works in which the researchers made predictions or built predictive models. It is also unfair to assess older research when obtaining enough computing power was a challenge. Also, my subjective bias can skew the findings.

Summary of Literature

Most of the studies in this review focused on either predicting the likelihood of a person donating or predicting the giving level or amount. An exception was the Greenlee and Trussel (2000) study of the financial stability of institutions. As Brittingham and Pezzullo (1990, 39) wrote about the predictive studies in fundraising, “Most of the studies are dissertations, and most are based on a single institution, most often a university. The results … do not support strong conclusions.” What was true in the 1990s remains true today. As we saw in the earlier sections, dissertations lare the second-most studies in applied analytics for fundraising.

Many dissertations followed a similar pattern: select variables based on literature, study each variable for correlations and significance, include selected variables for an estimation model, reject or accept the null hypothesis, and then present final results.

There are some challenges with this approach.

1. These studies are often limited to one institution; hence the results cannot be generalized.
2. This type of research primarily becomes about the application of a statistical technique to the researcher’s dataset and doesn’t contribute to knowledge advancement, either through the application of newer and different predictive methods or towards a unified theory of giving.
3. This type of framework can be templatized using a programming language.

While building local predictive models are useful for development offices, we need either groundbreaking research to significantly improve on the donor classification problem, or we need to find different fundraising problems to solve.

Many of these studies used the null hypothesis significance testing (NHST) to infer the answers to research questions. This is problematic for two reasons:

1. As Trafimow (2014) declared in his editorial of the Basic and Applied Social Psychology journal, “The null hypothesis significance testing procedure has been shown to be logically invalid and to provide little information about the actual likelihood of either the null or experimental hypothesis.” Then next year, while banning the null hypothesis significance testing procedure from the journal, Trafimow and Marks (2015) said, “\(p < .05\) bar is too easy to pass and sometimes serves as an excuse for lower quality research.”

2. As Gliner, Leech, and Morgan (2002) noted, “A common misuse of NHST is the implication that statistical significance means theoretical or practical significance.” In these surveyed studies, you can find examples of researchers interpreting statistically significant results mistaken for important findings.

While most researchers report on the overall accuracy of their prediction models, very few report on other evaluation measures, such as precision, recall, or specificity. Another challenge is the lack of comparison to baseline proportions. Since such measures or comparisons aren’t reported, it is hard to assess whether the new predictive models performed better than guessing.

For example, say our data had 5% donors and 95% non-donors. We built a predictive model that classified donors and non-donors. Let’s say that this model had an overall accuracy of 95%. Now, if were to evaluate the model only based on accuracy, we might be satisfied with its performance. But even if we guess every row as a non-donor, we achieve 95% accuracy.

Similarly, if the data has 45% donors and 55% non-donors, and the model had an overall accuracy of 50%, it did worse than the baseline. Even if the predictive models aren’t compared to other models, they should at least be compared with the baseline. As my colleagues and I reported in another paper, if the overall accuracy rate is close to the baseline, then the complex analysis can be replicated by a simple majority vote model (Nandeshwar, Menzies, and Nelson 2011).

One benefit of the research done over decades into the likelihood of a person’s donation is that we have a comprehensive list of attributes, attitudes, and values that could go into building new predictive models.

Opportunities and Future Direction

Today’s technological advancement offers fascinating paths to study various problems in fundraising. Here are some suggestions and ideas to build on our knowledge of applications of data science in nonprofit fundraising.

• Establish the baseline. In classification or numeric prediction models, use a majority vote or the mean value to compare the results against. Witten et al. (2016) call this model is called ZeroR. Also, consider using a simple, single-rule classification model known as 1R or OneR. Holte (1993) calculated the results from this simple model on many datasets and compared them with an advanced decision tree model and found that 1R was only “a few percentage points less accurate.”
• Use and report a wider set of evaluation metrics. As we saw earlier, reporting accuracy can be misleading. We can consider different evaluation measures shown in the equations below (Branco, Torgo, and Ribeiro 2016). For example, Rau (2014, 30) reported that “76.4% of cases are correctly classified,” but you can see in Table @ref(tab:raustudycf) that 73% of their study data contained non-donors, so simply predicting everyone a non-donor, our accuracy is 73%. The study predicted only 88 donors who were actual donors, making the recall or true positive rate of 20%. Thus, the model failed at correctly identifying donors. Similarly, the F-measure and balanced accuracy were low at 0.32 and 59%.

\[\begin{equation} \mathrm{Precision} = \frac{TP}{TP+FP} \end{equation}\] \[\begin{equation} \mathrm{Recall} = \frac{TP}{TP+FN} \end{equation}\] \[\begin{equation} \mathrm{Specificity} = \frac{TN}{TN+FP} \end{equation}\] \[\begin{equation} \mathrm{F-measure} = 2\times\frac{Precision \times Recall}{Precision + Recall} \end{equation}\] \[\begin{equation} \mathrm{Balanced Accuracy } = \frac{Recall + Specificity}{2} \end{equation}\]

• Consider selecting variables using feature subset selection (FSS). In his extensive study of feature subset selectors, Hall (1999) documented compared his feature (or variable) selector with other predictive techniques. He found that FSS removed redundant and irrelevant features, and in some cases, even improved the performance of the underlying predictive algorithms.
• Consider class balancing methods. When the number of rows for one class (such as non-donor) is higher than the rows for any other class (such as donor), class imbalance occurs. To overcome this problem, Kim, Chae, and Olson (2012) used undersampling to reduce the number of majority class rows. Some other approaches to achieve class balance: oversampling the minority class rows, synthetic generation of minority class rows, such as SMOTE and family (Chawla et al. 2002; Han, Wang, and Mao 2005), and cost-sensitive learning (Domingos 1999).
• Consider ensemble methods. Either combine various models (that is bagging, boosting, or stacking methods (see Witten et al. 2016, Section 8.1)) or compare various models and pre-processors. This type of comparison should be standard. Here’s pseudocode to explain this comparison:

For each dataset:
    Create P pre-processed datasets
    For each p in P:
        Divide p into ten cross-folds
        For each predictive learning technique t:
            Train t on 9-folds
            Test the model on the remaining folds
            Store results and the resulting model

"In 1975, Gates and Allen formed Micro-Soft, a blend of "micro-computer" 
and "software" (they dropped the hyphen within a year). Bill Gates Fact 
Card Microsoft’s Software for IBM PCs As the computer industry grew, 
with companies like Apple, Intel and IBM developing hardware and 
components, Gates was continuously on the road touting the merits 
of Microsoft software applications. Since stepping down from Microsoft, 
Gates devotes much of his time and energy to the work of the Bill & 
Melinda Gates Foundation."