Critical reflections on the USDA local food economics toolkit

In this paper we report the results of a field test of an economic impact toolkit recently commissioned by the U.S. Department of Agriculture (USDA). The toolkit was created as a guide for food systems organizations to frame issues and collect and analyze data in order to credibly measure economic and other benefits of their initiatives. To test the toolkit, we applied it to an economic contribution study of a local food-buying program in a large regional hospital in Vermont. Our findings indicate that by working with a dedicated and motivated community partner, we were able to agree on the scope and objectives of the project, obtain highquality data, and enter these data into an inputoutput model to measure broader economic contributions (Modules 1 though 6 of the toolkit). We experienced difficulty, however, in obtaining data from a sufficient number of the hospital’s vendors to modify the model from its default settings to better reflect local food system actors’ purchase patterns (the subject of Module 7). Our experience suggests that practitioners need to work with community partners and consider which stages of the analysis meet their project objectives; in particular, they should focus on the difficulty and expense of incorporating Module 7. Our implications focus on strategies for decreasing the cost and effort of data collection for Module 7.

output model to measure broader economic contributions (Modules 1 though 6 of the toolkit).We experienced difficulty, however, in obtaining data from a sufficient number of the hospital's vendors to modify the model from its default settings to better reflect local food system actors' purchase patterns (the subject of Module 7).Our experience suggests that practitioners need to work with community partners and consider which stages of the analysis meet their project objectives; in particular, they should focus on the difficulty and expense of incorporating Module 7. Our implications focus on strategies for decreasing the cost and effort of data collection for Module 7.

Introduction and Literature Review
"All models are wrong, but some are useful." -George Box The food system's impact on human well-being is vast, touching on a broad array of dimensions, from the economy, nutrition, and health to land use and the environment and to community and civic engagement (Conner & Levine, 2007;Institute of Medicine & National Research Council, 2015;Lyson, 2004).As governments, foundations, the private sector, and other funders continue to support and implement local food systems-related programs, it is important to be able to measure the economic impact of these programs in order to enable comparisons and analysis and to foster learning and adoption of effective practices.Many economic impact studies have used tools such as the IMPLAN input-output model to measure the economic impact or contribution of current or prospective food systems initiatives (Conner, Knudson, Hamm, & Peterson, 2008;Gunter, 2011;Haynes, 2009;Hughes, Brown, Miller, & McConnell, 2008;Jablonski, Schmit, & Kay, 2015;Swenson, 2006aSwenson, , 2010;;Tuck, Haynes, King, & Pesch, 2010).An input-output model is a matrix of economic multipliers (how many times a given dollar circulates in the economy before leaving or leaking out from the study area) associated with a given purchase.It measures the broader impact of how much income is generated by the recirculation of money spent in a given sector.Accurate measurements depend on many factors, including sound data, realistic assumptions about opportunity costs and tradeoffs (e.g., did the researchers account for foregone food purchases from out of state when measuring the impact of local food purchases), and models that accurately reflects conditions in the field.The USDA Agricultural Marketing Service (AMS) has commissioned a group of largely university-based scholars to develop a toolkit to allow recipients of funded projects and other stakeholders to more accurately measure economic impact and to address the issues above (Thilmany McFadden et al., 2016); co-author Conner of this paper was on the team that developed the toolkit.This study utilizes the methods and guidelines recommended by the toolkit and will contribute to the field by presenting methods and results for future comparisons.This article uses an economic contribution article (Becot, Conner, Imrie, & Ettman, 2016) to critically examine the use of the USDA toolkit.Our test case is the University of Vermont Medical Center (UVMMC), the largest hospital in the state of Vermont.We begin with an overview of previous economic contribution and impact studies, then briefly describe the context, methods, and results of our Vermont study, focusing on the degree to which we were able to incorporate the methods and recommendations of the toolkit.
Spending money locally can have large impacts on a local economy.A number of studies have used input-output models to calculate the direct and indirect economic benefit from increased purchases of local foods by consumers (Conner et al., 2008;Swenson, 2006aSwenson, , 2010)), farmers' markets (Hughes et al., 2008), food hubs (Jablonski, Schmit, & Kay, 2015), and farm-to-school programs (Gunter, 2011;Haynes, 2009;Tuck et al., 2010).On the whole, these studies find that depending on the size of geographic area and scope of changes in purchase behaviors, local food systems can add or contribute thousands of jobs and hundreds of millions of dollars to an economy.
Good economic impact studies depend on good data, good assumptions, and a sound, accurate model (Bauman & Tegegne, 2013).Secondary data sources may not exist or may inadequately reflect conditions in the field, yet primary data collection is time-consuming and difficult and methods are not well established (Conner et al., 2013;O'Hara & Pirog, 2013).Moreover, many economic impact studies rely on faulty assumptions and tend to overstate economic impact, and proponents are often eager to tout these studies to support their positions (Eathington & Swenson, 2007;Swenson, 2006b).It is also important to use careful and consistent semantics to reduce confusion and the misuse of economic analyses (Watson, Wilson, Thilmany, & Winter, 2007).
A crucial consideration for any economic impact analysis is the geographic area where the economic activity studied takes place.The larger the study area, the bigger the economic multiplier, while the smaller the study area, the larger local industries will appear (Watson, Wilson, Thilmany, & Winter, 2007).The geographic area needs to encompass an economic area that accurately represents the transactions germane to the study.
Researchers have found that the IMPLAN input-output model does not fully capture and/or accurately measure the impacts of smaller, diversified farms and other small to medium-scale operations that frequently participate in the localized food system (Lazarus, Platas, & Morse, 2002;Jablonski, Schmit, & Kay, 2015;Swenson, 2011).These researchers have found that farmers serving local and regional markets, as well as smaller-scale farmers, tend not only to spend more money locally, but also differently (e.g., more expenses on labor) than is assumed in IMPLAN.The farmers' economic activity, therefore, actually has a higher multiplier and greater impact than is assumed.To address these limitations of IMPLAN, which underestimate the impact of farmers serving local and regional markets, researchers are modifying the model by customizing the agricultural sector using secondary and primary data on spending and sales patterns of those farms (Gunter, 2011;Jablonski & Schmit, 2015) Overview of the Toolkit The toolkit (Thilmany McFadden et al., 2016) was authored by leading scholars in the economics of local food systems.It was motivated by the need for rigorous methods to be applied to the burgeoning interest, funding, and enterprise development around local food systems.The goal was to create food system assessment principles and economic indicators that communities can use to guide economic development discussions and strategies: "The goal of this Toolkit is to guide and enhance the capacity of local organizations to make more deliberate and credible measurements of local and regional economic activity and other ancillary benefits" (Thilmany McFadden et al., 2016, p. 1).The toolkit is composed of seven modules.Modules 1 to 4 describe how to frame the problem, engage stakeholders, and collect and analyze primary and secondary data.Modules 5 to 7 provided detail on how to use IMPLAN in a rigorous manner to provide a more robust economic impact measurement.Table 1 lists each module, its subject area, and its use in food systems economic studies.
Key recommendations from the toolkit (Thilmany McFadden et al., 2016) include: • Have a concise scope of the project and invested stakeholders who will guide and set the context for the study, in order to ensure useful results (Modules 1 and 4).• When possible, utilize secondary data sets, but be aware of their limitations (Module 2).• Collect primary data to overcome the limitations of secondary data (Module 3).(Buzalka, 2012;Fletcher Allen Health Care, 2014).In 2013, UVMMC served over two million meals, of which 15% were meals served to patients and 85% were meals served to visitors and hospital staff.Approximately 40% of the food served comes from local suppliers in Vermont, including food purchased directly from farmers and food manufacturers.The rest of the local food is purchased through wholesalers.More than 90% of the beef served is from Vermont.The hospital is currently purchasing food from about 70 Vermont farmers and producers as well as from one Vermont wholesaler and three larger-scale national wholesalers.

Overview of Methods
A more detailed account of the methods and results of this study have been published elsewhere (Becot et al., 2016).In brief, we gathered purchase data from UVMMC for use in IMPLAN analysis.We also interviewed vendors and surveyed customers to better understand their behaviors and motivations.Questions to vendors focused on the magnitude, motivations, and impacts of sales.Consumer questions focused on hospital employees, and in particular their motivations, rather than visitors and community members, since they purchase more meals there.
This essay critically reflects on this study, discussing future prospects for similar studies and how they can be improved.The AMS toolkit was under development while the Vermont hospital study was being conducted; one of the researchers of the Vermont study and authors of this paper (Conner) is also an author of the toolkit.Another author (Becot) had the benefits of both early drafts of the toolkit and discussions with two other toolkit authors, which helped to guide the Vermont study's methods.
Decisions made in accord with toolkit recommendations include: 1. Working closely with UVMMC's nutrition services director and obtaining all purchase data directly from her. 2. Working with stakeholders to define the geographic scope of the models.We determined that the functional economy should be defined as the whole state of Vermont in order to conform to the hospital's definition of local food and given that the hospital's food suppliers are located throughout the state.We excluded bottled soda, as this has no nutritional value and uses only local water, a common asset; we do not consider it to be local food as commonly understood.

Results
Key findings of our study include: • UVMMC spent US$1.784 million on Vermont food, representing 44.3% of its total food purchases, with 16.3% of the local purchases bought directly from farmers, 22.9% bought directly from food manufacturers, and 60.8% bought from wholesalers.In the case of the purchases from wholesalers, we only accounted for products that were grown or processed in Vermont.
• Additionally, two full-time positions representing US$95,058 in labor income were added at UVMMC in nutrition services due to an increase in the volume in meals served.The increase in volume was due largely to an increase in sales to hospital employees and the university community rather than increased patient meals.• On average, 9.3% of the vendors' production went to UVMMC.UVMMC also served as a gateway to other wholesale accounts for many vendors.• Depending on assumptions, we find the output multiplier to be 1.38 (every US$1 spent on local food by the hospital contributes another US$0.38 to the Vermont economy), where previous studies have found multipliers ranging from 0.65 to 1.82 (Gunter & Thilmany, 2012;Jablonski, Schmit, & Kay, 2015;Tuck et al., 2010).According to Hughes (2003) the probable range of multiplier is between 1.5 and 2.5, and the larger the size of the economy the larger the size of the multiplier.Additionally, Hughes (2003) cautions that a multiplier above 2.5 should be carefully examined.
For a more complete description of the methods and results, see Becot et al. (2016).

Discussion
It is important to note that in this study we focused only on the contribution of UVMMC local food purchases, including opportunity costs as they relate directly to the food supply.Since starting local purchases in 2006, UVMMC has seen an increase in the number of meals served while the number of inpatient and outpatient days has remained fairly constant.According to the nutrition services director, it seems that the increase in meals served can be attributed to two things: more UVMMC staff eating at the cafeteria instead of bringing in their lunch from home, and people from outside coming to the hospital to eat at the cafeteria (UVMMC is adjacent to the University of Vermont campus).Because the number of inpatients and outpatients has remained essentially constant, we hypothesize that the change in meals served due to patient visitors is negligible.A future study would then add the economic impact of the increase in meals served at UVMMC.In this case, the opportunity costs to consider are the lower sales for supermarkets and the food service provider (Sodexho).
It is also important to note that input-output models are not well equipped to handle opportunity costs.We were not able to examine key issues that in-depth, including what tradeoffs the hospital makes to afford higher-priced local food, such as smaller portions of proteins; impacts on staff training and preparation time; and whether farmers who are selling to UVMMC may be foregoing higher prices from direct markets.In this sense, the results of IMPLAN studies are (to echo Box's quotation above) "wrong"-even if useful -in that they do not address these opportunity costs well.
We wish to share a word of methodological caution.We originally intended to interview vendors to gauge their spending patterns and then conduct advanced IMPLAN analysis (as detailed in Module 7 of the USDA toolkit).Despite encouragement from UVMMC and a cash incentive, only eight out of 70 vendors responded to our interview requests.Of these eight, four were farmers and four were manufacturers.In order to modify IMPLAN as outlined in Module 7, we would have needed to assume that four vendors were representative of the farming and food manufacturing sectors in the state as a whole.Vermont has approximately 7,300 farms (USDA, National Agricultural Statistics Service, 2014) and at least 385 food manufacturers (Vermont Specialty Food Association, n.d.), so this sample cannot credibly claim to represent all 70 UVMMC food vendors, let alone the state as a whole.We did use qualitative results, however, from the vendor surveys to highlight vendors' perceived benefits of selling to UVMMC on the overall business.In these interviews, many vendors expressed that while the sales to the hospital were a relatively small portions of their overall sales (a mean of 9% and ranging from 0.04% to 33%), the sales had instrumental value in raising brand awareness and helping them gain experience meeting the logistical rigors of wholesale markets.
In contrast, a previous study (Jablonski & Schmit, 2015) had the resources to interview a much larger number of respondents.This data collection method requires very large time commitment to gain respondents' trust (due to the sensitive nature of sharing financial data) and pore over records.The limited budget of our project and time availability of the researchers, as well as lack of response from vendors, all precluded further data collection.As a result, we were not able to customize IMPLAN as we intended.We posit that survey fatigue in farmers and other supply chain actors, as well as tight funding environments, will be future challenges to this kind of work.

Reflections on the USDAAMS Toolkit
On the whole, we believe our study generally addressed the key recommendations from the toolkit mentioned earlier.These specifically include: • Have a concise scope of the project and invested stakeholders who will guide and set the context for the study, in order to ensure useful results.The study was conceived and crafted in close collaboration with the hospital's nutrition services director.• When possible, utilize secondary data sets, but be aware of their limitations.We utilized data already collected by the hospital.• Collect primary data to overcome the limitations of secondary data.We collected primary data to understand vendors' purchase patterns, as no secondary data set existed with this information • Input-output models like IMPLAN are useful but also have limitations.Careful estimation of opportunity costs and customization to reflect the behavior of local actors will create more realistic and defensible scenarios.We carefully addressed opportunity cost, particularly netting out purchases from distributors that were displaced by local purchases.We were unable to calibrate the model to reflect vendors' input purchase patterns due to the small sample size.However, in the end we believe that although we did not field-test all seven modules of the toolkit, the results we produced were sufficient to meet the goals of providing UVMMC Nutrition Services with a measurement (the output multiplier of 1.38) with which to promote its success, gain recognition from the hospital administration and broader community, and create momentum for and inform future program development.

Reflections on Ease of Use
What went well • We had solid buy-in from UVMMC staff and were able to agree quickly to the scope of the study.• The secondary data from UVMMC appeared to be complete and were very well-organized and easy to work with.
UVMMC staff provided all additional information requested by the research team.

Key Takeaways
• Engage stakeholders early and often to define the scope of work, obtain and vet available data, and vet and frame results to meet stakeholder needs.• Include on the research team one or more members with expertise with regional economic studies and IMPLAN; seek outside help as needed.• Using existing (secondary) data saves time, money, and effort.• Be flexible.Some data may not be available at all (such as, in our case, vendors' purchase data that we wanted to use to modify IMPLAN), or may not be of sufficient quality and quantity to allow for credible analysis or conclusions.Be honest and open about the shortcomings of the data and frame the results and implications accordingly.

Implications for Future Research
The major implication of our study revolves around the difficulty of obtaining farmers' and other supply chain actors' financial data.Conducting an analysis using Modules 1 to 6 is challenging and time-consuming; modifying IMPLAN (Module 7) is even more so.For some purposes, including ours, stopping after Module 6 or even earlier may be sufficient to meet the goals of the study.For those who wish to utilize Module 7, we offer a cautionary note and suggestions for how to modify IMPLAN to reflect purchasing patterns of businesses engaged in local food systems.Even when respondents do not feel that expenditure data are too personal or proprietary, the time and effort to collect them are significant.This is a challenge that will likely be faced by many stakeholders in local food initiatives working under tight budget and times constraints.We feel that addressing this is critical to advancing this emerging field."Improving the funding environment" is aspirational but likely not practical in the current environment.Some practical thoughts and ideas, which are not mutually exclusive and should build on each other, include:

Conclusion
This essay reflects on a field test of the USDA local food economics toolkit.We were able to follow most of the key recommendations, including development of clear objectives and scope with stakeholders, use of primary and secondary data, and use of the IMPLAN model (Modules 1 to 6).Through the research collaboration, we were able to publish results in a scholarly journal and fulfill other aforementioned stakeholder objectives (program promotion, recognition, and development).We emphasize, however, the difficulty of obtaining sufficient data to conduct credible modifications of IMPLAN.Specifically, we suggest that researchers and stakeholders create mutually agreed upon goals.It may be that the steps of Modules 1 through 6 (or even fewer) may be sufficient to meet these goals.If modification of IMPLAN is needed (Module 7), it is necessary to budget for adequate time and resources and to have a representative sample of vendors willing and able to provide the needed data.
Future directions of research in this area may focus on developing better, more efficient, and less burdensome methods to gain primary data (O'Hara & Pirog, 2013), by increasing producer participation and overcoming survey fatigue.As one of the first pilots of the USDA AMS toolkit, we hope our study motivates further investigation of the economic impacts of local food systems and encourages collaboration to improve methods and results.

Table 1 .
Overview of the USDA Toolkit • Input-output models like IMPLAN are useful but also have limitations.Careful estimation of opportunity costs and customization to reflect the behavior of local actors will create more realistic and defensible scenarios.It is important to have a team member with expertise in IMPLAN modeling on the research team, especially if modification are made (Modules 5 to 7).
Plan carefully, since data collected as the year progresses rather than retroactively at year's end will be more accurate.•Partner with farm viability experts such as extension educators who are already working with farmers and have access to their financial records.• Create and maintain a database with farm and supply-chain actor characteristics and expenditure patterns for use by researchers.Such a database should be sufficiently robust to guide credible extrapolation, yet protect respondents' confidentiality.