Designing an effective, scalable data collection tool to measure farmers market impacts

The need for an updated framework for all types of farmers markets and the varied levels of capacity to share the impacts of their work led to the development of the Farmers Market Metrics (Metrics) program at the Farmers Market Coalition (FMC), a nonprofit working to strengthen farmers markets across the country. This essay provides a timeline of the steps and partnerships that led to the creation of this program, including the exploration of existing data collection systems suitable for grassroots markets, observations from markets engaged in evaluation, feedback by pilot users of the Metrics system, and best practices and recommendations uncovered during the development of Metrics.


Introduction
Collecting data at farmers markets is not a new endeavor; reports on the impacts of farmers markets stretch back decades (Brown, 2002). However, in most instances the resulting reports were designed for use by stakeholders interested more in the market's role in the larger food system or in measuring one type of project impact within a market (McGuirt, 2011;Minaker, 2014;Racine, Vaughn, & Laditka, 2010;Oberholtzer, 2016;Sadler, 2013;Slocum, Ellsworth, Zerbib, & Saldanha, 2009). In reviewing research conducted at farmers markets since 2004, 1 FMC found that the market organizations rarely participated in the choice of data to be used or joined in the collection of the data. Yet in the last half decade, markets have consistently requested technical assistance from FMC in collecting and using market-level and network-level data. At the network level, state associations of farmers markets have reported frustration during FMC's State and Network Leader monthly calls on the lack of data available, which significantly reduces their ability to advocate for those markets at the state or the federal level. Also notable was the change to the request for applications for the Farmers Market and Local Food Promotion Program (FMLFPP) grant by the U.S. Department of Agriculture (USDA) Agricultural Marketing Service (AMS) in 2014, which now requires the collection of data for specific indicators for each grantee, including sales, customer, and business metrics. It also requires grantees to specify how they plan to collect this data.
Beginning in 2011, FMC began to identify common characteristics and impacts of programs at markets in order to provide technical assistance with the Center for Disease Control and Prevention's Communities Putting Prevention to Work program (CPPW) grantees. FMC's analysis at the end of this subcontract indicated a need for more research into evaluation resources and tools that could be used easily by understaffed market operators.
The multiyear, multipartner process to develop Metrics was meant to address these needs and to advance existing methodologies while building standardized, appropriate tools that allow even 1 The year 2004 was chosen because that year the USDA listed 3,706 farmers markets, slightly double the number of listed markets in 1994 and about half the number listed in 2012. Also, in 2004 markets began to expand their programming by adding the first wireless card processing machines in Philadelphia, Pennsylvania; Prescott, Arizona; and Lewiston, Maine. This made it a milestone year for the farmers market field in terms of numbers and programming. Thus more of the reports written after that year remain relevant to the structure and goals of markets today, allowing FMC to refine its search for reports to those written after that date. low-capacity farmers markets to collect and use data to improve market operations, share data with aligned research entities, and communicate market impacts to stakeholders.

Previous Data Collection Systems Suitable for Grassroots Markets
A range of farmers market-focused data collection methodologies have been used since the late 1990s, each adding to market organizations' efforts to measure their internal and external impacts in a consistent, comparable manner. At least three were designed expressly with farmers markets in mind: Rapid Market Assessment (RMA), Sticky Economic Evaluation Device (SEED), and FM Tracks. Each of these systems greatly expanded the potential for data collection within market organizations by adding new collection, entry, or reporting functions. However, the limited use of these tools by markets over the past decade (as evinced by the continued requests for technical assistance around evaluation and the lack of reports from markets using this data) was noted by FMC and its research partners. In response, FMC added evaluation functions beyond what the tools listed below had already offered, as outlined Table 1 and described in the next sections. The functionality that Metrics added to each approach is in a bulleted subsection.

Rapid Market Assessment
In 1998, Larry Lev, Linda Brewer, and Garry Stephenson devised a grassroots collection toolkit using three methods of data collection under the name Rapid Market Assessment (RMA), which has since been updated (Lev, Brewer, & Stephenson, 2008). The first two methods-Attendance Counts and Dot Surveys-could be completed by markets with a small amount of preparation and a minimum of staff training time. The third method is a qualitative method called Constructive Comments and Observations (CCO) to observe three main components of the market: 1. Physical characteristics of the site: Access, flow of people and traffic, liability issues, and organization; 2. Vendors and products: Product mix, product quality, signage, display, and customer service; 3. Market atmosphere: The "feel" of the market, shopper demographics, interactions and conversations, and educational and community activities.
The resources available for RMA include instructions for devising the team and collecting the data.
The Metrics adaptation of the RMA approach • The Dot Survey can offer markets a rich tapestry of data when gathering qualitative, discrete (yes/no) type questions, but when asking a series of detailed questions of a single respondent, the method is difficult. Metrics allows for the collection and entry of a series of data points for a single respondent, which allows for deeper analysis of types of shoppers (such as weekly versus monthly shoppers). • Markets often use only the Dot Survey without conducting attendance counts, which means the number of responses for the Dot Survey cannot be known to be representative of the market visitor population. Metrics includes training on the importance of conducting both analyses and resources to calculate the responses needed to ensure that the number and sample of responses are representative. • Without the CCO, the analysis lacks the contextual details of the market's physical space and organizing intention in relation to the data collected. Metrics requires the addition of profile data for each market whose data is being uploaded. • In terms of data management, the main drawback to the use of Dot Surveys is the time and skill necessary to transfer the answers to a spreadsheet or online form for analysis. The lack of preformed questions also adds more work for the market to refine questions suitable for coding. Metrics includes preformed survey questions refined by FMC's research partners and the ability to add answers directly into the online account using a smartphone or tablet.

Sticky Economics Evaluation Device (SEED)
In 2002 ( McCarthy, 2007). SEED used survey responses from market visitors to offer a snapshot of a market's economic contributions in a PDF report. Market organizations uploaded their raw data into their online SEED account, where they were run through an economic multiplier and formatted into standard reports with text and tables. The reports covered core market statistics, market visitation, market economic impact on vendors and nearby businesses, and total combined economic impact.
The Metrics adaptation of the SEED approach • Markets using SEED reported that they were frustrated with the inability to delete certain questions or to add others. Metrics allows for the addition or deletion of the survey questions. • In discussion with researchers during the development of Metrics, many reported that they were uncomfortable with the multiplier and seasonal calculations in SEED not being visible or editable. For now, Metrics data can be downloaded into Microsoft Excel spreadsheets for the market and its partners to calculate direct, indirect, and induced effects based on local multipliers and other economic calculations. • The SEED final report is only available in a PDF format, which requires a separate design phase in order to use the individual pieces of data on social media or in graphic form. Metrics offers a graphic format of the data either collected on a one-page summary report or printed or shared as individual metrics.

FM Tracks
For those markets using a centralized SNAP/EBT 2 system to process cards at the market booth, daily data is often required by the partner or funder of the matching SNAP incentive or other coupon program. FM Tracks was created to meet that need with development was led by Dr. Darcy Freedman through a partnership between Case Western Reserve University, the Prevention Research Center for Healthy Neighborhoods, and Wholesome Wave. FM Tracks includes an iOS application and website that work together to track data by market managers operating one or more markets and for healthy food incentive networks. This includes sales data, customer data, and market day information.
By engaging with SNAP customers at the market organization's booth, the shopper can be asked to complete a short survey while their card is processed and tokens are counted out. FM Tracks also gathers data from the transaction summary available through the processing system at the end of each market day, calculating metrics such as total dollars spent on SNAP, average dollar amount of the transaction, number of transactions, and, through a unique FM Tracks I.D. number, even tracking which shopper had used their card previously at the market. Once entered into the FM Tracks app, a visual dashboard of the data is made available to both the market and the network.
The Metrics adaptation of the FM Tracks approach • The FM Tracks tool was already in a later development stage during the early pilot testing of Metrics, and the two teams engaged in direct communication, which led to Metrics survey questions being included in FM Tracks and Metrics using nutrition program questions from FM Tracks. Still, the availability of two systems that track much of the same information has been a source of concern for users, so Metrics prioritized developing protocols for uploading data collected via FM Tracks into a Metrics account in batches. • The visual dashboard of data present in FM Tracks also influenced the addition of a data dashboard into Metrics. However, the FM Tracks tool does not offer detailed market-level reporting of the data collected, resources to aid markets in collecting other data, or resources for using data once it is verified. Metrics includes those as well as visual, dynamic dashboards of the data at the market level. • FM Tracks is most often used for collecting program data on nutrition incentives and is focused on aggregating that data for market networks and partners. As a result, the data collection and reporting functionality for market-or vendor-level analysis is still limited. Metrics is designed primarily for markets' use of the data, with markets as owners of the data, such that they are able to edit the reports and share specific data points with vendors or networks as needed. • For markets that are not managing a centralized processing system (i.e., where vendors process transactions directly or a partner manages the transactions on behalf of the market), FM Tracks is not as useful in gathering surveys. Metrics surveys and tools are designed to be useful at many different points during the market shopper's visit.

Other Added Functions of Metrics
• None of the systems allowed for vendor-level data to be included, even though most markets collect vendor demographic and production data via their annual application or renewal form. In response, Metrics added demographic questions that can be added easily to the annual renewal form, which are then uploaded into each vendor profile. Once in the system, that vendor can be added to other, separately managed markets, although the accompanying transaction data for each market day is accessed only by that market and the vendor. This solves the issue of multiple markets entering the same vendor more than once, which could lead to double counting any vendor-level data when aggregated across markets. • The lack of accompanying training materials on basic data-collection methods in all existing systems was identified as an issue by network leaders. Also identified were the lack of resources for aiding markets and networks of markets with building an evaluation plan, including the steps of finding volunteers, selecting metrics, and using the data. FMC has added resources to its site to respond to these needs and offers monthly calls with users. • Dynamic network-level directories that offer current locations, hours, and other profile infor-3 USDA AFRI award no. 2014-68006-21857; Principal Investigator Dr. Alfonso Morales. mation were also identified as vital by market stakeholders as more shoppers use online resources to seek out food and entertainment and app developers search for up-to-date directories to link to their search function. Metrics now includes this function.

Timeline of the Development of Farmers Market Metrics
To assist in identifying the most useful metrics, in 2011 FMC began work on market impacts with researchers and markets. Those collected metrics (around 130) became the basis of the metric selection over the next iterations of FMC's evaluation work with markets and researchers. In 2013, in collaboration with Dr. Alfonso Morales, professor of urban planning at the University of Wisconsin-Madison (UW), FMC began work on an ultimately successful grant application to the USDA National Institute of Food and Agriculture's Agriculture and Food Research Initiative (AFRI) 3 that would fund two components of the Metrics research: refining the collection methodologies and prototyping summary reports that would result from the data collected. In 2017, FMC tested the online site and automatic reporting functions now included in Metrics.

Pilot: Prototyping Reports
The value of a graphic summary report to incentivize data collection was tested through a sixmonth Knight Prototype Fund project with three steps: choosing 10-15 metrics from the list that could be pulled from past seasons' records (e.g., number of vendors, acreage in production) and would be of interest to current markets stakeholders; designing templates for recording and displaying the metrics in one-page reports with infographics; and piloting the metrics and templates during the 2014 spring market season. Test markets were selected from those that volunteered after the project requirements were shared in a national webinar. The markets were selected based on their size, age, location, and plans for using the reports with different stakeholders. The pilot markets were: Using 2013 data, summary reports were created, 4 with the pilot markets sharing them through their existing communication channels. Six of the eight markets used reports in their social media channels and within their market community (see an example in Figure 1). The other two did not use the data, reporting capacity challenges in entering and verifying the data. Once the project was completed, a survey was conducted to ask for feedback from the markets. The market representatives felt that sharing the summaries helped to strengthen their relationships with partners and funders and also encouraged customer loyalty. They reported that some vendors noted that seeing first-hand how that information would be used and its potential for increasing customer counts and sales helped them better understand the request for data.

Pilot: Refining Data Collection Methodology
In 2014, FMC partnered with Dr. Morales on the 4-year Indicators for Impact project funded by the USDA National Institute of Food and Agriculture's AFRI to test and refine methodologies to be used by market organizations during specific collection phases.
In  During the pilot project, the Mississippi Farmers Market used this graphic of its mileage calculation in print media to highlight the regional impact of its vendors.
vendors, setting (rural, suburban, urban), and staff size, as noted in Table 2. During the project, the markets participating in the indicator project offered feedback via monthly calls and emails on the process of collecting data using the resources developed by the project team. As the markets all had different levels of experience in collecting data, the feedback was extremely helpful in prototyping collection resources for a wide range of markets. 5 The participating markets were required to have operated for a full season, to name a specific contact person to participate in online and inperson meetings, to collect the data as instructed, and to collect the same four metrics (as well as the choice to collect others, too): • Number of visitors • Number of vendors • Acreage in production • Miles traveled (from production to market) These four metrics were deemed most useful for aggregation and to be of greatest interest to current market audiences across the U.S. By requiring only four metrics, each market was able to add two to four more unique metrics for their unique needs. The decision to encourage the markets to collect around six metrics was made by the UW/FMC team based on the findings from the earlier FMC Prototype Report project, which indicated that markets that were prone to selecting a long list of metrics often had difficulty in completing all the collection necessary.
The metrics chosen for this pilot were narrowed down to 37 by asking these questions: • Which current audiences want and will use the data: a municipality, a partner organization, or the market itself in an annual report? • Which group(s) within its market community does the market want to measure its impact upon: the farmers, shoppers, or the larger community? • What type of benefit is added to the market community: economic, social, intellectual, or ecological capital? • Can it be collected by low-capacity markets using one of three methods: (1) review of existing documents used by the market (such as card processing transaction summary reports), (2) conducting surveys, or (3) conducting tallies? These methods were chosen by the project team by conducting a literature review of existing methodologies 6 (Jeong, Morales, & Roubal, 2015) during this project.
The markets were given their own specific set of collection protocols based on the metrics chosen (Suerth, 2015) and an annual stipend to use as needed for data collection staff or to pay for staff time to do data entry and to attend meetings. All chose to enlist volunteers to collect the data, and only one (Ruston) used part of the stipend to hire a lead person, although some did supply incentives for their volunteers such as market tshirts, free drinks or food, or market gift certificates. One market (Hernando) used an organization that recruited corporate volunteers while another (Spotsylvania) enlisted a Boy Scout troop for some of the data collection and gave a donation to the troop in return. Two others drafted university students to serve as their data collectors. Three of the nine markets had trouble amassing enough volunteers on all of the selected days, requiring another day to be selected.
The data entry phase was problematic for almost all of the markets, with most unable to complete all of the data entry deadlines set by the project team. The reasons offered for these data entry challenges were reported by the markets as lack of time set aside for the market leader to complete this part of the process, issues with the data entry portal, or issues due to assigning a volunteer who did not complete tasks or made multiple mistakes in the data entry. 7

Vendor Response to Data Collection Pilot
During the Indicators project, current market vendors were generally supportive of the data collection project. Those who offered strong support were often current or past board members of the market organization and therefore were aware of the market's need for data to fundraise or to secure support from stakeholders. Even with that support, the markets that had not previously collected any data from vendors reported resistance during the project. Sales data was reported as the most difficult to collect, but markets also struggled with collecting complete demographic data. The main barrier to completing demographic information was reported as lack of time allotted for the market staff to collect and enter that data for each vendor.
In many markets across the U.S., collecting sales data from vendors is still seen as unlikely or even "impossible." For those that do collect it, it is often based on a stall payment system where vendors report daily sales to the market and a percentage of those sales 8 are calculated for the market fee. Other markets use either a flat fee system per market day or a single annual fee. Since four markets in this pilot already gathered weekly sales data, the markets that did not already collect this data had access to best practices and support by the early adopter markets. As those collecting it only for reporting purposes did not need individual vendor calculations, the project team introduced a system in use at other U.S. markets: a daily anonymous vendor slip. The slip is handed out at the beginning of each market, and once completed is either handed back to a market team member, who then checks that vendor's name off a list, or dropped into a canister at an unstaffed table and the vendor checks off their own name from the list. Having the vendors name checked off a list allows markets to remind those who have not submitted their slip, raising the response rate. Although not entirely anonymous, since the data is not entered next to any business name or vendor category and only the aggregate total for the day is used, most vendors are willing to submit this data. The market could tailor the slip to their own data needs, deleting fields that are not being collected at their market or adding fields such as the amount donated by that business to gleaning programs. One market had only one vendor refuse; another market found passive resistance to the collection (no outright refusals, but sales slips were promised and yet not turned in regularly); two other markets expected more resistance and so delayed asking for weekly sales until later in the season, which affected the completion since many vendors could not retrieve all the weekly data at that later time (Wolnik, 2016).

Data Use Process and Challenges in Data Collection Pilot
The markets in the Indicator pilot were then encouraged to share the data using graphics designed in the FMC's previous prototype pilot. The markets used the data in social media postings, emails, and website postings in the same manner and at the roughly the same rate as the Prototype Report pilot. Usage tended to cluster at the end of the season or the calendar year. The graphics were used during annual vendor meetings, usually held at the start of the next year. Many markets reported during the data usage phase that some of the metrics were not as useful as they had hoped and again offered an indication to FMC that helping markets with selecting the right number of metrics at the outset of the process is vital.

Pilot: Markets and Networks Test of Metrics Site
In 2017, FMC worked with 70 markets to beta test the Metrics resources and website; 58 of those were collecting data in a shared project. These shared projects were labeled as "networks" and offered a key update to the underlying data structure of the Metrics system; as projects shared across market organizations have become more common, the need to share aggregated data with a variety of partners has increased. In response, an account network category was created and defined in Metrics as organizations that need to aggregate data among a group of markets but are not engaged in the operation of the markets themselves. The participating networks included the Michigan Farmers Market Association (MIFMA) and the Virginia Farmers Market Association (VAFMA). Twelve markets in the 2017 pilot used Metrics to collect data for a single organization's use.

Process and Challenges Among Sample of Participants in the 2017 Pilot
The 12 individual markets who elected to use Metrics during the 2017 pilot had all collected data in the past, had specific ideas about what data they wanted to collect, were willing to offer input to the development of Metrics, and found out about the pilot via a national webinar.
The experiences of two of the participating markets and two networks who used Metrics in 2017 are highlighted below. The Countryside Farmers' Market at Howe Meadow Market in Peninsula, OH, and the Charlottesville, VA, market represent different market types and reasons for collecting data. MIFMA used Metrics to collect data on the economic impact of a program implemented at 15 market sites and to increase the comfort level with collecting data across all member markets. VAFMA chose seven markets in the Richmond area to aggregate data, with the goal of generating a data-driven social media marketing campaign in collaboration with bloggers and food writers.

The Countryside Farmers' Market at Howe Meadow
The Countryside Farmers' Market at Howe Meadow was founded in 2004 to provide a directto-consumer outlet for farmers in the growing Countryside Initiative program as well as regionally. The Countryside Initiative is an innovative land-use program created via a partnership between Countryside Conservancy and Cuyahoga Valley National Park to revive the agricultural heritage of the Cuyahoga Valley.

Process and challenges during the 2017 pilot
Countryside already collected the vendor sales and visitor count data that Metrics captures and shares aggregate weekly sales data directly with its vendors, comparing the previous year's data for the same week with vendors via email (as shown in Figure 2). The market did not previously collect or compile any vendor demographic data and focused their work with Metrics on adding that data. No significant challenges were uncovered by this market during the pilot, as the market had participated in data collection on its own. The market did request some additional technical assistance in the data entry phase from Metrics Program Manager Marian Weaver and participated in phone calls on the reporting phase with Darlene Wolnik, FMC's Senior Advisor.

Data use
The market organization wanted to use its data for advocacy, as their food and farming businesses largely have been overlooked as possible participants of incubator programs throughout Northeast Ohio. Having the combination of vendor sales, vendor demographics, and visitor count data, the market feels better equipped to encourage these programs to view markets and market vendors as critical components of the economic system. Metrics collected included: • Total number of vendors on each market day • Total number of farm vendors on each market day • Total market sales • Number of visitors • Vendor demographics Another of Countryside's reasons for using the Metrics program was the ability to create a communication package that could be customized to specific audiences to demonstrate the market's value to each audience's specific interests (Figure 3).
Since Metrics allows for individual metrics to be printed, the data can be easily added to a single email, social media post, or funder's report. Like most markets, Countryside has many stakeholders it wishes to communicate with, including its vendors and shoppers, Extension offices, public health partners, and municipal and regional governments.

Charlottesville City Market
Charlottesville City Market was founded in 1973 by local farmers, the Cason brothers, as a street-side farm stand. The market is now managed and operated by the city of Charlottesville and hosts over 100 farmers, bakers, and artisans on Saturday mornings.
The market needed a snapshot over a wide swath of activity at the current downtown market in preparation for the city's long-term planning process to determine the market's new location. The market itself collected data using Metrics to calculate all vendor-and market-level sales, examples of which are shown in Figure 3. The market was also supported with data collection by a group of students under the supervision of Dr. Paul Freedman, associate professor at the University of Virginia (UVA) and founding member of the UVA Food Collaborative. Freedman was an early supporter of the Metrics project and helped refine training exercises for Metrics during UVA's 2012 Morven Summer Institute course on farmers markets. Having researchers already familiar with Metrics collecting data using the same survey questions and methodology means more data can be managed by that market within Metrics.

Process and challenges during the 2017 pilot
This market was one of the first individual markets to enter all its vendor data (over 100 accounts) into the Metrics system. That level of data entry uncovered functional changes to the Metrics site that were communicated to the Metrics team by the Since this use of the Metrics system in Year 1 happened during one of the major development phases of Metrics, the market did not have access to the automatically generated reports until the late spring of 2018. In response, FMC volunteered to individually design the reports as needed and assist with finalizing the data that had been entered.

Data use
The data on the number of visitors and market sales have proven most useful to the Charlottesville City Market. Market Manager Justin McKenzie said, "The fact that we bring an average of 4,000-5,000 people to the area is astounding in a city with a population of just under 50,000." Visitor attendance was shared with and favorably received by staff at the departments of parks and recreation and economic development, the city manager's office, and market vendors at their 2018 annual meeting. Tracking vendor demographics ( Figure 4) and sales proved particularly useful this past year, as the city of Charlottesville experienced a traumatic event on August 12, 2017, when a white nationalist group precipitated a riot that descended on the downtown area just a block from the farmers market. That data offered a window into one indicator of recovery as the city struggled to recuperate from the tragedy.
McKenzie reported to the FMC project team, "We shared our sales numbers with the City Manager's Office and they were surprised and delighted to hear that our markets were performing well after the events of August 12 th when many of the downtown businesses saw steep declines in revenue."

Michigan Farmers Market Association Network
The Michigan Farmers Market Metrics program was designed to capture and evaluate economic contributions and market impacts to further the understanding of market outcomes. The program was intended to generate data for MIFMA to build advocacy messaging on behalf of Michigan markets. The association chose the metrics to be collected at fifteen participating sites through consultation with its Economic Impact Advisory Committee in partnership with Steve Miller, an economist at Michigan State University. Miller served as an advisor throughout the project and assisted with selecting metrics, identifying survey questions and collection methodology, and analyzing the data. Since vendor-level data proved difficult to obtain for some of the markets, total market sales were calculated instead based on visitor survey responses. The project team aided the markets with direct data collection and hosted monthly calls to ensure that the entry deadlines were being met. At the end of the season, summary reports were created for each market using the Metrics template, although MIFMA changed the design and layout slightly.
Participating markets for this data collection project included Allen Farmers Market, Bath Process and challenges during the 2017 pilot MIFMA expected resistance from the markets on collecting vendor sales data and so did not require those metrics during this project, instead using visitor surveys to capture economic impacts. The MIFMA advisory team also wanted the added functionality of calculations for data across the network, which is not yet included in Metrics, so those calculations were completed by the MIFMA team separately. MIFMA also supplied on-site staff to individual markets as needed to complete collection responsibilities.
Data use MIFMA reported that the summary reports were posted on its website, and it shared the reports with each participating market manager and with state agencies such as the Michigan Department of Agriculture and Rural Development.

Virginia Farmers Market Association
Established in 2011, VAFMA's mission is to support farmers markets through education initiatives while building opportunities for collaboration, networking, advocacy, and innovation that support the growth and sustainability of farmers markets statewide. The association needed data to help demonstrate the positive impact Virginia farmers markets have on their communities, so they joined FMC and the District of Columbia market network in a three-year grant to use the Metrics automated reporting features to share data with public officials, potential funding partners, members of the press, and other influencers. VAFMA chose to focus its efforts in the first year of the project on markets in the Richmond area. The VAFMA Metrics coordinator hired by VAFMA to oversee the project was located in Richmond and was already actively involved with Capital Area Farmers Market Association (CAFMA), which had been meeting regularly for several years. VAFMA selected the CAFMA markets based on location stability, years in business, interest in data collection, and to include a variety of market types. The participating markets were Birdhouse Farmers Market, Carytown Farmers Market, Chesterfield County Farmers Market, Goochland Fairgrounds Farmers Market, Lakeside Farmers' Market, and the Manakin Market.
The markets were asked to collect visitor attendance, number of vendors, miles to market, and acres in production, the same metrics that were required by FMC and UW during the Indicators project that were deemed most useful to a wide set of audiences and easily collected and aggregated. Based on FMC's earlier pilots with Metrics, VAFMA was encouraged to move more slowly in requiring sales data to be collected in Year 1 and also to work individually with each market organization to choose the number of metrics most likely to be used at the end of the season.

Process and challenges during the 2017 pilot
During the conference calls held by FMC and VAFMA with the markets at the beginning of this project, the issue of the data being visible to a network-level account was brought up by one of the markets. As the board of VAFMA includes market managers, some of the CAFMA pilot sites questioned the ability of those leaders to access others' individual data. FMC assured the markets that the network accounts do not allow access to individual vendor data and do not allow the network account to edit the market-level data. FMC added a privacy agreement for networks for this reason, which resolved the issue.
Near the end of the first year of this project, the Metrics site was updated to incorporate more vendor data as requested by many of the pilot markets across the system. As a result, the individual data collected and already entered by the VAFMA markets became incomplete, which was a source of frustration for VAFMA. The Metrics program manager resolved the problem, but that resolution was delayed until late spring 2018, pushing the data-use phase back at least one season. In 2018, markets across the state were allowed to join the pilot; more than a dozen markets were added. Meeting the needs of a larger number of markets across a wider geographical area has been a challenge for the VAFMA coordinator, as she was close to the 2017 market pilot sites and had participated in their monthly in-person meetings where the leaders discussed issues and shared strategies. The markets added in 2018 were self-selected, and the coordinator did not have previous relationships with their leaders. The self-selecting process also meant that the new markets have varied levels of comfort and experience with data collection, which has meant the need for more one-on-one training and calls with markets from FMC and the VAFMA coordinator.

Data use
The network-level calculation and use of the 2017 data are underway as of the writing of this paper, with some market leaders already reporting they used the data on social media posts and at events when presenting with other food and farming advocates (see Figure 5). VAFMA plans to share the data with food bloggers and writers to add meaningful data to posts and articles centered on local food and farming.

Observations from FMC's 2014-2017 Pilots
• Market leaders are endlessly curious about ways to measure and understand their markets, but have a list of tasks that often push that curiosity aside-especially in the summer. In response, FMC recommends creating assigned roles for the data collection team and choosing training materials for seasonal volunteers and interns to assist in setting expectations. It also recommends identifying suitable volunteers and presenting webinars and training for network leaders in the winter or early spring. • The pilot markets and network leaders reported a wide spectrum of reasons for collecting primary data, including producing annual reports, reporting program outcomes, and sharing shopper data with market vendors to propel more product development. • Collecting and managing a data collection team is a time-consuming process and cannot start in the busy part of the market or farming season, nor be built from scratch each year. • The skill level of the individual market leaders in using an online system to understand, plan, and use data is varied and has required FMC to offer more basic computer tutorials during calls with market staff. • During the 2017 season, many market staff began to request an integrated database for managing vendor applications, with that data feeding directly into their Metrics account. In response, FMC partnered with the online vendor management platform Farmspread to offer integration with vendor applications, thus cutting out the need for users to survey vendors for demographic data. In 2018, an additional subscription to Farmspread is available to all users, although it is not required to use Metrics. • The automatically generated graphics within Metrics have been found to be extremely appealing to markets and to network leaders, with both groups requesting more functionality in reporting visitor data and more vendor data in future iterations of the reports. • The graphics have been used primarily on social media or to meet the requirements of project reporting. In many cases, the individual metrics have been reported as more useful than the entire summary report, as many of the markets felt the final data was compelling only in some instances. • While many of the networks using the Metrics program in 2017 saw value in using Metrics in collaboration with an academic partner, none of the individual markets have yet reported sharing any of the collected data with researchers to receive further analysis. That lack of sharing indicates the need to create more resources and partnerships to facilitate markets and researchers working together to analyze and use the data collected by markets. • While understanding the value in aggregating data for regional impacts, market staff remained concerned about how the sharing of data with networks that included nearby markets could lead to misuse, such as poaching of vendors. As a result, they are often less eager to engage in projects that include other nearby markets. • Documentation that explicitly laid out the ways data were shared and the limitations to that sharing did not always assuage market staff at networks or vendors at markets. This may indicate the need for networks to move more slowly in requiring sensitive data from their markets or be aided by examples of data use by networks to build support for markets. • To add more usability to the reports, in 2017 a series of citations from other reports (FMC, 2018) were added that offered relevance to the data point collected. Those citations have been shared across many social media postings by markets including those not yet using Metrics.

Conclusion
The primary goal of FMC's Farmers Market Metrics is to foster an appropriate and responsive culture of data collection that encourages markets to employ easily understandable metrics, devise a clear collection strategy, and share the data among vendors and partners. The iterative process of testing the site with markets eager to solve their data collection dilemmas has helped FMC in this process, but has also led to frustration among early adopters, who were hoping for one complete tool that serves all their data management and reporting needs. Aiding those markets in becoming skilled evaluators of their many impacts will require more pilots, more resource development, ongoing customer service training, and "train the trainer" resources for network leaders and project partners.
It is also clear that gaining vendors' trust in sharing sensitive data remains the largest hurdle in implementing standard data collection methodologies across markets and regions. The rise of vendor management software, data structures that safeguard individual data, and technical assistance for vendors to compare their data to market numbers are the current development focuses of Metrics through its partnership with Farmspread. In many cases, the market manager position is already evolving to prioritize data collection and use, leaving market-day logistics to newer staff, trained volunteers, or interns. To support that evolution, funders should prioritize staffing support to allow market leaders more time to oversee data collection.
Funders and network leaders must also exercise patience and support for each market's level of capacity and comfort with data collection, and assist them analyzing and using the data. Continuing to advance tools and training that help markets choose the right (and the right number of) metrics that speak to many audiences will assist markets in limiting their choice of data points at the outset, which will reduce "survey fatigue" and encourage more disciplined data collection.
Aiding that process, the emergence of analysis tools such as the USDA's Local Foods Economic Toolkit, coupled with consistent support from academic partners will encourage market leaders to delve more deeply into economic data and to feel more confident sharing results. Once enough data is collected in standardized methods across market seasons, other toolkits that can measure metrics that show ecological, intellectual, or social capital benefits may follow. This is likely to add new metrics to the Farmers Market Metrics program.
Finally, even though the goal of a dynamic evaluation process being led by market organizations is closer to reality, it is only in the early stages of widespread acceptance. This conclusion, offered by farmers market founder and SEED developer Richard McCarthy in 2012, illustrates the situation then and now: If the farmers market field was mature, I would have an extensive roster of measurement tools to share with you. It is not mature. Rather, an unexpected and unorganized generation of actors in civil society are [sic] taking social entrepreneurial risks to reinvent the ancient mechanism of farmers markets as agents of social change in communities half-starved for products, people and place. The food revolution has only just begun. I am hopeful that measurement will follow shortly.