Project Topics
Get Complete Project Material File(s) Now! »
Results and Analysis
As described in Figure 3, the results from the collected data were used to create the model that is the goal of this study. In the following sections are first results from the data collection presented, followed by the model, its content and testing of it.
Analysis of secondary textual data
Several sources of secondary textual data were used to create an overall picture of the problem and as the basis when designing the evaluation model.
Referrals from the Swedish government to the council of legislation with suggested law concerning cultural heritage.
The analysis of the legislation gave an insight into what the government intended the public funded museums to be doing. From the legislation, it was possible to develop questions for the interviews and the survey.
Reports from the authority for cultural analysis.
By analysing the content of the annual report, it was possible to gain insight into the market the museums are working within in order to gain a deeper understanding. It also made it clear that the museums are collecting data and send it to the Agency for Cultural Policy Analysis. By using this information, it was possible to create questions for the survey and for the interviews.
Annual reports from Swedish museums
The authors collected and read several annual reports from museums dating back up to ten years. Analysis of these was done in order to understand how and what museums report to the public considering their activities. The findings from the reports gave input for questions for the survey and interviews.
Program declarations from Swedish Museums including vision and mission statements as well as organisational strategies.
As the mission was important in order to measure a museums performance it was necessary to understand how missions for museums are written. By analysing program declaration, the authors came to the understanding that museums get mission statements from the funders called uppdragsbeskrivning. Some museums use it to develop their own mission statement.
Survey
The survey was sent to 51 museum managers for central and regional museums. This was the total population of possible respondents for these types of museums and it was decided to send the survey to all of them to ensure that the sample is large enough to represent the population of interest.
28 of the 51 selected respondents completed the survey, giving a response rate of 55 percent. With a total population of 51 and the 28 responses being the sample size, the margin of error is calculated to 13 percent within a 95 percent confidence interval at a normal distribution of 50 percent.
To what extent do Swedish Museums measure and evaluate their performance today?
The first question of the survey was: Do you currently use any type of established, structured model to evaluate your operations and measure performance and goal fulfilment? The results show that 50 percent of the responding museums do not evaluate their performance and 50 percent use some sort of evaluation model as seen in Figure 4
The respondents answering that they did evaluate their performance received the follow- up question: Which model do you use? Where the answer options were: (1) An existing model, for example, The Balanced Scorecard or a modified version of an existing model and (2) A model developed specifically for our organisation. 36 percent of the evaluating museums are using an existing model and 64 percent are using a model developed specifically for their organisation.
Since these models developed specifically for individual organisations are unknown as well as the included factors and criteria being measured it could not be determined whether they fulfilled the formal requirements for a PMS as described in section 2.3.2. Therefore, it was decided that these responses could not be counted with the ones using an established evaluation method but had to be considered together with the ones where the organisations did not evaluate their performance.
When adding together the results from the first question with the results from the follow-up question is the outcome that only 18 percent of the museums evaluate according to what is considered as a proper evaluation model, this is illustrated in Figure 5.
Important factors and criteria to include in an evaluation model
Both respondents that evaluate and those that do not were asked which factors and criteria are important to include in an evaluation model. The respondents who are currently using some sort of evaluation model, including one developed for their organisation, were also asked to rank the importance of each factor and criteria on a scale from 1 to 6. The results of their responses, ranked in order of answered importance, are shown in Figure 6 where the colour codes each corresponding to one of the sections in the new museum law as follows and the perspectives in the balanced scorecard for museums illustrated in Figure 10.
Financial / Stewardship
Public Activities
Knowledge Acquisition
Collection Management
Respondents not currently evaluating performance were asked which factors would be important to include in an evaluation model of they were to start using one for the operations at their respective museum. The results ranked in order of the number of answers each factor/criterion received and are shown in Figure 7 As seen by the colour coding both groups of managers highly favour factors related to public activities and financial factors. Factors regarding knowledge acquisition and collection management have received a lower ranking in both groups of managers.
What is an acceptable result?
The respondents answering that they did evaluate their performance received the follow-up question: Is it within the framework of the model specified which measured values that represent an acceptable result? with yes and no as answer options. The results showed that out of the museums doing some sort of performance evaluation had 58 percent not specified what would be an acceptable result. If adding the museums that do not evaluate at all to that number, the result is that 81 percent of the respondents have not established what is an acceptable result for the operations as shown in Figure 8. This is a reasonable assumption as it would be impossible to determine an acceptable result without having evaluated performance in the first place.
Why not evaluate performance
The respondents that answered no to the first question of the survey, whether they currently use any type of established, structured model to evaluate your operations and measure performance and goal fulfilment, were asked why not. There were four pre-set options as well as a free text field. The answers from the free text field written by the respondents were formulated so they could be interpreted into the pre -set options and easier presented in the results as shown in Figure 9. None of the respondents chose the option: There is no demand for it from the board/owners.
The same respondents were also asked the question: Would it be interesting and useful for your organisation to start to evaluate your operations and measure performance and goal fulfilment if there was an appropriate model that was adapted to your operations and easy to use? With the answering options, yes and no. 100 percent of the managers answering that question answered yes.
Interviews
The unstructured interviews with selected specialists in the field served the purpose of proving relevance to the research and to verify that the design process was done using correct information and included appropriate factors.
The respondents were in forehand sent drafts of the report and did comment on content as well as providing additional information that had previously not been included. The result from the interviews supports the result from the survey that museum managers are mainly interested in performance related to public activities and less in knowledge acquisition and collection management. It was also pointed out that well-formulated evaluation criteria in these sectors would be useful.
Other results from the interviews were that even though there is a need for an evaluation model as the one being presented in this study and that there is a general interest to know more about it would implementation likely be difficult and could possibly face resistance from the affected organisations. As this report is delimited from implementation is this result of marginal importance for the study but nevertheless worth mentioning.
The final interview that was conducted was done with a museum manager as part of the evaluation process after having designed and tested the model which is described further in sections 4.5 and 4.6. The manager was presented with a brief presentation of the model and the results from the test museums. It was then discussed how this could be used both as a decision support tool and when reporting back to the board of the museum. This final interview concluded the design process and the feedback that was given was that it could be a useful tool for museum managers, both with public and private ownership.
Performance Evaluation Model adapted for museums
The literature review showed that there are criteria of what performance is considering PM and that efficiency and effectiveness were good indications of performance of an organisation. In order to construct a PMS, there are criteria to consider in order for the PMS to be viable. Paulus (2003) presents five design demands that must be fulfilled when designing a PMS for non-profit organisations. The literature review presents OM as a good management model for museums as it focuses on the value creation for the customer rather than purely financial decisions. The OM also fits well with prop. 2016/17:116 as both consider competencies to be of utmost importance. In Chapter 2, there are a couple of alternative PMS presented from which BSC has been researched if it is possible to use for museums. In section 2.3 is it described the research done on different KPIs for use within PMS for museums.
Using the definitions of what criteria is needed for a performance evaluation model as presented in Chapter 2, the Balanced Scorecard model was chosen as a basis. BSC was chosen since it fulfilled many of the criteria set up by Kennerly and Neely (2002) as well as Cocca and Alberti (2010), presented in section 2.3.2. The fact that Zorloni (2012) had presented a model using BSC for museums proved that it was viable to use it as the foundation for the model in this study.
As presented in Chapter 2, for an evaluation model to be valid it needs to be grounded in the strategy of the museum using it. As Zorloni (2012) points out, not all museums have or use a strategy and therefore in order to make a general BSC, the authors must find some other way to give the model a way to prove the museum are working the way the funder intended to. As presented in section 2.2, there will be a new law aimed for museums. Prop. 2016/17:116 will be the basis politicians funding museums will use when defining what they want the museums to be doing. This is often presented in an “Uppdragsbeskrivning” which translates into the English word, mission.
According to Slack and Lewis (2011), in order for an organisation, in this case, a museum, to achieve its mission, it is necessary to fulfil the customer demands. As presented in chapter two, the funder can be considered the customer of a museum. This makes the mission statement by the funders even more important for museums. As Matassoro (2007) points out, the arms-length distance and the fact that the politicians are buying for the public means that the mission statements are very general and hard to interpret for the managers. This is as, presented in chapter two, supported by Lindqvist (2007).
In order for the managers to properly be able to analyse how the museum is performing as well as getting as broad a picture as possible, it will be necessary to incorporate several variables into each KPI. By building the model in a calculation program like Excel it will be possible to synthesise the calculations, which is a criterion according to Paulus (2003).
The model
As presented in section 2.2, prop. 2016/17:116 intended for museums, is the starting point of how a museum’s mission is created. Therefore, it was decided to use three sections from prop. 2016/17:116 as three of the perspectives used in the BSC presented and illustrated in Figure 10. The reason for doing this is that many stake holders’ perspectives are incorporated into these sections. It also fulfils the criteria; the system should provide an easy to understand overview of the organisation’s performance, Kennerly
Neely (2002). The fourth perspective Financial/Stewardship is maintained as it was in the original model described in section 2.3.3. This was done since there will always be a financial aspect to organisations and for this type of organisations, a part of operations will be dedicated to administration and overhead and then covered in this perspective.
The model was built in excel where one page was used to add data needed called Data, and another page was built up using the four perspectives presented below called performance. The information in the Data page was designed to be inserted manually. In the performance page, the formulas presented in the BSC model were set up in such manner the model collected the right information from the Data page and made all calculations, without human interference. Paulus (2003) called this synthetize and it was part of one of the design demand for a PMS for museums, which can be found in section 2.3.2.
When designing the model an assumption was made, that the measurable period is one year as a standard and the KPIs were designed with this in mind.
Public activities (Publik verksamhet)
Relying on theories presented by Zorloni (2012) and prop. 2016/17:116 the KPIs used in this perspective should be based on the museum’s users and how the community perceives the museums.
Market share
The result of the survey shows that both the managers that do evaluate and those that do not evaluate the organisational performance, consider the number of attendees to the museum as a very important KPI and according to Gilhespy (1999) attendees can be used as a KPI measuring the effectiveness of a museum. The problem is that the attendance alone does not tell the museum if it is performing well or not. This can be corrected somewhat by looking at attendance over time. Even so, it is not possible for a museum to know if the changes in attendance are just normal variance or if the change to attendance is driven by external changes to the market such as tourism or populace growth, or if it is the museum’s activities that drive the changes. This problem is illustrated in Figure 11 where, when the tourism increases the attendance goes up, and when tourism decreases attendance to the museum goes down making it hard to decide if the change of attendees to the museum are a product of their activities (Jönköpings läns museum, 2005 – 2015). The number of accommodation nights has been reduced with 140 000 per year in order to make the picture clearer.
In order to minimise that problem and to try to measure the museum’s activities excluding external changes, the BSC model measures the market share of the museum. By using the formula presented below, it will be possible to calculate a market share in percentage for the museum which will decrease the impact of changes in tourism.
∗Populace used in the formula is acquired through Statistics Sweden (Statistiska Central Byrån) statistical database and based on the area the museum is meant to serve, for a regional museum the region is a rational choice. This KPI assumes that all inhabitants should attend the museum or one of their activities at least once during the measured period. For a national museum, the area that is plausible for the populace to have easy access to the museum is also necessary to decide. Just as regional museums the region the museum is situated in would be a good choice.
The variable nights spent are also acquired from SCBs statistical database using “Nights spent. All hotels, holiday villages, hostels, camping sites, commercially arranged private cottages and apartments by region/county Year”, from where it is possible to choose the period and region necessary. This KPI assumes that each night represents one person that is a presumptive attendee to the museum or one of its activities during the measured period.
The KPI called market share will give the managers a better view of how large part of the community they are intended to reach out to, they actually manage to reach.
Table of Contents
1 INTRODUCTION
1.1 Background
1.2 Problem
1.3 Research Questions and Purpose
1.4 Structure of the Report
2 FRAME OF REFERENCE
2.1 Operations
2.2 Kulturarvspolitik- Museilagen
2.3 Performance measurement
3 RESEARCH METHODS AND PROCESS
3.1 Design Science Research (DSR)
3.2 Research Design and Process
4 RESULTS AND ANALYSIS
4.1 Analysis of secondary textual data
4.2 Survey
4.3 Interviews
4.4 Performance Evaluation Model adapted for museums
4.5 The model
4.6 Model testing
5 DISCUSSION AND CONCLUSION
5.1 Answer to research question 1
5.2 Answer to research question 2
5.3 Answer to research question 3
5.4 Fulfilling the goal and purpose
5.5 Suggestions and implications .
6 CHALLENGES, LIMITATIONS AND FUTURE RESEARCH
6.1 Challenges of the Study
6.2 Limitations of the Study
6.3 Future Research
7 REFERENCES
GET THE COMPLETE PROJECT
Performance measurement and evaluation for cultural non-profit organisations.
