Issues and Challenges in Evaluating Community Impact Study on Knowledge Transfer Program: Researchers’ Reflection

Issues and Challenges in Evaluating Community Impact Study on Knowledge Transfer Program: Researchers’ Reflection

Zainal Madon1, Misni Surif2, Asnarulkhadi Abu Samah3, Mahazan Muhammad4 and Abdul Hadi Sulaiman2
1Faculty of Human Ecology, Universiti Putra Malaysia, 2Knowledge Transfer Centre, Universiti Sains Malaysia, 3Institute for Social Science Studies, Universiti Putra Malaysia, 4Faculty of Social Sciences & Humanities, Universiti Kebangsaan Malaysia
Email: asnarul@upm.edu.my

Abstract
Knowledge Transfer Program (KTP) is Malaysian government’s initiative to transfer the required skills and knowledge from higher education institutions to the community. Evaluation of the KTP is part of the agenda to examine the impacts of the KTP on society. This study is among the pioneer researches to analyze the issues in conducting the community impact study. The study adopted qualitative research design using researcher’s reflexivity approach. A total of 16 KTP communities’ projects for the first rolling (2011 – 2013) were examined. The findings indicate that there are five core issues that emerged through the reflexive process of conducting the study. They include absence of database, timeframe, multiple beneficiaries and poor assessment of community needs. These issues equally are important for stakeholders, researchers and practitioners. Few recommendations are also put forward to improve the community project evaluation in the future.
Keywords: Knowledge Transfer Program (KTP), Community Impact Study, Social Innovative Community, Reflective Approach

Introduction
Didactic and research oriented activities are the main functions of traditional higher education institutions (HEIs). However, with the passage of time, the needs and expectations from the society towards public universities became more diverse and demanding. Particularly, community’s issues shall be addressed in order to promote well being by cultivating positive lifestyle (Murphy, 2013; Osman, 2015). As modern educational institutions, it is universities’ obligation to deliver more innovative and meaningful outcomes as compared to traditional way of lecturing, research, publication and consultation. In other words, the society requires the university to play a significant role in improving living standard by increasing income and providing better well-being of the community (Albulescu and Albulescu, 2014; Jali et. al, 2016). To achieve these objectives, HEIs need to engaged community effectively through Knowledge Transfer Program (KTP). KTP is considered as a bridge towards building a knowledgable society to improve quality of life by nurturing social innovation (Critical Agenda Project Ministry of Education). Social innovation has been believed as a prime outcome from the proposed objective of active collaboration between government agencies, higher education institutions and beneficiaries. Their main agenda was to achieve higher percentage of employments, public health, education systems and country’s economic growth (Surif et. al, 2015).

KTP was initiated in 2011 as one of the Malaysian Government’s initiatives to promote the involvement of the public Universities in collaboration with the community and local industry. The main goals of the KTP are (1) to spread academia innovation, (2) to increase graduate employability, (3) to improve the communities’ livelihood and (4) to multiplicated industries’ revenues. Previously, under the 10th Malaysian Plan (2011-2015), the KTP grant has improved 459 organizations including businesses (ranging from small enterprise to medical pharmacology, machine construction, agriculture, computer technology, art and design) and educational institutions. In addition, KTP also contributed some portion of the grant to so called “disability groups”, such as blind people, disable people. The fund was used to establish certain skills development programs for disability group to enable them contributing towards nation building.

The need for conducting the impact study stems from the necessity of KTP’s key performance indicator (KPI) under Strategic Objective No. 3 (SO.3) to identify impact after sucessfull implementation of SO.1 (implementing the KTP project and monitoring process) and SO.2 (evaluating the project output or success). Impact study was planned to be employed two years after the completion of KTP project. The expert evaluators will meet the project’s beneficiaries in order to measure their project continuity, sustainability and degree of the improved quality of life. Moreover, the impact study will also serve as a guideline to economic planning unit (EPU)- the funder of these initiatives- in order to assess the results achieved through funded projects. However, being a pioneer in the field of knowledge transfer with a very structured based programme, the complexity and unpredictable nature of projects generate serious challenges for administrators to monitor implementation of the projects. Therefore, this research aims to examine the challenges and issues in carrying out impact studies on KTP.

The main objective of this paper is to assess problems and challenges existing in the process of evaluating the impact and effectiveness of the KTP. This is in line with the KTP Project “Rolling 1” that requires formulation of methods to improve the effectiveness of the KTP on the community and to further enhance the KTP program. Through this initiative, the KTP is expected to achieve the Government’s Transformation Agenda aimed at transforming the public university towards a more people-centric (Surif et. al, 2015). This paper focuses on sharing experiences, challenges and obstacles in carrying out studies on the impact and effectiveness of KTP community. Highlighting such pitfall and challenges during execution of impact study is very important as they provide essential information for the future program development, stakeholders and administrator (Chini, 2004; Howlett, 2011).





Literature Review
Knowledge Transfer
According to Van Der Spek and Spijkervet (1997), knowledge is “the whole set of insight, experiences and procedures that are considered correct and true, which guide the thought, behavior and communication of people”. Polanyi (1966) divided the term knowledge into two categories: tacit and explicit knowledge. As per his explanation, tacit knowledge refers to personal knowledge, experience and insights that are by nature difficult to quantify, store or share with others as they are developed in the mind of an individual. On the other hand, explicit knowledge is seen as a proper science where the personal information is able to be noted, read, analyzed and discussed by others. While, knowledge transfer (KT) refers to the process of transmitting information, knowledge (cognitive), skills (psychomotor) and even emotional feeling (affective) from a person or even a place or machine or ownership to another one (Kogut and Zander, 1992; Grant, 1996; Argote and Ingram, 2000; Liyanage et al. 2009). Within the literature, Nizam Jali et. al. (2016) has quoted several definitions that represent the current interpretation of knowledge transfer, as follows: (1) “ the process through which one unit for example an individual, group, department, division or organization is affected by the experience of another” (Argote and Ingram; 2000); (2) “ the process of transmission of knowledge resource from a source to recipient and the knowledge resource is then absorbed and improves the behaviour and performance of the knowledge recipient” (Szulanski et. al, 2004); and (3) “it is an activity of exchanging two knowledge resources namely, tacit and explicit knowledge between source and recipient. The knowledge resource is then transmitted, absorbs and applied that leads to changes in behaviour and performance of the recipient” (Kumar and Ganesh, 2009).

The process of KT starts when a prime source (knowledge creator) who has a better understanding about something meaningful in mind (tacit knowledge) shares the idea, knowledge or innovation with other/s through selected method (explicit knowledge). Philosopher Ikujiro Nonaka and Takeuchi (1998) developed a SECI Model (Socialization, Externalization, Combination and Internalization) to better understand how knowledge disseminates and transforms within the ecosystem as a place for learning occur (known as ‘Ba’) (as shown in Figure 1).

Figure 1: SECI Model


After this brief introduction, it can be seen that knowledge is a powerful tool for innovation, and invention. It also supports the idea that individuals equipped with better knowledge lead the institution, technology, industry and create own wealth or increase quality of life. For this reason, researchers are of the opinion that the implementation of KT will increase the performance as it involves individual, unit, division, department and organization. Besides that, concept of knowledge transfer isn’t confined to the organization. Several scholars found that interorganizational knowledge transfer through smartpartnership, mutual collaboration and strategic alliances are evocative due to the time source limitation. They prove that using external sources, such as universities, government agency, private R&D agency or key players in the industry are more significant, impressive, unique and impactful towards organization in discovering new knowledge creation.

Knowledge Transfer Program: Government-University-Community Initiative
Historically, the agenda of knowledge transfer in Malaysia has been practised since Malaya’s Independent through transfer of technologies between the specialized academia and specific beneficiary, such as agricultural industry and manufacturing industries. At that time, the focus was on transferring knowledge that can solve problem and increase the production of the industries or organization. Later, when the idea of KTP became significant in intensifying national growth among the industries, the policymakers decided to include the government as a key player in the relationship model (known as triple helix model of KTP). It aimed to support the agenda of transfering the technology towards cultivating social innovation paradigm. Triple helix model (as depicted in Figure 2) portrays the relationship between government, HEIs and industry as an example of how Malaysian model was operated.

Figure 2: The Triple Helix-University-Industry-Government Relations


The 10th Malaysian Plan (2011-2015) and the 11th Malaysian Plan (2016-2020) were recognized as strategic plans and a pathway towards future’s development for the country. This is parallel with the prime mission concerning economic achievement based on citizen’s social innovation and quality of life. In achieving the target, public must be equiped with a diverse set of knowledge, skills and attitude in order to generate innovation (as mentioned in Figure 3) (Nizam Jali et al. 2016). Therefore, in 2010, Malaysian Government through Economic Planning Unit (EPU) has launched Knowledge Transfer Program (KTP). KTP was designed in the Tenth Malaysia Plan. It adopted a New Economic Model (NEM) approach, which later been recognized as one of the Critical Agenda Projects (CAP) under the National Higher Education Strategic Plan (NHESP) (Surif et al. 2015). CAP underlined the need to increase the local talent in collaboration between universities and industry/community by optimizing the outcome of research and development (R&D) that has been generated in the universities. This is in line with the development in global higher education and the country’s desire to become a higher-income nation by 2020.

Figure 3: Strategic knowledge management processes and social innovation


At the same time, KTP also allows attentive parties, especially those in the field of science and technology to share their knowledge with the community in order to strengthen the knowledge society agenda and quality of life of ordinary people. It is timely to nurture the interactions among interested parties to achieve sustainable and productive partnerships. Therefore, the government initiates the KTP to shift the knowledge from public universities/higher education institutions (HEIs) to the community. Government’s seriousness was shown through huge monetary investment placed by the government to achieve the Government’s Transformation Agenda. This agenda aims at transforming the expertise and credible invention among HEIs to the selected beneficiaries. There are five (5) key result areas (KRA) implemented in the KTP programs, as follows:
a) Education (EDU)
b) Profit Economy (ECO)
c) Sustainability and Green Technology Iinitiatives (GT)
d) Disabilities Group (DG)
e) Development of Industry Relevant Curriculum (IRC)

Unlike other knowledge transfer initiatives, Malaysian KTP developed new innovative elements in implementing Knowledge Transfer Program (Osman, 2015). In Malaysia, KTP element involved three (3) key players, namely (a) academia; (b) graduate interns; (GI) and (c) beneficiaries (local industry or community). Academia is a group of academic experts in certain fields, which are expected to collaborate and update their superior knowledge resource that is valuable for the industry and the community. This knowledge transfer can be embedded into products, processes and services through teaching, learning, research and consultancy. GI is a group of graduate or post-graduate trainee who work as a fulltime assistant to their respective academia. Almost all of the GI are former students who have awareness (knowledge and skills) about KTP related projects granted to their academia. Besides having a proper job for a certain period of time, they will be able to improve their professionalism and personal development. While at the same time, they play a significant role as facilitators to transfer knowledge and technology to the targeted beneficiaries (Ismail et al., 2016). Beneficiaries are targeted community or local industry that face some problems in their organization due to limited expertise and resources. Hence, they need experts to solve the problems in organization (Contractor and Lorange, 2002). Both academia and GI play a crucial role in sharing knowledge with the beneficiaries as collaborating partners to improve and strengthen community capacity building and their economic activities. As a pioneer project, a total of 44 KTP projects were granted for KTP project rolling 1 (2011-2013), which consists of 16 community projects categories (as seen in Table 2) and 28 industry projects categories. From a total of MYR 6 Million grant, 32% was distributed for the KTP community project and remaining 61% was channel to implement KTP industry project (Surif et al., 2015).

Impact Study
The terminology of ‘Impact study’ has been used in many evaluation programs and its definition may differ depending on its purpose of application. Some scholars used term ‘impact assessment’ or ‘impact evaluation’ interchangeably. It is similar to ‘impact study’ that is rooted in the theory of evaluation umbrella (Ahmad et al. 2016; Brinkerhoff, 2005; Kirkpatrick and Kirkpatrick, 2009; Stufflebeam, 2007). Impact has been defined by Organizasion of Economic Co-operation and Develepment (OECD) as “the positive and negative, intended and unintended, direct and indirect, primary and secondary effects produced by an intervention”. It is clear that impact study includes the processes of analyzing, monitoring and managing the planned and unplanned outcomes (either positive or negative) through designated planned interventions (Rogers, 2012). Furthermore, impact study also helps an administrator to formulate the best-practice management. it also supports to investigate the cause of failure and to give a recommendation for further improvement. Since the KTP is a prominent instrument to improve community well-being, it is crucial to highlight the impact and effectiveness of KTP towards community.
It is worth mentioning that a complex and effective evalution model is required to investigate the impact of a complicated program (either in organization, community or national level). Hence, one of the difficult aspects in the evaluation process is to determine the most appropriate evaluation model to be adopted. Therefore, several model needs to be reviewed in order to select the most suitable model as a framework for the implementation of the impact study.
CIPP Model of evaluation developed by Stufflebeam (2007) focuses on context, input, process and product. In line with Kirkpatrick’s model, Stufflebeam insists that evaluation process must start from level one and move sequentially through levels two, three, and four. This is because all information from each prior level serves as a base for the next level’s evaluation (Zhang et al., 2011). On the level of context evaluations, it helps an organization to assess the needs, pertaining issues and problems, organization capacity and resources. It also discovers ‘space for improvement’ to help organization ‘fine-tune’ their objectives, priorities and expected outcomes. Next, at the input evaluation’s level, the assessment will focus on seeking alternative, contending previous/old action plans and financial strength towards achieving organizational prescribed objectives. Process evaluations level assess the implementation of the developmental project, performance and output produced at the end of the project implementation. Product evaluations level helps organization to analyze achieving important outcomes, which may be divided into assessments of impact, effectiveness or sustainability.
On top of it, CIPP model (as seen in Figure 4) is a useful and simple tool, which can help evaluators to design and implement evaluation easily by following all steps (in logical structure) as recommended by Stufflebeam. It focuses on the objective of each level, collecting information, organizing, analysing and reporting the finding of evaluation (Wiles and Bondi, 2002). However, Worthern, Sanders and Fitzpatrick (1997) realized that a major weakness of Stufflebeam’s model is that in time of uncertainty, evaluators can’t follow all guided question due to limitation of time and sourses.

Figure 4: Stufflebeam’s CIPP Model (2007)

Another comprehensive model to measure impact is known as Success Case Method (SCM) constructed by Brinkerhoff. The SCM was operated specifically to find ‘the very best project/s output’ producedfrom the program. These successful cases of evaluation will be used later as benchmark to compare with other related project. Those project categorized under failed project will be further investigated to find out their pitfalls, whereby suggestions will be proposed for further improvement (Brinkerhoff, 2005).

Issues and Challenges in Community Impact Study
A careful analysis of the literature discloses that during the conduction of impact study in organizations, institutions, evaluators and practitioners reveal their experience (issues and challenges). Even though they may have different scope of intervention, the following table shows issues and challenges in community impact study (Table 1).

Table 1: List of literatures on Community Impact Study
Authors Issues and Challenges
Smith and Straughn, 1983 1) Identifying potential effect of the project
2) Measurement
3) Separation from those other origins
Lima et al. (2015)

Impact Evaluation of Development Programs: Challenges and Rationale 1) Timing of Evaluation
2) Coordination between Managers and Evaluators
3) Counterfactual
4) Size of the Sample

Vaessen, J (2010)

Discussion Paper: Challenges in impact evaluation of development interventions: opportunities and limitations for randomized experiments 1) Delimitation
• Institutional versus beneficiary level effects
• Intended versus unintended effects
• Short-term versus long-term effects
2) Attribution ‘vs’ explanation
• Internal versus external validity
• Theory-based ‘vs’ multi-method evaluation
Type of design to measure impact
There are plenty of models for community impact study in the literature. However, organization must realize their own agenda, capacity and resources before planning to design their framework for impact study. At the next stage, organization needs to select research design, whether quantitative design using a questionnaire or site visit, interviews and expert evaluator inspections. In some cases, multi approach design is required to answer different objectives in the impact study.

Timeframe for implementing impact study
The most common issue in community impact study is deciding the appropriate time for final evaluation to measure the impact after the program in community is ended. Scholars hold different viewpoint on the suitable time to embark on impact study. The best solution to know the exact time is referring to the organizational need itself (Lima et al. 2015). For instance, in case of measuring the skills and competencies, six months to one year is required to assess the impact. While measuring economic gain needs up to three years, so that the fluctuation of the profit and losses can be seen throughout a period of time. However, social impact, such as well-being and quality of life may differ from time to time. Hence, early six months to three years would be better depending on organization needs.

What specific measurement is needed in community impact study?
While developing the measurement’s indicator, researchers must be very specific on their proposed objective. In case of misguided objectives, they shall be revised and amend according to the scope of the project. Measuring impact encompasses a variety of constructs, especially application and sustainability of the knowledge and learned skills, they may also have other indicators such as mentoring, influence others. Besides that, benefits, such as an economic gain and social life improvement indicators are measured as output of the program.

How to identify the impacts produced by under investigation program?
The most important aspect in impact study is its ability to confirm that the impact was from the intended program. However, community program always opens to various sources to improve their well-being. Therefore, it is very difficult to differentiate whether impacts received by the community are genuinely from the project funded by the organization or it’s influenced by the circumstances other than those introduced by a program (Smith and Straughn, 1983). Through this finding, it will help an organization to decide whether to continue or discontinue the program.

Budget Allocation
The budget allocation will identify how critical and crucial are the community impact study to the funder and organization. They often contain financial impact and efficiency rather than effectiveness. Without enough funding, the process of community impact study will not be productive as it effects choosing appropriate design, methodology and sample selection.

Participant
Some of the project leaders fail to manage a participant database, which will make hurdles for the evaluator. Some project leader didn’t report the list of participants who participated from the beginning till end of the project. The worst case is when some of the participants selection process was too lenient. Hence, most of them lose their interest to participate in certain projects. However, their names remain in the participant list, making it difficult for the evaluators to obtain the correct respondent in evaluating the success outcomes. In addition, some evaluators experienced failure to engage with the community because of they move to other places, change their contact number, or inactive email. In some cases, the participants have been observed not collaborating with the evaluators.

Ethics and Conflict of interest in community impact study
Most of the funder organizations will tend to hope that all funded projects must be successful by effectively impacting the community. However, scholars discussed the complexity of the issues in evaluation (including impact study) with regards to occurrences of ethical dilemmas that are rarely addressed. The simplest working definition states: “A conflict of interest is a situation in which financial or other personal considerations have the potential to compromise or bias professional judgment and objectivity” (Council on Government Relations, 2002). It is confirmed that conflict of interest commonly happens in situation when a person is being a; i) member investigators; ii) member holding significant financial interest in sponsorship of the research; iii) loyal to colleagues submitting for review; iv) member having close tie to area of research under review; v) having possible impact of decisions on member's own work (e.g., policy changes) and; vi) personal agendas. While on the institutional level, an organization may; i) pressure or desire to protect institution; ii) concern for institution’s reputation or prestige; iii) promote research vs protect subjects; iv) undervaluation of service; v) potentially liable; vii) institutional or community values; viii) pressure for speedy reviews; ix) have institutional equity or ownership and; x) review fees. (see Responsible Conduct of Research @ http://ccnmtl.columbia.edu/projects/rcr/index.html)

Methodology
Original idea of the implementation of the community impact study consisted of quantitative methodology with support of qualitative method, particularly on data gathering, data analysis and data reporting (Zhang et al. 2008). Due to the unique nature of KRAs, the researcher believes that using questionnaire would not be able to address the research objectives. In contrast, qualitative method is useful to examine the reality within its rich environment. Hence, it is a significant design to discover, analyze, and provide meaningful findings for the community impact study (Roche, 1999; Lee and Krauss, 2015). Therefore, an in-depth interview session was conducted with field observation to enrich the understanding and to evaluate the impact of each program on the community. The issues and challenges are studied through the lens of qualitative approach known as researcher’s reflexivity.
Researcher’s reflexivity addresses the subjectivity related to people and events that researcher encounters on the field. Reflexivity also addresses the subjective nature of the research account as a narrative constructed by researchers. Therefore, process of reflection was essential to generating meaningful data towards new knowledge creation (Cowan, 1998; Menon and Pfeffer, 2003). This study does not present findings from informants, but identifies core issues that emerged while conducting the impact study through the reflexive process among the researchers. The reflexive framework reflects in the analysis and interpretation across the entire research development. After one and a half year of field investigation, study produced these experiential results that reflect all related issues and challenges in conducting impact study. The 16 KTP community projects in Rolling 1 are as portrayed in Table 2.

Table 2: 16 KTP Community project for Rolling 1
Project Title Project KRA Type of Community
1. Knowledge & Economic Generation Program for Fish Farmers. Profit Economy Fish Farmers
2. Enhancing Knowledge and Economy for Livestock Farmers. Profit Economy Goat Farmers
3. Enhancing the Quality of Life through Ecotourism Development. Profit Economy Fisheries/Costal Community
4. Halal Certification Industry. Profit Economy Food Entrepreneur
5. E-Braille for All. Education NGO, Educators & Students
6. Enhancing Informal Science Learning Program. Education Science Teachers and Secondary School Students
7. Potentials for Sustainable Transformation (PST). Education Secondary School Students,
8. Community Engagement towards Better Health. Education Village Community
9. Training Brain Apprentice. Education Primary & Secondary School Students
10. Microscale and Green Chemistry. Education Secondary School Students
11. Vermicomposting & Green Technology. Sustainability & green technology initiatives Local Council Authority
12. Energy Efficiency and the Energy Audit. Sustainability & green technology initiatives Teacher Trainee
13. Senior Citizens & ICT. Disabilities group NGO & Senior Citizens
14. Home Nursing. Disabilities group NGO & Community
15. Knowledge on Methodology of Coaching. Industy Relevant Curriculum Teachers
16. Kit for Potentially Gifted Children. Industy Relevant Curriculum Kindergarten Teachers

Findings and Discussion
In KTP community impact study, there are several obstacles and challenges faced by researchers. Below are several pertinent issues, which obstructed the flow of the community impact study.

Instrument Measurement Parameter
At first stage of conducting KTP impact study, the researchers held a workshop to develop a research framework. It was decided that KTP projects will be masured with a combination of objective and social measurement. After the researchers reviewed the literature on impact assessment study and related tools, a framework for research model was developed based on model of Success Case Method (SCM) Brienkerhofff (2003), Bloom’s taxonomy, and well-being index in effort to determine relevance impact of KTP on community. This research consists of six main variables, (a) apply; (b) sustain; (c)influence; (d) mentoring; (e) objective benefit; and (f) subjective benefit. These were identified based on the combination of the literature and expert feedback from the research team. In addition, two subscales were introduced under the objective benefit (economic; and social Benefit) and three subscales were incorporated in subjective benefit (social well-being; personal development well-being; and psychological well-being).
Due to the diversity of the KTP-KRA (ECO; EDU; GT; DG; and IRC), the beneficiaries for each KTP Project are selected from variety of backgrounds (such as rural communities, farmers, fishermen, students, teachers, entrepreneurs, civil servants, people with disabilities and the elderly citizens) (see Table 2). Hence, the development of measurement parameter in this study is expected to justify the requirements of the objectives of the study.

Database of Beneficiaries Profile
In the early stages of data collection, the researchers contacted the academia (project leader) for each KTP community rolling 1 to get an overview of the their KTP, such as type of program, figures of the beneficiaries and community representative.
One of the biggest challenge seen in this impact study is the lack of beneficiaries profile of the community. Due to the absense of proper database in most of the rolling 1 project, demograhic profile of the beneficiaries, such as name and contact details are not made available. This makes the impact study of the KTP programs challenging. However, the representative of the beneficiaries (such as comunity leader, teachers or NGO) were the only connections to the community.
Based on researchers’ observation (as shown in Table 3), majority of the KTP project does not have a complete database of program participants. Moreover, since there were no proper lists of KTP participants, the community representatives were not able to oppoint appropriate presenters at community level. Consiquintly, the researchers faced difficulties to communicate with representatives as they changed contact numbers or withdrwan from the projects.

Table 3: Observation data - List of KTP Project with participant database issues
Project Title Observation Notes
Potentials for Sustainable Transformation. No database provided to researcher. Academia create Facebook page for the program, participant can join the group.
Community Engagement Towards Better Health. Project leader only kept community leader contact information. No proper database on participant was provided
Kit for Potentially Gifted Children. No database provided by project leader
E-braille For All. No specific database, however some of the participants registered in online system
Halal Certification Industry No spesific database. Databased from the organization/partner, however some of them was not participate in the program. (not clean database)

Timeframe
The wide gap of two or three years between the KTP project and impact study may lead to changes in the beneficiaries demographic profiling or interest. For example, when researchers tried to retrace beneficiaries in one of the KTP project that involves students, they found that most of the students were either graduated or kept continue their studies at a higher level. Similalry, in the case of beneficiaries among teachers, some of them were either promoted or transferred to different location.There are some cases where the previous community representative was no longer holding the post in community organizations. Thus, he or she does not has the authority to call or invite the respondents to be involved in the impact study. In fact, some respondents could not recall the program they previously participated in.

Singlular beneficiary vs multiple beneficiary
Its happen that KTP project beneficiaries can be divided in two categories. First category was focus on single individual representative from the community agency in the partnership. While some other project go beyond one group of community to implement the KTP. However, the problem rise when some of the KTP community project were fail to sustained the program within the community because they make a one-off knowledge transfer activities among beneficiaries and move to different location and different beneficiaries (see Table 4). Based on experience in data collection process, there are two out of 16 KTP projects running series of one-off knowledge transfer targeting students and community villagers. It was seen that colleting data from multiple beneficiaries is difficult to compared to the singular beneficiary (involved from the beginning until the completion of the program). For the impact study purposes, it is hard to measure the impact of the project on those beneficiaries who went through one learning session over the last two years of KTP project. Thus, they do not feel responsible to participate and cooperate in impact study’s data collection process.

Table 4: Observation data - List of KTP Project with participant multiple beneficiaries issues
Project Title Observation Notes
Potentials for Sustainable Transformation. More than 2 secondary school, with one-off knowledge transfer activities.
Community Engagement Towards Better Health. More than five village participated with different type of age group.
E-braille For All. All support system (facilitate blind people) can participate in the program such as parents, friends, volunteers, teachers trainee etc.

Community needs
Based on observations during the data collection process, some of the KTP project beneficiaries were less satisfied, as they believed that the knowledge transfer project did not meet their needs. In fact, some of them were not comfortable to be called for interview, while some used this impact study as a platform to express their dissatisfaction and anger toward the KTP projects.
Some beneficiaries expressed that most of the knowledge received through KTP is already known to them. Therefore, they are expecting from the academia to provide advance knowledge relevant to community development. Some of the beneficiaries added:

“…In my opinion, the program brought by the university does not give much impact. Because they come exactly to give a talk about fish culture... the things that we already know!!! However, the villagers still accept this information…
… What we want to learn actually is how we can grow fish faster in eight months… so presumably either food or other substance that can accelerate our fish grows (to production stage) without changing the taste of the fish... that right!!”
-Fish farmers project participant-


Recommendation
Based on the reflection approach analysis, the researcher realized that several challenges occurred during the KTP impact study. Therefore, several recommendations are designed to improve the implementation of KTP program. First, establishment of a proper database for the beneficiary’s profile must be a priority for every KTP community projects. Database must be standardized and can be used for all KTP projects. The detail of information that is required shall include names of participants, address, telephone, email and the information of community representatives for each project KTP. With a database, the information is easily accessible and useful for future purpose even if the KTP projects are no longer active. In addition, the database can also be used as evidence that the knowledge transfer process is conducted by academia. Besides that, for each KTP projects, the community is recommended to appoint KTP sub-committee from the participating beneficiaries themselves to ensure the smooth running of the knowledge transfer program. It will enable the academia and KTP secretariat to keep in touch with community in the future even if the duration of KTP project has ended.
KTP Office should reconsider the timeframe of two-year period before implement the impact study. Duration between nine months to one year is deemed appropriate to examine the frequency of the participant occupying the knowledge and skills learned in the program.
University can only achieve KTP ultimate goals when they provide a program that meet the needs of the community (Zhang et al. 2008). Academia members are highly recommended to fully analyze and understand the needs and identify the level of knowledge and experience possessed by the community beneficiaries. In order to meet the actual needs of the community, the design of a project should address effectively both parties’ needs (Zhang et al. 2011). At the same time, the academia members must have a solid knowledge, expertise, skills, and experience. This is to address the confidence level of the community in the process of receiving the knowledge and to generate the desirable outcome.

Conclusion
The KTP program is foreseen to produce a significant agenda towards nation building. It serves as a source of knowledge dissemination through creative and innovative mechanism that produce new product and improve tangible processes. Furthermore, this program will empower the community to enhance living standard. The impact study is an essential requirement of the program as it provides a guideline for EPU and Ministry of Higher Education (MoHE) to assess the impact and effectiveness of the granted projects. Realizing that KTP program is a new developmental driver, the researchers have faced many challenges in examining the impacts of KTP. Despite the existence of some flaws, it is expected that the findings of this study will help the KTP secretariat for better planning in the future. The secretariat should consider the proposed recommendations in developing future policies.

Funding: This work was supported by Economic Planning Unit from the Prime Minister’s Department Malaysia and Ministry of Higher Education [Knowledge Transfer Programme Grant].

References
Ahmad, A. R., Soon, N. K., Saiman, S. S., & Yusoff, R. M. (2016). The Impact of Knowledge Transfer Program on Malaysian Higher Education Strategic Planning. Information, 19(7), 2797–2801.
Albulescu, I., & Albulescu, M. (2014). ScienceDirect The University in the community. The university’s contribution to local and regional development by providing educational services for adults. Procedia-Social and Behavioral Sciences, 142, 5–11. 10.1016/j.sbspro.2014.07.578
Argote, L., & Ingram, P. (2000). Knowledge transfer: A basis for competitive advantage in firms. Organizational behaviour and human decision processes, 82(1), 150-169.
Brinkerhoff, R. O. (2005). The Success Case Method: A Strategic Evaluation Approach to Increasing the Value and Effect of Training. Advances in Developing Human Resources, 7(1), 86–101. 10.1177/1523422304272172
Chini, C. T. (2004). "Effective knowledge transfer in multinational corporations". Palgrave Macmillan. 10.1057/9780230005877
Contractor, F. J., & Lorange, P. (2002). The growth of alliances in the knowledge-based economy, International Business Review, 11(4), 485-502
Council on Government Relations. (2002). Recognizing and Managing Personal Conflicts of Interest. Winter. From http://www.cogr.edu
Cowan, J. (1998). On Becoming an Innovative University Teacher: Re?ection in Action. Buckingham: SRHE and OUP.
Grant, R. M. (1996) "Toward a knowledge-based theory of the firm", Strategic Management Journal, 17, 109-122.
Ismail, M. R., Arshad, M. A., Haris, I., & Ali, M. (2016). KNOWLEDGE TRANSFER PROGRAM ( KTP ): THE EXPERIENCES. In 3rd National conference on knowledge Transfer. Pulau Pinang.
J.Howlett, R. (2011). Innovation through Knowledge Transfer 2010. http://doi.org/10.1007/978-3-642-20508-8
Kirkpatrick, J., & Kirkpatrick, W. K. (2009). The Kirkpatrick Four Levels TM?: A Fresh Look After 50 Years 1959-2009. Kirkpatrick Partners, (April), 12.
Kogut, B., & Zander, U. (1992). "Knowledge of the Firm, Combinative Capabilities, and the Replication of Technology?, Organization Science, 3(3), 383-397.
Kumar, A. J., & Ganesh, L. S. (2009). Research on knowledge transfer in organizations: a morphology. Journal of Knowledge Management, 13(4), 161-174.
Lee, K., & Krauss, S. E. (2015). Why use qualitative research methods to understand the meaning of clients' experiences in healthcare research? International Journal of Public Health and Clinical Sciences. 2(4), 1-6.
Lima, L., Souza, A. P., & Figueiredo, D. (2015). Impact Evaluation of Development. eVALUAtion Matters, 18.
Liyanage, C., Elhag, T., Ballal, T., & Li, Q. (2009). Knowledge communication and translation-a knowledge transfer model. Journal of Knowledge Management, 13(3), 118-131.
Menon, T., & Pfeffer, J. (2003). Valuing internal vs. external knowledge: Explaining the preference for outsiders. Management Science, 49(4), 497-513.
Murphy, P. (2013). Tourism: A community approach (RLE Tourism). Routledge.
Jali, N. M., Abas, Z., & Ariffin, A. S. (2016). Addressing Social Innovation in the Malaysian Knowledge Transfer Program: Gaining a Preliminary Insight. IOSR Journal of Business and Management, 18(10), 56–64. http://doi.org/10.9790/487X-1810045664
Nonaka, I., & Konno, N. (1998) “The concept of Ba?: building a foundation for knowledge creation”, California Management Review, Vol.40, No.3, Spring, pp.40-54.
Osman, O. (2015). Driving innovative community engagement through Knowledge Transfer Program. In Enhancement and Innovations in Community Engangement (p. 1).
Perez?Nordtvedt, L., Kedia, B. L., Datta, D. K., & Rasheed, A. A. (2008). Effectiveness and efficiency of cross?border knowledge transfer: An empirical examination. Journal of Management Studies, 45(4), 714-744.
Polanyi, M. (1966). The logic of tacit inference. Philosophy, 41(155), 1-18. Responsible Conduct of Research @ http://ccnmtl.columbia.edu/projects/rcr/index.html)
Roche, Chris. (1999) Impact Assessment for Development Agencies: learning to value change. Oxford: Oxfam.
Rogers, P. J. (2012). Introduction to impact evaluation. Impact Evaluation Notes, 1(1), 1–21.
Smith, M. F., & Straughn, A. A. (1983). Impact Evaluation: A Challenge for Extension. Journal of Extension, September/October pp. 55-63.
Stufflebeam, D. L. (2007). CIPP EVALUATION MODEL CHECKLIST.
Surif, M., Ismail, M. R., Noordin, N., and Shamsudin, U. S (2015). Laporan Pemantauan KTP Rolling 1 (2011 – 2013). Sekretariat Pemantauan KTP: USM, Penang.
Szulanski, G., Cappetta, R., & Jensen, R. J. (2004). When and how trustworthiness matters: Knowledge transfer and the moderating effect of causal ambiguity. Organization Science, 15(5), 600-613.
Vaessen, J. (2010). Challenges in impact evaluation of development interventions: opportunities and limitations for randomized experiments IOB-discussion papers. http://www.enterprise-development.org/wp- content/uploads/Opportunities_Limitations_Randomized_Experiments.pdf
Van der Spek, R., & Spijkervet, A. (1997), Knowledge Management: Dealing Intelligently With Knowledge, in: Liebowitz, J., Wilcox, J. (Eds), Knowledge Management And Its Integrative Elements, CRC Press
Wiles, J., & Bondi J. (2002). Curriculum development-A guide to practice-. New Jersey: Merrill Prentice Hall.
Worthern, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1997). Program evaluation – Alternative approaches and practical guidelines-. New York: Longman, Inc
Zhang, G., Zeller, N., Griffith, R., Metcalf, D., Williams, J., Shea, C., & Misulis, K. (2011). Using the Context, Input, Process, and Product Evaluation Model (CIPP) as a Comprehensive Framework to Guide the Planning, Implementation, and Assessment of Service-learning Programs. Journal of Higher Education Outreach and Engagement, 15(4), 57–84.
Zhang, G., Zeller, N., Shea, C., Griffith, R., Metcalf, D., Misulis, K., Williams, J., and Knight, S. (2008). A 360° assessment of the multidimensional effects of a service-learning program in teacher education using mixed-methods research. Paper presented at the 8th International Research Conference on Service-Learning and Community Engagement, New Orleans, LA.

In-Text Citation: (Madon et al., 2021)
To Cite this Article: Madon, Z., Surif, M., Samah, A. A., Muhammad, M., & Sulaiman, A. H. (2021). Issues and Challenges in Evaluating Community Impact Study on Knowledge Transfer Program: Researchers’ Reflection. International Journal of Academic Research in Business and Social Sciences, 11(15), 229–244.

Copyright: © 2021 The Author(s)
Published by HRMARS (www.hrmars.com)
This article is published under the Creative Commons Attribution (CC BY 4.0) license. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this license may be seen at: http://creativecommons.org/licences/by/4.0/legalcode