Evaluation of International Organizations
the need, current efforts, qualitative and quantitative methods
- / -
Extract from: Report of a Preliminary Investigation of the Possibility of
Using Computer Data Processing Methods (1966).
Need for Evaluation
The need for evaluation of NGOs in general can be itemized as follows:
The need for self evaluation of an individual organization can be itemized as follows:
Current Efforts to Evaluate Non-Profit Organizations
International NGOs In 1954 a group of 30 NGOs in consultative status with Unesco undertook an enquiry with regard to techniques employed by NGOs in evaluating the results of their projects. The report showed that some organizations use travelling inspectors and statistical reports which can be compared from area to area. Some question persons who have been involved in their programs. Some evaluate on the basis of the increase in membership in a given area, the increase of the material resources in that area, and the resulting financial independence attained by the project. The costs of evaluation were shown under such a diversity of headings that it was impossible to give the cost as percentage of the expenditure on the project. 'Per capita' calculation of expenditure was frequently inacceptable as being unsuited to humanitarian and social work (Unesco, D1)
In 1956 a plan was drawn up for the evaluation of international NGOs. This plan was prepared by a committee brought together by the Union of International Associations under a contract from Unesco. The plan was published but did not result in any detailed evaluation. The committee rejected any quantitative evaluation and concentrated its attention on the techniques for qualitative evaluation of a group of NGOs (their aims, their program targets, the operation of the organizations), (Meynaud, Jean, D2).
Social Science and Voluntary Associations No other attempts have been made to evaluate international NGOs. In the United States a lot of work is being done in the social sciences on organizational effectiveness, efficiency, control and decision-making (see bibliography D). Much of this work has specifically dealt with voluntary non-profit organizations. Some of this material could be developed into useful techniques for comparative analyses of international NGOs, but many of the techniques are dependent on complex questionnaires which must be evaluated by experts. Some are not suited as a management tool or a convenient indication to members of the progress of the organization.
One such study was designed to test some "popular" assumptions, namely: non-governmental organizations are more efficient than governmental agencies, and profit-making organizations are more efficient than non-profit organizations. The results suggested that the assumptions were not supported by evidence. The author suggests that the nature of the popular assumptions may have had the unanticipated consequence of making such organizations highly efficiency conscious. The data from the study indicated that governmental and non-profit organizations do have slightly larger percentages of personnel directly concerned with accomplishment of the objectives of the organization than do non-govern mental and profitmaking organizations. (Haas, E., Hall, R.H. Johnson N.J., D3).
Accountancy and Non-Profit Organizations In the field of accountancy there is a continuing discussion of the problems of giving a true and fair account of the financial position of non-profit organizations. In the United States problems of satisfactory financial reporting by voluntary health and welfare organizations have, for example, concerned these organizations for many years. Most of these bodies have been urgently concerned with the adeauacy and accuracy of their public financial reporting, since malpractice by a few organizations reflects on all fund-raising organizations. A committee was set up to investigate this matter and produced a report on the "Accounting and Financial Reporting Procedures for Voluntary Health Agencies" (National Health Council, EI), which concentrated on the establishment of uniform accounting and financial reporting. A discussion of the committees work concluded that : "Public understanding of the operations of a non-profit health and welfare organization must be based on recognition from the outset that there are no financial reports by which the performance of an agency can be basically appraised comparable to the evaluation of a commercial organization by its income statement and balance sheet. Sound evaluation of the performances of a voluntary agency must be based substantially on criteria other than financial operations. The basic need of the contributing public was considered by the project committee to be full and complete accounting by agencies for resources received and used. The significance of voluntary agency ~financial reports to the public is to be found in their accounting for stewardship of resources rather than in their emphasis on operating results". (Withey, H.A. and Holman, G., E2).
The accounting studies have not been restricted to the health and welfare organizations but have dealt with voluntary non-profit organizations in general (see bibliography E). Hospitals, universities and churches have all received attention.
Productivity and Efficiency in Non-Profit Government Agencies Closely related to the accounting studies are the efforts to improve the efficiency of non-profit government agencies. Much of this work is relevant to the development of techniques of evaluating international NGOs (see bibliography E). This is much more concerned with the development of useful techniques for evaluation by management and the public.
Association Management The most importan attempt at evaluation is being made in the United States by the American Society of Association Executives. The society has published a number of articles and reports on various aspects of association management and evaluation. It has made use of 'operating ratios' in evaluating associations in a manner similar to that used with ratio analysis in commercial organizations. This work has not been adapted to international NGOs.
The value of NGOs has not yet been clearly established as will be seen from some of the critical comments listed below. The main reason for this is that no adequate techniques have yet been developed and applied to demonstrate the value of these organizations. The only evidence of a general nature is qualitative, and the fact that the number of these organizations continues to increase.
There are a variety of methods for judging the quality of service provided by an organization. Three levels of measurement of excellence can be identified. The impressionistic evaluation is entirely subjective, the presumptive arises when excellence is presumed because certain arrangements have been made to ensure that the service is good. The most valid form of evaluation occurs when some form of investigation is undertaken. A number of approaches are possible: the organization program can be evaluated to measure progress against stated goals; reliance can be placed on the judgment of other groups interacting with the organization; the productivity of the organization can be analysed; an expert can be appointed to investigate the organization: the organization executives can be appraised as managers; the cost of operations can be compared with those in other organizations; and the techniques and methods employed can be compared with those used in other organizations.
The qualitative evaluation of NGOs will be considered first and then progress toward the quantitative evaluation will be discussed.
Methods In this section the value and weaknesses of NGOs will be listed as viewed by those convinced of their importance and also by others who are less enthusiastic.
Favourable View of NGO Value One view, given in a report to Unesco on the value of NGO work, is that the specific values of international nongovernmental action reside in:
Favourable View of NGO Weaknesses A list of the general problems and weaknesses of NGOs is given below. This represents the remarks and suggestions made at a meeting of NGO executives in Paris in 1966:
One view itemized NGO weaknesses as including: lack of strong leadership; lack of good communication between the organization's leaders and its membership; lack of a permanent secretariat; and the -need to extend geographic representation further to become more truly international in character. The commonest and most serious weakness is inadequate financing (United States National Commission for Unesco 1964).
Other Views of NGO Value Many political scientists are convinced that in the current state of international relations, NGOs can only hope to modify slightly the acts of intergovernmental organizations. They consider that NGOs can do little to achieve any real o, r useful cooperation (Meynaud, J., A18, p. 385-405).
Another view expressed in technical terminology is that "different types of private international associations perform, in various contexts, the articulation, aggregation, and supervisory functions and provide for both innovation and boundary maintenance. The role of crossnational management and labour groups in the European integrative process is also a reflection of this". (Miles, E., D5).
Other Views of NGO Weaknesses In one approach to the study of international organization, international NGOs are specifically excluded because it is claimed that "their role in international system function performance is minimal". (Haas, M., D6).
Another view is that "despite the successes which international non -governmental organizations have had, problems remain. There are too many, some of them the result, possibly, of the efforts of 'idle women in search of celebrity .... politicians on the downward slope.... ambitions individuals, and ... sincere, determined but unrealistic men'...Many have large resources, many more few, resulting in a competition for funds from a limited number of sources... Finally, the very freedom of international nongovernmental organizations in a world characterized by ideological struggle is the gravest problem facing them." (Blaisdell, D., D p. 388
Methods In this section the problem of evaluating nonprofit organizations using quantitative techniques will be discussed. Much of this work has been done in the United States in connection with national voluntary agencies or indirectly through the question of government agency efficiency as a non-profit organization.
Financial Importance of Non-Profit Organizations Published data on the international NGOs in existence in 1958 show that the total of their budgets amounted to $ 300 million. The total budgets for all intergovernmental organizations at this time was $ 415 million.
In addition to the organization budget, there are the indirect economic effects of an international congress on the city and country in which it is held. Congress costs have been estimated at about $ 60-75 per delegate on average. Many congresses have from 200-800 delegates (Alkjaer, E., D7). There are from 3000-4000 congresses per year.
Data on national NGOs is relevant here since many international NGOs have programs which influence the expenditure of the national member organizations. The United States Internal Revenue Service, for example, estimated that there were over 100,000 national, regional and local voluntary health and welfare agencies 'in the country. Between 1940 and 1958, annual contributions to these agencies increased from $ 188 million to $ 1.5 billion. It is estimated that total contributions for philanthropic purposes will exceed $ 13 billion in 1970 (Hamlin, R.H., D8).
A count of American national conventions in 1957, showed 20,000 conventions of all types and a total attendance of more than 10 million. Delegate expenditure was estimated at $ 1.2 billion (Turner, J.S., D9).
Efficiency and Effectiveness: an important distinction There is a very important distinction to be made between the efficiency and the effectiveness of an organization. The distinction is that the former measures how well a program is carried out from the administrative and organizational point of view, whereas the latter measures the contribution made by the program towards the fulfillment of the objectives of the organization. A completely ineffectual program can be carried out with great efficiency, or a highly effective program can be carried out using most inefficient methods.
In many commercial organizations the efficiency of management can be measured directly by the profits made. The more profits made, the greater the efficiency. There is no comparable measure of performance for non-profit organizations which can be used to compare one organization with another.
In a commercial organization the effectiveness and the efficiency are closely related. When the objectives of the organization are profits, an increase in efficiency is equivalent to an increase in effectiveness. In a non-profit organization efficient low cost performance of a plan is not necessarily related to production of better results.
In a profit-oriented commercial organization, the financial resources used can be directly compared to the financial results. Where the commercial organization is not solely governed in terms of the profit criterion (i.e. where management are also concerned with public and employee welfare and other intangible factors), the situation resembles that of the non-profit organization.
A very common observation heard among the personnel of a non-profit institution is that their particular establishment is not expected to earn a profit. This statement has much broader connotations than is sometimes realized, either by the outsider or by those within the institution itself. To those who are commercially oriented, it often sounds like an excuse for inefficiency and, therefore, stirs up a feeling of mistrust. The institutional administrator, on the other hand understands this statement as an assertion of purpose, achievement of which cannot be measured in financial terms
Measurement of Efficiency Every organization has certain activities which require the regular expenditure of funds to achieve precise results. This may be the preparation of a journal or newsletter, publicity mailings, fundraising campaigns, etc. There are two methods of rating the efficiency of such activities individually. The costs of the activity can be compared with those of the previous month or year for the same activity. The costs can also be compared with those of other organizations, not necessarily NGOs, which perform the same activity. If the costs go down from period to period then the efficiency is increasing and the improvement can be plotted on a graph as a guide to the NGO executive.
This form of comparison does not give an absolute figure so that it is impossible to compare organizations as a whole in this way. For certain types of organization overall efficiency measures can be developed. In a membership organization, for example, the proportion of funds arising from membership fees could be checked against the amount arising from other sources, such as sale of publications. Increasing independence of membership funds could be equated with increased efficiency, if it is assumed that an efficiently run organization should be self-sufficient.
Measurement of Effectiveness There is clearly no point in managing an organization in order to achieve an increase in efficiency if the organization becomes increasingly ineffective. Some NGOs have registered suspicion of any attempts at making themselves efficient. The term efficiency has acquired during the past generation a number of unfortunate connotations which associate it with a mechanistic, profit-directed, stop-watch theory of administration. Criticisms of efficiency as a guide to administration have been frequent and vociferous. (Simon, H., D10). p. 180-182).
Some measure of effectiveness is therefore necessary to check that efficiency and effectiveness increase together. The measurement of effectiveness in NGOs is a much more difficult matter. Effectiveness is related to accomplishment of organization objectives as measured by some perceptible change in the part of the environment with which the organization is concerned. The problem is that the ultimate value objectives of international NGOs are seldom expressible in concrete terms. They may include key phrases such as 'increase international understanding in the field of education of youth', 'improve relations between documentalists in different countries', 'improve and extend inter-institute relationships', 'foster and facilitate exchanges...', etc.
It is not sufficient to concentrate on some intermediate goal, because there is a serious danger that decisions governed by the intermediate goal will persist even when that end is no longer appropriate to the realization of the ultimate value objective.
The problem is made more difficult when an organization has several value objectives to be balanced.
Use of Indexes to Represent Aspects of Efficiency and Effectiveness In order to avoid this difficulty it is possible for each organization to consider the part of its environment that it wants to modify and to define the factors which are sufficient to represent any change as a result of its activities. For example, an organization interested in the study of international NGOs might develop the following criteria. Its documentary activities might be checked by totalling the number of favourable and unfavourable reviews, the number of citations of its publications in research papers, number of supplementary written enquiries received, etc. Its activities as a study centre could be evaluated on the basis of number of enquiries, number of persons coming to use its documentation, number returning a second time, etc.
In this way, indexes can be set up for key activities and can be plotted on a graph for comparison from year to year. All indexes must be plotted with efficiency indexes on the same graph in order to check where the weak points of the organization are.
The danger is that the indexes will be taken to represent the goals of the organization. They are the best concrete representation, but great care must be taken that they are reviewed as indexes, to check whether some other index would not be necessary or more applicable as a measure of what the organization is attempting to achieve.
Comparison between NGOs
The measures of efficiency and effectiveness defined above are mainly for use as an assessment of the progress of a particular organization, An organization may, however, improve on previous years performance, but not be as efficient or effective as other organizations concerned with the same sort of problems. It is important for all groups concerned with the progress of a particular organization to realize any such differences and the reason for them. The executives of the organization, the members, and the potential fund-allocating bodies all need such information.
In the case of commercial organizations, comparison is basod on an analysis of the balance sheet and income statement for each organization. It is recognized that it is not meaningful (except for investment purposes) to compare organizations which operate in different ways, for example, an engineering consultancy firm and a shoe manufacturing firm. It is however useful to compare organizations in the same industry with approximately the same characteristics, e.g. heavy engineering firms, or leather products companies. Such comparison is based on a classification of the organizations and enables individual companies to determine in what respect their operations are better or worse than the average for the other companies in the industry. One isolated attempt to provide some form of analysis for trade associations has been made through the American Society of Association Executives. (Huppelsberg, R., D11). Apart from this, no attempt has been made to develop any such balance sheet analysis for non-profit organizations. The reason given is that the profit criterion is absent. Profit is however only one of the factors which are checked in any analysis. There are many others which are concerned with identifying the internal equilibrium and general health of the company which are not directly related to the existence of profits. In addition, the extent of the loss made, if any, by a non-profit organization does give useful information on the affairs of the NGO even if other criteria are of relatively greater importance.
A list of indexes is given below which could be usefully investigated further as a basis for comparison between organizations using data in their balance sheets.
Non-profit organizations may have departments or programs which produce publications for sale at a profit. Such departments can be assessed using many of the financial ratios in use in business. Fund management can be evaluated using:
Profitability related to sales can be evaluated using:
Other business ratios can be used to evaluate the organization as a whole, e.g.
These ratios are useless as absolute figures. They can only be useful when compared with the same ratios for the previous years to show a trend. When accounting statement data for different organizations can be made compatible, they can be used to compare the financial results of operations of organizations with similar activities.
There are two problems. The balance sheets of organizations are not yet standardized as was once the case for balance sheet data for companies within individual countries. This has to be done before a direct comparison is possible. A useful classification of NGOs has not been developed. It is essential to be able to compare the same sort of organization. The balance sheet analysis might prove to be a useful aid in grouping organizations with the same characteristics by supplying comparable financial ratios.
It is important to understand that any such evaluation of individual organizations or groups of organizations is only a guide for all the parties concerned with the organization. Evaluation in quantitative terms only supplies additional information which must be balanced against the qualitative, subjective evaluation, as an aid to decision-making. Evaluation can only help to improve the organization. Its absence will not necessarily prevent the organization from functioning effectively. It should help the NGO to make better use of its very limited financial resources.
D. Evaluation of Non -governmental Organizations
1. UNESCO. Evaluation methods employed by certain non-governmental organizations approved for consultative arrangements with Unesco. International Social Science Bulletin, 8, 1957, 4.
2. J. Meynaud (Ed), Plan for evaluating international non-govern mental organizations. Brussels, Union of International Associations, 1957, 32p.
3. E. Haas, et al. The size of the supportive component in organizations: a multi-organizational analysis. Social Forces, 42, October 1963, p. 9-17
4. G. P. Speeckaert. La valeur du travail international non-gouvernementale: ses méthodes et sa position à l'égard des organisations intergouvernementales. Texte d'une conférence, 1955
5. E. Miles. Organizations and integration in international systems. International Studies Quarterly, to be published in June 1968 issue
6. M. A. Haas. A functional approach to international organization. Journal of Politics, 27, 1965, pp. 498-517
7. E. Alkjaer and J. L. Eriksen. Location and economic consequences of international congresses. Copenhagen, Einar Harcks Forlag, 1967
8. R. H. Hamlin. Voluntary health and welfare agencies in the United States. Schoolmasters' Press, 1961, 88p. (distributed by the Rockefeller Foundation)
9. J. S. Turner. Conventions: an American institution. Cincinnati, International Association of Convention Bureaus, 1958, 138p.
10. H. A. Simon. Administrative behavior; a study of decision-making processes in administration organization. Coll ier-Macmi Ilan, 1965
11. R. Huppelsberg and E. Nitchie. Operating ratio report for associations. American Society of Association Executives, Management reprint, no. 203, 1967
12. P. Nokes. Purpose and efficiency in humane social institutions. Human Relations, 13, 1960, pp. 141-155
13. D. Blaisdell. International Organization. Ronald Press 1966
14. L. V. Blankenship. Community power and decisionmaking: a comparative evaluation of measurement techniques. Social Forces, 43, no. 2, December 1964, pp. 207-216
15. R. H. Bremner. The business spirit in philanthropy. Business Topics, 10, 1962, no. 2, pp. 8-15.
16. H. L. Costner. Criteria for measures of associations. American Sociological Review, 30, no. 3, June 1965, pp. 341-353.
17. P. Diesing. Noneconomic decision-making. Ethics, 66, no. 1, part 1, October 1955, pp. 18-35
18. B. S. Georgopoulos and A. S. Tannenbaum. A study of organizational effectiveness. American Sociological Review, 22, October 1957, pp. 534-540
19. W. A. Glaser and D. L. Sills. The government of associations; selections from the behavioral sciences, New Jersey, Bedminster Press, 1966, 264 p.
20. B. M. Gross. What are your organization's objectives?; a general-systems approach to planning. Human Relations, August 1965, pp. 195-216.
21. S. Hyman. Organisations and society. Management International Review, 8, 1968, no. 2-3, pp. 75-83
22. S. Klaw. The perils of running a non-profit: Stanford Research Institute was pursuing happily with a $ 50 million business until it began to ask what its business really was. Fortune, 74, November 1961, pp. 158-161.
23. J. L. Kuethe and B. Levenson. Conception of organizational worth. American Journal of Sociology, 70, no. 3, November 1964, pp. 324-348.
24. R. L. Lesher. Evaluation of the' role of independent non-university, non-profit research institutions with special attention to mid-western United States. Unpublished doctoral dissertation, Indiana University, 1963-64.
25. R. L. Lesher. The non-profit corporation; a neglected stepchild comes of age. The Business Lawyer, July 1967.
26. P. J. Lovenwell. Long range planning: a powerful tool for effective management. Association Management, 20, n 5, May 1968, pp. 15-18.
27. J. L. Price. Organizational effectiveness; an inventory of propositions. Irwin, 1968, 212 p.
28. S. Priestland. Association policies and practices survey report. Association Management, 20, no. 5, May 1968, pp. 40-44.
29. J. Rees. The standing and efficiency of international non-governmental organizations. International Associations, 1958, no. 9, pp. 625-642.
30. J. Rothenberg. The measurement of social welfare. Prentice-Hall, 1961.
31. C. G. Smith and A. S. Tannenbaum. Some implications of leadership and control for effectiveness in a voluntary association. Human Relations, 18, 1965, p. 265-272.
32. W. A. Songer. A business management appraisal of trade associations. Association Management, 16, September 1964.
33. A. S. Tannenbaum. Control and effectiveness in a voluntary organization. American Journal of Sociology, 67, July 1961, pp. 33-46.
34. P. Vasseur. Les difficultés des organisations internationales non gouvernementales. International Associations, 1958, no. 9, pp. 579-584.
35. H. V. Webb. A practical guide to self evaluation (by associations). Association Management, 20, no. 8, August 1968, pp. 28-31 + 76-77.
36. P. Wasserman. Measurement and evaluation of organizational performance: an annotated bibliography. Cornell University, Graduate School of Business and Public Administration, 1959.
E. Accounting in Non-Profit Organizations (Non-Goernmental)
l. National Health Council (U.S.A.). Accounting and financial reporting procedures for voluntary health agencies. (Washington?), the Council, 1961
2. H. A. Withey and G. Holman. Standards of accounting for voluntary health-welfare agencies. The Journal of Accountancy, 120, August 1965, pp. 47-53
3. W. L. Fluckiger. A philosophy of fund accounting. The Journal of Accountancy', 116, August 1963, pp. 66-71
4. R. E. Czarnecki. Selected applications of managerial and financial accounting to the Roman Catholic Church. Unpublished doctoral dissertation, Michigan State University 1965-1966.
5. L. Englander. Accounting principles and procedures of philanthropic institutions. The New York Certified Public Accountant, April-May 1958.
6. T. L. Holton. Reports on non-profit organizations: generally accepted accounting principles. The Journal of Accountancy, 107, April 1959, pp. 61-67.
7. W. J. Jacquette. Current practices in developing an association budget. Association Management, 17, 1965.
8. L. E. Larson. Church accounting. The Journal of Accoutancy, 103, no. 5, May 1957.
9. J. T. Reeve. Audits of nonprofit organizations (charitable). The Journal of Accountancy', 119, February 1965, pp. 63-67.
10. T. T. Shan. Interesting points relating to charitable foundations. (Report on a conference of charitable foundations on accounting definitions). The Journal of Accountancy, 112, November 1961.
11. R. W. Williams and R. L. Leonard. Financial reporting by nonprofit organizations. The Journal of Accountancy, 113, April 1962, pp. 46-59.
12. G. G. Yankee and L. Morey. Accounting in nonprofit enterprises. In: Backer, M. (Ed.). Modern accounting theory. Prentice-Hall, 1966, pp. 464-484.
this work is licenced under a creative commons licence.