Evaluating Court Performance : Findings From Two Italian Courts

This study is part of a wider research project aimed at developing and testing a Performance Measurement System (PMS) for courts based on a Balanced Scorecard (BSC) framework. The current study represents an initial effort to describe results of a performance measurement attempt that may suggest some challenges in developing a comprehensive PMS for courts. We have tried to assess the performance in two Italian courts focusing on three issues: efficiency measures (clearance rates, case turnover, and disposition time), culture assessment, and Information Systems (IS) success. Our findings provide some useful and interesting insight for researchers and practitioners.


Introduction
Courts have experimented with innovative management, such as greater autonomy for court administrators and new ways to work supported by Information and Communication.Over the last twenty years the Italian Judicial System (JS) has been facing a crisis of performance, such as the unacceptable length of proceedings, a large number of both pending civil and criminal proceedings and has had a significant amount of money invested 4 .As a consequence, the Italian Legislator is making efforts to realize a modernization process of the JS aimed at changing the organization of courts, management approach and performance measurement.Italian Technology (ICT); ICT is an important medium to spread managerial philosophy in the JS

5
. Italy has been one of the European Countries that has invested the most in ICTs to develop an "egovernment approach" for the JS (e-justice) 6 .
Despite the modernization process and the considerable investment in ICT, to date the results achieved have been very few and the Italian JS is still characterized by poor performance 7 .A managerial approach for courts, and the use of PMSs, in particular, could be useful for court administrators and presiding judges in order to monitor the court activities, the achievement of goals and thus to improve court efficiency and effectiveness.
The importance of the managerial approach and performance measurement is shown by the efforts made over recent decades by scholars of Court Management and Judicial Administration to produce Performance Measurement Systems (PMS) for US courts."Performance measurement is crucial to a court's ability to provide high quality yet cost-effective and efficient services to its customers.Court managers and presiding judges increasingly embrace the idea of systematically integrating performance measurement into the daily operations of the courts" 8 .
In order to overcome these limitations, BSC uses indicators to monitor targets achievement from four dimensions: financial, customer, internal process, and innovation and learning.More specifically, BSC is a management tool that provides a periodic and concise assessment of how well the organization is progressing towards achieving its strategic and operational goals.The model was created by Kaplan and Norton in the early 1990's and has grown in popularity ever since.To each strategic objective of the organization are associated some indicators on the four perspectives of the BSC 14 .Furthermore, BSC has been widely used to evaluate ICT investments in different kinds of organizations, both profit and non-profit oriented 15 .In particular, Wright and colleagues (1999) applied BSC to evaluate software performance including this aspect in the innovation and learning perspective.Rosemann and Wiese (1999), instead, used the BSC for the evaluation of IS tasks such as the process of implementation and operational use, adding a new project perspective to the traditional four dimensions.
The evaluation of effectiveness of ICT investments and the inclusion of this dimension within CPMS could be useful for court administrators and presiding judges because it provides the opportunity to understand if and how the ISs contribute to improve court performance 16 ."IS success or effectiveness is critical to our understanding of the value and efficacy of IS management actions and IS investments" 17 .
Many scholars recognized the need to evaluate IS success and a large number of system success measures exist.The most widespread and recognized indicator is user satisfaction 18 .An IS is a successful information system if it is capable of satisfying the information requirements of its users.
According to DeLone and McLean (1992), however, a single indicator is not sufficient to measure such a complex construct as the success of an IS.The authors therefore developed a model, known as the "IS success model", based on several dimensions, aimed to investigate "What causes IS success?" and, consequently, IS impact on individual and organizational performance."The IS Success Model" considers six dimensions: 1. system quality, characteristics of the information system itself which produces the information; 2. information quality, quality of the information that the system produces, primarily in the form of reports; 3. information system use, how many times people use IS; 4. user satisfaction, IS ability of satisfying the information requirements of its users; 5. individual impact, the contribute of IS on individual work performance, and 6. organizational impact, the effect of IS on organizational performance.In particular, system quality concerns the consistency of the user interface, the quality of documentation, and whether there are bugs in the system.Information quality concerns the relevance, timeliness and accuracy of information generated through the system.IS use concerns the usage of the system by user.User satisfaction, instead, is the sum of an individual's reactions to a set of factors affecting IS success.Finally, individual and organizational impacts are the outcomes attributed to user satisfaction and IS use.

A Performance Measurement System for Courts
Within the wider research project of the "International Laboratory for the Study of Judicial Systems", we have tried to define the CPMS based on BSC framework.Following the studies of Ostrom and colleagues (2008), we have chosen the BSC framework to measure court performance because BSC makes extensive use of qualitative and non-financial indicators to demonstrate the overall ability of the organization to adequately satisfy stakeholders.We decided to add another dimension to the traditional four of the BSC: the IS success dimension.Some authors, in fact, have argued that although the innovation and learning dimension of BSC could also provide indicators for the evaluation of IS performance, in some cases it is not suitable for this purpose 19 .Therefore, CPMS consists of five perspectives: customer, internal operating, financial, innovation and learning, and IS success.Figure 1 shows the CPMS.The five dimensions composing CPMS are the following: 1. customer perspective: the customer of a court is a person or an organization that receives the service provided by the court.Indicators included in this dimension measure the court's accessibility and treatment of customers in terms of fairness, equality, and respect.To define these measures, customers are asked to answer questions about how they are treated in court and whether the court's decision making process seems fair.These questions have been defined on the basis of the questions used by the CourTools; 2. internal operating perspective: indicators used in this perspective assess the court's ability to be efficient, controlling its internal procedures and environments.The efficiency indicators used could inform presiding judges and court administrators about how well resources are used to achieve intended goals in terms of case resolution.In this perspective we include the indicators proposed by the European Commission for the Efficiency of Justice (CEPEJ) to evaluate efficiency of European courts.These indicators are: "clearance rate", which is the number of cases resolved as a percentage of the number of incoming cases; "case turnover ratio" measured as the resolved cases divided by unresolved cases; "disposition time" calculated as 365 divided by "case turnover ratio".These indicators are a fundamental management tool that evaluate the length of time it takes a court to process cases; 3. financial perspective: the indicator included in this dimension is a cost indicator such as "cost per case".It is a measurement of the average cost of processing a single case, by case type (e.g., civil and/or criminal cases)."Cost per case" is an indicator developed in the CourTools.It could aid managers in decision-making about the resources allocation in order to improve cost effectiveness of courts; 4. innovation and learning perspective: we include in this dimension some indicators that could be useful to evaluate the contribution of human resources, information capital, and court culture to support innovation and learning.
Particularly, for human resources, we use the following indicators: number of administrative staff, number of professional judges, and number of IS end-users.Information capital, instead, is evaluated using the following indicators: ICT software and hardware investments.Although the values of these indicators are not a direct measure of performance, they can be interpreted as an approximation of the potential for innovation and learning of the court.For example, information about the number and types of human resources could be useful to understand if court staff is large enough to give court the possibility to carry out its activities, but also to innovate and learn.Finally, court culture is evaluated using a specific scale .The CCAI is based on two specific dimensions such as solidarity and sociability.Solidarity refers to how unified the court is and shows the degree to which judges and court personnel working are to shared goals, mutual interests, and common tasks in order to get common ends, while sociability refers to how closely knit are members of the court, highlighting the degree to which judges and court personnel work cooperatively as one in a cordial fashion.Combining these dimensions, the CCAI provides a classification scheme that systematically produces four distinguishable types of cultures (communal, networked, autonomous, and hierarchical) measured by five content dimensions such as case management style, judge-staff relations, change management, courthouse leadership, and internal organization.Communal culture (High Sociability -Low Solidarity) emphasizes the importance of getting along and acting collectively.Networked culture (High Sociability -High Solidarity) emphasizes collaborative work environment and effective court-wide communication.Autonomous culture (Low Sociability -Low Solidarity) is characterized by judges broad discretion in processing cases.Finally, Hierarchical culture (Low Sociability -High Solidarity) defines rules and procedures to meet clearly stated court-wide objectives 23 .

IS success perspective: as previously argued, the model of DeLone and McLean consists of six dimensions useful to
investigate a process understanding of IS and their impacts.The model analyses three components, such as creation, use, and consequences of system use, and "each of these steps is a necessary, but not sufficient, condition for the resultant outcome(s)" 24 .With reference to courts, this model promotes understanding of ICT applications, such as a case tracking system, used by court administrative staff for performing activities and their impacts.Understanding user's perception of these applications, as well as their usage and effectiveness, provides court managers important information for supporting strategic and operational decision making.

Research Methodology
Data was collected from two courts, Naples and Bari, by using a qualitative and quantitative methodology.In particular, a qualitative analysis was conducted by ethnographic interviews and document analysis in order to collect data useful to measure indicators of internal operating perspective.Overall, we conducted nine interviews for each court including a preliminary interview with the court administrator in order to obtain general information about courts and eight semistructured interviews with two court administrators, an IS manager, two chancellery officers and three administrative officers.Furthermore, through the analysis of court documents (Judiciary Administration Report; Directorate General for Automated Information Systems; CEPEJ reports) and data collecting we obtained some important information about the performance of the courts and calculated CPMS indicators.
Then, a quantitative analysis was carried out by using a structured questionnaire, in Italian, completed by the administrative staff of Bari and Naples courts during the period from the 20 th of October 2009 to the 15 th of January 2010.The questionnaire was aimed at capturing respondent profile information and to measure court culture and IS success by using two scales developed by research.In particular, these scales were translated from English and then back-translated to check the reliability of the translation using the procedure suggested by Brislin (1970) 25 .Furthermore, we conducted a pre and pilot test to validate the measures and in the first instance we obtained feedback from representatives of the court administrative staff.Findings of the pre-test confirmed the reliability and consistency of the scales used.
Then, we gave questionnaires to all the administrative staff of Bari and Naples courts.Of 620 court administrative staff, 212 were from the court of Bari and 408 from the court of Naples, we received a total of 321 completed questionnaires: 59 from Bari (r.r.27.83%) and 262 from Naples (r.r.64.21%).To minimize data entry errors, the collected data was checked for consistency.As a result, 314 valid responses were collected.
With reference to measurement, court culture was measured using the CCAI scale developed by Ostrom and colleagues (2007).This scale allows both current and preferred cultural conditions to be measured by assessing five key dimensions of court culture.For Case Management Style an example indicator is: Judges are committed to use case flow management with the support of administrative and courtroom staff.Written court rules and procedures are applied uniformly by judges.For Judicial and Court Staff Relations, an example is: Judges value and promote a diverse workforce and diversity of ideas; act to enhance professional administrative and courtroom staff development; seek to treat all staff with fairness and respect.For Change Management, an example used is: Judges and court managers seek input from a varied set of individual and measure court user preferences concerning policy changes [..].For Courthouse Leadership, an example used is: Judicial and administrative staff leaders seek to build an integrated justice system community [..].For Internal Organization, an example used is: Judges and administrators seek a shared court-wide view of what needs to be accomplished [..].Each dimension is composed by four sets of statements; responders should divide 100 points among these four statements giving a higher number of points to the statement that is most often emphasized.
Finally, IS success consists of six dimensions measured as follows.System quality, information quality, and IS use was measured using respectively two-item, seven-item and single item of Rai and colleagues ' (2002) 26 scales.Examples of items used to measure variables are respectively: Is system user friendly?Does system provide the precise information you need?How many am I dependent on system?Twelve items were used to measure user satisfaction based on Doll and Torkzadeh 's (1988) 27 end-user computing satisfaction scale.An example used is: Are you satisfied with the system?Finally, individual impact was measured using Etezadi-Amoli and Farhoomand's (1996) 28 user performance four-item scale.An example used is: How successful has system been in improving the quality of your work?Organizational impact was not measured because it could be inferred from indicators related to other CPMS perspectives.All the IS success dimensions were measured by five-point scale ranging from "Strongly disagree" (1) to "Strongly agree" (5), except for IS use and individual impact that were measured by seven-point scale ranging from "Strongly disagree" (1) to "Strongly agree" (7).

Results
Concerning the internal operating perspective, we calculated the clearance rate, the case turnover ratio, and the disposition time indicators as well as a summary of civil cases management for Bari and Naples courts.On the contrary, concerning the innovation and the learning and IS success perspectives, data was collected by the "t-test" statistical method.The t-test is a statistical test common used to assess whether the means of two groups are statistically different from each other.In this regard, the t-test is very useful for our study because it allows us to compare court culture and IS success dimensions among two courts, showing the significant differences at p≤0.010 value.
With reference to the first perspective, the clearance rate for Bari court for the year 2010 is 114.6%, while it was 107.1% for Naples court.The longitudinal analysis has highlighted a general increase in productivity of the two courts, there is only a slight decrease in production capacity on Naples court from 2008 to 2009 (Fig. 2).

Fig. 2 -Clearance Rate
The case turnover ratio for Bari court in 2010 was equal to 0.43, while for Naples court it was 0.47.Moreover, the longitudinal analysis has shown that the value of case turnover ratio for Bari court increased from 2007 to 2008 and then decreased.Instead, it is almost constant during the observed period for Naples (Fig. 3).With reference to innovation and learning perspective, the results have shown that the Bari court is composed of 106 judges and 252 administrative staff.Investment in ICT hardware amounted to € 40,000, of which € 35,000 for the purchase of 50 computers and € 5,000 for other computer equipment.Investment in ICT software, by contrast, is not quantifiable because these investments are often provided by the central government without any indication of cost.The court of Naples is composed of 438 judges and 674 administrative staff.Data relating to ICT hardware and software investments is not available.Furthermore, using CCAI 29 we assessed the organizational culture of the two courts and compared results in order to verify differences.Thus, firstly, we compared the current and preferred cultural types within each court in order to understand the gaps among cultural models.After, we compared the culture models of Bari and Naples courts in order to understand the difference between two courts about current and preferred organizational cultures. 29See note 17 supra.Table 1 shows the results of comparative analysis between current and preferred culture for each court, while Table 2 shows the results of comparative analysis of culture types between two courts.
Tab. 1 -Current and preferred culture comparative analysis for each court

Tab. 2 -Bari and Naples Courts comparative analysis for current and preferred court culture
The results of comparative analysis within two courts has shown that current and preferred court culture types are often statistically different from each other (Sig.≤0.010).On the contrary, the results of comparative analysis between two courts have shown that cultural types, both current and preferred, are not statistically different form each other (Sig.≥0.010), apart from some exceptions.To achieve a better understanding of CCAI results, we built the following graphs, developed from two previous tables, in order to better show the gaps among current and preferred court culture types for each of the five dimensions and for each court.Figure 6 shows the results of CCAI for the Case Management Style dimension.With reference to Case Management Style, the court of Bari is currently characterized by autonomous and hierarchical cultural models, while communal and networked are preferred.Moreover, autonomous and communal current cultural models prevail in the court of Naples, while communal and networked are preferred.

Fig. 6 -Case Management Style
In both courts, despite a general agreement on court's goals, judges are relatively free to make their own determinations on case flow management.Moreover, the court of Bari is also characterized by more rules and procedures to meet clearly stated court-wide objectives, while the other court emphasizes the importance of group involvement and mutually agreed norms rather than established rules and procedures.
Finally, both court personnel prefer a collaborative work environment and effective court-wide communication aimed to involve people and to decide on policy guidelines.Figure 7 shows the results of CCAI for judge-staff relations.

Fig. 7 -Judge-Staff Relations
With regard to judge-staff relations, the CCAI results have shown that the court of Bari is currently characterized by communal and autonomous cultural models, while it prefers communal and hierarchical types.On the contrary, the court of Naples is characterized by communal and autonomous cultural types, but it prefers communal and networked cultural archetypes.In this regard, in both courts, judges seek to involve and to collaborate with the administrative staff in a flexible way, such as norms, rather than established rules and firm lines of authority.Whereas, the court of Bari prefers the current model, but its personnel would also like to use evaluation systems and performance appraisals in order to obtain rewards, promotions, and merit recruitment.The court of Naples's personnel, instead, also prefers working in a collaborative environment characterized by more flexibility, rather than its traditional environment characterized by more authority and wide discretion of judges.Figure 8 shows the results of CCAI for the change management.

Fig. 8 -Change Management
With reference to change management, the CCAI results have shown that both courts are mainly characterized by autonomous and hierarchical cultural models, while the hierarchical type is even more preferred by administrative staff of the courts.Thus, judges seek individual ways to change management resisting a rule and process bound organizational setting rather than centralized change initiatives.On the other hand, judges are inclined to use technology, new ways of working and interaction inspired by principles of management in order to improve the timeliness of case processing and accuracy of record keeping.In both courts, judges who are perceived as good coordinators and organizers and who seek to achieve the advantages of order and efficiency are even more preferred by administrative staff.Using new technologies and principles of management court personnel could improve their individual performance promoting widespread benefits for the whole organization.Figure 9 shows the results of CCAI for the courthouse leadership.

Fig. 9 -Courthouse leadership
With regard to courthouse leadership style, the CCAI results have highlighted that both courts are characterized by hierarchical and autonomous cultural models, while they would prefer to go to other, opposite, cultural models such as communal and networked.In both courts, presiding judge leadership is inhibited because each judge prefers to work with a few corresponding staff members of their own choice.Furthermore, each judge establishes rules and directives to guide court operations and uses their own channels to get things done.On the contrary, the personnel of two courts would like to emphasize human relationships in order to mutually agree upon the court performance goals, to obtain more job satisfaction, and to build an integrated court system community.Finally, figure 10 shows the results of CCAI for the internal organization.

Fig. 10 -Internal Organization
With reference to the internal organization, both courts are currently characterized from the prevalence of the autonomous cultural model.However, the court of Bari prefers communal and networked cultural types, while the court of Naples would like to go to networked and hierarchical models.Thus, in both courts the internal organization is autonomous so that each judge decides how to organize their own work and has wide discretion to get things done.Two courts emphasize stability and slow change of practices, while the confrontation is minimized.On the contrary, the court of Bari prefers the collegiality and teamwork pointing out the role of personal relations in workplace.Court personnel prefer informal channels to communicate and to share information among judges and administrative staff and to work collaboratively to perform case processing.The court of Naples, like the court of Bari, prefers internal organization based on collaborative work, but also would like to have a clear division of duties and formalized roles.
Regarding IS success perspective, as previously mentioned we conducted a comparative analysis of IS success dimensions between the two courts in order to understanding where ISs are perceived to be more effective for performance improvement.The results of the comparative analysis are presented in Table 3.  Findings have highlighted that both system's quality and information quality are positively perceived by court personnel because their responses' mean is higher than 2.5 (Bari court: means 3.57 and 3.35; Naples court: means 3.32 and 3.32).
Regarding user satisfaction and IS user dimensions, findings have shown that court personnel are satisfied with the system (Bari court: mean 3.47; Naples court: mean 3.34) because there is a fit between job requirements and IS functionality.Consequently, court personnel use it to perform the court activities (Bari court: mean 5.71; Naples court: mean 5.54).
Regarding individual impact, results have highlighted that within two the courts, administrative staff perceive positively the benefits occurring at the level of individual performance, because their responses' mean is higher than 3.5 (Bari court: mean 4.91; Naples court: mean 4.48).
Overall, comparing the dimensions of the IS success model between the two courts, our findings have shown that Bari and Naples courts are similar with regard to IS success dimensions, except for the individual impact dimension (Sig.≤0.010).In this regard, Bari court personnel perceived more benefits deriving from IS compared to Naples court personnel.

Conclusions
Overall, three main conclusions emerge from our analysis.
Firstly, despite that over the recent decades the Italian JS has been characterized by a dramatic crisis of performance, our findings have shown an opposite trend from 2007 to 2010 for Bari and Naples courts.More generally, internal operating perspective's indicators have shown a good civil case management for both courts and for Bari in particular.Results have highlighted that both Bari and Naples courts have been able, not only to justify the demands of justice during the year, but also to reduce the backlog and the number of pending cases.These results may have arisen from the modernization process of the JS introduced by the Italian Legislature, tough management approach and ICTs investment.However, despite these results appearing positive when compared with those achieved by other Italian courts, they cannot be compared with the performance of most European courts.According to research, this gap could arise from the limited autonomy and empowerment of Italian court managers, but also from other inadequate organizational characteristics of courts30 .The Italian Legislature has enacted three basic principles, such as autonomy, responsibility and evaluation in order to improve the performance of PA.With regard to the JS, perhaps the action on these three principles is still small and thus unable to generate the desired improvements.We believe that PMSs, like CPMS, could be useful to increase the effort on the level of the evaluation, also by delegating more responsibility to court personnel and so improving the internal and external accountability of courts.However, presiding judges and court administrators require greater autonomy and specific managerial skills.
Second, with reference to court culture our findings have shown that prevailing current cultural models of court personnel are hierarchical and autonomous, while the cultural models preferred by court personnel are communal and networked.Italian JS is characterized by a bureaucratized governance model and, thus, has a strong resistance to change that hinders the modernization process.As research suggested the resistance to change is one of the peculiar characteristics of the bureaucratic governance model of Italian Public institutions and in particular the JS.This seems principally due to the autonomy that the law allows to the judicial branch.However, our analysis also shows the willingness of court personnel to change their ways of working through the adoption of new cultural and governance models, similarly to what happens in other sectors of PA.
Finally, our findings have also shown that administrative staff of both courts positively perceived the benefits arising from IS on individual performance.ISs have encouraged the digitalization of documents and the streamlining of organizational processes, allowing the integration of existing databases and to explore the possible uses of ICT to improve the data exchange in the JS 31 .Using these applications, both administrative staff and judges could access legal information timely and without time and spatial limits.In this regard, these applications are useful for receiving information in a timely fashion, reducing the resolution time of legal cases and improving individual and court performance.Furthermore, findings of comparative analysis on IS success between two courts have also identified that Bari court personnel perceived more benefits derived from IS compared to Naples.The literature agrees that user satisfaction and IS use are two determinants of the individual impact dimension 32 .In Bari court, these dimensions are, on average, higher than Napoli court; consequently, for Bari court personnel, IS contributes most to the improvement of their performance. 31Contini F., Cordella A. (2007), Information System and Information Infrastructure Deployment: the Challenge of the Italian e-Justice Approach, The Electronic Journal of e-Government, 5(1). 32See note 14 supra.

Fig. 5 -
Fig. 5 -Civil Cases of Naples and Bari Courts Culture is an important element enabling court performance because it encompasses and makes coherent […] values, expectations, and assumptions about how work gets done in a particular court" 20, the Court Culture Assessment Instrument (CCAI), which has been19See note 12 supra Martinsons M.G. (1992); Rosemann M., Wiese J. (1999).20OstromB.J., Ostrom Jr. C.W., Hanson R.A., Kleiman M. (2007), Trial Courts as organizations, Philadelphia: Temple University Press.adapted by a researcher of NCSC from the Organizational Culture Assessment Instrument (OCAI) scale 21 ."