THE IMPACT OF ATTORNEYS ON JUDICIAL DECISIONS: EMPIRICAL EVIDENCE FROM CIVIL CASES

This article analyses the impact of attorneys on the outcome of judicial decisions in civil cases. We currently have little quantitative information about the effect of attorneys on the outcome of civil cases due to (i) the nonrandom pairing of attorneys and cases and (ii) the difficulty in accurately defining what a favorable decision in a civil case is. The Office of the Solicitor General of the Union in Brazil presents a unique research opportunity, since it assigns cases among its attorneys on a random basis and has standardized rules to record outcomes of civil cases. We analyzed the work performed by 386 Federal Attorneys and their impact on 30,821 judicial decisions. Significant win-rate differences among attorneys were detected in half of the 70 teams surveyed. The fact that attorneys achieve different outcomes, despite working in the same type of cases, indicates how judicial decisions can be affected by the work of a attorney in the civil area. No statistical correlation between attorney experience and outcome of civil cases was detected.


Introduction
In ideal circumstances, court cases should be decided based solely on their merits, analyzing evidence introduced by the parties and the law applicable to the case. However, litigation does not proceed separately from external social factors. Several factors may help to explain a litigation outcome, such as judge or jury bias, regional influence and the nature and resources of plaintiffs and defendants (Eisenberg, 1989).
Attorneys also play a key role in the outcome of court proceedings. However, analyzing the impact of attorneys on case outcomes is not a simple endeavor. The attorney often selects personally the cases he will take, making it difficult to determine whether the results are attributable to that attorney or simply to the characteristics of his cases (Anderson & Heaton, 2012). Due to attorney matching, an attorney may have a high trial success rate that has nothing to do with his skill (Shinall, 2010). Because of this case-selection effect, it is usually impossible to isolate and measure the magnitude of the effect of the attorney on the outcome of the case.
In the civil area, researches using reliable methodology are rare. After reexamining the quantitative literature about how much difference legal representation makes in the outcome of civil cases, Greiner and Pattanayak (2012) concluded that we know almost nothing about the effect of representation in civil proceedings.
The objective of this work is to analyze the impact of attorneys on civil cases outcomes. In this sense, the Office of the Solicitor General of the Union -SGU (Procuradoria-Geral da União -PGU) in Brazil offers a unique research opportunity, once its civil cases are assigned to Federal Attorneys on a random basis, which can be described as a natural experiment. We investigate if attorneys that work on the same type of cases (randomly assigned) achieve different outcomes. As it will be demonstrated, in many cases judicial decisions varied depending on the attorney who worked on the civil case, which shows how judges can be sensitive to differences in the work done by those professionals. This article will proceed in the following order. Section 2 brings the theoretical framework of the issue. Section 3 describes the organization where the research was conducted, presents the data used and the statistical analysis applied. Section 4 reports its findings. Section 5 discusses the results. Section 6 concludes.

Theoretical framework
Practically all studies regarding the impact of attorneys that used a random design were conducted in the criminal area. The study carried out by Iyengar (2007) show that defendants with court-appointed private attorneys are more likely to be found guilty and to receive longer sentences than defendants with public defenders. Huang, Chen and Lin (2010) found that while public defenders and government-contracted legal-aid attorneys are about equally effective, they tend to adopt different litigation strategies, which will in turn affect their clients' fates. Specifically, the defendants represented by public defenders tend to have higher conviction rates, but shorter sentences if they are convicted. Abrams and Yoon (2007) pursued this further and investigated which attorneys' characteristics could affect case outcomes. Analyzing the dataset provided by the Clark County Office of the Public Defender in Nevada, which randomly assigns criminal cases among public defenders, they found that defendants with more experienced attorneys obtain lower sentences than defendants with less experienced attorneys, while they did not find that the attorney's legal educational background affects case outcomes.
However, we currently have little quantitative information about the effect of attorneys on the outcome of civil cases. Greiner andPattanayak (2012, p. 2123) discovered that in spite of this issue having been studied multiple times "almost all such studies suffer from methodological problems so severe as to render their conclusions untrustworthy, which (we hasten to emphasize) is different from wrong". This finding is surprising if we remind ourselves that civil cases are much more present in day-to-day lives than criminal proceedings.
If civil litigation is so prevalent, what are the reasons for the paucity of knowledge about the impact of attorneys on the outcomes of civil cases? Two major reasons seen to be the difficulty of finding an environment where the cases are randomly assigned and the complexity in accurately defining what a favorable decision in the civil area is. In the criminal area, the outcome of cases is most evident: the defendant is either convicted or is acquitted; when convicted, the penalty is set mostly in the form of incarceration time. This clarity, however, is not as common in the civil area.
According to Schwab and Eisenberg (1987), there are many definitions of success in the civil area, which can range from a pure economic analysis to more subjective approaches. An economic analysis would involve the calculation of how much has been invested in the case and what his payback is. On the other hand, the subjective analysis seems to be related to the expectations the parties have of the case, which makes it quite difficult to identify the winners and losers in a litigation (Grossman et al. 1999). Abrams andYoon (2007, p. 1155) illustrate this perception with the following example: "If Firm A sues Firm B for $100 million and they settle for $30 million, is this a victory for Firm A or Firm B?".

The Office of the Solicitor General of the Union (SGU)
In Brazil, all legal issues involving the Federal Government are under the responsibility of the Office of the Attorney General of the Union (Advocacia-Geral da União -AGU). Within AGU structure, the division responsible for defending the Federal Government in judicial cases is the Office of the Solicitor General of the Union (SGU), which manages three million lawsuits nationwide. In the year 2015 alone, judicial decisions rendered accounted for almost five billion dollars against the Federal Government.
SGU presents a unique research opportunity, since most of its cases are assigned to Federal Attorneys on a random basis and it has internal regulations that define clearly the outcome of a court decision. When a lawsuit is filed against the Federal Government, the court sends the case to one of the 69 SGU units. In small units, attorneys are assigned all types of cases. In larger units, specific teams are formed by case or legal proceeding type. The case assignment within each team usually is based on the final digit of the court case number, in a mechanical fashion (no previous human evaluation is made before the distribution of cases to the attorneys). The system works as follows: Attorney 5

31%
It is this system that ensures that cases are randomly assigned, controlling a number of factors that could affect the court decision: the difficulty of the cases, the judge who will decide the case, the attorney of the opposing party, and practically all other variables linked to the lawsuit, including hidden variables.

Data
The data released by the SGU were the 93,263 defenses made by 953 Federal Attorneys across the country during the year 2011. At the beginning of this study, 190 teams were identified.
The first step in analyzing data was to identify the number of defenses made by each attorney. Some attorneys presented a low number of defenses, which may indicate a short passage in a unit or even a registration error. We deemed it inappropriate to assess the impact of the attorney's performance from a minuscule number of activities. Thus, all attorneys who presented fewer than 20 defenses were eliminated from the dataset.
In sequence, we associate defenses with judicial decisions. Only cases with decisions classified in one of the three basic outcomes (favorable, unfavorable and partially favorable) were considered. As other outcomes do not allow verifying the impact of the attorney in the court decision, such cases were eliminated from the database. The next step of the research was to calculate the win rate per Federal Attorney, i.e., the percentage of favorable decisions connected to their defenses. As we considered defenses produced in 2011 with judgments rendered by February 2014, we had a sufficient time base to access attorneys´ performance outcomes. However, in some groups only 20% or 30% of cases have had a court decision rendered. To overcome this problem, a new limit has been set: we considered only attorneys with 15 or more judged cases. In addition, in order to ensure greater comparability among professionals, we considered only teams with at least three attorneys.
After completion of all these steps, several teams were reduced or disappeared from the analysis. The final database is composed of the following numbers: 70 teams, 386 Federal Attorneys and 30,821 defenses with judicial decision.

Checking the similarity of cases in each team
The procedure used until this point should have been sufficient to identify teams of attorneys that work in cases with the same characteristics, since most SGU units use random assignment of cases within their teams. However, it is necessary to verify if the randomization mechanism is really working, checking the main assumption of a natural experiment: whether cases (units upon which a treatment operates) were really randomly assigned among attorneys (treatment) of each team.
The credibility of this step depends on whether relevant control variables (also called pretreatment covariates) have been collected and whether sufficient balance has been achieved between the treatments (Ho and Rubin, 2011). Greiner (2008) explains that the control variables are those prior to treatment and that it is important to distinguish the control variables (not affected by treatment) of the outcome variables (affected by treatment).
In this verification, the control variable used was the judge appointed to the case. This is a valid control variable because it is not affected by treatment: the judge´s appointment predates the attorney assignment to the case. It follows that if an attorney has a higher concentration of cases from a particular judge than another attorney of his team, their outcomes cannot be compared. The balance would be the confirmation that the processes were randomly assigned.
Therefore, a study into the balance of judges in relation to attorneys within each team was performed. An analysis was made, using k-means, as to whether each group of attorneys would have its homogeneity improved if divided into subgroups according to the judges of their cases. This was done by determining the value of k that minimizes the standard error of the gap statistic (Tibshirani et al. 2000). Bootstrapping was then conducted, clustering by k-means with k varying between all possible integers for that group. If the k value that minimizes the gap standard error is one, this indicates that the team is receiving cases from various judgments homogeneously.
After applying this methodology, results show that 95.26% of teams can be in fact considered homogeneous, which reinforces the hypothesis of random distribution of cases within the teams. One can note that the procedure adopted does not constitute a hypothesis test, but the findings are consistent with conference call conversations with the heads of SGU units, meaning that cases are randomly assigned within each team of Federal Attorneys.

The differences among win rates: statistical analysis
Once identified the win rate of each attorney and checked the similarity of cases in each team, we decided to verify if the differences in the outcome of attorneys of a same group were statistically significant, i.e., whether they were not simple coincidences. Each court decision presents a binary result: a favorable decision or not (which includes unfavorable and partially favorable decisions). Given this dichotomous variable, we opted for the use of logistic regression to access the significance of the results.
Logistic regression is a statistical technique designed to predict the likelihood of the occurrence of an event. In this study, we calculated the probability of an attorney reaching a favorable decision. After that, we compared these probabilities among attorneys of the same team in order to determine if there were statistically significant differences among them. Logistic regression captures the observed data and sets a mathematical model that goes beyond empirical observations. As the parameters of this model are adjusted by the least squares method, they tend not to be biased.
Then we checked if the model as a whole presents solid quality adjustment (goodness of fit). Afterwards, we verified whether the model better predicts the probability of favorable decisions of each attorney than the simple average of the group, and if this improvement is statistically significant. In this sense, we performed an analysis of variance.

Correlation between attorney experience and win rate
The next step of the research was to investigate a factor that could explain the differences in attorneys' outcomes. Undoubtedly, experience is the most researched predictor of attorney performance. On the one hand, several studies, including those that used the random distribution of cases as research design, indicate the existence of a positive correlation between the attorney´s experience and case outcome (Abrams & Yoon 2007;Iyengar 2007;Boylan & Long 2005). Other studies, however, did not identify this relationship (Hinkle 2007;Norberg & Compo 2007). Wright & Peeples (2013) found a nonlinear relationship: the first years of experience have a positive impact on the results of judgments, but the following years have a negative impact.
As is common in many public organizations, seniority plays a central role at SGU. In every promotion cycle, half of the vacancies are based on the years since attorneys have joined the organization. Regarding transfers from a unit to another, this is almost the only criterion used. While great importance is given to seniority at SGU, it is not known whether it affects attorneys' outcomes.
To make a fair comparison of attorneys' outcomes, we cannot consider the mere observed outcome of all professionals in the country. As they are part of different units and work on different cases, comparing their percentage of favorable decisions would be unfair. Thus, we decided to standardize the results of the attorneys. Firstly, we considered the mean of each team as zero standard deviation and then we calculated the positive and negative differences between the mean and the attorney outcome in terms of standard deviations (standardized outcome). Once the weighted outcome of each attorney was calculated, we tested possible correlation of this index with the attorney's experience, measured in terms of years since joining the organization (the same criterion used by Abrams and Yoon, 2007). For this test, we considered only the teams that showed a statistically significant difference (<0.05) in outcomes among their members. Table 1 presents the results of the 70 teams surveyed: it shows if the difference between the attorneys with the lowest and the highest percentage of favorable decisions in each team is statistically significant (Column Difference Sig.), the model significance to each of the teams (Column Model Sig.), and the chances the attorney with the highest percentage has in obtaining a favorable decision in relation to the attorney with the lowest percentage (Column Max Odds): Source: authors using data provided by SGU Table 1 shows statistical significance in the results and respective models of approximately half of the 70 groups. If we consider the sig. <0.05 standard level, 30 groups (43%) present a significant difference among the outcomes of their attorneys, a number that increases to 41 groups (59%) if we consider the sig. <0.10 level. Interestingly, there was a statistically significant difference in groups of various sizes (in groups from 3 to 29 attorneys), from all regions of the country and which were in units of all types (Regional Units, State Units and Local Units), even if we consider the stricter sig. <0.05 level.

Results
The 30 groups in which significant (<0.05) differences were found congregate 212 of the 386 surveyed attorneys (55%), which means that a little over half of SGU attorneys are part of teams in which statistical differences outcomes were found, in a model also statistically significant (sig. <0.05). Considering the sig. <0.10 level, the 41 groups congregate 267 attorneys (69%), in a model also statistically significant (sig. <0.10).
The correlations between the weighted outcome and experience are statistically insignificant, according to Pearson's product-moment correlation test. In other words, attorney seniority does not seem to work as a predictor of performance in the researched organization. This conclusion contradicts most of the literature written so far on the subject and, arguably, common sense.

Discussion
Different judges are expected to render similar judgments in similar cases. This is the philosophical basis of Justice. Therefore, attorneys working on similar cases are expected to achieve similar outcomes. In the organization surveyed, additional factors induce attorneys to reach similar judicial decisions.
Each Federal Attorney works in dozens of cases per year. The vast majority of lawsuits filed against the Federal Government are not unprecedented cases and the judges usually follow the precedents set by superior courts. In this context, a competent or mediocre performance by Federal Attorneys should produce low impact on the final case outcome.
The Federal Attorneys are submitted to an extremely rigorous selection process. Generally, less than 1% of applicants are approved, and this small group tend to have a very homogenous professional profile. After entering public service, there are no economic stimulus for an attorney to perform better than his colleagues. The remuneration of all Federal Attorneys is fixed -do not vary in accordance to the individual success rate. Wage differences are small (basically motivated for seniority reasons) and job security is high. Finally, all attorneys work within the same organization environment -the physical structure, support team and access to systems are almost identical. Therefore, it is no surprise that half of the 70 teams presented no significant differences in the outcome of their attorneys.
The surprise is finding significant differences in attorneys' outcomes in the other half of the 70 groups. If virtually all factors induce homogeneous attorney behavior, what are the possible causes for such heterogeneous outcomes? A first hypothesis can be linked to the existing stability in public service, where dismissal for poor performance is virtually nonexistent. This could induce some people to perform poorly, making their results fall below their peers. There is a possibility that the quality of the work done is so poor that it affects the outcomes even of cases that should be decided in a similar way. This can happen because cases can be decided based not only on the rules of law, but also on the specific facts involving the situation. If legal professionals do not work on the facts properly, they tend to lose the case. On the other hand, it is possible that, despite incentives to the contrary, other professionals still maintain high internal motivation, always seeking the best outcomes, which can result in a high win rate.
We suspect, however, that the impact of attorneys on the outcome of civil cases is higher than that found in this study. The methodology used in this survey focused only on the general outcome of the judicial decision, i.e., if it was favorable or not. However, in many civil cases, the main concern of attorneys is to reduce the monetary award (as sometimes the main concern of a criminal attorney is to reduce incarceration time). In order to measure this effect, it would be necessary to compare the initial amount requested by the plaintiff with the final amount paid by the Federal Government. Unfortunately, these more refined data are not reliably registered, which makes this investigation difficult. In any event, if differences in outcomes of attorneys have been detected using more general data (overall outcome of the judicial decision), these differences would probably become more evident if more refined data were used (such as the reduction in the value paid).
One must remember that this study analyzed attorneys with similar profiles. If it were feasible to investigate attorneys with different backgrounds working in the same kind of cases (like many criminal studies did, comparing private and public defenders), it is possible that a broader variety of outcomes would be found.
One hypothesis to explain the absence of a relation between a attorney's experience and his outcome can be the trade-off between experience and motivation. Usually, during the first years in the organization, motivation is high, but experience is low. Over the years, the experience increases, but motivation diminishes.

Conclusion
This article made an empirical investigation about the impact of attorneys on outcomes of civil cases. In accordance with what was presented, attorneys can achieve different outcomes, despite working on the same type of cases. The judicial decision varied depending on the attorney who worked on the civil case, which shows how judges can be sensitive to differences in the work done by those professionals. If the magistrates were impervious to the work done by attorneys, the case outcomes should be quite similar.
If, on the one hand, the differences in the outcomes of attorneys indicate that the judicial decisions may be influenced by the attorney's performance, the reverse is not necessarily true, i.e., equality of outcome in judicial decisions does not mean that attorneys have no impact. It may be that, for example, all attorneys have very similar performances and are impacting the magistrates similarly.
However, we caution against broad generalizations. The answer to the question of whether counsel affects case outcomes is by nature jurisdiction-based. Nevertheless, the results of our study provide certain references for further research, considering it is one of the first to analyze the impact of attorneys on outcomes of civil cases using a solid methodological framework.
The scarcity of empirical studies analyzing the impact of attorneys on the outcome of civil cases evidences the gap to be filled in the construction of scientific knowledge in this field. Identifying the underlying characteristics of high performance attorneys in civil area remains a challenge for future legal empirical research.