Is the Judge or the Clerk Making the Decision? Measuring the Influence of Judicial Assistants via an Experimental Survey among Dutch District Court Judges

Today, law clerks and judicial assistants have an important position in the judicial decision making process. Yet, the legitimacy of their role is regularly questioned. In the discussion on the legitimacy, it is widely assumed that judicial assistants are actively involved in adjudication and influence the outcome of judicial decisions. However, there is still little empirical evidence to substantiate these assumptions. This article contributes to the knowledge regarding the role of judicial assistants in adjudication. It does so by means of conducting an experimental survey among Dutch administrative law district court judges (N = 80). In this survey, we measure the role and influence of judicial assistants in adjudication in three different steps. We demonstrate that judicial assistants are actively involved in adjudication via soundboarding, preparing bench memos and drafting judgments and have a self-reported and experimentally established influence on the outcome of court cases.


Introduction
During the past decades of rising caseloads in courts all over the world, legally trained, nonjudge court personnel (law clerks, staff lawyers, judicial assistants, etc.) are assigned a more prominent role in the adjudicative process. While employing judicial assistants is often not novel in these judiciaries, the number of assistants and the duties assigned to them have increased. 1 This provides these assistants with a more prominent position at courts. Regarding the US, Posner expressively speaks of 'the age of the law clerk '. 2 Most of our knowledge about judicial assistants is based on US literature, with most studies focusing on law clerks at the US Supreme Court. However, interest in judicial assistants of both practitioners and academics in other jurisdictions is on the rise. 3 For example, the Consultative Council of European Judges (CCJE) (representing the 47 member states of the Council of Europe) recently issued an Opinion on the role of court clerks and legal assistants in courts and their relationships with judges. 4 This Opinion reveals that judicial assistants also have an important position in the courts of many European states. Judicial assistants can relieve judges of non-judicial tasks, help courts to reduce backlogs and assist judges in producing high quality judgments. 5 However, the CCJE Opinion also raises concerns regarding judicial assistants' involvement -and potential influence -in the process of adjudication. Such concerns are also found in academic publications. Various authors 6 suggest that the extensive employment of judicial assistants or clerks changes the way judges adjudicate and has undesirable effects on the quality of judicial decisions. According to these authors, drafting a judgment serves as a natural avenue to reconsider one's initial judgment 7 and to fully consider the plurality of claims by the parties in reaching a decision. 8 Having assistants involved in adjudication potentially gives rise to a diminishing sense of personal responsibility for judgments, 9 leads to judgments based on policy preferences instead of legal arguments 10 and results in more monocular adjudication. 11 In order to have a normative discussion about the legitimacy of the judicial assistants' contribution, it should first be asked whether these assistants are in fact actively involved in adjudication and influence the outcome of judicial decisions. However, these pressing questions remain largely unanswered by current research. 12 In this paper, we address this empirical question.
While the question of the role and influence of judicial assistants in adjudication is most pressing, it is also a particularly difficult one to answer. Complications in gaining access to the decision making process as well as confidentiality agreements for judges and judicial assistants make it challenging to conduct research on judicial assistants. 13 Most of the current knowledge on the influence of judicial assistants is therefore based on first-person accounts, interviews and -to a lesser extend -surveys. 14 The determination of the involvement and influence of judicial assistants are thereby exclusively based on judges' and assistants' selfreported level of influence.
For this article, we have conducted an experimental survey among Dutch district court judges at the administrative law division. To the best of our knowledge, it is the first large academic survey on judicial assistants' influence outside of the US as well as the first study based on experimental behaviour. In our study we will establish the following: 1) through which duties judicial assistants may wield influence; 2) what the overall perceived influence of an assistant is in a concrete court case; and 3) the extent to which judges' decisions are influenced by the suggestions of judicial assistants in their bench memoranda (henceforth 'bench memos'). We establish the latter using an experimental vignette. The experimental vignette enables us to establish the extent to which judges' decisions are affected by assistants' suggestions to decide a case in a particular way. This experimental vignette provides an indication of the extent to which judges delegate part of their responsibility to judicial assistants that is not based on self-reporting.
In the next section, we give an account of the research that has already been conducted on judicial assistants' role in adjudication and of the questions that remain unanswered so far. After explaining the methodology in section three, we present our findings in section four. In the final section we discuss the implications and limitations of our study and draw conclusions.

Previous research on the role and influence of judicial assistants in judicial decision making
The tasks and responsibilities of judicial assistants vary between jurisdictions (for example, differences can be noted regarding common law versus civil law jurisdictions), court levels and legal domains. 15 Still, based on the existing research, four universal duties can be distinguished that judicial assistants share in all these contexts: screening, sound boarding, preparing bench memos and drafting judgments or opinions. In the following, we describe the current state of knowledge regarding the influence that assistants may wield via these four duties as well as their overall influence.

a. Screening
The first set of tasks through which judicial assistants may wield influence is found in a screening duty, for instance regarding the selection of cases to be reviewed by a court. Many higher and/or constitutional courts have systems of discretionary review, granting only small percentages of requests for review. about 2-3%. 16 With the enormous load of petitions for review, memos (or reports) prepared by assistants are indispensable for handling these requests. 17 In a survey (N = 133) conducted by Ward and Weiden, 18 former US Supreme Court law clerks self-reported this stage as the one in which they are most influential. The process of selecting cases for judicial review at the US Supreme Court (also known as 'examining to grant or reject writs of certiorari') has been studied extensively. 19 Stras 20 examined the agreement of Supreme Court Justices with the proposed way of handling petitions for certiorari (requests for review) by the law clerks in four different years. He found the level of agreement to be 98 to 99% in all decisions to grant or deny a petition for certiorari, depending on the year, and the level of agreement in granted cases was 66-74%, depending on the year. 21 Building on the work of Stras, 22 Black and Boyd 23 also found influence of the certiorari memo on judges' voting when controlling for the judges' likely action in the absence of a clerk's advice. Additionally, they found that the greater the distance was between the political attitudes of a judge and the clerk who wrote the certiorari memo, the smaller the likelihood was of the judge following the recommendations of the clerk.
Many lower courts (and also some highest courts, such as the Dutch Supreme Court) do not have a system of discretionary review. Hence, judicial assistants in these courts do not participate in selecting cases for review. However, when a case is filed at these courts, various other screening decisions must be made, such as whether a case is referred to a specialised court and whether it will be handled by a single judge, a jury or a judicial panel. Through these screening decisions, judicial assistants can indirectly influence the manner in which cases are handled by the court, e.g. by determining how much time the court takes to hear a case or by deciding how many judges are involved in the decision making. 24 Thus, several studies indicate that screening influences judicial decision making at the US Supreme Court. There are indications that screening also occurs at other courts, but sound empirical evidence for influence on the outcome of cases in those settings is lacking. 16 Table 9. It makes sense to particularly pay attention to the granted cases as only a few percentages of all petition for certiorari are granted and, thus, those are the cases in which most dissimilarity is expected. The negative association between the political attitudes of a judge and the clerk who wrote the certiorari memo and the likelihood of the judge following the recommendations of the clerk might be an indication of assistants' influence. However, it is also possible that the causal order is reversed; clerks may simply seek affirmation by preparing proposals they expect the judges to agree with. 22 Ibid. 23

b. Sound boarding
A second way in which judicial assistants can wield influence is by acting as a sounding board. 25 In many courts it is common for judges to discuss the merits of a case with judicial assistants. Ward and Weiden as well as Kromphardt note that judges can particularly benefit from judicial assistants' familiarity with the latest academic insights on legal discussions (this is particularly true for law clerks at the US Supreme Court). 26 These authors also emphasise the importance of understanding the role of judicial assistants as advisors to judges, presenting some qualitative evidence that judges employ clerks for their advisory role. In the study by Ward and Weiden, 27 former US law clerks do not solely report that they would discuss cases with justices and tell them what they believe should be decided but also state that they would try to convince the justices of their positions. In the Rehnquist court (1986)(1987)(1988)(1989)(1990)(1991)(1992)(1993)(1994)(1995)(1996)(1997)(1998)(1999)(2000)(2001)(2002), more than half of the respondents (strongly) agree with the statement 'I have often attempted to convince the Justice of my position on a case or issue'. Judges may discuss cases with their assistants informally, or the deliberation can be more institutionalised. The latter occurs when assistants are present and permitted to participate in the discussion in the deliberation room. There is anecdotal evidence of judicial assistants trying to persuade judges and of judges changing their views after discussing a case with an assistant. 28 However, no systematic empirical evidence exists as to the extent to which judicial assistants actually influence judges' decisions through discussions. This is probably due to the complexity of isolating the influence of judicial assistants' input in legal discussions. Deliberation sessions are also difficult to study as these discussions are usually confidential.

c. Preparing bench memos
A third duty through which judicial assistants can wield influence is by assisting judges in their preparations for hearing a case. Various US states' high court judges note that conducting research on the case records and on the law is the most important task of judicial assistants. 29 Documents in which judicial assistants report their findings of this research are in the US commonly referred to as 'bench memoranda'. Various judges qualify these memos as ' a roadmap to the case'. 30 Some memos mainly summarise the legal facts of a case, whereas others include suggestions for questions to ask during a hearing, advice on how to handle the procedural aspects of a case and/or advice on how to decide a case. 31 Research on cognitive heuristics and biases in (judicial) decision making 32 has revealed that judges are influenced by information that is presented to them prior to a hearing even if that 25  information is completely irrelevant to the outcome of a case. 33 Holvast argues that a bench memo can also function as an anchor (a, sometimes unconscious reference point for the judgment) in the decision making of the judge. 34 The extent to which this occurs depends on how the bench memo is used. An anchoring effect is particularly expected when the memo is used as a roadmap for understanding the case. Cohen, 35 who studied the interaction between US appellate judges and their clerks, makes a distinction between judges who are front-loaders and those that are back-loaders. Front-loaders are highly involved in the preparations of the bench memo. Conversely, back-loaders characteristically get truly involved in cases only after the oral argument and thus rely predominantly on the bench memo for the preparation of the oral argument. In a study on Dutch district judges, Holvast 36 finds that some judges canat times -be classified as front-loaders. Judges mentioned that they occasionally consciously decided to first read the court files before reading the memo to avoid becoming biased. Yet, most respondents fell into the category of back-loaders. They started their preparation with reading the memo, using it as a roadmap for reading the case files.
Studying the influence of judicial assistants via bench memos is challenging as these memos are generally not made public. An exception are the memos written by law clerks of Justice Blackmun, who, after his retirement from the bench, opened his archive to the public. In a small sample of 40 of Blackmun's cases, Johnson, Stras and Black 37 examined whether the questions that law clerks proposed to ask during the oral argument in their bench memos were asked by Justice Blackmun. Of all 230 proposed questions, the justice had asked 94 (40.9%). He only occasionally added additional questions of his own. This small sample suggests that judges (or at least this judge) use the bench memo to prepare for a hearing and that their behaviour during a hearing might be influenced by it.
Thus, judges self-report that they use bench memos as 'roadmaps to cases'. The research on bench memo usage by Justice Blackmun also suggests that memos influence judges' behaviour in court. Yet, a systematic test of the prevailing idea that bench memos influence judges' decisions is lacking.

d. Drafting judgments/opinions
Probably the most controversial duty of judicial assistants is their role as drafters of judicial decisions and opinions. 38 It has become common practice for judges in different jurisdictions (e.g. US, Germany, the Netherlands, Switzerland) to delegate to their judicial assistants the duty of writing a first draft. 39  US federal appellate court judges delegate the drafting of judicial opinions. In Holvast's study on Dutch district courts, all of the 62 studied judgments had been drafted by an assistant. 41 However, this delegation is not common practice in all courts. In some jurisdictions it appears to be considered improper to do so. 42 Rosenthal and Yoon 43 as well as Carlson, Livermore and Rockmore 44 studied the writing style of large numbers of judicial opinions by US Supreme Court Justices spanning the entire history of the Supreme Court. They found that the writing style of judicial opinions varied more as the number of law clerks that justices had at their disposal grew. This variation in writing style indicates that opinions have multiple authors, thereby suggesting that justices delegate at least parts of the opinion writing to their clerks. Similar results were found by Bodwin, Rosenthal and Yoon 45 in a study on the Canadian Supreme Court. However, the fact that judicial assistants prepare a first draft of a judgment does not automatically imply that they also influence the outcome of cases.
The extent to which judicial assistants wield influence through the drafting of judgments also depends on the interaction between judges and their assistants during the writing process. Do judges, for instance, provide detailed instructions for writing the draft? And/or do they drastically revise the first draft of an assistant, turning it truly into their own judgment? Holvast 46 showed that Dutch district court judges differ greatly in how much instruction they provide to judicial assistants and how much they revise their drafts. Ward and Weiden 47 distinguish the following three types of opinion writing at the US Supreme Court: 1) retention -the justice writes the opinion, and the clerk provides citations, footnotes and editing; 2) collaboration -the justice and clerk work in a tandem towards the constitution of a single opinion; 3) delegation -the justice delegates the task of writing the opinion to the clerk and revises the draft. In their study from 2006, Ward and Weiden find that delegation has surpassed retention in being the most common way of drafting opinions at the US Supreme Court. This suggests that by taking the lead in drafting opinions the opportunities for assistants to influence the opinions have increased.
Hence, in various jurisdictions, assistants seem to have important responsibilities in drafting judgments or opinions. It is expected that drafting judgments influences the content thereof. Research reveals that the writing style does seem to be affected by the involvement of assistants in drafting judgments or opinions. However, whether the drafting also affects the outcome of court cases is currently unstudied.

e. The overall influence of judicial assistants
Several surveys among judges and/or judicial assistants focus on the perceived overall influence of judicial assistants on the outcome of court cases. Swanson and Wasby 48 conducted a survey among US state high court judges (N = 81), in which most judges reported that law clerks influence adjudication: 28.4% of judges consider law clerks' influence to There is also some indirect evidence that law clerks influence the outcome of judicial decisions that is not based on judges' perceptions. Drawing on a survey of the political ideology of 532 former Supreme Court law clerks, Peppers and Zorn 51 studied the influence of clerks on voting on the merits of a case by taking into account the political attitudes of both the law clerks and the judges. They found an independent effect of the political attitude of the law clerk on the votes of the justice, which is about one-third to one-half of the effect of the judges' political attitudes. 52 Also focusing on the influence of the political ideology of US Supreme Court law clerks, Bonica, Chilton, Goldin, Rozema and Sen 53 provide evidence for a modest effect of clerks' ideology on justices voting on cases. They found that 'replacing a judicial chambers composed of the justice's annual set of most liberal clerks with a judicial chambers composed of the justice's annual set of the conservative clerks would result in a 4% increase in the number of conservative votes by the justice during the term'. 54 This influence, however, varies per case and is larger when it concerns high profile and legally significant cases.
This means that there is some evidence that assistants influence judicial decisions. However, this evidence is either based on judges' self-reporting or derived indirectly from the impact of the political attitude of the assistant. What is missing, is a direct assessment of assistants' influence that is independent from the self-reporting of judges and does not derive from the assistant's political attitude.
Although sound evidence is scarce, the existing research provides various indications that judicial assistants may wield influence on adjudication via a multitude of duties. Based on these findings we test the role and influence of judicial assistants in adjudication in three consecutive steps. In the first step, we distinguish the involvement that judges report assistants to have via performing their universal duties. However, the reported involvement in these different duties does not reveal the influence of assistants on the outcome of court cases. This is why, in step 2, we measure the extent to which judges attribute influence to assistants in a concrete court case. However, since the measurement in step 2 is based on self-reported influence, the results may be distorted by judges' ideas regarding the level of influence they deem appropriate. Furthermore, it does not capture influences of which judges are not self-aware. For this reason, in step 3, we measure the influence by means of an experiment in which we test the influence of bench memos on the judges' decision. 49 Ibid., p. 32. 6.2% of judges did not answer the question. 50

Research context
In order to contextualize our findings, we briefly discuss the role of judicial assistants in the administrative law divisions of Dutch district courts. In Dutch district courts, judicial assistants are not assigned to specific judges. Rather, judicial assistance work in a pool for multiple judges. Employment as a judicial assistant is not a position for a set number of years, but can be a lifelong career. Commonly, one judicial assistant is assigned to assist a judge or a panel of judges in a particular case from the beginning of the process up to the writing of the judgment. Administrative law assistants perform all four duties through which assistants may influence judges' decisions. First, a special group of judicial assistants screen cases; they take preliminary decisions regarding the time scheduled to hear cases and whether a case is allocated to a single judge or a panel. Second, the assistant assigned to a case normally prepares a bench memo for the judge(s). Third, the assistant is also present during deliberationsessions and regularly functions as a sounding board in the discussion in the deliberation room. Fourth, it is custom that assistants prepare the first draft of a judgment, which can then be altered by the judge(s). 55

Research population
Our targeted research population consisted of administrative law judges working at district courts in the Netherlands. We targeted judges (and not judicial assistants) as we can only learn whether judges are influenced by focusing on changes in their decision making. We limited our study to administrative law for three reasons. First, administrative law judges are approached less often to participate in research than, for example, criminal law judges are. Second, we had the best contacts within this legal domain. Third, Holvast 56 has shown that administrative law assistants perform all four duties (screening, sound boarding, preparing bench memos and drafting judgments) through which they may influence judges' decisions. For these reasons, Dutch administrative courts provide an appropriate setting to study judicial assistants' influence on judicial decision making.

Data
Data were collected by way of an experimental survey. Approval for our research was granted first by the Netherlands Council of the Judiciary (Raad voor de rechtspraak) and subsequently by the National Board of Administrative Law Judges (Landelijk overleg vakinhoud bestuurs recht). Data were collected in two rounds. The first round took place at the end of 2018. Team leaders of all Dutch administrative law judges (272 Full Time Equivalent (FTE)) were asked to distribute an email among their team members containing an invitation to complete the online questionnaire. 57 After two weeks, a second email was distributed to remind judges to complete the questionnaire. The first round yielded 31 questionnaires that were filled in completely. In order to boost the response, we were granted permission by the National Board of Administrative Law Judges to conduct a second round of data collection. During this second round, we personally visited the five administrative law divisions of district courts in the Netherlands -Rotterdam, Amsterdam, Haarlem, Utrecht and Arnhem -to briefly present our research to the judges at team meetings, to answer possible questions about the questionnaire and to invite them to complete our questionnaire. 58 Additionally, the team leader of administrative law judges in Zwolle offered to personally bring the questionnaire to the attention of the judges in his team and collect and post the completed questionnaires to us. This second round of data collection increased the response to a total of 90, of which 67 respondents also completed the experimental vignette that was included in the questionnaire. Ten questionnaires were deleted because they were only partly completed. At the start of our data collection, in total 272 FTE were working as an administrative law judge, while on average judges work 0.925 FTE. 59 This means the response rate was approximately 27% (80/ (272 x 1.081)). This rate is lower than the average response rate of 35.7 percent that Baruch and Holtom found for studies utilizing data from organizations, but it is within the range of one standard deviation of 18.8 that they established in their study. 60 In comparison to the total population of judges in the Dutch courts, the average age of the sample is similar (50.5 (population) vs 50.1 (sample) years), while males are overrepresented in the sample (36.9% (population) vs 51.4% (sample)). 61 We are unable to determine whether the (lack of) representativeness in our sample has had an impact on the test results of our hypotheses.

Measurement
Judicial assistants' role in adjudication was measured in three steps. First, we measured the involvement that judges attribute to judicial assistants in performing different duties that are related to the decision making process. In the survey, judges were asked about nine different aspects of their duties. Given that sound boarding is the most varied duty, four of the questions relate to that duty. Preparing bench memos and drafting judgments are more straightforward duties. Hence, only two questions relate to the duty of preparing bench memos and three to the duty of drafting judgments. No questions were included to measure the screening duty as most judges are not involved in the screening of cases and, therefore, cannot report on judicial assistants' influence in this part of the decision making process. Each question consisted of a statement on the perceived judicial assistant's involvement. Judges had to indicate the percentage of their cases in which that type of involvement actually occurred. The percentages were divided into the following five categories: 1) 0-20%, 2) 21-40%, 3) 41-60%, 4) 61-80% and 5) 81-100%. Second, the overall influence that judges attribute to assistants in the outcome of a concrete court case was measured. Judges were asked to recall a recent court case in which they had collaborated with a judicial assistant and to report the extent to which the assistant had contributed to the outcome of the court case. The answer categories were the following: 1) 'none', 2) 'very inconsiderable', 3) 'inconsiderable', 4) ' considerable' and 5) 'very considerable'. The court cases vary in content and pertain, for example, to family reunion, real estate appraisal, denial of social welfare payments and the licensing of electric vehicle charging posts.
Third, the extent to which judges' decisions are influenced by judicial assistants' bench memos was measured. Judges were asked to read one of three versions of a fictional but realistic court case and bench memo and to determine an outcome for the case. The bench memos were composed in collaboration with an experienced administrative law judge and 58 The judges could either fill-in the questionnaire online or on paper. 59 According to the judiciaries annual report over 2018. judicial assistant. With their input, we created a vignette that offered a certain amount of discretion on how to judge the case.
The court case for this experiment concerned a preliminary injunction case regarding an eviction order of a mother with three children by a mayor because drugs were found in the woman's home. A preliminary injunction case was selected as, in those cases, judges have to reach a decision that is important for the civil litigant in a short amount of time and based on relatively little information. Such cases are well suited for a survey which aims to include a short but realistic case. The case description was identical in all versions, but the input of the judicial assistant via the bench memo was manipulated. In one version, the control version, no bench memo was included. In the two other versions the judicial assistant either provided suggestions to uphold the mayor's decision to evict the family from their home or to suspend the mayor's decision by imposing a preliminary injunction. This was established by emphasising different elements of the case in the two memos. These elements strengthened the argument to either uphold or to suspend the mayor's decision. We decided to contrast the versions on these more subtle aspects rather than have assistants give concrete recommendations to the judge about what to decide in the memo because the latter is not common in practice. Furthermore, judges were aware of the subject of the survey. We wanted to prevent them from adjusting their behaviour by, for instance, altering their judgment in order to avoid the appearance of being influenced by the assistant. Therefore, a randomiser in the online survey program Qualtrics was used to assign respondents to one of the three versions of the vignette. 62 In total, 30.4% of the judges responded to the version of the vignette that did not contain a bench memo, 38.0% responded to the version wherein the judicial assistant suggested to suspend the municipality's decision to evict the family and 31.6% responded to the version wherein the judicial assistant suggested to uphold the municipality's decision to evict the family. Respondents were asked: 'How likely is it that you would suspend the decision of the mayor to evict the mother and her children from their home?' The answer categories ranged from 1 to 10.
Even though respondents were randomly assigned to one of the three versions, we included several control variables in the analysis due to the modest size of the sample. Without these controls, the outcome of the judicial decisions might have been affected by differences between the categories of judges who responded to the three versions. In a preliminary analysis, we controlled for a substantial number of factors that have been linked to the influence of judicial assistants on judicial decision making in previous research. As we describe in a separate article, 63 these factors pertain either to judges' perceived risks of involving judicial assistants in judicial decision making or to contextual factors. In the final analysis, we included only the control variables that correlated significantly with judges' decisions about the vignette and excluded those that did not correlate significantly with their decisions. 64 In the following, we report how we measured the control variables which we included in the final analysis.
The first control variable pertains to the perceived risks and benefits accompanied by involving judicial assistants in the decision making process. This variable was included based on the expectation that assistants have more influence on judges' decisions when judges are more convinced that the benefits of involving assistants in the decision making process outweigh the risks of doing so. Respondents were asked, for instance, whether the contribution of 62 The assignment process was double blind: neither the respondents nor the researchers knew which version was assigned to whom. 63 See Mascini & Holvast supra note 60. 64 The control variables that were excluded from the analysis were the perceived trustworthiness of the judicial assistant, judges' managerial role orientation, gender and age. The results of the analysis that also includes the control variables that did not correlate significantly with judges' decisions did not differ significantly from the results we present here and are available upon request.
judicial assistants to judicial decision making 'Harms the reputation of the judiciary', 'Enables the judge to focus on his/her core tasks' or 'Dilutes the ultimate responsibility of the judge for decision making'. This risk-benefit perception was measured with nine Likert items, with answer categories ranging from ' disagree strongly' to ' agree strongly'. Together they constitute a reliable scale (Cronbach's α = 0.70). The second control variable concerns a rule-of-law role orientation. This variable captures a role orientation by judges in which they consider judges to be the personification of the judiciary -the adjudicative branch of government. Judges who adhere to this role orientation emphasise that judges are entrusted to adjudicate due to the unique safeguards related to their function which assure their independence, impartiality, integrity and competence. This control variable was included due to our expectation that the more judges have internalised a rule-of-law role orientation, the less influence they will allow judicial assistants in the decision making process given that the latter do not possess the same level of training and safeguards as judges do. The rule-of-law role orientation is measured with eight Likert items with the following answer categories: 1) ' disagree strongly', 2) ' disagree', 3) 'neither disagree nor agree', 4) ' agree', 5) ' agree strongly' and 6) ' don't know'. Examples of the items measuring the rule-of-law role orientation are 'The use of algorithms in judicial decision making jeopardises the quality of the judiciary', 'Politicians interfere too much with individual court cases' and 'Judgment by lay people is undesirable because it undermines the quality of the judiciary'. The reliability of the scale for the rule-of-law role orientation is sufficient (Cronbach's α = 0.69). 65 The last control variable concerns the courts in which respondents were employed. This characteristic may affect the influence of judicial assistants because of differences in interaction patterns between assistants and judges in the various courts. Employment in the different courts was categorised as follows: 1) Amsterdam (13.2%), 2) Gelderland/Overijssel (25.0%), 3) Limburg/East-Brabant/Zeeland-West-Brabant (6.6%), 4) Middle-Netherlands (14.5%), 5) North-Holland/North Netherlands (21.1%) and 6) Rotterdam (19.7%). 66

Analysis
The analyses of the first two steps in the measurement of judicial assistants' influence concern simple descriptive statistics, such as frequencies, means and standard deviations. For the last step of measuring judicial assistants' influence, a covariance analysis was used (ancova). This technique was chosen because it enables the comparison of the means of categorical variables, when controlling for the influence of numerical variables. In this analysis, the means of the decisions made by the judges who were assigned to one of the three versions of the vignette were compared when controlling for the risk-benefit perception, the rule-of-law role orientation and the courts in which respondents were employed. A post hoc test was performed to test whether the differences between the means of the three categories were significant. For this test a Sidak correction was used, which provides an estimate of the means adjusted for the effects of the covariates.

Results
In this section, we report the results of the measurement of influence in the three consecutive steps. For step 1, Table 1 shows the reported involvement in adjudication via the three different duties (sound boarding, preparing bench memos and drafting judgments). 67 The data collected via this first step demonstrate that judges report that judicial assistants regularly participate in adjudication via all of the three key duties. First, assistants are particularly actively involved as a sounding board, with assistants being heavily involved in discussing the content of cases with judges (mean 4.7, with scores ranging from 1 '0-20%' to 5 '81-100%') as well as being involved in the deliberation sessions (mean 4.78). In the deliberation sessions, assistants are regularly the first person to be invited to share their views (mean 2.85). Nevertheless, it is exceptional that a decision made during the deliberation session is thereafter altered on the initiative of an assistant (mean 1.28). Thus, assistants act as sounding boards in judicial decision making, but the decision making is not entirely taken over by the assistant. Second, regarding bench memos, it is common that assistants provide judges with their advice regarding the outcome of the case (mean 4.37), which normally occurs via the bench memo. Yet, judges will hardly ever prepare for the hearing by exclusively reading this memo (mean 1.16). Last, assistants are involved in drafting judgments on the facts and procedural aspects of a case (it occurs regularly that judges adopt these aspects without changing them, mean 2.60). However, judges will not completely rely on assistants regarding the judgment drafting (mean 1.39), nor will they delegate the drafting without providing instructions (mean 1.35).
Thus, judicial assistants are involved in all these key duties. However, no duties are completely delegated to assistants without any oversight by the judges. On the contrary, the Table 1: Attributed influence of assistants by judges via three different duties (N = 80), scale 1-5.

Sound boarding
The judge and judicial assistant discuss the content of a case.
2 5 4.7 0.68 The judicial assistant participates in the discussion during the deliberation session.
The judicial assistant is the first person to be invited to share his/her views on a case in the deliberation session.
1 5 2.9 1.22 The decision made in the deliberation session is afterwards (partly) changed on the initiative of the judicial assistant.

Bench memo
The judge prepares for a hearing by exclusively reading the bench memo prepared by the judicial assistant.

Judgment drafting
The judge adopts the facts and procedural information that the judicial assistant provides in the draft without modifications. decision making process appears to be very much a joint effort of the judge and the assistant.
Since the decision making occurs in collaboration, it is difficult to isolate the contribution of the assistant and, thereby, to extricate how much influence on outcomes they actually have via these duties. The data collected via the second step provide knowledge regarding the perceived influence of judicial assistants on the outcome of adjudication in concrete cases. Table 2 shows the results of the question in which we asked judges about the influence that an assistant had in a recent court case.
The table demonstrates that, except for one, all judges report that the assistant had some level of influence. Most judges (61.3%) report that this influence was '(very) considerable'. A substantial portion (37.5%) also report that assistants had '(very) inconsiderable' influence. Thus, the level of reported influence varies greatly. This makes sense as judges reported about dissimilar cases that were adjudicated under varying circumstances. Still, the median and average reported influence demonstrate that this perceived influence on concrete cases is considerable.
For step 3, Table 3 shows the extent to which assistants influence the outcome of judicial decisions that is independent of the perception of judges. It shows that judges indeed decide differently depending on which version of the vignette they read. Judges who read the vignette with the memo in which the judicial assistant suggested to suspend the eviction of the family by the mayor were more likely to decide accordingly than those judges who were not presented with a memo in the vignette. At the same time, judges who read the version without a memo were more inclined to suspend the mayor's decision than judges who read  the vignette with the memo wherein the judicial assistant suggested to uphold the decision to evict (subsequent averages of pro 7.0, neutral 5.3 and contra 5.0; p < 0.01). The content of the vignette explains 16.7% of the total variance in judges' decisions. Table 4 reports the significant test of the differences between the means of the decisions between the three versions of the vignette adjusted for the effect of the covariates. The test shows that the mean decisions of the vignette containing the bench memo wherein the assistant recommends to suspend the mayor's decision to evict the family differ significantly from the decision means of each of the two other versions of the vignette (subsequently, p < 0.05 and p < 0.01). However, the mean of the decisions of the judges who responded to the vignette without a bench memo does not differ significantly from the mean decisions of the judges who responded to the vignette with the bench memo wherein the assistant recommends upholding the mayor's decision to evict.
To sum up, even though the difference between the neutral and contra eviction vignette is not significant, the findings are that: first, the pro eviction vignette differs significantly from the other two vignettes, second, the means of the three vignettes are ranked as expected and, third, the type of vignette is significantly associated with judges' decision (it explains 16.7% of the total variance in judges' decisions). All of these findings indicate that the recommendations judicial assistants make in bench memos influence the outcome of judges' decisions.

Discussion and conclusion
This article contributes to the evidence that law clerks or judicial assistants are actively involved in adjudication and that this involvement influences the outcomes of judges' decisions. In this article, we focused on the judicial assistants' role in adjudication in Dutch district courts. We measured the involvement and influence of these assistants in three steps. The results from the measurements in each step also have their limitations. In the following, we first discuss the implications and limitations of the data collected via each of the three steps. Thereafter, we reflect on the overall significance of the research. Lastly, we discuss questions for further research.
Step 1. Decision making as a joint effort First, we find that judicial assistants can be involved in the judicial decision making process by performing three key duties of sound boarding, writing bench memos and drafting judg ments. Of the three duties, we discovered that assistants are most actively involved in sound boarding. Furthermore, for all three duties we found that assistants do not perform these duties independently from the judge but in collaboration. Adjudication can, in that sense, be considered a truly joint effort. This is consistent with the findings of Holvast's qualitative study on judicial assistants in Dutch district courts. 68 The collaborative characteristic of the interaction between judges and law clerks is also stressed in research on the US Supreme Court. 69 A fourth duty by which assistants can wield influence, screening, was deliberately not included in our measurement. Our respondents consist of all administrative law judges at district courts. Most of these judges are not aware of the screening decisions made by assistants and cannot, therefore, report on the involvement of assistants therein. Even though other studies have reported screening to influence the outcome of judicial decisions, 70 this is less likely to be the case in the setting of our study. After all, unlike, for instance, at the US Supreme Court, Dutch district courts do not have a system of discretionary review.
Bonica et al. 71 distinguish two pathways for wielding influence: delegation and persuasion. They find that the influence of law clerks at the US Supreme Court is mainly wielded via persuasion. The results of this first step also suggest that persuasion is the most likely pathway for wielding influence at Dutch district courts. The judges in our study report high levels of involvement of assistants by means of discussing cases and giving advice, providing ample room for influence via persuasion. And, while some delegation of duties does occur (for example, the drafting of judgments), the judges emphasise their monitoring role in this regard: they will not completely delegate their decision making responsibilities to judicial assistants.
Whereas step 1 shows that assistants are actively involved in adjudication via the different duties they perform, it does not determine that this involvement influences the outcomes of concrete court cases.
Step 2. Influence in a recent court case In step 2, judges report that judicial assistants have quite considerable influence on concrete cases. Only one judge reported no influence at all. Most judges (61.3%) reported '(very) considerable' influence of the assistant. Our results point in a similar direction as a study by Swanson and Wasby among US states' high court judges. 72 The majority of judges in their study reported law clerks having important/substantial (28%) or some/moderate (49%) influence on adjudication. 73 Similar to the study by Swanson and Wasby, 74 our data demonstrate a large variation in answers; a substantial group of judges in our study also reported '(very) inconsiderable' levels of influence (37.5%). We assume that this disparity is related to the enormous variation in the cases on which the judges report.
Even though step 2 indicates that assistants' overall influence on concrete cases is considerable, the fact that this finding is based on judges' self-reporting may imply that it is biased to some extent. Normative views of judges regarding the appropriate role of assistants may have affected this finding as well as other factors they are unaware of.
Step 3. Influence of the bench memo In the third step, we measured influence independently from the perceptions of judges by means of an experiment. We tested the influence of a bench memo prepared by a judicial assistant on the judgment that judges make in a fictitious but realistic court case. The results display a significant effect. We find that the content of the memo explains 16.7% of the total variance in judges' decisions. The experiment conducted in this third step shows that the involvement of assistants in writing bench memos influences judicial decision making. Though, the experiment does not provide insight into the question as to whether the other duties that assistants perform -screening, sound boarding and drafting judgments/opinions -also influence adjudication.

Significance of the research
In this article, we have established that judicial assistants are actively involved in adjudication and have a self-reported and experimentally established influence on the outcome of court cases. Each of these three indicators makes a unique contribution to the assessment of the role and influence of assistants in adjudication, but each of these indicators also has its specific limitations. This means that although the different measurements are complementary, they do not provide a complete picture. This is not only because of the limitations of the different measurements but also because assistants can have influence in many different ways and via different duties. Still, our measurement of the role of assistants in adjudication is more versatile and less dependent on the self-reporting of judges than the measurements conducted in the existing literature.
Nonetheless, this conclusion warrants some caution due to the limitations in the measurements of each of our three steps, as well as the response rate of about 27%. We know that the average age of our sample is comparable to that of the general population of Dutch judges and that males are overrepresented. We do not know if our sample is representative with regards to other factors. We are also unable to determine whether the (lack of) representativeness in our sample has had an impact on the results of our research.
Although the setting in which judges and judicial assistants collaborate differs considerably per jurisdiction and level of court, our literature study reveals that screening, sound boarding, preparing bench memos and drafting judgments are duties which assistants perform in diverse settings. Therefore, we expect our findings regarding to the involvement of assistants in adjudication, and conceivably also their influence on the outcome of judicial decisions, to be generalisable to other settings that are also characterised by intense judge-assistant working relationships, such as high-level courts in the US.

Suggestions for further research
A first logical avenue for follow-up research concerns the normative question of how to value the role of assistants in adjudication. The extent to which this role could be problematised may be conditional, for instance, on the safeguards surrounding the employment of assistants, the part of the decision making process in which assistants wield influence and the transparency of their involvement. This normative question regarding judicial assistants' influence deserves a more systematic analysis.
Second, it would be valuable to also conduct additional experimental research to test the influence on judicial decision making that is not based on self-reports. More particularly, it would be worth conducting experiments that assess the extent to which assistant wield influence via duties other than preparing bench memos. The influence wielded by memos in the process of reviewing requests for review (at the US Supreme Court petitions for certiorari) could, for instance, be a worthy focus for an experiment. Additionally, it would be valuable