Jan Kuylenstierna Abstract The author and his colleagues have for several years used chess as tool for research intended to be relevant for the improvement of military command and control. The paper summarizes the main results from this research and also discusses the external validity of the findings. The major finding is that the ability to benefit from an information quality advantage - in terms of more current information or longer range of vision of the board - decreases as the information quality for both sides is reduced, even if the advantage is left unaffected. If this result from experiments with chess should be valid also for military command and control it would contradict a current hypothesis in that area which implies that military information systems do not have to deliver perfect information, it is sufficient if they can deliver more and/or better information than those of the enemy. This, in turn, would mean that measures other than investments in advanced information technology may be needed in order to establish superior command and control under conditions of imperfect information. It is finally concluded that findings from experiments with chess can have external validity if there is a theory connecting them to the target situation, and that there are no reasons to believe a priori that findings from abstract laboratory research have less external validity than findings from field studies in natural settings. Introduction In spite of its innocent appearance and use as a parlour game, chess is a several thousand year old wargame. It involves a struggle of will and it also contains what has been termed the essentials of fighting - to strike, to move and to protect. It is assumed to have evolved from an ancient Indian game known as Chaturanga and it has for centuries been used by the military as a tool for teaching the basics of military thinking [9]. Several military variants of chess have also been developed, known collectively as "military chess" or "war chess". The most renowned of these variants was called "the King's" game or Koenigspiel. [9] Chess also has a long history as a research tool. For more than half a century it has been used for the study of complex problem solving in general and expertise in particular. The reason for this is the great skill differences between novices and Grandmasters, its well-established rating scale (developed by the Hungarian mathematician Árpád Élõ) and its method of recording games. These features make chess an ideal task environment for the study of skilled performance and it has therefore at times been referred to as the Drosophila, or fruit fly, for cognitive psychology (see [5] and [7]). Another probable reason for its popularity as a research tool is that it is convenient to use, many people can play chess and experiments can be carried out at a low cost. The author and his colleagues at the Swedish National Defence College have merged these two traditions of use into one. Chess is used to carry out research intended to be relevant for the improvement of military command and control. The aim of this paper is twofold: First to summarize the main results from this research and second to discuss the external validity of the research findings. The research A current recipe for establishing superior command and control is to invest in advanced information systems with the goal of establishing an information advantage vis-a-vis an adversary. The information advantage is assumed to result in superior command and control which in turn leads to superiority in the battle space (information superiority) [1]. Ideally the information system should always provide for perfect information, but there seems to be an almost total agreement within the research area that such a goal is unrealistic and that the "fog" of war can be expected to remain and lead to an unclear view of the battle space also in the future [1]. However, according to a widespread view this need not be very serious for the possibilities to create an effective information advantage [1]. This view states that it does not matter if a commander in battle has a good or bad information position (to what extent the information can meet his needs), what counts is how good his information position is in comparison with the adversary's information position. That is, all he needs is a "relative information advantage". Therefore military information systems need not deliver perfect information, to result in superior command and control it is sufficient that they can deliver more and better information than those of the enemy. An information system can excel in primarily five crucial dimensions; completeness, correctness, currency, accuracy and consistency [1]. An information advantage can therefore be established by exceeding the adversaries system along one or several of these dimensions. The author and his colleagues have tested the hypotheses of relative information advantage with the aid of chess. In the first experiment ([10], [13]) an information advantage was created by providing one of the players in a chess game with more current information than that of the opponent.
In the experiment, pairs of participants played chess under one of two conditions. In the first condition (0/1) favoured player in each pair had real time information about the situation on the board, whereas the disadvantaged player had to make his or her moves without knowledge about the favoured player's latest move. In the second condition (2/3) the corresponding delays of information about the opponent's latest moves were two moves and three moves for the favoured and disfavoured player respectively. The delays were enabled by the use of two chess boards, an opaque screen between them and a game leader who moved the pieces, kept track of the time and recorded the moves. This set-up is illustrated in Figure 1 (more recent experiments have had a corresponding arrangement with the aid of two computers). In both conditions the favoured player in each pair had a relative information advantage corresponding to one move less of delay, but in the second condition the information for both players was more delayed than for the players in the first condition. Each player had 15 minutes to spend and if a player reached this time limit he or she lost the game. For each game the favoured player was scored one point for winning, half a point for a draw and zero points for losing. The results showed that the players with the information advantage in the first condition won an overwhelming majority of their games, whereas the value of the advantage disappeared completely in the second condition. The difference between the two conditions was highly significant. A variant on the first experiment with similar information conditions but where the participants could chose between making zero, one, two ore three moves for each turn, lead to the same result [15]. An experiment has also been made where in one condition (4/2) the favoured player in each pair could see four squares around each piece whereas his or her opponent only could see two squares. In the other condition (2/1) the corresponding figure where two versus one square. The information advantage in terms of the ratio between the opposing players' ranges of vision was the same in both conditions, but the ranges of vision were shorter in the second condition for both players. The results showed that the outcome for the players with the advantage was significantly worse in the second condition than in the first one [14]. This finding, concerning the value of superior range of vision, has been replicated in an experiment employing a computerized wargame called DKE [17]. To summarize, none of the experiments carried out so far lend any support to the hypothesis of relative information advantage. Instead the value of having an information advantage decreases as the quality of information decreases. A natural question to ask is if some other type of command and control related advantage1 can work better under conditions of imperfect information. Current research is investigating if an advantage in terms of higher ranking can be effective under such conditions. However, if the findings should generalize to military command and control it would mean that investments in advanced information technology may prove insufficient to establish superior command and control under conditions of less than perfect information and that other measures have to be taken as well. But can findings from laboratory experiments with chess really generalize to military command and control? Would it not be better to instead use case studies in natural settings such as studies of large scale field exercises, wars, or operations-other-than-war? The next section will discuss answers these two questions? The external validity of the research findings A wide spread view seems to be that results from abstract laboratory experiments are unlikely to generalize to the "real world" whereas results from case studies in natural settings can be indiscriminately trusted to have validity for the external target situation in question. This is a misconception. All thinking involves abstraction, and so also thinking about war. This is true for case studies as well; you do not describe everything, you abstract out what you are interested in. Chess is a concrete realization of an abstraction of thoughts about war. The question is if this abstraction captures what is essential; a question of theory, which you can ask also about a case study. Without a theory it is impossible to decide what experiments/case studies are relevant for the target situation. The theory must tell the investigator which variables to look for and how to operationalize them [4]. To clarify; the external validity of field studies is just as dependent on the veracity of these underlying theoretical assumptions as is the external validity of laboratory research. Not realizing that there are limits also to the validity of case studies can have grave consequences; just think of what happened when the French generalized their conclusions from the First World War to the Second World War, as revealed in May 1940. There is furthermore no evidence that results from case studies more often lead to valid conclusions than findings from laboratory studies. Attempts to empirically compare case studies and laboratory studies with respect to generality give no ground to give precedence for one over the other [2]. The choice of a field study rather than a laboratory study must therefore be based on the theoretical and practical suitability for the problem at hand rather than on erroneous a priori beliefs about external validity. In the end, external validity of findings can only be empirically determined. As pointed out above, the external validity of an experimental situation is related to the extent to which it contains those aspects of the task that are pertinent to the target situation and relevant for the hypotheses to be tested. What these variables should be, must be guided by theoretical considerations ([4], [6], and [8]). The theoretical basis for the author's and his colleagues' choice of chess as a research tool for command and control is as follows: First, the situation facing a commander in battle is an example of a "dynamic decision situation" and such situations have four properties [4].
Chess also has these features, the player must make a series of decisions how to move, the moves he makes put restrictions on subsequent moves, the situation on the board changes as a result of both players actions and the time limit means that time was an important factor. Military decision situations also contain complexity [3]. Complexity in a situation is related to the number of variables that the decision maker must take into consideration and how they are interrelated. In a battle task the complexity is among other things related to the number of units involved and how they can interact. A consequence of complexity is that the decision maker must take side effects into consideration. Chess also captures this aspect, it has many pieces and when moving a piece into a more offensive position a player the protection of another piece may be lost. Finally, the military decision situation contains opacity [3]. Opacity refers to a lack of transparency about the situation. One source of opacity is the enemy's active attempts to hide assets and locations; another is feedback delays which makes it impossible for the commander to have relevant information about the state of the battle. As described above, opacity was introduced into the game by the delay of information or by hiding parts of the board. This means that although the physical appearance of chess does not have much in common with the situation facing a military commander, a checkered board and stylized pieces, its "psychological fidelity" for the target situation ([6], [8]) from the theoretical perspective sketched above, apparently is quite good. If the theoretical assumptions are correct there are therefore good reasons to believe that the findings will have external validity. Two questions were asked in the previous section: Can results from laboratory experiments with chess have external validity and are case studies in natural settings a better alternative from that perspective? The answer is that if there is a theory connecting chess to a target situation results from a chess experiment can have external validity and there seems to be little reason to believe that field studies automatically lead to more valid conclusions. There is therefore no foundation for claims that it is better to use case studies instead of laboratory experiments in research aiming to improve military command and control. Concluding remarks In conclusion it can be worth pointing out that the strategy of using simple laboratory set-ups in development work is a well-established method within the industry. Model planes in wind tunnels and model boats in water tanks are regularly used to test design concepts in the early stages of the development of new airplanes and ships respectively. In this way, mistakes that would be costly to correct in later stages can be detected in time and at a low cost. It is difficult to see why this strategy of conducting simple laboratory tests of design concepts during the early stages of the development process should not also lead to the same benefits in the development of new military command and control systems. References [1] Alberts D. S., Garstka J. J., Hayes R. E. & Signori D. A. (2001). Understanding information age warfare . Washington DC: CCRP. [2] Berkowitz L. & Donnersten E. (1982). External validity is more than skin deep. Some answers to the criticisms of laboratory experiments. American Psychologist , 37 (3), 245-257. [3] Brehmer, B. (2000). Dynamic decision making in command and control. In C. McCann & R. Pigeau (Eds.), The human in command (pp. 233-248). New York: Kluwer Academic/Plenum Publishers. [4] Brehmer, B. (2004). Some reflections on microworld research. In Schiflett S. G., Elliot L. R., Salas, E. & Coovert M. D. (Eds.), Scaled worlds: Development, validation and applications (pp. 22-36). Aldershot, Hampshire, UK: Ashgate [5] Charness N. (1991). Expertise in chess: The balance between knowledge and search. In Ericsson K. A. & Smith J. (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 39-63). New York, NY: Cambridge university press [6] Elliot L. R., Dalrymple M. A., Schiflett S. G. & Miller J. C. (2004). Scaling scenarios: Development and application to C4ISR susteained operations resarch. In Schiflett S. G., Elliot L. R., Salas, E. & Coovert M. D. (Eds.), Scaled worlds: Development, validation and applications (pp. 119-133). Aldershot, Hampshire, UK: Ashgate [7] Frensch P.A. & Sternberg R. J. (1991). Skill-related differnces in game playing. In Sternberg R. J. & Frensch P.A., Complex problem solving: Principles and mechanisms (pp. 343-381). Hillsdale, NJ: Lawrence Erlbaum Associates [8] Kozlowski S. W. J. & DeShon R. P. (2004). A psychological fidelity approach to simulation-based training: Theory, research and principles. In Schiflett S. G., Elliot L. R., Salas, E. & Coovert M. D. (Eds.), Scaled worlds: Development, validation and applications (pp. 75-99). Aldershot, Hampshire, UK: Ashgate [9] Perla P. P. (1990). The art of wargaming: A guide for professionals and hobbyists. Annapolis, Md.: Naval institute press [10] Kuylenstierna, J., Rydmark, J. & Fahraeus, T. (1999). Värdet av informationsöverläge vid olika nivaer pa osäkerhet om läget i stridsrummet. Informationsbehov i framtida krig: Experimentrapport fran ett forskningsprojekt, Rapport nr 2. OpI/FoU, Försvarshögskolan, Stockholm, (FHS Skr 21 105: 60723). [11] Kuylenstierna, J., Rydmark, J. & Fahraeus, T. (1999). Värdet av styrkeöverläge vid olika nivaer pa osäkerhet om läget i stridsrummet . Informationsbehov i framtida krig: Experimentrapport fran ett forskningsprojekt, Rapport nr 3. OpI/FoU, Försvarshögskolan, Stockholm, (FHS Skr 21 105: 60856). [12] Kuylenstierna, J., Rydmark, J. & Fahraeus, T. (2000). Värdet av att ha tempoöverläge vid olika nivaer pa osäkerhet om läget i stridsrummet. Forskningsprojektet Informationsbehov i framtida krig: Experimentrapportering, Rapport nr 5. OpI/FoU, Försvarshögskolan, Stockholm, (FHS Skr 21 105: 60929). [13] Kuylenstierna, J., Rydmark, J. & Fahraeus, T. (2000). The Value of Information in War: Some Experimental Findings. Proceedings of the 5th International Command & Control Research & Technology Symposium (ICCRTS). 24-26 okt 2000, Canberra, Australien. [14] Rydmark, J., Kuylenstierna, J., Fahraeus, T. & Sandström, H. (2001). Värdet av av överlägsen sensorräckvidd vid olika nivaer pa osäkerhet om läget i stridsrummet . Forskningsprojektet Informationsbehov i framtida krig: Experimentrapport, Rapport nr 7. OpI/FoU, Försvarshögskolan, Stockholm, (FHS Skr 21 105: 61692). [15] Kuylenstierna, J., Rydmark, J., Fahraeus, T. & Sandström, H. (2002). Informationsöverläge och val av tempo i striden . Forskningsprojektet Informationsbehov i framtida krig: Experimentrapport , Rapport nr 9. KVI/FoU, Försvarshögskolan, Stockholm, (FHS Skr 21 105: 61021). [16] Kuylenstierna, J., Rydmark, J., Fahraeus, T. & Sandström, H. (2002). Överläge i rörlighet och val av tempo i striden . Forskningsprojektet Informationsbehov i framtida krig: Experimentrapport, Rapport nr 10. KVI/FoU, Försvarshögskolan, Stockholm, (FHS Skr 21 105: 61252). [17] Kuylenstierna, J., Rydmark, J. & Sandström, H. (2003). Betydelsen av överlägsen sensorräckvidd vid olika nivaer pa osäkerhet om läget i stridsrummet. Forskningsprojektet Informationsbehov i framtida krig: Experimentrapport, Rapport nr 12. KVI/FoU, Försvarshögskolan, Stockholm, (Bilaga 2 till FHS Skr 21 105: 61252). 1 Four experiments have also been carried out in order to investigate if advantages not pertaining to command and control can work under conditions of imperfect information. The value of advantages of material strength ([11], [13]), longer reach of the pieces [16] and opportunity to make more moves per turn ([12], [13]) have been investigated under two information delay conditions, (0/0) and (2/2). The value of the advantages was worse in the latter condition although less so for the opportunity to make more moves. |
Back
© ZMNE BJKMK 2006.