By Lin Zhong, March 2016
"Scientists discover the world that exists; engineers create the world that never was." by Theodore Von Karman
Broadly speaking, my research is about building computer systems. Toward building a computer system, one must first understand related existing systems, either those in use or early prototypes, and then design it. Many researchers take intellectual satisfaction from understanding systems or designing systems. Over the years, I have realized that most of my scholarly satisfaction comes from actually building systems. In this article, I want to explain the role of building systems in the greater context of science.
According to Wikipedia, Science "is a systematic enterprise that creates, builds and organizes knowledge in the form of testable explanations and predictions about the universe." The essence of this enterprise, in my opinion, is about discovering testable knowledge about the universe (or world, to use Von Karman's term).
Computer system builders are both engineer and scientist. The quote from Von Karman conveniently glorifies engineers by contrasting them against scientists. However, we can revise it into: Scientists discover the world that exists; engineers discover the world that is possible, by creating it.
It is often said that engineering is about applying scientific knowledge to solve practical problems. In my opinion, this emphasizes the practical aspect of engineering. However, it importantly misses the scientific aspect of engineering. Engineering, especially engineering research, is also about discovering testable knowledge, about discovering a world that is possible, by creating it. That possible world may be practical (at the time of discovery and the definition of practical is debatable) or not but the world must not have existed at the time of discovery. In contrast to the world discovered by natural scientists, that by engineers is artificial, with man-made artifacts.
At the dawn of Computer Science, many computing pioneers argued that discovering an artificial world is also science. This argument helped catapult Computer Science into a major scientific discipline today . Indeed, the title of this article pays homage to one of the important works from that era .
I look around and realize scholars make testable discoveries in three very different ways, often by combining them.
By reasoning in logic or mathematics, theoreticians proves theorems of which correctness can be tested with reason.
By observing the world, natural scientists discover phenomena that can be verified by further observation; they often revise/propose new theories that are supported by available observations and of which correctness can be tested with further observations.
By creating the world, engineers, including system builders, discover testable knowledge about how to build things that can be tested by future building activities.
System building, or creating the world, is particularly important for computers because computer systems are often not amenable to formal methods, at least not without many simplifying assumptions. More importantly, observations made from existing systems and shoehorned prototypes may not necessarily generalize for a possible system with a radically new design. Ironically, a novel discovery must be not true for any existing systems by definition: you can only make a novel discovery by building a novel system.
Having established above that system building is about making discoveries very much like other scientific disciplines, I would like to elaborate the unique aspects of system building as science.
An important type of discovery common from system building is that there is a previously unknown hole in existing knowledge. This type is often in the form of questions like: what explains this observation? how to mathematically reason about the data collected? Or in the form of needs: "A tool that does A and B will make the job a lot easier"; "we need a formal specification of the interface''.
The more interesting type of discovery is, however, about building itself. The computer systems literature prides abstractions, principles, rules, hints about computer designs  . Some even call it art , in an attempt to emphasize the human creativity required while ignoring the scientific aspect of it. See references below. Many seminal papers in systems clearly articulate their contributions in the form of principle and rule.
These abstractions, principles, rules, and hints are forms of empirical knowledge that are useful for building systems. They can also be tested by applying them to building new systems.
Such empirical knowledge is quite unlike the laws in many physical sciences, in two ways. First, it is hardly universal, meaning that one can often find cases where it does not work. Unlike in physical sciences where a contradictory observation usually leads to an update of the knowledge (in the form of theory and law), system builders rarely update the principles/rules/hints, at least rarely do so explicitly: those that fail often were simply forgotten. Such empirical knowledge is considered valuable if it works quite often, instead of always. Indeed, people use different choices of word to indicate their confidence in how often empirical knowledge works, e.g. principles work more often than rules and hints should be applied with extreme caution. The root reason is that the object of system building (computer systems) has become extremely complex while the subject (the human builders) has remain the same bounded in their rationality: for mysterious unknown reasons, we have not been able to systemize computer systems as we have systemized physics.
Second, related to the first, the empirical knowledge discovered from system building is as much about the systems as about the human builders. The famous KISS principle is a loud attestament to our own bounded rationality. Indeed, one can argue much of the empirical knowledge is about knowing us human designers, programmers, administrators, and users of systems.
The sciences of system building are actually close to other empirical sciences involving complex objects that have not been systemized into math or logic. Psychology seeks knowledge about our mental processes without a formal, complete picture of how our brain is built. Political sciences, economics and finance investigates complex human, social, organizational activities without systematic, universal laws. These sciences, like the sciences of system building, feature empirical knowledge that works often but not always.
Following the argument above, I posit that the scientific contribution from system building is about the discoveries from building, rather than the system built. Indeed, I argue that the system built should be considered a validation of the discoveries. When the system happens to serve some practical purposes, we say the system is useful. Building useful systems is, however, an engineering contribution.
As system-building scholars, we must carefully decide what a system to build. If you agree with my position above, we should make the decision based what scientific contribution may be made from building. We should care more about the usefulness of the discoveries from building than about the usefulness of the system built.
Ode to crazy systems: If we assume people who built existing systems are as smart as we are, the chance for us to make any new discovery by building a system the same as or close to existing ones is low. Henceforth, if you want to make significant scientific contributions, try to build a very different system, a crazy, outrageous one.
Compliment or Insult: There is a joke that mathematicians often feel insulted when told their results are useful. I argue that system-building scholars should similarly feel insulted when told that the system they have just built is useful. Personally I take as compliment when told the discoveries we have made by building a system are useful (for building other systems).