Plat_Forms: A Web Development Platform Comparison by an Exploratory Experiment Searching for Emergent Platform Properties

Background: For developing Web-based applications, there exist several competing and widely used technological platforms (consisting of a programming language, framework(s), components, and tools), each with an accompanying development culture and style. Research question: Do Web development projects exhibit emergent process or product properties that are characteristic and consistent within a platform, but show relevant substantial differences across platforms or do team-to-team individual differences outweigh such differences, if any? Such a property could be positive (i.e., a platform advantage), negative, or neutral, and it might be unobvious which is which. Method: In a nonrandomized, controlled experiment, framed as a public contest called “Plat_Forms,” top-class teams of three professional programmers competed to implement the same requirements for a Web-based application within 30 hours. Three different platforms (Java EE, PHP, or Perl) were used by three teams each. We compare the resulting nine products and process records along many dimensions, both external (usability, functionality, reliability, security, etc.) and internal (size, structure, modifiability, etc.). Results: The various results obtained cover a wide spectrum: First, there are results that many people would have called “obvious” or “well known,” say, that Perl solutions tend to be more compact than Java solutions. Second, there are results that contradict conventional wisdom, say, that our PHP solutions appear in some (but not all) respects to be actually at least as secure as the others. Finally, one result makes a statement we have not seen discussed previously: Along several dimensions, the amount of within-platform variation between the teams tends to be smaller for PHP than for the other platforms. Conclusion: The results suggest that substantial characteristic platform differences do indeed exist in some dimensions, but possibly not in others.