Evaluation Methods and Replicability of Software Architecture Research Objects

Abstract

Our paper at the 19th IEEE International Conference on Software Architecture (ICSA 2022) started by noticing that Software Architecture (SA) as research area experienced an increase in empirical research. Empirical research builds a sound foundation for validity and comparability. A current overview of the evaluation and replicability of SA research objects could help to discuss our empirical standards as a community. However, no current overview existed. We assessed the current state of practice of evaluating SA research objects and replication artifact provision in full technical conference papers from ICSA and the European Conference on Software Architecture (ECSA) 2017–2021. We first developed a categorization schema for SA research object evaluation and artifact provisioning. In a systematic literature review with 153 papers, we then classified the papers according to that schema. From our findings we derive and describe four proposals for improving the state of practice in evaluating SA research objects.

BibTeX

@inproceedings{konersmann_evaluation_2023,
    author = {Konersmann, Marco and Kaplan, Angelika and Kühn, Thomas and Heinrich, Robert and Koziolek, Anne and Reussner, Ralf and Jürjens, Jan and al-Doori, Mahmood and Boltz, Nicolas and Ehl, Marco and Fuchß, Dominik and Großer, Katharina and Hahner, Sebastian and Keim, Jan and Lohr, Matthias and Sağlam, Timur and Schulz, Sophie and Töberg, Jan-Philipp},
    title = {Evaluation Methods and Replicability of Software Architecture Research Objects},
    booktitle = {Software Engineering 2023},
    year = {2023},
    editor = {Engels, Gregor and Hebig, Regina and Tichy, Matthias} ,
    pages = { 75-76 },
    publisher = {Gesellschaft für Informatik e.V.},
    address = {Bonn}
}