**The IEEE International Conference on Software Architecture (ICSA) 2022
is on the way! Please note that deadlines are earlier than usual:
https://icsa-conferences.org/2022/
**
**
*Artifact Evaluation Track – ICSA 2022*
**
*----------------------------------------------------*
*
CFP ICSA 2022 - Artifact Evaluation Track
12-15 March | Honolulu, Hawaii (USA)
https://icsa-conferences.org/2022/
----------------------------------------------------
**
ICSA 2022 will be the second ICSA featuring an Artifact Evaluation
Track. An artifact evaluation track aims to review and promote the
research artifacts of accepted papers.
Artifacts can be software systems, scripts, or datasets. High quality
artifacts of published research papers are a foundation for the research
results to be reproduced by other researchers and are thus a desirable
part of the publication itself.
The artifact evaluation system is based on the upcoming NISO Recommended
Practice on Reproducibility Badging and Definitions, which is supported
by our publisher, IEEE, and the evaluation criteria are also inspired by
ACM’s artifact review and badging system as well as the criteria used by
ICSE 2021.
Who can submit?
------------------------------------
Authors of the papers accepted to the following ICSA 2022 tracks are
invited to submit an artifact for evaluation for the Research Object
Reviewed (ROR) – reusable badge and the Open Research Object (ORO)
badge: Technical Track, Journal-First Track, Software Architecture in
Practice Track, and New and Emerging Ideas Track. All authors of papers
related to the topics mentioned in the call for papers of the ICSA
technical track are invited to submit studies for the Results Reproduced
(ROR-R) and Results Replicated (RER) badges.
In addition, authors of any prior software architecture research are
invited to submit an artifact for evaluation as Results Reproduced
(ROR-R) and Results Replicated (RER).
Please note that we require one author of each artifact submission to
peer-review 2-3 other artifacts!
Candidate Artifacts
------------------------------------
Artifacts of interest include (but are not limited to) the following:
- Software, which are implementations of systems or algorithms
potentially useful in other studies.
- Data repositories, which are data (e.g., logging data, system traces,
survey raw data) that can be used for multiple software engineering
approaches.
- Frameworks, which are tools and services illustrating new approaches
to software engineering that could be used by other researchers in
different contexts.
This list is not exhaustive, so the authors are asked to email the
chairs before submitting if their proposed artifact is not on this list.
Further information on data sharing principles and approaches are
further introduced along an introduction of the general notion of open
science in the book chapter Open Science in Software Engineering by
Méndez, Graziotin, Wagner, and Seibold: https://arxiv.org/abs/1904.06499.
The best artifact selected by the reviewers will be awarded the best
artifact award.
For accepted ICSA 2022 papers, we plan to integrate the badge on the
paper in the official IEEE proceedings.
Evaluation Criteria and Badges
------------------------------------
Evaluation criteria for badges and additional information have been
taken from ICSE 2021, which are based on the ACM policy on Artifact
Review and Badging Version 1.1, and from the upcoming NISO Recommended
Practice on Reproducibility Badging and Definitions, which is supported
by our publisher, IEEE.
The ICSA artifact evaluation track uses a single-blind review process.
The reviewers for the artifacts will be a combination of artifact
authors and ICSA program committee members.
Artifacts will be evaluated using the criteria in the last row of the
table below. The goal of this track is to encourage reusable research
products.
In order to obtain any badge every artifact will be required to be
Functional, that is, it must be properly documented, consistent,
complete, exercisable, and include appropriate evidence of verification
and validation. The badges to be awarded as well as the evaluation
criteria are the following ones:
RESEARCH OBJECT REVIEWED (ROR) – REUSABLE: Functional + very carefully
documented and well-structured to the extent that reuse and repurposing
is facilitated. In particular, norms and standards of the research
community for artifacts of this type are strictly adhered to.
OPEN RESEARCH OBJECT (ORO): Functional + placed on a publicly accessible
archival repository. A DOI or link to this repository along with a
unique identifier for the object is provided.
(this matches the ACM “Available” badge)
RESULTS REPRODUCED (ROR-R): ROR + ORO + main results of the paper have
been obtained in a subsequent study by a person or team other than the
authors, using, in part, artifacts provided by the author.
RESULTS REPLICATED (RER): ROR + ORO + the main results of the paper have
been independently obtained in a subsequent study by a person or team
other than the authors, without the use of author-supplied artifacts.
Papers with such badges contain functional and reusable products that
other researchers can use to bootstrap their own research. Experience
shows that such papers earn increased citations and greater prestige in
the research community.
General information and further suggested reading about artifact
evaluation can be viewed in a public draft standard by Baldassarre,
Ernst, Hermann and Menzies.
Important Dates
------------------------------------
Artifact Evaluation Registration Deadline: January 11, 2022 (mandatory)
Artifact Evaluation Submissions Deadline: January 14, 2022
Artifact Evaluation Notification: January 31, 2022
Submission and Review
Submission and Review
------------------------------------
Please, see details at:
https://icsa-conferences.org/2022/conference-tracks/artifact-evaluation-track/
Organization
------------------------------------
Elena Navarro, University of Castilla-La Mancha, Artifact Evaluation Chair
Peng Xin, Fudan University, Artifact Evaluation Chair
*
--
*Dalila Tamzalit*
**/Maître de Conférences HC HdR - Associate-Professor/
Page web <http://dalila-tamzalit.com/fr/accueil/> - Home page
<http://dalila-tamzalit.com/>
Laboratoire des Sciences du Numérique de Nantes - CNRS UMR 6004
Équipe NaoMod <https://www.ls2n.fr/equipe/naomod/>
4 rue Alfred Kastler, 44307 Nantes Cedex 3 - FRANCE
33 (0)2.51.85.80.54
<http://www.ls2n.fr>
<http://www.ls2n.fr>
IUT Nantes - Département Informatique
3, rue du Maréchal Joffre
44 041 Nantes cedex 1 - FRANCE
Tel : 33 (0)2.40.30.60.57
_______________________________________________
hol-info mailing list
hol-info@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/hol-info
