With the explosion of robotics analysis, staying present in fields like Studying from Demonstration (LfD) is a monumental problem. Is AI the answer to the “paper deluge,” or is it a part of the issue? Learn the article preview under to be taught extra!
Obtain the complete paper: Surviving the Paper Deluge.
Authors: Aude Billard, Renaud Detry, Nadia Figueroa, Maximilian Foriest, Dongheui Lee, Kunpeng Yao
Contributions: The 5 senior authors (A.B, R.D, N.F, D.L and Ok.Yao) collectively designed the examine, learn the papers, carried out the qualitative and quantitative evaluation and writing of the paper. M. F. contributed scripts for LLM evaluation and took part in LLM-Human comparability.
Abstract
Scientists are anticipated to learn newly revealed papers of their subject to remain present and hold their work related. Nevertheless, when confronted with the huge variety of publications, it might appear an awesome activity to learn all these papers, even when one have been to cut back this to solely a fraction associated to at least one’s personal space of analysis. For instance, in 2024 alone, IEEE revealed at least 46,968 papers on “robotics” or “automation”, and IEEE publications signify solely a fraction of the full analysis accessible on-line
To evaluate the magnitude of this problem, in addition to to judge how a lot real progress is reported in right this moment’s publications, we undertook precisely this effort. For the duty to be affordable, we decreased our search to at least one specific subarea, studying from demonstration (LfD), that’s strategies whereby robots are taught by human consultants. We monitor progress via each quantitative and qualitative metrics, providing a evaluation on present traits and notable contributions. We additionally delineate areas of significance, however that appear to obtain little consideration and supply suggestions for selling.
Our evaluation was based each on a human-eye evaluation of all papers. We additionally explored the usage of AI and different computing instruments to do that activity in our place. Whereas scripts and enormous language fashions (LLMs) can be utilized pretty faithfully to supply normal quantitative evaluation, they fail in relation to assessing the true significance of the analysis. They can’t acknowledge a paper revisiting a piece that already had options. They fail to acknowledge when the summary or claims of the paper are overstatements over the true contribution reported within the paper.
Our general evaluation led us to conclude that from a deck of greater than 300 papers, solely about 20% of the papers might be certified as providing extremely notable contributions, whereas the rest of the papers provided quite a lot of incremental enhancements over current strategies, or new domains of functions. The notable contributions didn’t correlate essentially with a better variety of downloads or citations. Discovering these gems is, nonetheless, important to cut back the danger that novel work goes unnoticed and cut back duplication of efforts. We provide a couple of ideas on find out how to finest mix direct studying of the literature with automated approaches (scripts and LLMs) to streamline the evaluation course of. We shut with a couple of suggestions: a) develop a analysis engine that restores the pure significance of labor finished by journal and convention editorial boards to rank papers based mostly on analysis scores and peer-reviewed standing, rather than Google Scholar or IEEEXplore, that place all publications on equal footing, disregarding peer reviewing and the popularity of journals and conferences, b) think about establishing a blind publication mannequin and topic-based social media posting, the place authors’ identify and establishment are downplayed and develop into accent to the paper to make sure that focus be on the content material of the publication reasonably than secondary facets, c) take a holistic method to make use of of LLM in assist of reviewing literature, utilizing them for what they excel at, particularly summarizing a bit of labor and gathering exact quantitative info, however taking into consideration that, whereas right this moment the instruments can’t match professional capability to evaluate true novelty, ought to they obtain this sooner or later, this may increasingly have repercussion on our personal means to supply stated experience.
Publications development
Over the previous decade, the variety of submissions to robotics journals has grown steadily on a yearly foundation, with an explosive development in 2023 (26%) and 2024 (31%), probably attributable to various factors, together with rising curiosity in the private and non-private sectors and to the supply of AI instruments supporting the writing of papers and code. The variety of revealed papers has carefully adopted this development, regardless of all efforts made by editorial boards to comprise the expansion by lowering acceptance charges. Conferences have adopted the identical development. As an illustration, ICRA doubled the variety of papers it revealed in ten years, reaching roughly 1,800 in 2024. Concurrently, the sturdy strain exerted by the neighborhood to publish quickly has led to a 50% lower within the time window between the submission of a paper and its publication. The phenomenon isn’t specific to IEEE publications, and journals and conferences resembling IJRR, RSS and CoRL have adopted the identical development.
Clearly, it might be unrealistic to anticipate any researcher to learn all of those publications. One may argue that researchers are usually occupied with solely a subset of the literature, for example a selected area or methodology, and would subsequently learn solely a fraction of all revealed papers. But even this narrower scope might show unmanageable. To evaluate how possible it’s for a researcher to remain present inside their very own space of experience, we undertook the duty of studying a big fraction of all papers revealed in our area – studying from demonstration – over the course of a single 12 months (2024).
This text initially appeared on IEEE RAS.
tags: IEEE RAS
IEEE Robotics and Automation Society (RAS)
strives to advance innovation, schooling, and basic and utilized analysis in robotics and automation