Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Peter Brown, Yaoqi Zhou*, Aik Choon Tan, Mohamed A. El-Esawi, Thomas Liehr, Oliver Blanck, Douglas P. Gladue, Gabriel M.F. Almeida, Tomislav Cernava, Carlos O. Sorzano, Andy W.K. Yeung, Michael S. Engel, Arun R. Chandrasekaran, Thilo Muth, Martin S. Staege, Swapna V. Daulatabad, Darius Widera, Junpeng Zhang, Adrian Meule, Ken HonjoOlivier Pourret, Cong Cong Yin, Zhongheng Zhang, Marco Cascella, Willy A. Flegel, Carl S. Goodyear, Mark J. van Raaij, Zuzanna Bukowy-Bieryllo, Luca G. Campana, Nicholas A. Kurniawan, David Lalaouna, Felix J. Hüttner, Brooke A. Ammerman, Felix Ehret, Paul A. Cobine, Ene Choo Tan, Hyemin Han, Wenfeng Xia, Christopher McCrum, Ruud P.M. Dings, Francesco Marinello, Henrik Nilsson, Brett Nixon, Konstantinos Voskarides, Long Yang, Vincent D. Costa, Johan Bengtsson-Palme, William Bradshaw, Dominik G. Grimm, Nitin Kumar, Elvis Martis, Daniel Prieto, Sandeep C. Sabnis, Said E.D.R. Amer, Alan W.C. Liew, Paul Perco, Farid Rahimi, Giuseppe Riva, Chongxing Zhang, Hari P. Devkota, Koichi Ogami, Zarrin Basharat, Walter Fierz, Robert Siebers, Kok H. Tan, Karen A. Boehme, Peter Brenneisen, James A.L. Brown, Brian P. Dalrymple, David J. Harvey, Grace Ng, Sebastiaan Werten, Mark Bleackley, Zhanwu Dai, Raman Dhariwal, Yael Gelfer, Marcus D. Hartmann, Pawel Miotla, Radu Tamaian, Pragashnie Govender, Oliver J. Gurney-Champion, Joonas H. Kauppila, Xiaolei Zhang, Natalia Echeverría, Santhilal Subhash, Hannes Sallmon, Marco Tofani, Taeok Bae, Oliver Bosch, Páraic O. Cuív, Antoine Danchin, Barthelemy Diouf, Tuomas Eerola, Evangelos Evangelou, Fabian Filipp, Hannes Klump, Lukasz Kurgan, Simon S. Smith, Olivier Terrier, Neil Tuttle, David B. Ascher, Sarath C. Janga, Leon N. Schulte, Daniel Becker, Christopher Browngardt, Stephen J. Bush, Guillaume Gaullier, Kazuki Ide, Clement Meseko, Gijsbert D.A. Werner, Jan Zaucha, Abd A. Al-Farha, Noah F. Greenwald, Segun I. Popoola, Shaifur Rahman, Jialin Xu, Sunny Y. Yang, Noboru Hiroi, Ozgul M. Alper, Chris I. Baker, Michael Bitzer, George Chacko, Birgit Debrabant, Ray Dixon, Evelyne Forano, Matthew Gilliham, Sarah Kelly, Karl Heinz Klempnauer, Brett A. Lidbury, Michael Z. Lin, Iseult Lynch, Wujun Ma, Edward W. Maibach, Diane E. Mather, Kutty S. Nandakumar, Robert S. Ohgami, Piero Parchi, Patrizio Tressoldi, Yu Xue, Charles Armitage, Pierre Barraud, Stella Chatzitheochari, Luis P. Coelho, Jiajie Diao, Andrew C. Doxey, Angélique Gobet, Pingzhao Hu, Stefan Kaiser, Kate M. Mitchell, Mohamed F. Salama, Ivan G. Shabalin, Haijun Song, Dejan Stevanovic, Ali Yadollahpour, Erliang Zeng, Katharina Zinke, C. G. Alimba, Tariku J. Beyene, Zehong Cao, Sherwin S. Chan, Michael Gatchell, Andreas Kleppe, Marcin Piotrowski, Gonzalo Torga, Adugna A. Woldesemayat, Mehmet I. Cosacak, Scott Haston, Stephanie A. Ross, Richard Williams, Alvin Wong, Matthew K. Abramowitz, Andem Effiong, Senhong Lee, Muhammad B. Abid, Cyrus Agarabi, Cedric Alaux, Dirk R. Albrecht, Gerald J. Atkins, Charles R. Beck, A. M.J.J. Bonvin, Emer Bourke, Thomas Brand, Ralf J. Braun, James A. Bull, Pedro Cardoso, Dee Carter, Robin M. Delahay, Bernard Ducommun, Pascal H.G. Duijf, Trevor Epp, Eeva Liisa Eskelinen, Mazyar Fallah, Debora B. Farber, Jose Fernandez-Triana, Frank Feyerabend, Tullio Florio, Michael Friebe, Saori Furuta, Mads Gabrielsen, Jens Gruber, Malgorzata Grybos, Qian Han, Michael Heinrich, Heikki Helanterä, Michael Huber, Albert Jeltsch, Fan Jiang, Claire Josse, Giuseppe Jurman, Haruyuki Kamiya, Kim de Keersmaecker, Erik Kristiansson, Frank Erik de Leeuw, Jiuyong Li, Shide Liang, Jose A. Lopez-Escamez, Francisco J. Lopez-Ruiz, Kevin J. Marchbank, Rolf Marschalek, Carmen S. Martín, Adriana E. Miele, Xavier Montagutelli, Esteban Morcillo, Rosario Nicoletti, Monika Niehof, Ronan O'Toole, Toshihiko Ohtomo, Henrik Oster, Jose Alberto Palma, Russell Paterson, Mark Peifer, Maribel Portilla, M. C. Portillo, Antonia L. Pritchard, Stefan Pusch, Gajendra P.S. Raghava, Nicola J. Roberts, Kehinde Ross, Birgitt Schuele, Kjell Sergeant, Jun Shen, Alessandro Stella, Olga Sukocheva, Vladimir N. Uversky, Sven Vanneste, Martin H. Villet, Miguel Viveiros, Julia A. Vorholt, Christof Weinstock, Masayuki Yamato, Ioannis Zabetakis, Xin Zhao, Andreas Ziegler, Wan M. Aizat, Lauren Atlas, Kristina M. Bridges, Sayan Chakraborty, Mieke Deschodt, Helena S. Domingues, Shabnam S. Esfahlani, Sebastian Falk, J. L. Guisado, Nolan C. Kane, Gray Kueberuwa, Colleen L. Lau, Dai Liang, Enwu Liu, Andreas M. Luu, Chuang Ma, Lisong Ma, Robert Moyer, Adam D. Norris, Suresh Panthee, Jerod R. Parsons, Yousong Peng, Ines M. Pinto, Cristina R. Reschke, Elina Sillanpää, Christopher J. Stewart, Florian Uhle, Hui Yang, Kai Zhou, Shu Zhu, Mohamed Ashry, Niels Bergsland, Maximilian Berthold, Chang Er Chen, Vito Colella, Maarten Cuypers, Evan A. Eskew, Xiao Fan, Maksymilian Gajda, Rayner Gonzálezlez-Prendes, Amie Goodin, Emily B. Graham, Ewout J.N. Groen, Alba Gutiérrez-Sacristán, Mohamad Habes, Enrico Heffler, Daniel B. Higginbottom, Thijs Janzen, Jayakumar Jayaraman, Lindsay A. Jibb, Stefan Jongen, Timothy Kinyanjui, Rositsa G. Koleva-Kolarova, Zhixiu Li, Yu Peng Liu, Bjarte A. Lund, Alexandre A. Lussier, Liping Ma, Pablo Mier, Matthew D. Moore, Katja Nagler, Mark W. Orme, James A. Pearson, Anilkumar S. Prajapati, Yu Saito, Simon E. Tröder, Florence Uchendu, Niklas Verloh, Denitza D. Voutchkova, Ahmed Abu-Zaid, Joaira Bakkach, Philipp Baumert, Marcos Dono, Jack Hanson, Sandrine Herbelet, Emma Hobbs, Ameya Kulkarni, Narendra Kumar, Siqi Liu, Nikolai D. Loft, Tristan Reddan, Thomas Senghore, Howard Vindin, Haotian Xu, Ross Bannon, Branson Chen, Johnny T.K. Cheung, Jeffrey Cooper, Ashwini K. Esnakula, Karine A. Feghali, Emilia Ghelardi, Agostino Gnasso, Jeffrey Horbar, Hei M. Lai, Jian Li, Lan Ma, Ruiyan Ma, Zihang Pan, Marco A. Peres, Raymond Pranata, Esmond Seow, Matthew Sydes, Ines Testoni, Anna L. Westermair, Yongliang Yang, Masoud Afnan, Joan Albiol, Lucia G. Albuquerque, Shimon Amir, Eisuke Amiya, Rogerio M. Amorim, Qianli An, Stig U. Andersen, John D. Aplin, Christos Argyropoulos, Yan W. Asmann, Abdulaziz M. Assaeed, Atanas G. Atanasov, David A. Atchison, Simon V. Avery, Paul Avillach, Peter D. Baade, Lars Backman, Christophe Badie, Alfonso Baldi, Elizabeth Ball, Olivier Bardot, Adrian G. Barnett, Mathias Basner, Jyotsna Batra, O. M. Bazanova, Andrew Beale, Travis Beddoe, Melanie L. Bell, Eugene Berezikov, Sue Berners-Price, Peter Bernhardt, Edward Berry, Theolis B. Bessa, Craig Billington, John Birch, Randy D. Blakely, Mark A.T. Blaskovich, Robert Blum, Marleen Boelaert, Dimitrios Bogdanos, Carles Bosch, Thierry Bourgoin, Daniel Bouvard, Laura M. Boykin, Graeme Bradley, Daniel Braun, Jeremy Brownlie, Albert Brühl, Austin Burt, Lisa M. Butler, Siddappa N. Byrareddy, Hugh J. Byrne, Stephanie Cabantous, Sara Calatayud, Eva Candal, Kimberly Carlson, Sònia Casillas, Valter Castelvetro, Patrick T. Caswell, Giacomo Cavalli, Vaclav Cerovsky, Monica Chagoyen, Chang Shi Chen, Dong F. Chen, Hao Chen, Hui Chen, Jui Tung Chen, Yinglong Chen, Changxiu Cheng, Jianlin Cheng, Mai Chinapaw, Christos Chinopoulos, William C.S. Cho, Lillian Chong, Debashish Chowdhury, Andre Chwalibog, A. Ciresi, Shamshad Cockcroft, Ana Conesa, Penny A. Cook, David N. Cooper, Olivier Coqueret, Enoka M. Corea, Antonio Costa, Elisio Costa, Carol Coupland, Stephanie Y. Crawford, Aparecido D. Cruz, Huijuan Cui, Qiang Cui, David C. Culver, Amedeo D'Angiulli, Tanya E.S. Dahms, France Daigle, Raymond Dalgleish, Håvard E. Danielsen, Sébastien Darras, Sean M. Davidson, David A. Day, Volkan Degirmenci, Luc Demaison, Koenraad Devriendt, Jiandong Ding, Yunus Dogan, X. C. Dong, Claudio F. Donner, Walter Dressick, Christian A. Drevon, Huiling Duan, Christian Ducho, Nicolas Dumaz, Bilikere S. Dwarakanath, Mark H. Ebell, Steffen Eisenhardt, Naser Elkum, Nadja Engel, Timothy B. Erickson, Michael Fairhead, Marty J. Faville, Marlena S. Fejzo, Fernanda Festa, Antonio Feteira, Patrick Flood-Page, John Forsayeth, Simon A. Fox, Steven J. Franks, Francesca D. Frentiu, Mikko J. Frilander, Xinmiao Fu, Satoshi Fujita, Ian Galea, Luca Galluzzi, Federica Gani, Arvind P. Ganpule, Antonio García-Alix, Kristene Gedye, Maurizio Giordano, Cecilia Giunta, Paul A. Gleeson, Cyrille Goarant, Haipeng Gong, Diop Gora, Michael J. Gough, Ravinder Goyal, Kathryn E. Graham, Ana Grande-Pérez, Patricia M. Graves, Harm Greidanus, Darren Grice, Christoph Grunau, Yosephine Gumulya, Yabin Guo, Vsevolod V. Gurevich, Oleg Gusev, Elke Hacker, Steffen R. Hage, Guy Hagen, Steven Hahn, Dagmar M. Haller, Sven Hammerschmidt, Jianwei Han, Renzhi Han, Martin Handfield, Hapuarachchige C. Hapuarachchi, Timm Harder, Jennifer E. Hardingham, Michelle Heck, Marcel Heers, Khe F. Hew, Yohei Higuchi, Cynthia St Hilaire, Rachel Hilton, Enisa Hodzic, Andrew Hone, Yuichi Hongoh, Guoku Hu, Heinz P. Huber, Luis E. Hueso, Judith Huirne, Lisa Hurt, Helena Idborg, Kazuho Ikeo, Evan Ingley, Philip M. Jakeman, Arne Jensen, Hong Jia, Husen Jia, Shuqin Jia, Jianping Jiang, Xingyu Jiang, Yi Jin, Daehyun Jo, Andrew M. Johnson, Marie Johnston, Karen R. Jonscher, Philippe G. Jorens, Jens O.L. Jorgensen, Johan W. Joubert, Sin Ho Jung, Antonio M. Junior, Thomas Kahan, Sunjeev K. Kamboj, Yong Kook Kang, Yannis Karamanos, Natasha A. Karp, Ryan Kelly, Ralph Kenna, Jonathan Kennedy, Birgit Kersten, Roy A. Khalaf, Javaria M. Khalid, T. Khatlani, Tarig Khider, Gregor S. Kijanka, Sarah R.B. King, Tomasz Kluz, Paul Knox, Tatsuya Kobayashi, Karl Wilhelm Koch, Maija R.J. Kohonen-Corish, Xiangpeng Kong, Deborah Konkle-Parker, Kalevi M. Korpela, Leondios G. Kostrikis, Peter Kraiczy, Harald Kratz, Günter Krause, Paul H. Krebsbach, Søren R. Kristensen, Prerna Kumari, Akira Kunimatsu, Hatice Kurdak, Young D. Kwon, Carl Lachat, Malgorzata Lagisz, Brenda Laky, Jan Lammerding, Matthias Lange, Mar Larrosa, Andrew L. Laslett, Gozewijn D. Laverman, Elizabeth E. Leclair, Kyung Woo Lee, Ming Yih Lee, Moon Soo Lee, Genyuan Li, Jiansheng Li, Klaus Lieb, Yau Y. Lim, Merry L. Lindsey, Paul Dag Line, Dengcai Liu, Fengbin Liu, Haiyan Liu, Hongde Liu, Vett K. Lloyd, Te Wen Lo, Emanuela Locci, Josef Loidl, Johan Lorenzen, Stefan Lorkowski, Nigel H. Lovell, Hua Lu, Wei Lu, Zhiyong Lu, Gustavo S. Luengo, Lars Gunnar Lundh, Philippe A. Lysy, Angela Mabb, Heather G. Mack, David A. Mackey, S. R. Mahdavi, Pamela Maher, Toby Maher, Sankar N. Maity, Brigitte Malgrange, Charalampos Mamoulakis, Arduino A. Mangoni, Thomas Manke, Antony S.R. Manstead, Athanasios Mantalaris, Jan Marsal, Hanns Ulrich Marschall, Francis L. Martin, Jose Martinez-Raga, Encarnacion Martinez-Salas, Daniel Mathieu, Yoichi Matsui, Elie Maza, James E. McCutcheon, Gareth J. McKay, Brian McMillan, Nigel McMillan, Catherine Meads, Loreta Medina, B. Alex Merrick, Dennis W. Metzger, Frederic A. Meunier, Martin Michaelis, Olivier Micheau, Hisaaki Mihara, Eric M. Mintz, Takuo Mizukami, Yann Moalic, D. P. Mohapatra, Antonia Monteiro, Matthieu Montes, John V. Moran, Sergey Y. Morozov, Matthew Mort, Noriyuki Murai, Denis J. Murphy, Susan K. Murphy, Shauna A. Murray, Shinji Naganawa, Srinivas Nammi, Grigorios Nasios, Roman M. Natoli, Frederique Nguyen, Christine Nicol, Filip van Nieuwerburgh, Erlend B. Nilsen, Clarissa J. Nobile, Margaret O'Mahony, Sophie Ohlsson, Oluremi Olatunbosun, Per Olofsson, Alberto Ortiz, Kostya Ostrikov, Siegmar Otto, Tiago F. Outeiro, Songying Ouyang, Sabrina Paganoni, Andrew Page, Christoph Palm, Yin Paradies, Michael H. Parsons, Nick Parsons, Pigny Pascal, Elisabeth Paul, Michelle Peckham, Nicoletta Pedemonte, Michael A. Pellizzon, M. Petrelli, Alexander Pichugin, Carlos J.C. Pinto, John N. Plevris, Piero Pollesello, Martin Polz, Giovanna Ponti, Piero Porcelli, Martin Prince, Gwendolyn P. Quinn, Terence J. Quinn, Satu Ramula, Juri Rappsilber, Florian Rehfeldt, Jan H. Reiling, Claire Remacle, Mohsen Rezaei, Eric W. Riddick, Uwe Ritter, Neil W. Roach, David D. Roberts, Guillermo Robles, Tiago Rodrigues, Cesar Rodriguez, Jo Roislien, Monique J. Roobol, Alexandra Rowe, Andreas Ruepp, Jan van Ruitenbeek, Petra Rust, Sonia Saad, George H. Sack, Manuela Santos, Aurore Saudemont, Gianni Sava, Simone Schrading, Alexander Schramm, Martin Schreiber, Sidney Schuler, Joost Schymkowitz, Alexander Sczyrba, Kate L. Seib, Han Ping Shi, Tomohiro Shimada, Jeon Soo Shin, Colette Shortt, Patricia Silveyra, Debra Skinner, Ian Small, Paul A.M. Smeets, Po Wah So, Francisco Solano, Daniel E. Sonenshine, Jiangning Song, Tony Southall, John R. Speakman, Mandyam V. Srinivasan, Laura P. Stabile, Andrzej Stasiak, Kathryn J. Steadman, Nils Stein, Andrew W. Stephens, Douglas I. Stewart, Keith Stine, Curt Storlazzi, Nataliya V. Stoynova, Wojciech Strzalka, Oscar M. Suarez, Taranum Sultana, Anirudha V. Sumant, Mathew J. Summers, Gang Sun, Paul Tacon, Kozo Tanaka, Haixu Tang, Y. Tanino, Paul Targett-Adams, Mourad Tayebi, Reema Tayyem, Christoph C. Tebbe, Evelyn E. Telfer, Wolfram Tempel, Julita A. Teodorczyk-Injeyan, Gert Thijs, Sally Thorne, Amanda G. Thrift, Celine Tiffon, Philip Tinnefeld, Daryono H. Tjahjono, Fabrice Tolle, Ervin Toth, Andria L. Del Tredici, Apostolos Tsapas, Konstantinos Tsirigotis, Ayse Turak, George Tzotzos, Edet E. Udo, Toshiaki Utsumi, Subramanian Vaidyanathan, Michel Vaillant, Armand Valsesia, Roosmarijn E. Vandenbroucke, Feliciano H. Veiga, Marc Vendrell, Peter A. Vesk, Paul Vickers, Victor M. Victor, Richard Villemur, Marie Claude Vohl, Christian R. Voolstra, Anne Vuillemin, Steven Wakelin, Levi Waldron, Laurence J. Walsh, Amanda Y. Wang, Fuan Wang, Yun Wang, Yoichi Watanabe, Andreas Weigert, Jet Chau Wen, Carol Wham, Ethan P. White, Jan Wiener, Gottfried Wilharm, Simon Wilkinson, Raffaella Willmann, Coralie Wilson, Brunhilde Wirth, Timothy R. Wojan, Mathieu Wolff, Bryan M. Wong, Tzu Wei Wu, Hanno Wuerbel, Xiangshu Xiao, Dong Xu, J. W. Xu, Jianping Xu, Bin Xue, Suayib Yalcin, Hong Yan, En Cheng Yang, Shiqi Yang, Wei Yang, Yuzhen Ye, Zhi Qiang Ye, Jari Yli-Kauhaluoma, Hiroshi Yoneyama, Ying Yu, Guo Cheng Yuan, Chiou Hwa Yuh, Manuela Zaccolo, Chen Zeng, Branko Zevnik, Chi Zhang, Li Zhang, Yingkai Zhang, Yusen Zhang, Zhiyong Zhang, Zhong Yin Zhang, Yuan Zhao, Min Zhou, Torsten Zuberbier, Carmen M. Aanei, Rafi Ahmad, Manar Al-Lawama, Alexandre Alanio, Judith Allardyce, David Alonso-Caneiro, John M. Atack, Dirk Baier, Abhisheka Bansal, Yannick Benezeth, Colette Berbesque, Frederik Berrevoet, Peter H.W. Biedermann, Erik Bijleveld, Florian Bittner, Fabian Blombach, Wouter van den Bos, Shellie A. Boudreau, Adam D. Bramoweth, Oliver Braubach, Yufeng Cai, Matthew Campbell, Zanxia Cao, Thibault Catry, Xin Chen, Shuiqin Cheng, Hee Jung Chung, Miguel A. Chávez-Fumagalli, Aaron Conway, Bruno M. Costa, Normand Cyr, Lorraine T. Dean, Martin S. Denzel, S. V. Dlamini, Kevin J. Dudley, Maeva Dufies, Thorsten Ecke, Denitsa Eckweiler, Elisenda Eixarch, Hosny El-Adawy, Julius V. Emmrich, Alex J. Eustace, Christine M. Falter-Wagner, Ramin Farhoudi, Johannes Fuss, Jianzhao Gao, Martin R. Gill, Liz Gloyn, Robert Goggs, Usha Govinden, Garrett Greene, Victor Greiff, D. S. Grundle, Patrick Grüneberg, Nicksy Gumede, Gbaguidi Haore, Pille Harrison, Xavier Hoenner, Diego Hojsgaard, Hikaru Hori, Maria P. Ikonomopoulou, Patrick Jeurissen, Daniel M. Johnson, Dhiraj Kabra, Koji Kamagata, Chandan Karmakar, Olga Kasian, Linda K. Kaye, Murad M. Khan, Yong Min Kim, J. K. Kish, Sebastian Kobold, Gary Kohanbash, Gregor Kohls, Jan Michael Kugler, Gyanendra Kumar, Jon Lacy-Colson, Asam Latif, Volker M. Lauschke, Bingling Li, Chinten J. Lim, Fang Liu, Xiaodong Liu, Jin Jian Lu, Qiang Lu, Poornima Mahavadi, Ugo Marzocchi, Christine A. McGarrigle, Tom van Meerten, Rogier Min, Iain Moal, Massimiliano Molari, Lucas Molleman, Saiful R. Mondal, Thea van de Mortel, W. N. Moss, Othonas A. Moultos, Maheswari Mukherjee, Kazuhiko Nakayama, Edward Narayan, Navaratnarajah, Philipp Alexander Neumann, Jiyun Nie, Yingjiu Nie, Frank Niemeyer, Fiona Nolan, Ogueri Nwaiwu, Wendy H. Oldenmenger, Emmanuel Olumayede, Jianhong Ou, Menuka Pallebage-Gamarallage, Simon P. Pearce, Tuula Pelkonen, Maria C. Pelleri, Joana L. Pereira, Mpho Pheko, Karina A. Pinto, Allison Piovesan, Michael Pluess, Illya M. Podolsky, Julie Prescott, Dongchen Qi, Xingshun Qi, Vaia D. Raikou, Andreas Ranft, Johanna Rhodes, Jean Yves Rotge, Anna D. Rowe, Manish Saggar, Robert A. Schuon, Shaouli Shahid, Vahid Shalchyan, Prasad Shirvalkar, Oleg Shiryayev, Jugpreet Singh, Michael J. Smout, António Soares, Chunjiao Song, Kshitij Srivastava, Rupesh K. Srivastava, Jim Sun, Attila Szabo, Wiktor Szymanski, Chan N.P. Tai, Hisashi Takeuchi, S. Tanadini-Lang, Fei Tang, Wanyin Tao, G. Theron, Chang F. Tian, Yu Shi Tian, Lisa M. Tuttle, Anna Valenti, Pierre Verlot, Mirella Walker, Jun Wang, Danielle Welter, Matthew Winslade, Dalei Wu, Yi Rui Wu, Han Xiao, Beisi Xu, Juan Xu, Ziyue Xu, Dongdong Yang, Mingjun Yang, Patricio Yankilevich, Yuyi You, Chenglong Yu, Jian Zhan, Gong Zhang, Kai Zhang, Tuo Zhang, Yi Zhang, Guoyan Zhao, Jing Zhao, Xiaofan Zhou, Zhenxing Zhu, Penelope A. Ajani, Udunna C. Anazodo, Saeed A. Bagloee, Kasia Bail, Ido Bar, Joe Bathelt, David Benkeser, Meghan L. Bernier, Adam M. Blanchard, Dominic W. Boakye, Vasileios Bonatsos, Michele H. Boon, George Bouboulis, Elizabeth Bromfield, Joshua Brown, Kim C.M. Bul, Kathryn J. Burton, Eugene G. Butkowski, Grace Carroll, Fengqing Chao, Elisabeth E. Charrier, Xiaoyin Chen, Yu Chih Chen, Chenguang, Jane R. Choi, Tore Christoffersen, João C. Comel, Cyril Cosse, Yanru Cui, Pieter van Dessel, Dhaval, Daria Diodato, Maelle Duffey, Avik Dutt, Luis G. Egea, Mohammed El-Said, Martin Faye, Beatriz Fernandez-Fernandez, Kieran G. Foley, Luria L. Founou, Fan Fu, Rabea A. Gadelkareem, Evgeny Galimov, Gulcan Garip, Alison Gemmill, Quentin Gouil, James Grey, Zoya Gridneva, Michel J. Grothe, Théophile Grébert, Fabricio Guerrero, Léo Guignard, Marco J. Haenssgen, David Hasler, Joan Y. Holgate, Ancheng Huang, Amanda M. Hulse-Kemp, Claire Jean-Quartier, Sang Min Jeon, Yangyang Jia, Catherine Jutzeler, Panagiotis Kalatzis, Masud Karim, Kathrin Karsay, Anne Keitel, Andreas Kempe, Jeremy R. Keown, Chin M. Khoo, Nyil Khwaja, Rogier A. Kievit, Sasha Kosanic, Dimitrios A. Koutoukidis, Paul Kramer, Dilip Kumar, Nükhet Kiraǧ, Giuseppe Lanza, Thuc D. Le, Jung W. Leem, Daniel Leightley, Andreia Leite, Lukas Lercher, Ying Li, Renly Lim, Luiz R.A. Lima, Li Lin, Tong Ling, Yuchen Liu, Zhonghua Liu, Yao Lu, Fok M. Lum, Hang Luo, Jatin Machhi, Angus Macleod, Isaac Macwan, Hanumantha R. Madala, Nima Madani, Nicola de Maio, Kalina Makowiecki, Daniel J. Mallinson, Ruta Margelyte, Caracausi Maria, Y. Markonis, Luca Marsili, Suzanne Mavoa, Lorna McWilliams, Moa Megersa, Caetano S.M. Mendes, Julia Menichetti, Rebecca Mercieca-Bebber, Jack J. Miller, David Paul M. Minde, Alexander Minges, Eleanor Mishra, Virendra R. Mishra, Carly Moores, Nicola Morrice, Alexander E. Moskalensky, Nicolò Navarin, Edessa Negera, Philippe Nolet, Ana Nordberg, Rickard Nordén, Jessica P. Nowicki, Nelly Olova, Pawel Olszewski, Robert Onzima, Chih Long Pan, Charny Park, Dong I. Park, Seyoung Park, Chandrashekhar D. Patil, Sansoa A. Pedro, Samuel R. Perry, Jessica Peter, Brent M. Peterson, Andrea Pezzuolo, Ilya Pozdnyakov, Siyu Qian, Lei Qin, Ali Rafe, Ishier Raote, Ali Raza, Henrike Rebl, Osama Refai, Tim Regan, Tambi Richa, Mark F. Richardson, K. R. Robinson, Luca Rossoni, Romain Rouet, Soroush Safaei, Pierre H.H. Schneeberger, Daniela Schwotzer, Agata Sebastian, Jennifer Selinski, Stefanie Seltmann, Feng Sha, Nir Shalev, Jin Long Shang, Josef Singer, Mandeep Singh, Taylor Smith, Emma Solomon-Moore, Lijuan Song, Samuele Soraggi, Ryan Stanley, Nico Steckhan, Frederic Strobl, Lorenzo Subissi, Irwan Supriyanto, Chinmay R. Surve, Tomo Suzuki, Caitlin Syme, Karl Sörelius, Young Tang, Marwa Tantawy, Sumudu Tennakoon, Serafino Teseo, Christine Toelzer, Nikola Tomov, Miguel Tovar, Linh Tran, Sushil Tripathi, Anil M. Tuladhar, Azubuike C. Ukubuiwe, Carolina O.L. Ung, Kaspar Valgepea, Hamid Vatanparast, Arnau Vidal, Fang Wang, Qing Wang, Ricky Watari, Rebecca Webster, Ruth Webster, Junnian Wei, David Wibowo, Tanja S.H. Wingenbach, Rose M. Xavier, Shumin Xiao, Peng Xiong, Shicai Xu, Shilin Xu, Ruifeng Yao, Wen Yao, Qinan Yin, Yongbo Yu, Masayoshi Zaitsu, Zian Zeineb, Xiao Yong Zhan, Jilei Zhang, Rongqiang Zhang, Wei Zhang, Xianglilan Zhang, Shan Zheng, Bailing Zhou, Xiaoyan Zhou, Haroon Ahmad, Sayo A. Akinwumi, Gregory F. Albery, Ahmed Alhowimel, Junaid Ali, Mansour Alshehri, Mohammed Alsuhaibani, Andrey Anikin, Samuel O. Azubuike, Anders Bach-Mortensen, Lior Baltiansky, Martin Bartas, Kiflemariam Y. Belachew, Vivek Bhardwaj, Karin Binder, Nicholas S. Bland, Michael Boah, Benjamin Bullen, Giovanna E. Calabrò, Tiffany J. Callahan, Bing Cao, Kelsey Chalmers, Wei Chang, Zhengping Che, Andrew T.Y. Chen, Haimin Chen, Huaming Chen, Youning Chen, Zhao Chen, Youngrok Choi, Mohiuddin A.K. Chowdhury, Martin R. Christensen, Robert S.C. Cooke, Marzia Cottini, Natalie V. Covington, Catriona Cunningham, Julien Delarocque, Lucie Devos, Aurup R. Dhar, Ke Feng Ding, Kexian Dong, Zheng Dong, Niklas Dreyer, Chelsea Ekstrand, Tanguy Fardet, Berhanu E. Feleke, Thomas Feurer, Angela Freitas, Tian Gao, Gebremedhin, Francesco Giganti, Piotr Grabowski, José R. Guerra-Mora, Chengying Guo, Xinyi Guo, Himanshu Gupta, Shuonan He, Marloes Heijne, Stephanie Heinemann, Alexander Hogrebe, Zhengping Huang, Sophinese Iskander-Rizk, Lavanya M. Iyer, Yasmin Jahan, Ameh S. James, Emmanuel Joel, Bastian Joffroy, Clara Jégousse, George Kambondo, Priyanka Karnati, Cihan Kaya, An Ke, Daniel Kelly, Rob Kickert, Peter E. Kidibule, Jennifer P. Kieselmann, Hyeon J. Kim, Takeshi Kitazawa, Aniek Lamberts, You Li, Huakang Liang, Sabrina N. Linn, Thomas Litfin, Wang Liusuo, Vasiliki Lygirou, Ajay K. Mahato, Zhi Ming Mai, Rupert W. Major, Samira Mali, Panagiotis Mallis, Wenzhi Mao, Katie Marvin-Dowle, Leanda D. Mason, Ben Merideth, Maria J. Merino-Plaza, Britt Merlaen, Rossella Messina, Anand K. Mishra, Junaid Muhammad, Conrad Musinguzi, Afroditi Nanou, Amreen Naqash, Joe T. Nguyen, Thi T.H. Nguyen, Duan Ni, Nida, Shirli Notcovich, Barnabas Ohst, Quinn R. Ollivier, Daniël F. Osses, Xiangda Peng, Arnoud Plantinga, Michael Pulia, Muhammad Rafiq, Ayush Raman, Raucher-Chéné, Rafal Rawski, Asit Ray, Lubna A. Razak, Kevin Rudolf, Peter Rusch, Margaux L. Sadoine, Axel Schmidt, Roey Schurr, Stephen Searles, Saurab Sharma, Barry Sheehan, Chunhu Shi, Belal Shohayeb, Andrew Sommerlad, Jan Strehlow, Xianbao Sun, Raghav Sundar, Ghazaleh Taherzadeh, Nur D.M. Tahir, Jun Tang, Jean Testa, Zhiqi Tian, Qian Tingting, Geert P. Verheijen, Casey Vickstrom, Teng Wang, Xiaomin Wang, Zhenxing Wang, Pan Wei, Alex Wilson, Wyart, Abdul Amir Yassine, Abbas Yousefzadeh, Asma Zare, Zhen Zeng, Chengrong Zhang, Haowen Zhang, Linxing Zhang, Tongchuan Zhang, Weijia Zhang, Zhe Zhang, Jianyu Zhou, Dongjie Zhu, Vincenzo Adamo, Adebolajo A. Adeyemo, Maria Aggelidou, Adi M. Al-Owaifeer, Arwa Z. Al-Riyami, Saeed K. Alzghari, Vibeke Andersen, Kathryn Angus, Muhammad Asaduzzaman, Hadi Asady, Dai Ato, Xiaoyong Bai, Rebecca L. Baines, Maghan Ballantyne, Bo Ban, Jill Beck, Walid Ben-Nafa, Emma Black, Antoine Blancher, Ron Blankstein, Neil Bodagh, Paulo Borges, Anastasia Brooks, Josue Brox-Ponce, Arturo Brunetti, Colin D. Canham, Piero Carninci, Richard Carvajal, Shun C. Chang, Jie Chao, Pranab Chatterjee, He Chen, Liming Chen, Yi Chun Chen, Adnan K. Chhatriwalla, Ibrahim Chikowe, Trees Juen Chuang, Rosane G. Collevatti, Diego A.V. Cornejo, Ana Cuenda, Myriam Dao, Delphine Dauga, Zaian Deng, Kiran Devkota, Lisa V. Doan, Yaser H.A. Elewa, Dongsheng Fan, Mohammed Faruk, Shi Feifei, Trevor S. Ferguson, Francesco Fleres, Emma J. Foster, Stephen Foster, Tzvi Furer, Yibo Gao, Enid J. Garcia-Rivera, Adi Gazdar, Ronald B. George, Sayantan Ghosh, Elena Gianchecchi, Joshua M. Gleason, Allan Hackshaw, Adam Hall, Richard Hall, Paul Harper, William E. Hogg, Guangqun Huang, Kylie E. Hunter, Adriaan P. Ijzerman, Carlos Jesus, Gao Jian, James S. Lewis, Souha S. Kanj, Harsheen Kaur, Shona Kelly, Fayez Kheir, V. S. Kichatova, Musa Kiyani, Reinhild Klein, Tom Kovesi, Jennifer L. Kraschnewski, Addanki P. Kumar, Dmitry Labutin, Alejandro Lazo-Langner, Guy Leclercq, Maoteng Li, Qingchun Li, Tangliang Li, Yongzhe Li, Wei Ting Liao, Zheng Yin Liao, Jessica Lin, J. Lizer, Giambattista Lobreglio, Cher Lowies, Cheng Lu, Haroon Majeed, Adam Martin, Luis Martinez-Sobrido, Edwin Meresh, Marianne Middelveen, Alireza Mohebbi, Jorge Mota, Zahra Mozaheb, Ley Muyaya, Amar Nandhakumar, Sheryl H.X. Ng, Monther Obeidat, Deog Hwan Oh, Mohammed Owais, Pia Pace-Asciak, Ajay Panwar, Caroline Park, Chris Patterson, Felipe Penagos-Tabaree, Paolo T. Pianosi, Valentina Pinzi, Clare Pridans, Anna Psaroulaki, Ravi K. Pujala, Leonardo Pulido-Arjona, Peng Fei Qi, Proton Rahman, Nayanjot K. Rai, Tienush Rassaf, Julie Refardt, Walter Ricciardi, Olaf Riess, Alexandros Rovas, Frank M. Sacks, Sherif Saleh, Christopher Sampson, Axel Schmutz, Robert Sepanski, Neeraj Sharma, Manisha Singh, Paul Spearman, Mehala Subramaniapillai, Ritu Swali, Cher M. Tan, Juan I. Tellechea, Lisa Marie Thomas, Xin Tong, Demetrios G. Vavvas, Ralf Veys, Veronica Vitriol, Horng Dar Wang, Jinhui Wang, Jiucun Wang, Jason Waugh, S. A. Webb, Brendan A. Williams, Alan D. Workman, Tingxiu Xiang, Li Xin Xie, Jun Xu, Taosheng Xu, Chongjun Yang, Jihoon G. Yoon, Christina M. Yuan, Arno Zaritsky, Yao Zhang, Haochen Zhao, Hannah Zuckerman, Ran Lyu, Wayne Pullan, RELISH Consortium

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical science.

Original languageEnglish
Pages (from-to)1-67
Number of pages67
JournalDatabase
Volume2019
DOIs
Publication statusPublished - 1 Jan 2019

Cite this

Brown, P., Zhou, Y., Tan, A. C., El-Esawi, M. A., Liehr, T., Blanck, O., ... RELISH Consortium (2019). Large expert-curated database for benchmarking document similarity detection in biomedical literature search. Database, 2019, 1-67. https://doi.org/10.1093/database/baz085