Skip to content

Conversation

@Siva-Sai22
Copy link

Closes #14421

Implemented web search capability for ScienceDirect by integrating the Scopus Search API, since the ScienceDirect Search API requires institutional access that most users don't have. Added a new ScopusQueryTransformer to convert JabRef search queries to Scopus query syntax, and rewrote the ScienceDirect class to implement PagedSearchBasedParserFetcher interface, enabling paginated search results with proper JSON response parsing, this allows users to search academic literature via the "Web search" feature in JabRef, which was previously not available for ScienceDirect.

Steps to test

  • Go to settings -> preferences -> websearch.
  • Add the ScienceDirect API key and test it.
  • Then search using the ScienceDirect (Example query: Einstein)
Screenshot_20251208_151456 Screenshot_20251208_151516 Screenshot_20251208_151747

Mandatory checks

Siva-Sai22 and others added 4 commits December 7, 2025 16:30
…PI v2

Migrate ScienceDirect from FulltextFetcher-only to also support
PagedSearchBasedFetcher interface using Elsevier's API v2 with PUT method.

Changes:
- Implement PagedSearchBasedFetcher interface for search functionality
- Use HTTP PUT method with JSON request body (API v2 native approach)
- Add performSearchPaged() for paginated search results
- Add JSON request/response parsing for API v2 format
- Support pagination via offset/show parameters (max offset: 6000)
- Add proper error handling for 401/400 responses
- Parse authors, DOI, journal, volume, issue, pages from JSON response

The ScienceDirect Search API v1 was deprecated in December 2018.
This implementation follows Elsevier's recommended API v2 approach.

API docs: https://dev.elsevier.com/
@github-actions
Copy link
Contributor

github-actions bot commented Dec 8, 2025

Hey @Siva-Sai22! 👋

Thank you for contributing to JabRef!

We have automated checks in place, based on which you will soon get feedback if any of them are failing.

After all automated checks pass, a maintainer will also review your contribution. Once that happens, you can go through their comments in the "Files changed" tab and act on them, or reply to the conversation if you have further inputs.

Please re-check our AI Usage Policy to ensure that your pull request is in line with it. It also contains links to our contribution guide in case of any other doubts related to our contribution workflow.

@Siva-Sai22 Siva-Sai22 changed the title Fix/#14421 feat(fetcher): implement SearchBasedFetcher for ScienceDirect using scopus search API Dec 8, 2025
@github-actions github-actions bot added the status: changes-required Pull requests that are not yet complete label Dec 8, 2025
@github-actions github-actions bot removed the status: changes-required Pull requests that are not yet complete label Dec 8, 2025
Copy link
Member

@koppor koppor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to https://dev.elsevier.com/technical_documentation.html Scopus is not ScienceDirect.

I see two options:

  1. To avoid confusion, ScienceDirect should be renamed to Elsevier. But then, we have a "big ball of mud", because the methods in that class are not coupled, but only semnatically correlated
  2. Add new functionality to Scopus - this should make maintainence easier. This fetcher should be added to the list of fetchers in the "Web Search" prefernces

Please do option 2 to have the fetcher class small.

  • API key should renamed from "Scopus" to "Esevier", since it covers both.

Future work: Split up API key configuration and fetcher enablement.

grafik

@github-actions github-actions bot added the status: changes-required Pull requests that are not yet complete label Dec 8, 2025
@Siedlerchr Siedlerchr added status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers and removed status: changes-required Pull requests that are not yet complete labels Dec 8, 2025
Comment on lines +227 to +231
String response = new BufferedReader(new InputStreamReader(inputStream))
.lines()
.collect(Collectors.joining(OS.NEWLINE));

JSONObject jsonObject = new JSONObject(response);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There must be a method to convert an inputstream to a JSON object directly. But maybe, not mit dem JSONObject.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

** JSONObject response = JsonReader.toJsonObject(inputStream);
**

@koppor
Copy link
Member

koppor commented Dec 8, 2025

Another view - from

grafik

@koppor koppor added status: changes-required Pull requests that are not yet complete and removed status: devcall status: ready-for-review Pull Requests that are ready to be reviewed by the maintainers labels Dec 8, 2025
@Siva-Sai22
Copy link
Author

Siva-Sai22 commented Dec 9, 2025

  • API key should renamed from "Scopus" to "Esevier", since it covers both.

@koppor
But the ScienceDirect is accessing the api key using the FETCHER_NAME (ScienceDirect). If we rename it to Esevier we have to change the fetcher name also
importerPreferences.getApiKey(getName())

I think it would be better to take a scopus api key also.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

first contrib status: changes-required Pull requests that are not yet complete

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make ScienceDirect a SearchBasedFetcher to be able to use an API key in the UI

3 participants