[FEATURE] New Extension Template Repo

[stephen-crawford]

Is your feature request related to a problem?

With the SDK, we offer support for creating custom extensions. As part of this process, a user needs to create a new repo and configure various settings to be compatible with OpenSearch.

I propose we create a new extension template where people can use the template for creating a new extension and not need to go through the process of grabbing the different configuration files.

I am curious to hear what defaults the maintainers would like to be included. I will submit a basic version for use shortly.

[dbwiddis]

Can you be more specific on what you want in this template?

I created a basic CRUD extension on my personal repo here: https://github.com/dbwiddis/CRUDExtension
And the process of creating it is fully documented here: https://github.com/opensearch-project/opensearch-sdk-java/blob/main/CREATE_YOUR_FIRST_EXTENSION.md

I'd be happy to move "my" code or some version of it somewhere as a template, or an other sample similar to hello world, etc. but it's not clear to me what you're asking for that's not covered by that document or the hello world example.

[stephen-crawford]

Hi @dbwiddis, something like your basic CRUD extension would be great. I am hoping to have a template where someone can use the GitHub repository template option to autofill a significant portion of the reusable code.

With the template, I think we can skip over at least the dependency declaration and port assignment steps of the CREATE_YOUR_FIRST_EXTENSION document. I understand why they are there but I am trying to get a template in place where a user could reasonably create a basic extension to complete some task within an hour or so.

Even just your CRUD extension would be a great step towards this as it gives the framework for us to be able to direct users to "put your code here." I think this would give us an "express" extension creation pattern that would be valuable for different community events like workshops, hackathons, and conferences.

Basically, I am looking to make a template where as much generic code as possible is already present to allow for reuse in local development scenarios. I.e. I am user and just testing the extension against my own fork for fun.

[stephen-crawford]

So this is what I came up with: https://github.com/scrawfor99/test_extension.

I took your CRUD Extension then swapped to gradle and used the gradle tasks framework to write a couple different scripts. The first lets someone rename the extension base components. Is you run ./gradlew setExtensionName the templateAction and templateExtension files will be swapped to <newName>Action and <newName>Extension. Similarly, the package will be swapped, the old one will be deleted, and the internal references to the past extension name (template in the default case) will be replaced with the new name.

I also created three generic starting points for a new extension. The first is just your CRUDExtension. The second is a BasicOutline where the templateAction file becomes replaced with a generic outline of the file:

package template;

import java.util.List;
import java.util.function.Function;

import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.rest.NamedRoute;
import org.opensearch.rest.RestRequest;
import org.opensearch.rest.RestRequest.Method;
import org.opensearch.rest.RestResponse;
import org.opensearch.core.rest.RestStatus;
import org.opensearch.sdk.ExtensionsRunner;
import org.opensearch.sdk.rest.BaseExtensionRestHandler;

public class templateAction extends BaseExtensionRestHandler {

    private OpenSearchClient client;

    public templateAction(ExtensionsRunner extensionsRunner) {
        this.client = extensionsRunner.getSdkClient().initializeJavaClient();
    }

    @Override
    public List<NamedRoute> routes() {
        return List.of(
                // Add custom routes here...
                new NamedRoute.Builder().method(Method.POST)
                        .path("/custom-route")
                        .uniqueName("custom_extension:custom-route")
                        .handler(customHandler)
                        .build()
        );
    }

    // Placeholder for Custom Handler
    Function<RestRequest, RestResponse> customHandler = (request) -> {
        // Implement your custom logic here...
        return createEmptyJsonResponse(request, RestStatus.OK);
    };

    // Add other placeholder methods for different types of handlers (create, read, update, delete).

    // Helper method to create a JSON response
    private RestResponse createJsonResponse(RestRequest request, RestStatus status, String key, Object value) {
        // Implement your JSON response logic here...
        return null;
    }
}

finally I also made a word counter extension

package template;

import java.io.IOException;
import java.util.List;
import java.util.function.Function;

import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch._types.OpenSearchException;
import org.opensearch.client.opensearch._types.Result;
import org.opensearch.client.opensearch.core.DeleteRequest;
import org.opensearch.client.opensearch.core.DeleteResponse;
import org.opensearch.client.opensearch.core.GetRequest;
import org.opensearch.client.opensearch.core.GetResponse;
import org.opensearch.client.opensearch.core.IndexRequest;
import org.opensearch.client.opensearch.core.IndexResponse;
import org.opensearch.client.opensearch.core.UpdateRequest;
import org.opensearch.client.opensearch.core.UpdateResponse;
import org.opensearch.client.opensearch.indices.CreateIndexRequest;
import org.opensearch.client.opensearch.indices.ExistsRequest;
import org.opensearch.client.transport.endpoints.BooleanResponse;
import org.opensearch.rest.NamedRoute;
import org.opensearch.rest.RestRequest;
import org.opensearch.rest.RestRequest.Method;
import org.opensearch.rest.RestResponse;
import org.opensearch.core.rest.RestStatus;
import org.opensearch.sdk.ExtensionsRunner;
import org.opensearch.sdk.rest.BaseExtensionRestHandler;

/**
 * MyCustomAction: A custom OpenSearch extension action for handling custom routes.
 */
public class templateAction extends BaseExtensionRestHandler {

    private OpenSearchClient client;

    public templateAction(ExtensionsRunner extensionsRunner) {
        this.client = extensionsRunner.getSdkClient().initializeJavaClient();
    }

    @Override
    public List<NamedRoute> routes() {
        return List.of(
                new NamedRoute.Builder().method(Method.POST)
                        .path("/count-words")
                        .uniqueName("template_extension:word-counter")
                        .handler(wordCounterHandler)
                        .build(),
                // Add more custom routes here...
        );
    }

    // Custom Handler for Word Count
    Function<RestRequest, RestResponse> wordCounterHandler = (request) -> {
        RestResponse response;
        String text = request.content().utf8ToString();
        int wordCount = countWords(text);

        String responseJson = "{\"word_count\": " + wordCount + "}";
        return createJsonResponse(request, RestStatus.OK, "success", responseJson);
    };

    // Placeholder for Custom Create Handler
    // Replace this with your own custom logic
    Function<RestRequest, RestResponse> customCreateHandler = (request) -> {
        // Implement your logic here...
        return createEmptyJsonResponse(request, RestStatus.OK);
    };

    // Placeholder for Custom Read Handler
    // Replace this with your own custom logic
    Function<RestRequest, RestResponse> customReadHandler = (request) -> {
        // Implement your logic here...
        return createEmptyJsonResponse(request, RestStatus.OK);
    };

    // Placeholder for Custom Update Handler
    // Replace this with your own custom logic
    Function<RestRequest, RestResponse> customUpdateHandler = (request) -> {
        // Implement your logic here...
        return createEmptyJsonResponse(request, RestStatus.OK);
    };

    // Placeholder for Custom Delete Handler
    // Replace this with your own custom logic
    Function<RestRequest, RestResponse> customDeleteHandler = (request) -> {
        // Implement your logic here...
        return createEmptyJsonResponse(request, RestStatus.OK);
    };

    // Helper method to count words in the text
    private int countWords(String text) {
        if (text == null || text.isEmpty()) {
            return 0;
        }

        String[] words = text.split("\\s+");
        return words.length;
    }
}

Each of these basic starting points can be triggered with ./gradlew useCRUDExtension, ./gradlew useBasicOutline, and ./gradlew useWordCounter accordingly.

So someone can first pick a starting template then automatically set all the values for the name of their extension from there. You can also continually rename the extension with the extension naming script. Once you rename it, running the use<target> commands will not work however since they overwrite the generic templateAction file.

[stephen-crawford]

It would also be nice if we could create a path for connecting a frontend. I will look into how this may be accomplished as well as what other scripts would be useful for getting started.

[dbwiddis]

Great work, @scrawfor99 ! I love the script idea.

One small tweak, I'm not sure we're going to keep the client initialization via SDK. I've argued for its removal (See #625 and linked issues there).

From a "create your first extension" standpoint it's easy, we just point people to the docs, but with a template you're going to need to actually import the documentation code. (That's really all the SDK does, btw, is copy one of those code bits.)

[dbwiddis]

Hey @scrawfor99 having just created a new plugin based on https://github.com/opensearch-project/opensearch-plugin-template-java I think this is probably the way to go. WDYT?