Skip to content

Docs update for PR #5 on Promptless/ray-test #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
promptless wants to merge 1 commit into from
Closed

Docs update for PR #5 on Promptless/ray-test #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 1 addition & 7 deletions doc/source/serve/model_composition.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,12 +113,7 @@ Note how the response from the `Adder` handle passes directly to the `Multiplier

## Streaming DeploymentHandle calls

You can also use `DeploymentHandles` to make streaming method calls that return multiple outputs.
To make a streaming call, the method must be a generator and you must set `handle.options(stream=True)`.
Then, the handle call returns a {mod}`DeploymentResponseGenerator <ray.serve.handle.DeploymentResponseGenerator>` instead of a unary `DeploymentResponse`.
You can use `DeploymentResponseGenerators` as a sync or async generator, like in an `async for` code block.
Similar to `DeploymentResponse.result()`, avoid using a `DeploymentResponseGenerator` as a sync generator within a deployment, as that blocks other requests from executing concurrently on that replica.
Note that you can't pass `DeploymentResponseGenerators` to other handle calls.
You can also use `DeploymentHandles` to make streaming method calls that return multiple outputs. To make a streaming call, the method must be a generator and you must set `handle.options(stream=True)`. Then, the handle call returns a {mod}`DeploymentResponseGenerator <ray.serve.handle.DeploymentResponseGenerator>` instead of a unary `DeploymentResponse`. You can use `DeploymentResponseGenerators` as a sync or async generator, like in an `async for` code block. Similar to `DeploymentResponse.result()`, avoid using a `DeploymentResponseGenerator` as a sync generator within a deployment, as that blocks other requests from executing concurrently on that replica. Note that you can't pass `DeploymentResponseGenerators` to other handle calls. If you have a use case requiring this feature, please file a feature request on GitHub.

Example:

Expand All @@ -127,7 +122,6 @@ Example:
:end-before: __streaming_example_end__
:language: python
```

## Advanced: Pass a DeploymentResponse in a nested object [FULLY DEPRECATED]

:::{warning}
Expand Down
83 changes: 41 additions & 42 deletions doc/source/serve/tutorials/batch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ This function must also be `async def` so that you can handle multiple queries c
```python
@serve.batch
async def my_batch_handler(self, requests: List):
pass
pass
```

The batch handler can then be called from another `async def` method in your deployment.
Expand All @@ -43,15 +43,15 @@ they were a normal function call:

```python
class BatchingDeployment:
@serve.batch
async def my_batch_handler(self, requests: List):
results = []
for request in requests:
results.append(request.json())
return results

async def __call__(self, request):
return await self.my_batch_handler(request)
@serve.batch
async def my_batch_handler(self, requests: List):
results = []
for request in requests:
results.append(request.json())
return results

async def __call__(self, request):
return await self.my_batch_handler(request)
```

:::{note}
Expand Down Expand Up @@ -97,20 +97,20 @@ import numpy as np

@ray.remote
def send_query(text):
resp = requests.get("http://localhost:8000/?text={}".format(text))
return resp.text
resp = requests.get("http://localhost:8000/?text={}".format(text))
return resp.text

# Use Ray to send all queries in parallel
texts = [
'Once upon a time,',
'Hi my name is Lewis and I like to',
'My name is Mary, and my favorite',
'My name is Clara and I am',
'My name is Julien and I like to',
'Today I accidentally',
'My greatest wish is to',
'In a galaxy far far away',
'My best talent is',
'Once upon a time,',
'Hi my name is Lewis and I like to',
'My name is Mary, and my favorite',
'My name is Clara and I am',
'My name is Julien and I like to',
'Today I accidentally',
'My greatest wish is to',
'In a galaxy far far away',
'My best talent is',
]
results = ray.get([send_query.remote(text) for text in texts])
print("Result returned:", results)
Expand All @@ -124,23 +124,23 @@ query independently, Ray Serve evaluates them in batches.
(pid=...) Our input array has length: 4
(pid=...) Our input array has length: 4
Result returned: [
'Once upon a time, when I got to look at and see the work of my parents (I still can\'t stand them,) they said, "Boys, you\'re going to like it if you\'ll stay away from him or make him look',
'Once upon a time, when I got to look at and see the work of my parents (I still can\'t stand them,) they said, "Boys, you\'re going to like it if you\'ll stay away from him or make him look',

"Hi my name is Lewis and I like to look great. When I'm not playing against, it's when I play my best and always feel most comfortable. I get paid by the same people who make my games, who work hardest for me.",
"Hi my name is Lewis and I like to look great. When I'm not playing against, it's when I play my best and always feel most comfortable. I get paid by the same people who make my games, who work hardest for me.",

"My name is Mary, and my favorite person in these two universes, the Green Lantern and the Red Lantern, are the same, except they're two of the Green Lanterns, but they also have their own different traits. Now their relationship is known",
"My name is Mary, and my favorite person in these two universes, the Green Lantern and the Red Lantern, are the same, except they're two of the Green Lanterns, but they also have their own different traits. Now their relationship is known",

'My name is Clara and I am married and live in Philadelphia. I am an English language teacher and translator. I am passionate about the issues that have so inspired me and my journey. My story begins with the discovery of my own child having been born',
'My name is Clara and I am married and live in Philadelphia. I am an English language teacher and translator. I am passionate about the issues that have so inspired me and my journey. My story begins with the discovery of my own child having been born',

'My name is Julien and I like to travel with my son on vacations... In fact I really prefer to spend more time with my son."\n\nIn 2011, the following year he was diagnosed with terminal Alzheimer\'s disease, and since then,',
'My name is Julien and I like to travel with my son on vacations... In fact I really prefer to spend more time with my son."\n\nIn 2011, the following year he was diagnosed with terminal Alzheimer\'s disease, and since then,',

"Today I accidentally got lost and went on another tour in August. My story was different, but it had so many emotions that it made me happy. I'm proud to still be able to go back to Oregon for work.\n\nFor the longest",
"Today I accidentally got lost and went on another tour in August. My story was different, but it had so many emotions that it made me happy. I'm proud to still be able to go back to Oregon for work.\n\nFor the longest",

'My greatest wish is to return your loved ones to this earth where they can begin their own free and prosperous lives. This is true only on occasion as it is not intended or even encouraged to be so.\n\nThe Gospel of Luke 8:29',
'My greatest wish is to return your loved ones to this earth where they can begin their own free and prosperous lives. This is true only on occasion as it is not intended or even encouraged to be so.\n\nThe Gospel of Luke 8:29',

'In a galaxy far far away, the most brilliant and powerful beings known would soon enter upon New York, setting out to restore order to the state. When the world turned against them, Darth Vader himself and Obi-Wan Kenobi, along with the Jedi',
'In a galaxy far far away, the most brilliant and powerful beings known would soon enter upon New York, setting out to restore order to the state. When the world turned against them, Darth Vader himself and Obi-Wan Kenobi, along with the Jedi',

'My best talent is that I can make a movie with somebody who really has a big and strong voice. I do believe that they would be great writers. I can tell you that to make sure."\n\n\nWith this in mind, "Ghostbusters'
'My best talent is that I can make a movie with somebody who really has a big and strong voice. I do believe that they would be great writers. I can tell you that to make sure."\n\n\nWith this in mind, "Ghostbusters'
]
```

Expand All @@ -156,8 +156,7 @@ to the Python script `tutorial_batch.py`:
```python
from ray.serve.handle import DeploymentHandle

handle: DeploymentHandle = serve.run(generator)
)
handle: DeploymentHandle = serve.run(generator, _local_testing_mode=True)
```

Generally, to enqueue a query, you can call `handle.method.remote(data)`. This call
Expand All @@ -166,15 +165,15 @@ retrieve the result. Add the following to the same Python script.

```python
input_batch = [
'Once upon a time,',
'Hi my name is Lewis and I like to',
'My name is Mary, and my favorite',
'My name is Clara and I am',
'My name is Julien and I like to',
'Today I accidentally',
'My greatest wish is to',
'In a galaxy far far away',
'My best talent is',
'Once upon a time,',
'Hi my name is Lewis and I like to',
'My name is Mary, and my favorite',
'My name is Clara and I am',
'My name is Julien and I like to',
'Today I accidentally',
'My greatest wish is to',
'In a galaxy far far away',
'My best talent is',
]
print("Input batch is", input_batch)

Expand All @@ -189,4 +188,4 @@ Finally, run the script.
$ python tutorial_batch.py
```

You should get an output similar to the previous example.
You should get an output similar to the previous example.