Deploying to gh-pages from @ dstackai/dstack@40cb4a0 🚀

Author: peterschmidt85

assets/images/social/examples.png

@@ -1056,6 +1056,19 @@
     </span>
   </a>
 
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#router" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        <span class="md-typeset">
+          Router
+        </span>
+      
+    </span>
+  </a>
+  
 </li>
 
           <li class="md-nav__item">
@@ -3884,6 +3897,19 @@
     </span>
   </a>
 
+</li>
+        
+          <li class="md-nav__item">
+  <a href="#router" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        <span class="md-typeset">
+          Router
+        </span>
+      
+    </span>
+  </a>
+  
 </li>
 
           <li class="md-nav__item">
@@ -4045,12 +4071,10 @@
 
 
 <h1 id="gateways">Gateways<a class="headerlink" href="#gateways" title="Permanent link">&para;</a></h1>
-<p>Gateways manage the ingress traffic of running <a href="../services/">services</a>,
-provide an HTTPS endpoint mapped to your domain, handle auto-scaling and rate limits.</p>
-<blockquote>
-<p>If you're using <a href="https://sky.dstack.ai" target="_blank">dstack Sky <span class="twemoji external"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m11.93 5 2.83 2.83L5 17.59 6.42 19l9.76-9.75L19 12.07V5z"/></svg></span></a>,
-the gateway is already set up for you.</p>
-</blockquote>
+<p>Gateways manage ingress traffic for running <a href="../services/">services</a>, handle auto-scaling and rate limits, enable HTTPS, and allow you to configure a custom domain. They also support custom routers, such as the <a href="https://docs.sglang.ai/advanced_features/router.html#" target="_blank">SGLang Model Gateway <span class="twemoji external"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m11.93 5 2.83 2.83L5 17.59 6.42 19l9.76-9.75L19 12.07V5z"/></svg></span></a>.</p>
+<!-- > If you're using [dstack Sky :material-arrow-top-right-thin:{ .external }](https://sky.dstack.ai){:target="_blank"},
+> the gateway is already set up for you. -->
+
 <h2 id="apply-a-configuration">Apply a configuration<a class="headerlink" href="#apply-a-configuration" title="Permanent link">&para;</a></h2>
 <p>First, define a gateway configuration as a YAML file in your project folder.
 The filename must end with <code>.dstack.yml</code> (e.g. <code>.dstack.yml</code> or <code>gateway.dstack.yml</code> are both acceptable).</p>
@@ -4094,6 +4118,42 @@ <h3 id="backend">Backend<a class="headerlink" href="#backend" title="Permanent l
 <p>Gateways in <code>kubernetes</code> backend require an external load balancer. Managed Kubernetes solutions usually include a load balancer.
 For self-hosted Kubernetes, you must provide a load balancer by yourself.</p>
 </details>
+<h3 id="router">Router<a class="headerlink" href="#router" title="Permanent link">&para;</a></h3>
+<p>By default, the gateway uses its own load balancer to route traffic between replicas. However, you can delegate this responsibility to a specific router by setting the <code>router</code> property. Currently, the only supported external router is <code>sglang</code>.</p>
+<h4 id="sglang">SGLang<a class="headerlink" href="#sglang" title="Permanent link">&para;</a></h4>
+<p>The <code>sglang</code> router delegates routing logic to the <a href="https://docs.sglang.ai/advanced_features/router.html#" target="_blank">SGLang Model Gateway <span class="twemoji external"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m11.93 5 2.83 2.83L5 17.59 6.42 19l9.76-9.75L19 12.07V5z"/></svg></span></a>.</p>
+<p>To enable it, set <code>type</code> field under <code>router</code> to <code>sglang</code>:</p>
+<div editor-title="gateway.dstack.yml">
+
+<div class="highlight"><pre><span></span><code><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">gateway</span>
+<span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">sglang-gateway</span>
+
+<span class="nt">backend</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">aws</span>
+<span class="nt">region</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">eu-west-1</span>
+
+<span class="nt">domain</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">example.com</span>
+
+<span class="nt">router</span><span class="p">:</span>
+<span class="w">  </span><span class="nt">type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">sglang</span>
+<span class="w">  </span><span class="nt">policy</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">cache_aware</span>
+</code></pre></div>
+
+</div>
+
+<div class="admonition info">
+<p class="admonition-title">Policy</p>
+<p>The <code>router</code> property allows you to configure the routing <code>policy</code>:</p>
+<ul>
+<li><code>cache_aware</code> &mdash; Default policy; combines cache locality with load balancing, falling back to shortest queue. </li>
+<li><code>power_of_two</code> &mdash; Samples two workers and picks the lighter one.                                               </li>
+<li><code>random</code> &mdash; Uniform random selection.                                                                    </li>
+<li><code>round_robin</code> &mdash; Cycles through workers in order.                                                             </li>
+</ul>
+</div>
+<blockquote>
+<p>Currently, services using this type of gateway must run standard SGLang workers. See the <a href="../../../examples/inference/sglang/">example</a>.</p>
+<p>Support for prefill/decode disaggregation and auto-scaling based on inter-token latency is coming soon.</p>
+</blockquote>
 <h3 id="public-ip">Public IP<a class="headerlink" href="#public-ip" title="Permanent link">&para;</a></h3>
 <p>If you don't need/want a public IP for the gateway, you can set the <code>public_ip</code> to <code>false</code> (the default value is <code>true</code>), making the gateway private.
 Private gateways are currently supported in <code>aws</code> and <code>gcp</code> backends.</p>

assets/images/social/partners.png

@@ -4486,12 +4486,13 @@ <h2 id="apply-a-configuration">Apply a configuration<a class="headerlink" href="
 <p>However, you'll need a gateway in the following cases:</p>
 <ul>
 <li>To use auto-scaling or rate limits</li>
+<li>To enable a support custom router, e.g. such as the <a href="https://docs.sglang.ai/advanced_features/router.html#" target="_blank">SGLang Model Gateway <span class="twemoji external"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m11.93 5 2.83 2.83L5 17.59 6.42 19l9.76-9.75L19 12.07V5z"/></svg></span></a></li>
 <li>To enable HTTPS for the endpoint and map it to your domain</li>
 <li>If your service requires WebSockets</li>
 <li>If your service cannot work with a <a href="#path-prefix">path prefix</a></li>
 </ul>
-<p>Note, if you're using <a href="https://sky.dstack.ai" target="_blank">dstack Sky <span class="twemoji external"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="m11.93 5 2.83 2.83L5 17.59 6.42 19l9.76-9.75L19 12.07V5z"/></svg></span></a>,
-a gateway is already pre-configured for you.</p>
+<!-- Note, if you're using <a href="https://sky.dstack.ai">dstack Sky klzzwxh:0160{ .external }</a>{:target="_blank"},
+a gateway is already pre-configured for you. -->
 <p>If a <a href="../gateways/">gateway</a> is configured, the service endpoint will be accessible at
 <code>https://&lt;run name&gt;.&lt;gateway domain&gt;/</code>.</p>
 <p>If the service defines the <code>model</code> property, the model will be available via the global OpenAI-compatible endpoint 

docs/concepts/gateways/index.html

@@ -3862,13 +3862,14 @@ <h2 id="usage">Usage<a class="headerlink" href="#usage" title="Permanent link">&
 <div class="termy">
 
 <div class="highlight"><pre><span></span><code>$<span class="w"> </span>dstack<span class="w"> </span>server<span class="w"> </span>--help
-Usage:<span class="w"> </span>dstack<span class="w"> </span>server<span class="w"> </span><span class="o">[</span>-h<span class="o">]</span><span class="w"> </span><span class="o">[</span>--host<span class="w"> </span>HOST<span class="o">]</span><span class="w"> </span><span class="o">[</span>-p<span class="w"> </span>PORT<span class="o">]</span><span class="w"> </span><span class="o">[</span>-l<span class="w"> </span>LOG_LEVEL<span class="o">]</span><span class="w"> </span><span class="o">[</span>-y<span class="o">]</span><span class="w"> </span><span class="o">[</span>-n<span class="o">]</span>
-<span class="w">                     </span><span class="o">[</span>--token<span class="w"> </span>TOKEN<span class="o">]</span>
+Usage:<span class="w"> </span>dstack<span class="w"> </span>server<span class="w"> </span><span class="o">[</span>-h<span class="o">]</span><span class="w"> </span><span class="o">[</span>--host<span class="w"> </span>HOST<span class="o">]</span><span class="w"> </span><span class="o">[</span>-p<span class="w"> </span>PORT<span class="o">]</span><span class="w"> </span><span class="o">[</span>-d<span class="w"> </span><span class="p">|</span><span class="w"> </span>-l<span class="w"> </span>LOG_LEVEL<span class="o">]</span><span class="w"> </span><span class="o">[</span>-y<span class="o">]</span>
+<span class="w">                     </span><span class="o">[</span>-n<span class="o">]</span><span class="w"> </span><span class="o">[</span>--token<span class="w"> </span>TOKEN<span class="o">]</span>
 
 Options:
 <span class="w">  </span>-h,<span class="w"> </span>--help<span class="w">            </span>Show<span class="w"> </span>this<span class="w"> </span><span class="nb">help</span><span class="w"> </span>message<span class="w"> </span>and<span class="w"> </span><span class="nb">exit</span>
 <span class="w">  </span>--host<span class="w"> </span>HOST<span class="w">           </span>Bind<span class="w"> </span>socket<span class="w"> </span>to<span class="w"> </span>this<span class="w"> </span>host.<span class="w"> </span>Defaults<span class="w"> </span>to<span class="w"> </span><span class="m">127</span>.0.0.1
 <span class="w">  </span>-p,<span class="w"> </span>--port<span class="w"> </span>PORT<span class="w">       </span>Bind<span class="w"> </span>socket<span class="w"> </span>to<span class="w"> </span>this<span class="w"> </span>port.<span class="w"> </span>Defaults<span class="w"> </span>to<span class="w"> </span><span class="m">3000</span>.
+<span class="w">  </span>-d,<span class="w"> </span>--debug<span class="w">           </span>Enable<span class="w"> </span>debug<span class="w"> </span>logging<span class="w"> </span>level<span class="w"> </span><span class="o">(</span>same<span class="w"> </span>as<span class="w"> </span>-l<span class="w"> </span>debug<span class="o">)</span>
 <span class="w">  </span>-l,<span class="w"> </span>--log-level<span class="w"> </span>LOG_LEVEL
 <span class="w">                        </span>Server<span class="w"> </span>logging<span class="w"> </span>level.<span class="w"> </span>Defaults<span class="w"> </span>to<span class="w"> </span>INFO.
 <span class="w">  </span>-y,<span class="w"> </span>--yes<span class="w">             </span>Don<span class="s1">&#39;t ask for confirmation (e.g. update the config)</span>

docs/concepts/services/index.html

@@ -1520,6 +1520,17 @@
     <nav class="md-nav" aria-label="Root reference">
       <ul class="md-nav__list">
 
+          <li class="md-nav__item">
+  <a href="#router" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        router
+      
+    </span>
+  </a>
+  
+</li>
+        
           <li class="md-nav__item">
   <a href="#certificate" class="md-nav__link">
     <span class="md-ellipsis">
@@ -3776,6 +3787,17 @@
     <nav class="md-nav" aria-label="Root reference">
       <ul class="md-nav__list">
 
+          <li class="md-nav__item">
+  <a href="#router" class="md-nav__link">
+    <span class="md-ellipsis">
+      
+        router
+      
+    </span>
+  </a>
+  
+</li>
+        
           <li class="md-nav__item">
   <a href="#certificate" class="md-nav__link">
     <span class="md-ellipsis">
@@ -3902,8 +3924,17 @@ <h6 class="reference-item" id="domain"><code>domain</code> - (Optional) The gate
 <h6 class="reference-item" id="public_ip"><code>public_ip</code> - (Optional) Allocate public IP for the gateway. Defaults to <code>True</code>.<a class="headerlink" href="#public_ip" title="Permanent link">&para;</a></h6>
 <h6 class="reference-item" id="_certificate"><a href="#certificate"><code>certificate</code></a> - (Optional) The SSL certificate configuration. Defaults to <code>type: lets-encrypt</code>.<a class="headerlink" href="#_certificate" title="Permanent link">&para;</a></h6>
 <h6 class="reference-item" id="tags"><code>tags</code> - (Optional) The custom tags to associate with the gateway. The tags are also propagated to the underlying backend resources. If there is a conflict with backend-level tags, does not override them.<a class="headerlink" href="#tags" title="Permanent link">&para;</a></h6>
+<h3 id="router"><code>router</code><a class="headerlink" href="#router" title="Permanent link">&para;</a></h3>
+<div class="tabbed-set tabbed-alternate" data-tabs="1:1"><input checked="checked" id="sglang-model-gateway" name="__tabbed_1" type="radio" /><div class="tabbed-labels"><label for="sglang-model-gateway">SGLang Model Gateway</label></div>
+<div class="tabbed-content">
+<div class="tabbed-block">
+<h6 class="reference-item" id="type"><code>type</code> -  The router type. Must be <code>sglang</code>.<a class="headerlink" href="#type" title="Permanent link">&para;</a></h6>
+<h6 class="reference-item" id="policy"><code>policy</code> - (Optional) The routing policy. Options: <code>random</code>, <code>round_robin</code>, <code>cache_aware</code>, <code>power_of_two</code>. Defaults to <code>cache_aware</code>.<a class="headerlink" href="#policy" title="Permanent link">&para;</a></h6>
+</div>
+</div>
+</div>
 <h3 id="certificate"><code>certificate</code><a class="headerlink" href="#certificate" title="Permanent link">&para;</a></h3>
-<div class="tabbed-set tabbed-alternate" data-tabs="1:2"><input checked="checked" id="lets-encrypt" name="__tabbed_1" type="radio" /><input id="acm" name="__tabbed_1" type="radio" /><div class="tabbed-labels"><label for="lets-encrypt">Let's encrypt</label><label for="acm">ACM</label></div>
+<div class="tabbed-set tabbed-alternate" data-tabs="2:2"><input checked="checked" id="lets-encrypt" name="__tabbed_2" type="radio" /><input id="acm" name="__tabbed_2" type="radio" /><div class="tabbed-labels"><label for="lets-encrypt">Let's encrypt</label><label for="acm">ACM</label></div>
 <div class="tabbed-content">
 <div class="tabbed-block">
 <h6 class="reference-item" id="type"><code>type</code> -  Automatic certificates by Let's Encrypt. Must be <code>lets-encrypt</code>.<a class="headerlink" href="#type" title="Permanent link">&para;</a></h6>