Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added Tutorial/docs/assets/webapp/create_dataset.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Tutorial/docs/assets/webapp/create_dataset_2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Tutorial/docs/assets/webapp/create_scenario.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Tutorial/docs/assets/webapp/created_dataset.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Tutorial/docs/assets/webapp/created_scenario.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Tutorial/docs/assets/webapp/empty_dataset.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Tutorial/docs/assets/webapp/empty_scenario.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added Tutorial/docs/assets/webapp/running_scenario.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
73 changes: 73 additions & 0 deletions Tutorial/docs/tutorials/hidden/adx-database.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Azure Data Explorer (ADX)

Azure Data Explorer (ADX) is a fast, fully managed data analytics service optimized for real-time analysis of large volumes of data.

## Purpose in CosmoTech Platform

ADX serves as the **data warehouse** for CosmoTech simulation results, storing:

- **ProbesMeasures** - Time-series data from simulation probes
- **ScenarioRun** - Execution details and run metadata
- **ScenarioMetadata** - Scenario configurations and parameters
- **SimulationTotalFacts** - Aggregated metrics and KPIs

## Key Benefits

- **High Performance** - Query billions of records in seconds
- **Real-time Ingestion** - Data flows from Event Hub to ADX automatically
- **Time-Series Optimized** - Built for temporal data analysis
- **KQL Language** - Powerful Kusto Query Language for data exploration
- **Power BI Integration** - Direct connection for visualization
- **Scalability** - Handles massive simulation datasets efficiently

## Data Flow

1. **Simulation runs** and generates output data
2. **Event Hub** receives and streams the data
3. **ADX** ingests data in real-time via connectors
4. **Data is stored** in structured tables with compression
5. **Users query** the data using KQL or visualize in Power BI

## Common Use Cases

1. **Analyze Simulation Results** - Query and compare scenario outcomes
2. **Track Metrics Over Time** - Monitor KPIs across multiple runs
3. **Generate Reports** - Create custom analytics dashboards
4. **Data Exploration** - Investigate simulation behavior and patterns
5. **Performance Analysis** - Identify bottlenecks and optimization opportunities

## Example Query

```kql
ProbesMeasures
| where SimulationRun == "run-abc123"
| where Timestamp >= ago(1h)
| summarize avg(Value) by bin(Timestamp, 5m), ProbeName
| render timechart
```

This query retrieves probe measurements from the last hour and displays average values in a time chart.

## Database Configuration

ADX databases are automatically created when a workspace is deployed with the following naming convention:

```
<organization_id>-<workspace_key>
```

Configuration includes:
- **Retention policies** - Default 365 days
- **Permissions** - User access controls
- **Initialization scripts** - Table schemas and mappings
- **Event Hub connectors** - Automatic data ingestion

## Integration with Power BI

ADX connects directly to Power BI, enabling:
- Real-time dashboard updates
- Interactive data exploration
- Custom visualizations
- Scenario comparison reports

Connect using the Azure Data Explorer connector in Power BI Desktop with your cluster URL and database name.
284 changes: 284 additions & 0 deletions Tutorial/docs/tutorials/hidden/babylon.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,284 @@
# Babylon

Babylon is CosmoTech's proprietary command-line tool for deploying, configuring, and managing platform resources.

## Overview

Babylon is a deployment orchestration tool developed by CosmoTech that simplifies the process of creating and managing solutions, workspaces, datasets, and other platform resources. It uses declarative YAML manifests to define infrastructure and configurations, similar to Kubernetes.

## Key Features

- **Declarative Configuration** - Define resources in YAML manifests
- **State Management** - Track and manage resource states across environments
- **Idempotent Operations** - Apply configurations multiple times safely
- **Multi-Environment Support** - Manage dev, staging, and production environments
- **Version Control Friendly** - Store configurations in Git repositories
- **Template Support** - Use variables and templates for reusable configurations
- **Dependency Management** - Handle relationships between resources automatically

## Why Babylon?

### Before Babylon

Without Babylon, deploying CosmoTech resources required:
- Manual API calls for each resource
- Complex scripts to coordinate deployments
- Difficult tracking of resource states
- Error-prone configuration management
- No easy way to replicate environments

### With Babylon

Babylon provides:
- Single command deployment of entire environments
- Automatic handling of resource dependencies
- Consistent configuration across environments
- Easy rollback and version management
- Template-based configuration for reusability

## Core Concepts

### Namespaces

Namespaces organize and isolate resources:

- **`remote`** - Environment identifier (dev, staging, production)
- **`state_id`** - Unique identifier for this deployment state
- **`context`** - Logical grouping of related resources
- **`platform`** - Target CosmoTech platform URL

### Payload

The payload contains the actual resource configuration that will be sent to the CosmoTech API.

## Resource Types

### Solution

Define and deploy solutions:

```yaml
kind: Solution
namespace:
remote: production
state_id: "solution-v1"
context: "manufacturing"
platform: https://api.cosmotech.com
metadata:
selector:
organization_id: o-a1b2c3d4e5f6g7
spec:
payload:
key: manufacturing-sim
name: Manufacturing Simulation
repository: myregistry.azurecr.io/manufacturing-sim
version: 2.1.0
sdkVersion: "11.3"
tags:
- manufacturing
- optimization
parameters:
- id: ProductionRate
varType: number
runTemplates:
- id: Simulation
name: Run Simulation
computeSize: basic
```

### Workspace

Define workspaces with all configurations:

```yaml
kind: Workspace
namespace:
remote: production
state_id: "workspace-main"
context: "manufacturing"
platform: https://api.cosmotech.com
metadata:
selector:
organization_id: o-a1b2c3d4e5f6g7
solution_id: s-a1b2c3d4e5f6g7
spec:
sidecars:
azure:
adx:
database:
create: true
retention: 365
eventhub:
create: true
powerbi:
workspace:
name: Manufacturing Analytics
payload:
key: main-workspace
name: Main Workspace
description: Production workspace for manufacturing
useDedicatedEventHubNamespace: true
datasetCopy: false
```

### Dataset

Manage datasets:

```yaml
kind: Dataset
namespace:
remote: production
state_id: "dataset-reference"
context: "manufacturing"
platform: https://api.cosmotech.com
metadata:
selector:
organization_id: o-a1b2c3d4e5f6g7
spec:
payload:
name: Reference Dataset
description: Base data for simulations
connector:
id: c-ADTConnector
parametersValues:
AZURE_DIGITAL_TWINS_URL: https://myadt.api.weu.digitaltwins.azure.net
```

## Using Templates and Variables

### Variable Substitution

Babylon supports Jinja2-style variable substitution:

```yaml
spec:
payload:
key: {{workspace_key}}
name: {{workspace_name}}
description: {{workspace_description}}
solution:
solutionId: {{services['api.solution_id']}}
```

### Variable Files

Define variables in separate files:

**variables.yaml:**
```yaml
workspace_key: production-workspace
workspace_name: Production Workspace
workspace_description: Main production environment
organization_id: o-a1b2c3d4e5f6g7
powerbi_workspace_name: Analytics Workspace
```

### Environment-Specific Variables

Organize variables by environment:

```
config/
├── common.yaml
├── dev/
│ └── variables.yaml
├── staging/
│ └── variables.yaml
└── production/
└── variables.yaml
```
## State Management

### State Files

Babylon maintains state files to track deployed resources:

```
.babylon/
├── states/
│ ├── production/
│ │ ├── solution-v1.state
│ │ └── workspace-main.state
│ └── dev/
│ ├── solution-v1.state
│ └── workspace-dev.state
```

### State Commands

```bash
# List states
babylon state list

# Show state details
babylon state show production/solution-v1

# Remove state (without deleting resource)
babylon state rm production/solution-v1

# Import existing resource into state
babylon state import solution my-solution --organization o-a1b2c3d4e5f6g7
```

## Sidecars

Sidecars are Azure resource configurations managed alongside platform resources:

### Azure Data Explorer (ADX)

```yaml
sidecars:
azure:
adx:
database:
create: true
retention: 365
permissions:
- email: user@company.com
role: Admin
scripts:
- id: init
name: 00-Initialization.kql
path: "adx/scripts"
```

### Event Hub

```yaml
sidecars:
azure:
eventhub:
create: true
consumers:
- displayName: adx
entity: ProbesMeasures
connectors:
- table_name: ProbesMeasures
consumer_group: adx
connection_name: ProbesMeasures
database_target: "{{organization_id}}-{{workspace_key}}"
format: JSON
compression: Gzip
```

### Power BI

```yaml
sidecars:
azure:
powerbi:
workspace:
name: Analytics Workspace
reports:
- name: Dashboard
type: scenario
path: "powerbi/Dashboard.pbix"
tag: main
parameters:
- id: ADX_Cluster
value: "https://mycluster.westeurope.kusto.windows.net"
permissions:
- email: user@company.com
role: Admin
```
Loading