DEVELOPER KNOWLEDGE BASE

NEXUS AI Documentation & CLI Guide

Learn how to compile models with TensorRT-LLM, serve high-throughput endpoints via vLLM, and integrate with the Python SDK.

1. Install the NEXUS CLI

The NEXUS CLI interacts directly with your local NVIDIA drivers to inspect GPU VRAM and launch quantized container runtimes.

# Install via Homebrew or curl
curl -sSL https://nexus-ai.com/install.sh | bash

# Authenticate with your Pro API key
nexus auth login --token nex_live_xxxxxxxx

2. Launch Instant vLLM Server

Run any model from the NEXUS / Hugging Face catalog with continuous batching and PagedAttention:

docker run --gpus all -p 8000:8000 --ipc=host vllm/vllm-openai:latest   --model meta-llama/Meta-Llama-3.1-70B-Instruct   --quantization fp8   --max-model-len 16384   --tensor-parallel-size 2   --gpu-memory-utilization 0.95

3. Compile to TensorRT-LLM Engine

TensorRT-LLM provides maximum FLOPS utilization on NVIDIA Hopper (H100) and Blackwell (B200) architectures:

# Convert weights to TensorRT-LLM checkpoint format
python3 convert_checkpoint.py --model_dir ./Meta-Llama-3.1-70B   --output_dir ./tllm_checkpoint   --dtype fp8

# Build the optimized binary engine
trtllm-build --checkpoint_dir ./tllm_checkpoint   --output_dir ./engine   --gemm_plugin auto