SAMLLMHost
SAM model of the LLMHost kind, generated from the Smarter platform’s JSON Schema.
Do not edit: run scripts/generate_sam_models.py. See smarter.models.sam.
- class smarter.models.sam.llmhost.SAMDependency(*, kind: str, name: str, **extra_data: Any)[source]
Bases:
SAMModelA resource that depends on the resource that a manifest describes.
- kind: str
Kind[String]: The SAM kind of the resource that depends on this resource, e.g. LLMClient.
Read only.
- model_config = {'extra': 'allow', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- class smarter.models.sam.llmhost.ApiFormat(*values)[source]
Bases:
EnumLLMHost.spec.engine.apiFormat[str]: the wire protocol.
One of: [‘openai_compatible’, ‘huggingface’, ‘ollama_native’, ‘custom’].
- class smarter.models.sam.llmhost.Name(*values)[source]
Bases:
EnumLLMHost.spec.engine.name[str]: the inference server.
One of: [‘vllm’, ‘tgi’, ‘sglang’, ‘llama_cpp’, ‘ollama’, ‘tei’, ‘custom’].
- class smarter.models.sam.llmhost.SAMLLMHostEngineConfig(*, apiFormat: ApiFormat | None = 'openai_compatible', args: list[str] | None = None, command: list[str] | None = None, contextLength: Annotated[int, None, Interval(gt=None, ge=1, lt=None, le=None), None] | None = None, dtype: str | None = None, env: dict[str, str] | None = None, gpuMemoryUtilization: Annotated[float, None, Interval(gt=0.0, ge=None, lt=None, le=1.0), None, None] | None = None, image: str | None = None, maxConcurrentRequests: Annotated[int, None, Interval(gt=None, ge=1, lt=None, le=None), None] | None = None, name: Name, port: Annotated[int, None, Interval(gt=None, ge=1, lt=None, le=65535), None] | None = None, tensorParallelSize: Annotated[int, None, Interval(gt=None, ge=1, lt=None, le=16), None] | None = None, trustRemoteCode: bool | None = False)[source]
Bases:
SAMModelSpec.engine: the inference server.
The common settings, e.g.
contextLength, are translated into each engine’s own arguments, e.g. vLLM’s--max-model-lenand llama.cpp’s--ctx-size.argsandenvare passed to the engine as they are, for everything else.- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- apiFormat: ApiFormat | None
LLMHost.spec.engine.apiFormat[str]: the wire protocol.
One of: [‘openai_compatible’, ‘huggingface’, ‘ollama_native’, ‘custom’].
- args: list[str] | None
LLMHost.spec.engine.args[list]: additional arguments, passed to the engine as they are.
- command: list[str] | None
LLMHost.spec.engine.command[list]: overrides the image’s entrypoint.
For custom engines.
- contextLength: conint(ge=1) | None
LLMHost.spec.engine.contextLength[int]: the maximum context the server allocates, in tokens.
- gpuMemoryUtilization: confloat(le=1.0, gt=0.0) | None
LLMHost.spec.engine.gpuMemoryUtilization[float]: the fraction of GPU memory the server may use.
- image: str | None
LLMHost.spec.engine.image[str]: the container image.
Defaults to the engine’s official image. Pin a version in production. Required for custom.
- maxConcurrentRequests: conint(ge=1) | None
LLMHost.spec.engine.maxConcurrentRequests[int]: the requests that one replica serves at once.
- name: Name
LLMHost.spec.engine.name[str]: the inference server.
One of: [‘vllm’, ‘tgi’, ‘sglang’, ‘llama_cpp’, ‘ollama’, ‘tei’, ‘custom’].
- port: conint(ge=1, le=65535) | None
LLMHost.spec.engine.port[int]: the container port.
Defaults to the engine’s port. Required for custom.
- class smarter.models.sam.llmhost.SAMLLMHostHealthCheck(*, path: str | None = None, periodSeconds: Annotated[int, None, Interval(gt=None, ge=1, lt=None, le=300), None] | None = 10, startupTimeoutSeconds: Annotated[int, None, Interval(gt=None, ge=60, lt=None, le=21600), None] | None = 3600)[source]
Bases:
SAMModelSpec.healthCheck: the probes that decide when a replica is ready.
- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- path: str | None
LLMHost.spec.healthCheck.path[str]: the HTTP health path.
Defaults to the engine’s, e.g. /health.
- class smarter.models.sam.llmhost.SAMLLMHostMetadata(*, annotations: None | list[dict[str, str | int | float | bool | date | datetime | UUID | bytes | list | dict[str, Any]]] = None, description: str | None, name: str, tags: list[str] | None = None, version: str | None, **extra_data: Any)[source]
Bases:
SAMModelSmarter API LLMHost Manifest - Metadata class.
- annotations: None | list[dict[str, str | int | float | bool | date | datetime | UUID | bytes | list | dict[str, Any]]]
The manifest annotations.
- Used for storing arbitrary metadata as
key-value pairs. Example: [{‘smarter.sh/test-manifest/project-name’: ‘Scooby dooby do’}]. The key should be a valid url-friendly string. The value accepts multi-line string values (YAML block scalars) and various scalar types including str, int, float, bool, datetime.date, datetime.datetime, decimal.Decimal, uuid.UUID, bytes, list, dict.
- tags: list[str] | None
The tags of the manifest.
Used for generic resource categorization and search. Example: [‘tag1’, ‘tag2’]
- model_config = {'extra': 'allow', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- class smarter.models.sam.llmhost.Quantization(*values)[source]
Bases:
EnumLLMHost.spec.model.quantization[str]: the weights’ precision.
One of: [‘none’, ‘fp32’, ‘fp16’, ‘bf16’, ‘fp8’, ‘mxfp4’, ‘int8’, ‘int4’, ‘gguf’, ‘awq’, ‘gptq’].
- class smarter.models.sam.llmhost.Source(*values)[source]
Bases:
EnumLLMHost.spec.model.source[str]: where the weights come from.
One of: [‘huggingface’, ‘ollama’, ‘s3’, ‘url’, ‘pvc’].
- class smarter.models.sam.llmhost.Task(*values)[source]
Bases:
EnumLLMHost.spec.model.task[str]: what the model does.
One of: [‘text-generation’, ‘embedding’].
- class smarter.models.sam.llmhost.SAMLLMHostModelCapabilities(*, functionCalling: bool | None = False, reasoning: bool | None = False, streaming: bool | None = True, vision: bool | None = False)[source]
Bases:
SAMModelWhat the model can do.
Informational, for discovery and for the LLMClients that use it.
- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- class smarter.models.sam.llmhost.SAMLLMHostNetwork(*, allowUnauthenticated: bool | None = False, apiKeySecret: str | None = None, hostname: str | None = None, ingress: bool | None = False)[source]
Bases:
SAMModelSpec.network: how the LLMHost is reached.
It is always reachable inside the cluster, at its Service. With
ingress, it is also reachable athttps://<hostname>, with a TLS certificate from cert-manager. A public endpoint requires an API key, unlessallowUnauthenticated.- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- allowUnauthenticated: bool | None
LLMHost.spec.network.allowUnauthenticated[bool]: permit an Ingress for an engine that does not authenticate requests, e.g. tgi and ollama.
Anyone who knows the hostname can use the model.
- apiKeySecret: str | None
LLMHost.spec.network.apiKeySecret[str]: the name of a Smarter Secret with the API key that clients must send as a Bearer token.
If omitted, Smarter generates one when the LLMHost is launched, and stores it in the Secret llmhost_<name>_api_key.
- class smarter.models.sam.llmhost.SAMLLMHostResources(*, cpu: str | None = None, gpuCount: Annotated[int, None, Interval(gt=None, ge=0, lt=None, le=16), None] | None = 0, gpuResource: str | None = 'nvidia.com/gpu', memory: str | None = None, nodeSelector: dict[str, str] | None = None, serviceAccountName: str | None = None, shmSize: str | None = '8Gi', tolerations: list[dict[str, Any]] | None = None, vramRequiredGb: Annotated[int, None, Interval(gt=None, ge=0, lt=None, le=None), None] | None = None)[source]
Bases:
SAMModelSpec.resources: the compute that each replica, one pod, requests.
These are the pod’s resource requests, not a node’s capacity. They must fit a node of the LLMHost’s compute, which also determines the GPU model.
- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- gpuCount: conint(ge=0, le=16) | None
LLMHost.spec.resources.gpuCount[int]: the GPUs per replica.
0 runs on CPU.
- gpuResource: str | None
LLMHost.spec.resources.gpuResource[str]: the Kubernetes extended resource, e.g. nvidia.com/gpu.
- memory: str | None
LLMHost.spec.resources.memory[str]: the pod’s memory request and limit, e.g. 24Gi.
- nodeSelector: dict[str, str] | None
LLMHost.spec.resources.nodeSelector[dict]: additional node labels.
- serviceAccountName: str | None
LLMHost.spec.resources.serviceAccountName[str]: e.g. for IRSA access to an S3 model source.
- class smarter.models.sam.llmhost.SAMLLMHostScaling(*, replicas: Annotated[int, None, Interval(gt=None, ge=1, lt=None, le=16), None] | None = 1)[source]
Bases:
SAMModelSpec.scaling: the replicas.
- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- class smarter.models.sam.llmhost.SAMLLMHostStatus(*, accountNumber: str, apiKeySecret: str | None = None, created: datetime, dependencies: list[SAMDependency] | None = None, deployedAt: datetime | None = None, endpoint: str | None = None, estimatedCost: float | str | None = None, healthy: bool | None = None, hostStatus: str | None = None, lastCheckedAt: datetime | None = None, message: str | None = None, modified: datetime, publicUrl: str | None = None, readyReplicas: int | None = None, recordLocator: str, replicas: int | None = None, uptimeHours: float | None = None, username: str, **extra_data: Any)[source]
Bases:
SAMModelSmarter API LLMHost Manifest - Status class.
Read only. Besides ownership, it reports the LLMHost’s lifecycle state, where it can be reached, and its accumulated cost, as of its most recent status check.
- apiKeySecret: str | None
LLMHost.status.apiKeySecret: the Smarter Secret with the API key, if any.
Read only.
- created: datetime
The date in which this resource was created.
Read only.
- dependencies: list[SAMDependency] | None
Dependencies[List]: The resources that depend on this resource, and that must be deleted, or stop referring to it, before it can be deleted.
Lists only the resources that you can read. Read only.
- endpoint: str | None
LLMHost.status.endpoint: the base URL inside the cluster, e.g. for LLMClients.
Read only.
- estimatedCost: float | str | None
LLMHost.status.estimatedCost: uptimeHours times replicas times each replica’s share of its compute’s node price. Read only.
- hostStatus: str | None
LLMHost.status.hostStatus: the lifecycle state, e.g. pending, active or error.
Read only.
- lastCheckedAt: datetime | None
LLMHost.status.lastCheckedAt: when its status was last checked.
Read only.
- model_config = {'extra': 'allow', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- message: str | None
LLMHost.status.message: why the LLMHost is in its state, e.g. a container error.
Read only.
- modified: datetime
The date in which this resource was most recently changed.
Read only.
- class smarter.models.sam.llmhost.AccessMode(*values)[source]
Bases:
EnumLLMHost.spec.storage.accessMode[str]: One of: [‘ReadWriteOnce’, ‘ReadWriteMany’, ‘ReadOnlyMany’].
- class smarter.models.sam.llmhost.SAMLLMHostStorage(*, accessMode: AccessMode | None = 'ReadWriteOnce', existingClaim: str | None = None, retain: bool | None = True, size: str | None = '50Gi', storageClass: str | None = None)[source]
Bases:
SAMModelSpec.storage: the model volume, a PersistentVolumeClaim mounted at /models.
It caches the weights, so that a restarted server does not download them again. The cluster’s storage class provisions its PersistentVolume.
- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- accessMode: AccessMode | None
LLMHost.spec.storage.accessMode[str]: One of: [‘ReadWriteOnce’, ‘ReadWriteMany’, ‘ReadOnlyMany’].
- existingClaim: str | None
LLMHost.spec.storage.existingClaim[str]: an existing PersistentVolumeClaim to mount, instead of creating one.
Required when model.source is pvc.
- class smarter.models.sam.llmhost.SAMLLMHostModel(*, architecture: str | None = None, capabilities: SAMLLMHostModelCapabilities | None = None, contextWindow: Annotated[int, None, Interval(gt=None, ge=0, lt=None, le=None), None] | None = None, embeddingDimensions: Annotated[int, None, Interval(gt=None, ge=0, lt=None, le=None), None] | None = None, file: str | None = None, license: str | None = None, parameterCount: Annotated[int, None, Interval(gt=None, ge=0, lt=None, le=None), None] | None = None, quantization: Quantization | None = 'none', repository: str, revision: str | None = None, servedName: str | None = None, source: Source | None = 'huggingface', task: Task | None = 'text-generation', tokenSecret: str | None = None)[source]
Bases:
SAMModelSpec.model: what is served, and where its weights come from.
- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- capabilities: SAMLLMHostModelCapabilities | None
LLMHost.spec.model.capabilities[object]: what the model can do.
- contextWindow: conint(ge=0) | None
LLMHost.spec.model.contextWindow[int]: the model’s maximum context, in tokens.
- embeddingDimensions: conint(ge=0) | None
LLMHost.spec.model.embeddingDimensions[int]: for embedding models.
- file: str | None
LLMHost.spec.model.file[str]: a single weights file within the repository, e.g. a GGUF file for llama_cpp.
For ollama with a Hugging Face source, the quantization tag, e.g. Q4_K_M.
- parameterCount: conint(ge=0) | None
LLMHost.spec.model.parameterCount[int]: the number of parameters.
- quantization: Quantization | None
LLMHost.spec.model.quantization[str]: the weights’ precision.
One of: [‘none’, ‘fp32’, ‘fp16’, ‘bf16’, ‘fp8’, ‘mxfp4’, ‘int8’, ‘int4’, ‘gguf’, ‘awq’, ‘gptq’].
- repository: str
LLMHost.spec.model.repository[str]: the model, in the source’s terms: a Hugging Face repository id, e.g. Qwen/Qwen3-8B; an Ollama tag, e.g. llama3.1:8b; an S3 prefix, e.g. s3://bucket/models/qwen3; a URL; or, for pvc, a path within the volume.
- revision: str | None
LLMHost.spec.model.revision[str]: a Hugging Face branch, tag or commit SHA.
Pin a commit SHA for reproducible deployments. Defaults to main.
- servedName: str | None
LLMHost.spec.model.servedName[str]: the model name that clients send in their requests.
Defaults to the LLMHost’s name. Ollama always uses repository.
- source: Source | None
LLMHost.spec.model.source[str]: where the weights come from.
One of: [‘huggingface’, ‘ollama’, ‘s3’, ‘url’, ‘pvc’].
- class smarter.models.sam.llmhost.SAMLLMHostSpec(*, compute: str | None = None, engine: SAMLLMHostEngineConfig, healthCheck: SAMLLMHostHealthCheck | None = None, model: SAMLLMHostModel, network: SAMLLMHostNetwork | None = None, resources: SAMLLMHostResources | None = None, scaling: SAMLLMHostScaling | None = None, storage: SAMLLMHostStorage | None = None)[source]
Bases:
SAMModelSmarter API LLMHost Manifest LLMHost.spec.
- model_config = {'extra': 'forbid', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- compute: str | None
LLMHost.spec.compute[str]: the LLMHostCompute that the LLMHost runs on, e.g. gpu_l4_1x.
Defaults to the cheapest one that its resources fit.
- engine: SAMLLMHostEngineConfig
LLMHost.spec.engine[object]: the inference server.
- healthCheck: SAMLLMHostHealthCheck | None
LLMHost.spec.healthCheck[object]: probes.
- model: SAMLLMHostModel
LLMHost.spec.model[object]: what is served.
- network: SAMLLMHostNetwork | None
LLMHost.spec.network[object]: reachability.
- resources: SAMLLMHostResources | None
LLMHost.spec.resources[object]: compute.
- scaling: SAMLLMHostScaling | None
LLMHost.spec.scaling[object]: replicas.
- storage: SAMLLMHostStorage | None
LLMHost.spec.storage[object]: the model volume.
- class smarter.models.sam.llmhost.SAMLLMHost(*, apiVersion: str, kind: str, metadata: SAMLLMHostMetadata, spec: SAMLLMHostSpec, status: SAMLLMHostStatus | None = None, **extra_data: Any)[source]
Bases:
SAMModelSmarter API Manifest - LLMHost.
- model_config = {'extra': 'allow', 'populate_by_name': True, 'protected_namespaces': (), 'regex_engine': 'python-re', 'use_attribute_docstrings': True}
Configuration for the model, should be a dictionary conforming to [`ConfigDict`][pydantic.config.ConfigDict].
- metadata: SAMLLMHostMetadata
LLMHost.metadata[obj]: Required, the LLMHost metadata.
- spec: SAMLLMHostSpec
LLMHost.spec[obj]: Required, the LLMHost specification.
- status: SAMLLMHostStatus | None
LLMHost.status[obj]: Optional, Read-only.
Stateful status information about the LLMHost.