DataBrain User Manual
An AI-native sensitive-data classification engine — given any text or structured column, it automatically discovers the sensitive data (PII) within and outputs confidence-scored classification. This manual covers everything from installation and API integration to operations and troubleshooting.
How to use this manual: for first-time integration, start from "Quick Start" at the top of 02 · Installation & Startup to run a scan end-to-end in 5 minutes; for production deployment see the same chapter; to integrate the engine into your business system see 03 · API Integration Guide. The contents on the left let you jump anywhere.
01Hardware & Software Requirements
DataBrain ships as a single Docker image. On startup it auto-detects the hardware and selects the appropriate tier — no manual tuning required.
1. Base prerequisites
| Component | Requirement |
|---|---|
| Operating system | Linux x86_64 (Ubuntu 20.04+ / CentOS 8+ / RHEL / Rocky / Alma) |
| Docker Engine + compose v2 plugin | Installed (the command is docker compose, not the legacy docker-compose) |
| curl | Installed |
GPU mode additionally requires: NVIDIA GPU driver (kernel module) + NVIDIA Container Toolkit. CPU mode can skip this.
The image bundles the entire runtime (Python / torch / transformers / CUDA runtime, etc.), so no CUDA Toolkit install is needed; after docker load it connects to no external network. The full locked dependency list is in requirements.lock inside the delivery package.
Docker install example (only if Docker is not yet installed on the host)
Ubuntu / Debian:
sudo apt-get update && sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo usermod -aG docker $USER # log out and back in to take effect
CentOS / RHEL / Rocky / Alma 8 (CentOS 8 is EOL; needs vault redirection + runc conflict resolution + SELinux relaxation):
sudo sed -i -e 's|^mirrorlist=|#mirrorlist=|' -e 's|^#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|' /etc/yum.repos.d/CentOS-*.repo
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y --allowerasing docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo setenforce 0 || true; sudo sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
sudo usermod -aG docker $USER; sudo systemctl enable --now docker
GPU: after installing Docker, install nvidia-container-toolkit (Ubuntu apt / RHEL dnf), then sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker.
Verify: docker --version && docker compose version.
2. Recommended configuration
| Mode | GPU | Memory | CPU | Disk |
|---|---|---|---|---|
| GPU production (recommended) | NVIDIA, VRAM ≥ 8 GB | ≥ 16 GB | ≥ 4 cores | ≥ 15 GB |
| CPU | None | ≥ 16 GB (8 minimum) | ≥ 4 cores | ≥ 15 GB |
On startup the tier is selected automatically by VRAM: ≥ 8 GB → GPU-large; has GPU but < 8 GB → GPU-small; no GPU → CPU. All three tiers have identical accuracy — only speed differs (CPU is roughly 1/19–1/33 of GPU). The active tier is shown in the log as profile=….
WSL2 / VMs without SMBIOS: the GPU generally cannot be passed through (automatic fallback to CPU), suitable for CPU functional validation / CI; the machine fingerprint (the hardware ID reported as the *machine code* in logs and CLI output) is stable — zero drift across upgrades, reboots, and container rebuilds.
3. Ports
| Port | Purpose |
|---|---|
| 18000 | Inference API (/v1/text/scan, /v1/health); port 8000 inside the container |
| 18001 | Management console WebUI (HTTPS, self-signed certificate — see 02 §Management console); port 8001 inside the container (independent from inference) |
To change a port: at install time DATABRAIN_PORT=<port> bash <databrain>.sp, or edit the ports in docker-compose.prod.yml.
02Installation & Startup
The .sp file is a self-extracting installer: bash <file>.sp is a single command that performs loading, CPU/GPU detection, startup, health gating, and sample verification.
Two packages (the actual filenames in your delivery may differ; test builds may be siipulse-beta_*):
siipulse-rel_baseimage_1.0.sp(~2.9 GB): the shared Docker dependency base, installed once per machine, reused by all versions / products.siipulse-rel_databrain_<ver>.sp(~0.7 GB): the application package, re-issued every release. The version number = the image tag = the trailing digits of the filename.
Quick start
# 0) One-time host preparation (needs sudo; creates the install directory + opens firewall 18000/18001)
sudo bash siipulse-rel_databrain_<ver>.sp --prepare-host
# 1) Install (enter the install directory created in step 0 — its path is printed by --prepare-host; place both .sp files there; the app package auto-installs BaseImage first, then starts the service)
cd <install-dir>
bash siipulse-rel_databrain_<ver>.sp
# 2) Activate the license (first install prints the machine code; send it to the vendor to obtain license.json)
bash databrain-ops.sh fingerprint
bash databrain-ops.sh activate license.json
# 3) Confirm readiness + first scan
bash databrain-ops.sh status
curl -X POST http://127.0.0.1:18000/v1/text/scan -H 'Content-Type: application/json' \
-d '{"text":"Contact John Doe, email john.doe@google.com, phone +1-415-555-0142."}'
Multilingual: DataBrain handles Chinese input the same way —
curl -X POST http://127.0.0.1:18000/v1/text/scan -H 'Content-Type: application/json' \
-d '{"text":"联系人张伟,邮箱 zhangwei@google.com,手机 +86 13800138000。"}'If the license is not activated on first install, the installer reports "Awaiting license activation" (exit code 0, not a failure) and prints the machine code — take it to step 2; after activation the service becomes ready in ~60s automatically, no reinstall needed.
Optional environment variables: DATABRAIN_PORT, DATABRAIN_FORCE_CPU=1, DATABRAIN_FORCE_GPU=1 (the latter two are mutually exclusive). One-command smoke test: databrain-ops.sh verify.
Upgrade / rollback
bash siipulse-rel_databrain_<new-version>.sp # installer path: auto-reuses the on-host BaseImage; auto-rolls back on failure
bash databrain-ops.sh upgrade <new-version> # safe in-place upgrade when the image is already loaded (multi-step guard chain, auto-rollback on any failure)
bash databrain-ops.sh revert # undo the most recent upgrade
The license lives in the license/ volume and is never touched by an upgrade, so no re-activation is needed.
Upgrade vs first install: an upgrade preserves the license volume and existing install state (keeping the deployment rollback-safe); a first install initializes this state from scratch.
⚠️ Do not bypass the installer: never work around the install/upgrade flow by hand-editing installation state files — this breaks the rollback and consistency guarantees and can leave the deployment unable to roll back or falsely reporting success. Always use databrain-ops.sh upgrade <ver> or bash <new-version>.sp (validation and automatic rollback are guaranteed by the installer).
License (offline authorization)
DataBrain uses offline authorization. Scanning endpoints (/v1/text/scan, /v1/values/scan) require an activated license first, otherwise they return 403 license_required; /v1/health and /v1/license/status are always open.
Activation (commands are in "Quick start" step 2 above): collect the machine code from this host → send it to Siipulse in exchange for license.json → load and activate it.
The license file is named license.json, but its content is a single-line authorization text issued by the vendor (not a JSON object); do not hand-edit or fabricate it — hand it to activate as-is; malformed files are rejected.
- The license is bound to this machine; upgrades, reboots, and container rebuilds on the same machine require no re-activation.
- When replacing the host or migrating to another machine, you must re-collect the machine code on the new machine and re-issue the license.
- Verify:
curl http://127.0.0.1:18000/v1/license/statusshould returnvalid:true.
Management console (port 18001, HTTPS)
After the container starts, the console is served at https://<host>:18001/ on the same host, with default credentials admin/admin (you are prompted to change the password on first login). It shares the same license as the inference API. The console provides:
- Dashboard: run overview and key metrics.
- System management: system status / license import /
.spversion management / restart. - Self-learning: new-type discovery and corpus-bundle management.
- Playground: live test scans.
- Audit: detection-log review / human labeling / statistics and pagination.
- Troubleshooting: diagnostic bundling.
- Top-bar health alerts: persistent status banner.
HTTPS notes (enabled by default since 2026-08-21)
- Certificate mechanism: the console is served via native TLS with a self-signed certificate generated automatically on first start (ECDSA P-256, 10-year validity, SAN covers the host IP + localhost, determined by
DATABRAIN_TLS_HOST_IPwritten to.envby the installer). - First-visit warning: private-IP hosts cannot obtain a public CA signature, so the browser shows "Your connection is not private" — click Advanced → Proceed to continue.
- Certificate persistence: the certificate lives in the
console-data/volume; container upgrades/rebuilds keep it, no re-trusting needed. - Eliminating the warning: import
console-data/tls/console.crt(in-container/app/webui/tls/console.crt) into the OS / enterprise certificate trust store. - Falling back to HTTP: set
DATABRAIN_CONSOLE_TLS=0in.env(local debugging only). - Public deployments: still recommended to front nginx to terminate TLS (with a proper domain certificate) + an IP allow-list.
Operations commands (databrain-ops.sh)
start | stop | restart | status | logs -f | health | license | verify | fingerprint | activate <file> | upgrade <ver> | rollback <ver> | revert | clear | deps | diagnose | version
Usable from any directory: add the install directory to your PATH (export PATH=<install-dir>:$PATH). For more troubleshooting see 07-Operations & Troubleshooting; for response field semantics see 03-API Integration Guide.
Common issues
| Symptom | Resolution |
|---|---|
/v1/text/scan returns 403 license_required | License not activated; follow Quick start step 2 |
license/status valid:false (fingerprint_mismatch) | Host hardware changed, or moved to another machine → re-run fingerprint and re-issue; expired → renew; revoked → contact the vendor |
Reports base image siipulse-baseimage:1.0 is not loaded | Put the BaseImage .sp and the app package in the same directory and re-run the app package |
Container exits immediately with No space left on device | Use the delivered compose (container shared memory is already configured correctly); do not run bare docker run |
Health check ready:false | warming = model loading (wait 30–70s); license_required = not activated |
| Needs GPU but runs CPU | Host missing nvidia-container-toolkit, or DATABRAIN_FORCE_CPU=1 was set during reinstall |
| Port 18000 / 18001 already in use | DATABRAIN_PORT=<port> only changes the inference port; the console port 18001 must be changed manually in the compose ports |
ops-agent (optional · WebUI remote upgrade / restart)
Once installed, the console's "System Management" page can trigger restarts and .sp version upgrades remotely over a local secure channel (the service is unavailable for ~30–90s during container rebuild, with automatic rollback on failure). Once per machine (the app must already be installed):
sudo bash siipulse-rel_databrain_<ver>.sp --install-ops-agent
03API Integration Guide
DataBrain exposes its sensitive-data discovery capability over HTTP. Choose the integration path based on the shape of your data:
| Data shape | Recommended path | Entry point |
|---|---|---|
| Free text / documents (email bodies, support transcripts, contract clauses, logs…) | Document-level POST /v1/text/scan | Discover entities within "one text" |
| Structured columns (database fields, table columns, CSV column values…) | Value-level POST /v1/values/scan | Classify a "batch of values" |
/v1/text/scan and /v1/values/scan share the same recognition engine and produce identical decisions. The only difference is the data source: /v1/text/scan locates entities within a text and returns character offsets; /v1/values/scan judges a batch of independent values one by one and returns confidence.
Every decision carries decision-provenance fields (method / role, always returned) and optional decision evidence (evidence, returned on request), making each decision traceable and auditable. Field semantics are in section 3.
1. Document-level scan POST /v1/text/scan
For unstructured free text. The engine locates all sensitive entities within a text and returns their type, character offsets, confidence, and decision provenance.
1. Health check
GET /v1/health
{"status":"ok","ready":true}
| Field | Meaning |
|---|---|
status | "ok" service ready / "warming" model loading |
ready | true means the AI model has finished loading and the service is callable |
2. Sensitive-data scan
POST /v1/text/scan
Content-Type: application/json
{
"text": "Text to scan (1–100,000 characters)",
"min_confidence": 0.5,
"return_evidence": false
}
Request parameters
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
text | string | yes | — | Text to scan, length 1–100,000 characters (returns 422 if exceeded) |
min_confidence | number | no | 0.5 | Emission threshold, range 0–1; see guidance below |
return_evidence | bool | no | false | Whether to include decision evidence in results (see section 3) |
min_confidence guidance
0.5(default, balanced tier): returns only results likely to be real sensitive data; fits mainstream production scenarios.0.0(high-recall tier): returns all candidates, each with a confidence score, prioritizing recall and letting downstream filter by confidence; for audit, data inventory, and other "better safe than sorry" scenarios.
Response
{
"results": [
{
"value": "zhangwei@example.com",
"pii_type": "EMAIL",
"start": 11,
"end": 31,
"confidence": 0.99,
"needs_review": false,
"method": "truth_table",
"role": "reference"
}
]
}
| Field | Type | Description |
|---|---|---|
value | string | The matched original text span |
pii_type | string | Sensitive-data type (see 04-PII Type Catalog, e.g. EMAIL/PHONE/ID_CARD) |
start / end | number | Character start/end offsets in the original text; usable for highlighting and redaction |
confidence | number | Confidence 0–1, calibrated as "the true probability of being that type" |
needs_review | bool | true means the value was detected but the type assignment is under-confident (e.g. two types are neck-and-neck); manual or rule-based review is recommended |
method | string | Decision path (always returned); see section 3 |
role | string | Semantic role (always returned); see section 3 |
evidence | object | Decision evidence, returned only when return_evidence=true |
A needs_review result is still an emitted, valid prediction (already counted in accuracy statistics), not a "failed detection" — it only flags that the item warrants review. In production you may trust it directly or route it to a review flow, depending on business tolerance.
Error codes
| HTTP status | Meaning |
|---|---|
200 | Success |
403 | License not activated (error: license_required); diagnostic endpoints are unaffected |
422 | Invalid request body (e.g. text empty, over 100,000 characters, min_confidence out of range) |
503 / 504 | Service not ready or timed out (check ready) |
2. Value-level batch classification POST /v1/values/scan
For scanning structured column values such as database tables, CSV / Excel columns, or data-lake fields. Operates on a "batch of values", judging each value independently and returning confidence.
POST /v1/values/scan
Content-Type: application/json
{
"values": [
{"value": "zhangwei@example.com", "value_id": "r1", "label_hint": "email"},
{"value": "110101199003077334", "value_id": "r2", "label_hint": "id_card"},
{"value": "13800138000", "value_id": "r3", "container_path": "customer.phone"}
],
"return_evidence": false
}
Request parameters
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
values | array | yes | — | List of values to classify, 1–1000 items |
values[].value | string | yes | — | A single value, length 1–10000 characters |
values[].value_id | string | no | auto-generated | Caller-side correlation ID, echoed back as-is; if omitted, generated as customer:request:index |
values[].label_hint | string | no | last segment of container_path | Column / field-name hint; significantly improves accuracy (see section 4); an explicit value takes precedence over container_path |
values[].container_path | string | no | — | Column path, e.g. users.email; when label_hint is not supplied the system auto-derives the hint from its last segment (see section 4) |
values[].surrounding_text | string | no | "" | Text surrounding the value (≤ 2000 characters; should contain an occurrence of the value itself — otherwise only the context channel is inert, other signals are unaffected); improves accuracy for context-dependent types |
return_evidence | bool | no | false | Whether to include decision evidence in results (see section 3) |
The value-level endpoint returns all decisions (including values judged "non-sensitive", whose pii_type is null), each with a confidence score; acceptance is up to the caller, which filters by confidence (there is no min_confidence emission threshold at value level, unlike document level).
Optional request header X-DataBrain-Consent (data-reflow consent statement): it only affects the local detection log and self-learning capture (semantics in 06-Security & Privacy §9) and does not change decisions.
Response
{
"results": [
{"value_id": "r1", "value": "zhangwei@example.com", "pii_type": "EMAIL", "confidence": 0.99, "needs_review": false, "is_mock": false, "method": "truth_table", "role": "subject"},
{"value_id": "r2", "value": "110101199003077334", "pii_type": "ID_CARD", "confidence": 0.92, "needs_review": false, "is_mock": false, "method": "truth_table", "role": "identifier"}
]
}
| Field | Type | Description |
|---|---|---|
value_id | string | Echoed input ID, for alignment |
value | string | Original value |
pii_type | string | null | Judged sensitive type (see 04-PII Type Catalog); null means judged non-sensitive |
confidence | number | Confidence 0–1, calibrated as "the true probability of being that type" |
needs_review | bool | true recommends manual review |
is_mock | bool | Whether the value looks like sample / test fake data (e.g. 000-00-0000) |
method | string | Decision path (always returned); see section 3 |
role | string | Semantic role (always returned); see section 3 |
evidence | object | Decision evidence, returned only when return_evidence=true |
Error codes
| HTTP status | Meaning |
|---|---|
200 | Success |
403 | License not activated (error: license_required); diagnostic endpoints are unaffected |
422 | Invalid request body (e.g. missing values, empty list, over 1000 items, a single value too long) |
503 / 504 | Service not ready or timed out (check ready) |
3. Decision-provenance fields (method / role / evidence)
Every DataBrain decision carries provenance information, making the decision process transparent, verifiable, and auditable — not a black-box output.
method — decision path
Identifies which engine path produced this result, reflecting the signal strength and confidence tier behind the decision:
| Value | Meaning |
|---|---|
truth_table | High-confidence direct: judged by the engine's combined assessment, with sufficient confidence — accept directly |
needs_review | Borderline / low-confidence: detected as this type but below the direct-emit threshold; manual or rule-based review recommended (the needs_review field is true in this case) |
mock_filter | Sample-data filter: judged as suspected sample / test fake data |
non_sensitive_filter | Pre-filter exclusion: judged clearly non-sensitive by rule pre-filtering and never entered the main recognition flow (only non-sensitive results of /v1/values/scan may show this) |
The vast majority of high-confidence results take the truth_table path; results whose confidence is borderline or in a blind spot (two types neck-and-neck, sparse borderline samples) take the needs_review path. The engine performs fully local inference (it calls no external model and makes no outbound network request).
Closed set: the method value set (the four values above) is part of the external contract — when engine evolution adds a value it will be registered in this table with the release. Downstream consumers should not match values outside the table (treat unmatched as unknown).
role — semantic role
Inferred from the column / field name (label_hint), indicating the value's semantic role in the business, to assist downstream policy decisions:
| Value | Meaning |
|---|---|
subject | The value is itself the sensitive subject (e.g. column named email / phone / passport) |
identifier | An identifying / reference key (e.g. column named id / account_no / ref) |
reference | Default reference role (no label_hint, or the column name carries no clear semantics) |
Document-level /v1/text/scan has no column context, so role is always reference; at value level /v1/values/scan, role is determined by the supplied label_hint. For results judged non-sensitive (pii_type is null), role is likewise null.
evidence — decision evidence (on request)
evidence is not returned by default; it is attached only when return_evidence=true is requested. It carries the objective evidence supporting the decision, for example:
| Evidence key | Meaning |
|---|---|
provider / provider_conf | Recognized brand / issuer (e.g. bank-card brand) and its confidence |
country / subtype keys | Recognized country / region / subtype |
ns_reject_reason | Reason for rejection when judged "non-sensitive" |
evidence can carry a lot of detail, so it is designed as opt-in: keep the default false in normal integration to keep responses lean; enable return_evidence=true for audit, forensics, and policy-tuning scenarios that need the full rationale.
4. label_hint notes (value-level only)
label_hint is the column / field-name hint supplied in the request (e.g. email, id_card, passport). It is a strong clue that assists the engine's decision and notably improves accuracy on pure-digit, format-similar identifiers (e.g. national ID vs passport vs social-security number).
- How to supply: set the
label_hintfield on each object invalues[]; - When
label_hintis not supplied, the system auto-derives it from the last segment ofcontainer_path(e.g.customer.phone→phone,users.email→email) — so as long as you put the real column / field name intocontainer_path, you get the same boost as an explicitlabel_hint, with no duplicate entry; - It only takes effect when the column name matches a known type word (e.g.
phone/email/ssn/id_card/passport/dob); meaningless names (e.g.col_0,value1) are safely ignored and do not affect the decision; - An explicit
label_hintalways takes precedence; if neither is supplied, the engine still judges normally; - Production tip: database field names and CSV headers are naturally high-quality column-name hints — pass them in via
container_pathorlabel_hint.
5. Complete call examples
curl
English sample:
curl -X POST http://127.0.0.1:18000/v1/text/scan \
-H 'Content-Type: application/json' \
-d '{"text":"Order #A12345, customer john.doe@google.com, phone +1-415-555-0142.","min_confidence":0.5}'
Chinese sample (same engine, multilingual):
curl -X POST http://127.0.0.1:18000/v1/text/scan \
-H 'Content-Type: application/json' \
-d '{"text":"订单 #A12345,客户 john.doe@google.com,电话 +1-415-555-0142。","min_confidence":0.5}'
Python (HTTP)
import requests
resp = requests.post(
"http://127.0.0.1:18000/v1/text/scan",
json={"text": "Customer john.doe@google.com, phone +1-415-555-0142.", "min_confidence": 0.5},
timeout=30,
)
for m in resp.json()["results"]:
print(m["pii_type"], m["value"], m["confidence"], m["method"])
Java
// Uses JDK 11+ HttpClient
import java.net.http.*;
import java.net.URI;
var body = "{\"text\":\"Customer john.doe@google.com, phone +1-415-555-0142.\",\"min_confidence\":0.5}";
var req = HttpRequest.newBuilder()
.uri(URI.create("http://127.0.0.1:18000/v1/text/scan"))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(body))
.build();
var resp = HttpClient.newHttpClient().send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(resp.body());
C# (.NET)
using var http = new HttpClient();
var body = """{"text":"Customer john.doe@google.com, phone +1-415-555-0142.","min_confidence":0.5}""";
var content = new StringContent(body, System.Text.Encoding.UTF8, "application/json");
var resp = await http.PostAsync("http://127.0.0.1:18000/v1/text/scan", content);
Console.WriteLine(await resp.Content.ReadAsStringAsync());
Go
body := []byte(`{"text":"Customer john.doe@google.com, phone +1-415-555-0142.","min_confidence":0.5}`)
resp, err := http.Post("http://127.0.0.1:18000/v1/text/scan",
"application/json", bytes.NewReader(body))
// ...read resp.Body
Value-level batch (/v1/values/scan)
import requests
resp = requests.post(
"http://127.0.0.1:18000/v1/values/scan",
json={"values": [
{"value": "john.doe@google.com", "label_hint": "email"},
{"value": "123-45-6789", "label_hint": "ssn"},
]},
timeout=30,
)
for r in resp.json()["results"]:
print(r["value_id"], r["pii_type"], r["confidence"], r["method"], r["role"])
With decision evidence (return_evidence=true)
curl -X POST http://127.0.0.1:18000/v1/values/scan \
-H 'Content-Type: application/json' \
-d '{"values":[{"value":"4111 1111 1111 1111","label_hint":"credit_card"}],"return_evidence":true}'
More runnable examples are in the examples/ directory.
6. Integration best practices
- Startup dependency check: probe
/v1/healthbefore calling, and only send requests onceready:true, to avoid 503 during cold start. - Timeout: a single
/v1/text/scanclient timeout should be ≥ 30 seconds (the first request includes model warm-up); in steady state a single text is usually millisecond-level. - Large batches of text: one HTTP request per text; for huge document volumes, concurrency on the caller side is recommended (the server-side concurrency cap depends on the hardware tier, see 05-Performance & Capacity); for very large batches of structured data use value-level
/v1/values/scan(call in batches of ≤1000 per request). - Handling
needs_review: set policy by business — risk / compliance scenarios should do manual or secondary rule-based review; general scenarios can trust the type label directly. - Use provenance:
methodlets you track decision-path distribution and the share of borderline cases; enablereturn_evidencewhen you need the full rationale, for audit, forensics, and policy tuning. - Idempotency: the same input always yields the same output — feel free to retry, cache, and build comparison baselines.
- Offset usage:
start/endcan be used directly for front-end highlighting and redaction (e.g. replace the[start,end)span with***).
04PII Type Catalog
DataBrain supports 32 types of sensitive data, grouped into four categories by sensitivity and detection reliability. The pii_type field returned by the system takes its values from the "Type" column in the tables below.
Category overview
| Category | Meaning |
|---|---|
| I-A Strong identifiers | High-sensitivity identifiers with mathematical validation |
| I-B Identifiers | Identifiers with a clear format or validation rule |
| II-A Common PII | Context-dependent common personal information |
| II-B Intrinsic attributes | Intrinsic attributes of a person (name, demographics, occupation) |
I-A / I-B mostly carry "mathematical / format validation" and are the most reliably decided; II-A / II-B rely more on AI context understanding.
I-A · Strong identifiers (4 types, with mathematical validation)
| Type | Meaning | Example |
|---|---|---|
CREDIT_CARD | Bank card number | 4111 1111 1111 1111 |
IBAN | International bank account number | DE89 3704 0044 0532 0130 00 |
SECRET | Keys / access credentials (cloud-vendor API keys, access tokens, private keys, etc.) | AKIA… / ghp_… / sk_live_… |
AADHAAR | Indian national ID (Aadhaar) | 12 digits |
I-B · Identifiers (16 types, with format / validation)
| Type | Meaning | Example |
|---|---|---|
ID_CARD | Identity card / national ID card | 110101199003077334 |
TAX_ID | Tax / VAT number | DE123456789 |
SOCIAL_SECURITY_NUMBER | Social security number | 123-45-6789 |
PASSPORT | Passport number | — |
SWIFT_BIC | Bank identifier code (SWIFT/BIC) | DEUTDEFF500 |
IP_ADDRESS | IP address (IPv4/IPv6) | 192.168.1.1 |
MAC_ADDRESS | Network interface physical address | 00:1A:2B:3C:4D:5E |
IMEI | Mobile equipment identity | 35-209900-176148-1 |
VEHICLE_ID | Vehicle identification number (VIN) | 17 chars |
CRYPTO_ADDRESS | Cryptocurrency wallet address | 1A1zP1eP… / 0x… |
UPI | Indian unified payment address | name@oksbi |
IFSC | Indian bank branch code | SBIN0001234 |
CN_LICENSE_PLATE | Mainland-China motor-vehicle license plate | 京A12345 |
CN_TRAVEL_PASS | Mainland Travel Permit for Hong Kong/Macau Residents (Home Return Permit) | H12345678 |
CN_SECURITIES_ACCOUNT | China securities account number (A/B shares / funds) | A123456789 |
GEO_LOCATION | GPS latitude/longitude coordinate pair (location trace) | 39.9042, 116.4074 |
II-A · Common PII (9 types, context-dependent)
| Type | Meaning | Example |
|---|---|---|
EMAIL | Email address | john@example.com |
PHONE | Phone / fax | +86 138 0013 8000 |
ADDRESS | Postal address | 123 Main St, Springfield / 北京市朝阳区… |
BANK_ACCOUNT | Bank account number | — |
DRIVER_LICENSE | Driver's license number | — |
DATE_OF_BIRTH | Date of birth | 1990-03-07 |
URL | Web URL | https://example.com |
PASSWORD | Password | — |
USERNAME | Username / account name | john_doe |
II-B · Intrinsic attributes (3 types, AI semantic recognition)
| Type | Meaning | Example |
|---|---|---|
NAME | Person name | John Smith / 张伟 |
DEMOGRAPHIC | Demographic attributes (gender, age, height, eye color, etc.) | Male, 32 / 男,32 岁 |
EMPLOYMENT | Employment info (job title, employer) | Product Manager @ Acme |
Coverage breadth
- Languages: the AI recognition model is based on multilingual pre-training and covers 100+ languages (Chinese, English, Japanese, Korean, Hindi, many European languages, etc.).
- Regions: national IDs cover 30+ country formats and tax IDs cover 20+ country formats (per the built-in type definitions), including mainland-China national ID / license plate / travel permit / securities account, Indian Aadhaar/PAN/UPI/IFSC, Japan My Number, various European/American IDs, etc.
- Regulatory alignment: the types above cover the sensitive-data categories regulated by mainstream data-protection laws (see the brochure's "Regulatory alignment" section).
If a type you need is not in the tables above, contact Siipulse to evaluate adding it — DataBrain has a self-learning capability of "discover a new type → generate samples → retrain" (see the brochure's "Three-loop self-learning" section).
05Performance & Capacity
This chapter gives measured reference performance and hardware-sizing guidance, to help estimate throughput and capacity.
All numbers are reference measurements (test hardware: NVIDIA RTX 5070 GPU, single machine). Actual performance depends on hardware model, text length, and load characteristics.
1. Latency & throughput (reference)
Document-level (HTTP /v1/text/scan, single text)
| Metric | GPU tier (host direct) | In-container (GPU tier) |
|---|---|---|
| Avg latency per text | ~8.6 ms/text | ~9.8 ms/text |
| Single-stream throughput | ~116 texts/sec | ~100 texts/sec |
Value-level (batch classification benchmark)
| Hardware tier | Per-value latency | Throughput |
|---|---|---|
| GPU tier | ~1.6 ms/value | ~640 values/sec |
| CPU tier | ~29.5 ms/value | ~34 values/sec |
CPU and GPU are identical in accuracy; the only difference is speed — the CPU tier is roughly 1/19–1/33 of the GPU tier, with the bottleneck on AI inference compute.
2. Concurrency & throughput notes
- Per-tier concurrency caps: CPU tier 4 / GPU-small 8 / GPU-large 16;
- Multiple requests can overlap concurrently; effective throughput rises with concurrency until GPU compute saturates;
- Callers are advised to do client-side concurrency control according to the hardware tier's cap, to avoid queuing under overload.
3. Sizing guidance
| Scenario | Recommended tier | Notes |
|---|---|---|
| Production, latency/throughput sensitive | GPU-large (≥ 8 GB VRAM) | Best price/performance |
| Medium traffic, limited VRAM | GPU-small | Auto-applied |
| Functional validation / low frequency / no GPU | CPU tier | Usable; speed ~1/19–1/33 of GPU |
| Massive structured-data batch scan (e.g. full-DB redaction) | GPU-large + value-level /v1/values/scan | Batch value classification, high throughput |
4. Capacity estimation example
Estimated with GPU-large, document-level single-stream ~100 texts/sec:
| Daily volume | Single-machine time (serial) | Notes |
|---|---|---|
| 1 million texts | ~2.5 hours | Single-stream serial; concurrency shortens this significantly |
| 10 million texts | ~25 hours | Sharding + horizontal scale-out recommended |
Longer text and denser PII slightly increase per-text cost. Value-level batch-scan throughput is notably higher than document-level (no text-segmentation overhead).
5. VRAM & memory footprint
- AI model: ~0.7 GB (multilingual sensitive-data recognition model, delivered encrypted inside the image, loaded into VRAM at startup);
- Dependency layer: ~2.7 GB (AI runtime stack, disk footprint, not resident in VRAM);
- Runtime peak: VRAM ≥ 8 GB and system memory ≥ 16 GB recommended;
- Disk: image ~3.4 GB; recommend reserving ≥ 15 GB (including 2-version rollback headroom). The detection log rotates daily with a default 180-day retention and is protected by a low-disk-space guard (auto-stops writing below 2 GB free; see 06-Security & Privacy).
06Security & Privacy
This chapter describes DataBrain's security design across deployment topology, data handling, supply chain, and offline authorization.
1. No data egress (offline deployment)
- Fully offline operation: the image is loaded from an offline package; at runtime it does not connect to Docker Hub, does not connect to any external source, and makes no outbound network request.
- Data never leaves the customer network: all sensitive-data scanning and judgment happens entirely inside the customer's own server/container; results are returned only to the caller.
- No telemetry, no callbacks: the system neither collects nor reports any scanned content or statistics to Siipulse or any third party.
That is, from the moment data enters DataBrain to the moment the result is produced, it stays within the customer's controllable environment.
2. Data persistence posture
- Scanning is real-time compute by default: input text is recognized in memory and the result emitted, with no on-disk persistence of the scanned content itself; operational logs (startup, health, errors) do not log PII.
- Detection log (default-on): the decision stream is written to disk for the console audit page (review and human labeling); values in records are redacted by default (only a few leading/trailing characters kept), rotated daily with a default 180-day retention and automatic cleanup on expiry.
- Low-disk-space guard: if free disk space drops below 2 GB, detection-log writing pauses automatically (keeping the classification service alive); it resumes once space recovers to about 3 GB; the stopped state is surfaced via
/v1/healthand console alerts. - Data-reflow exception (default-on, see §9): when a request carries valid consent, the system retains an encrypted sample of the PII on the customer's local disk (for human review + offline model retraining, AES-256-GCM encrypted), bounded by a dual gate + retention period.
Ordinary scans (no consent header) leave no plaintext on disk — detection-log values are redacted. Two exceptions exist: console-submitted text enters encrypted self-learning capture by default (see §4), and data reflow is a local encrypted retention the customer explicitly enables via consent (see §9). Neither leaves the network.
3. Container security
- Non-root: the container runs as a dedicated low-privilege user
databrain, not root; - Dependency isolation: the container bundles its entire runtime dependency set (AI inference framework, GPU runtime, etc.), isolated from the host's native environment, with no mutual interference;
- Only shared resource: in GPU mode the host and container share only the GPU driver (kernel module); no other system component is shared.
4. Data control rights
| Concern | Notes |
|---|---|
| Data storage location | Ordinary scans store no plaintext; the detection log is stored on-disk redacted (local); data reflow retains an encrypted corpus on the customer's host (output/corpus, no egress) |
| Data retention period | Detection log defaults to 180 days (daily rotation with automatic cleanup, configurable); data-reflow corpus bundles default to a 90-day retention (bundle_retention_days, configurable); labelable detection-log records from the console (submission text retained as encrypted samples for labeling) live in a separate console-playground bucket (see below) |
| Console Playground capture (default-on) | Console-submitted text enters encrypted self-learning capture (console-playground bucket) for audit-page labeling; opt-out and export are described below |
| Access control | Container / host network and permission controls; data-reflow review endpoints require a license + reviewer token + reviewer header (see §9) |
| Data destruction | Data reflow supports Art17 withdrawal (/v1/human-review/withdraw, soft delete); stopping / deleting the container clears the runtime state |
| Response-field minimization | Decision evidence is not returned by default (opt-in via return_evidence=true only); the response contains only the decision and provenance signals (method/role) by default, following data-minimization |
Console Playground capture: text submitted by operators in the console Playground enters the self-learning capture pipeline by default (console-playground customer bucket, encrypted retention), feeding the audit page for human labeling corrections. Per-request opt-out: check "do not record" when submitting (request body no_audit: true; the request is then not captured). Channel-wide off: deployment environment variable DATABRAIN_CONSOLE_CONSENT=false (default true). The bucket's retention cleanup and export are explicit ops actions (GET /v1/learning/export-corpus?customer=console-playground); it does not appear in the console's default corpus view.
5. Input limits (abuse prevention)
/v1/text/scancaps a single text at 100,000 characters (exceeding returns 422), preventing oversized-payload memory abuse;min_confidencelets the caller control the emission threshold, tuning between "high precision" and "high recall" per scenario.
6. Compliance posture
DataBrain's detection capability covers the sensitive-data types regulated by mainstream data-protection laws (GDPR, CCPA, China PIPL, HIPAA, PCI-DSS, SOC 2, etc.), with type-level mappings established.
Note: DataBrain provides a "sensitive-data discovery" capability that assists with the data discovery and inventory step of compliance requirements. Whether you are "compliant" depends on the overall compliance system (processes, people, controls, etc.); DataBrain is one component, not a compliance certification in itself. See the brochure's "Regulatory alignment" section.
7. Upgrade & supply-chain security
- Upgrade packages (
.sp) are offline incremental packages verified before loading with ECDSA signature validation + payload SHA-256 integrity comparison (the signature is bound to the payload hash, preventing "keep the signature, swap the content"); packages uploaded through the console's version-management page go through the same verification chain; - The upgrade script has built-in path-traversal / absolute-path defenses and version-format validation, rejecting tampered packages;
- Versions are always pinned to an explicit tag (no
:latest), for audit and precise rollback.
8. Offline license authorization
DataBrain uses fully offline authorization: all authorization checks happen locally on the customer side, with no outbound network request, no callback to Siipulse, and no online attestation, deployable in strict air-gapped environments.
- Host binding: the license is bound to the machine — bound to the host, not the container, so container rebuilds on the same machine do not affect authorization; each deployment instance must activate its own license. Replacing the host or migrating to another machine requires re-activation (see 02 "License (Offline Authorization)").
- Gate scope:
/v1/text/scan,/v1/values/scanrequire a valid license;/v1/health,/v1/license/statusare always open for diagnostics. - Validity & grace period: a license has an expiry date, followed by a grace period (per authorization type); after the grace period the scan endpoints return
403, while the health / status endpoints stay open; renew before expiry — no reinstall needed.
For the activation flow see 02 "License (Offline Authorization)"; for authorization status and renewal see 07-Operations & Troubleshooting.
9. Data reflow (self-learning data collection)
Data reflow makes DataBrain more accurate over time: it samples "needs-review" verdicts from production traffic → human annotation → export of an encrypted corpus → hand-off to offline model retraining. Sampled data stays encrypted on the customer's own host and never leaves the network.
Not enabling it: no changes needed
Without the request headers below, the scan APIs behave exactly as usual, with zero collection. Collection is guarded fail-closed by two gates (an encryption key auto-generated at deployment + a per-request consent header); if either is missing, nothing is collected at all — the capability ships pre-installed and is activated per request only via the headers.
Enabling collection: add one header to scan requests
X-DataBrain-Consent: {"ts":"<YYYY-MM-DD>","crown_pii":true}
- This header only affects local sampling and detection logs; it never changes detection results
crown_pii: trueconsents to collecting the PII original values in that request; omitting the header = no collection for that request
Review annotation and corpus management are performed by operators in the console (/self-learning and the review UI); no caller involvement is required. Data subjects exercising erasure rights should contact the operator.
07Operations & Troubleshooting
One-click self-check (online diagnostics, preferred)
The console's Diagnostics page (host port 18001 → Diagnostics) provides a one-click self-check that runs 9 checkpoints (in 6 groups) across deployment, startup, and runtime — each with a status light: 🟢 green = normal / 🟡 yellow = abnormal but core functions unaffected / 🔴 red = core functions impaired / ⚪ grey = not applicable (e.g. dependencies that cannot be assessed while the inference service is unreachable).
- Automatic: runs once after startup (first install / upgrade / manual restart); retries during model loading until a meaningful result converges.
- Manual: click the "Self-check" button at any time.
- Check groups: service status (inference service / ops agent) → licensing (license, distinguishing expiring / grace-period / pending-activation states) → model assets (integrity of the recognition-model components) → core function (built-in sample-detection verification, consistent with the pre-release verification criteria) → pipeline health (audit persistence / audit index) → data persistence (storage writes). Resource capacity (memory / disk / GPU), quality gates, drift, and p95 latency are on the dashboard and health-alerting pages.
- Remediation guidance: every yellow/red light carries handling advice and a jump link (e.g. license activation, audit page, restart from system administration); the raw data is expandable.
- Proactive alerting: when the overall result is red/yellow, the console's top alert bar shows "Self-check found issues" and can be pushed out via webhook (if
DATABRAIN_ALERT_WEBHOOK_URLis configured). - Result retention:
output/selfcheck/(history included), exported together with the diagnostic bundle; on a fresh install without an activated license, a yellow "pending activation" light is the expected state.
Diagnostic bundle (offline fallback)
When the container or the console fails to start and you need to send diagnostic information to support, run on the host, from the install directory:
bash databrain-ops.sh diagnose
- Output:
<install-dir>/diagnostics/diag_<timestamp>.zip(send to the vendor). - Collected: version / machine code / health / license status / container process logs (
docker compose logs) / docker info /nvidia-smi, all best-effort — even if the container is down or docker is missing, a usable partial diagnostic bundle is still produced. - Redaction: automatically strips sensitive environment-variable values; the license file and key material of any kind are never bundled.
- When the container is running normally, it is preferable to use the console's Troubleshooting page (host port 18001) for one-click bundling — its content is more complete (includes audit samples / metric series / model version), with the same redaction.
After unzipping you can run python analyze.py (shipped inside the zip) for 6 quick checks (license/health/GPU/error logs/need_review ratio/type confidence); exit code 0 = OK, 1 = warnings present.
1. Daily operations commands
The databrain-ops.sh commands below and throughout this chapter are run from the install directory (or add the install directory to your PATH first — see the tip below).
| Command | Description |
|---|---|
bash databrain-ops.sh status | Single-screen overview: container status + ready + license + profile + ports |
bash databrain-ops.sh logs | Recent logs (non-following by default, --tail 200) |
bash databrain-ops.sh logs -f | Live-follow logs (Ctrl+C to exit) |
bash databrain-ops.sh restart | Restart |
bash databrain-ops.sh stop | Stop |
bash databrain-ops.sh start | Start (auto-detects CPU/GPU, uses the current version) |
bash databrain-ops.sh health | Health check |
bash databrain-ops.sh license | Authorization status |
bash databrain-ops.sh verify | Sample-scan smoke test (includes EMAIL/PHONE) |
bash databrain-ops.sh version | App tag + base tag + CLI version |
bash databrain-ops.sh deps | Self-checks base-image third-party components (BOM diff) |
bash databrain-ops.sh fingerprint | Collect the machine code (also writes machine-code.txt) |
bash databrain-ops.sh activate license.json | Load license + restart + verify (use after renewal/re-issue) |
bash databrain-ops.sh diagnose | Offline fallback: bundle version/health/license/logs/docker-info/nvidia-smi into ./diagnostics/ (use when the container won't start, see "Diagnostic bundle") |
bash databrain-ops.sh rollback <old-version> | Switch to a specified old version (version switching & cleanup: see §4) |
bash databrain-ops.sh revert | Undo the most recent upgrade (back to the pre-upgrade version) |
bash databrain-ops.sh clear [--yes] | DANGER: clean from scratch (keeps license/+.sp/base) |
Skip the long prefix: after a one-time echo 'export PATH="<install-dir>:$PATH"' >> ~/.bashrc && source ~/.bashrc, all the above can be abbreviated to databrain-ops.sh … (for multi-user sharing, write it into a system-wide profile file).
2. Monitoring essentials
| Metric | How to check | What to watch |
|---|---|---|
| Service ready | GET /v1/health → ready | Should be true |
| Authorization status | GET /v1/license/status → valid | Should be true; watch days_remaining, grace_active; renew before expiry |
| Container health | docker ps (STATUS column healthy) | Docker auto-probes every 30s |
| Detection-log disk space | GET /v1/health → audit block / console top-bar alert | Below 2 GB free the detection log stops writing (audit_disk_stopped alert); it resumes after disk cleanup brings free space above ~3 GB |
| GPU utilization | nvidia-smi (GPU mode) | VRAM and utilization, to spot overload |
| Resource usage | docker stats databrain | Abnormal CPU/memory |
| Active tier | startup log profile=... | Confirm it is running the intended GPU tier |
Docker's built-in health check treats ready:true as healthy: it takes 3 consecutive probe failures (30s interval) to mark unhealthy, with a 60-second grace period after start.
3. Version management
- Always pin an explicit TAG (e.g.
TAG=1.0.1); do not use:latest, for audit and precise rollback; - Keep the latest 2 application versions (current + previous) for instant rollback;
- Never manually delete the dependency layer (base) — it is large but changes rarely (about once a year), and it is key to rollback; only manually clean up "orphan" dependency layers that no application version references;
- Upgrades are handled end-to-end (signature verification → load → health gate → rollback on failure) by the application siipulse package (
bash siipulse-rel_databrain_<new-version>.sp), reusing the on-host BaseImage;.sppackages uploaded through the console's version-management page go through the same signature and integrity verification (see 06-Security & Privacy §7).
4. Rollback / revert / cleanup / dependency self-check
4.1 Roll back to a specific version rollback
# Switch back to any old version still in the local image store (dependency layer + old app image still present)
bash databrain-ops.sh rollback <old-version>
A rollback completes within seconds (a tag switch + restart + ~30–70 s model reload).
4.2 Undo the most recent upgrade revert
# No need to remember the version: returns to the pre-upgrade version (auto-recorded by the installer)
bash databrain-ops.sh revert
Difference between revert and rollback: rollback <ver> requires an explicit version number and can switch to any old version; revert takes no argument and specifically "undoes the last upgrade", relying on the previous version recorded by the installer. Neither touches the license volume.
4.3 Clean from scratch clear (dangerous)
For extreme troubleshooting or before migrating to a new machine, you can fully clean this installation back to a "only delivery packages left" state:
bash databrain-ops.sh clear # interactive confirm (type YES)
bash databrain-ops.sh clear --yes # skip confirm (for scripts / automation)
clear will: stop & remove containers → delete all siipulse-databrain:* application images → delete compose and the installation state files. Kept: license/ (the authorization volume, sparing re-activation), *.sp delivery packages, siipulse-baseimage:1.0 (the base, sparing a re-load). After cleanup, reinstall with bash siipulse-rel_databrain_<version>.sp.
Note: keeping the license and base is intentional — after cleanup a reinstall needs no re-activation flow and no 2.9 GB base re-load. To also delete the base, manually docker rmi siipulse-baseimage:1.0.
4.4 Dependency self-check deps
Confirm whether the current base image contains all runtime dependencies (the dependency list is in requirements.lock inside the delivery package):
bash databrain-ops.sh deps
It checks the installed version of each key component (torch / transformers / fastapi, etc.) inside the base image, marking missing ones as MISS. Use it at delivery acceptance or when troubleshooting to answer "is the base complete?".
5. Troubleshooting table
| Symptom | Possible cause | Resolution |
|---|---|---|
/v1/health license_required (ready:false) | License not activated; AI model not loaded | Activate the license per 02 "License (offline authorization)" |
/v1/health warming and stuck for a long time | Model load failed (insufficient VRAM/memory, corrupt image) | Check logs docker compose logs databrain; confirm VRAM ≥8GB, memory ≥16GB |
Audit page stops receiving new records + console audit_disk_stopped alert | Free disk below 2 GB; detection-log write-stop protection | Clean up disk (the detection log rotates daily — old files can be removed); writing resumes once free space recovers to ~3 GB, no restart needed |
| Exits immediately on start (Exited) | Config / port / permission issue | Check logs; check whether port 18000 is occupied, whether the compose file is complete |
Needs GPU but shows profile=cpu | nvidia-container-toolkit not installed, or DATABRAIN_FORCE_CPU=1 was set | Confirm the host has the toolkit installed (nvidia-smi works in-container); check install-time environment variables |
| Intermittent 503/504 | Cold-start window or momentary overload | Probe health before calling; client timeout ≥30s; control concurrency per tier |
422 returned | Invalid request body | Check text is non-empty and ≤100,000 chars, min_confidence within 0–1 |
Scan returns 403 license_required | Not activated / license invalid | Activate per 02 "License (Offline Authorization)"; check reason in /v1/license/status |
license/status valid:false (expired) | License expired | Renew & re-issue (reuse the original machine code if the host is unchanged); still usable within the grace period |
license/status valid:false (fingerprint_mismatch) | Host hardware changed | Re-collect the machine code bash databrain-ops.sh fingerprint, then re-issue |
license/status valid:false (revoked) | License revoked | Contact the issuer to re-issue or update the authorization |
license/status valid:false (missing_asset_key / empty_asset_key / malformed_asset_key) | The license file was mis-issued, or a license from another product was used | Contact the vendor to verify and re-issue |
| Actual tier below hardware capability | Limited by the license tier | This is normal; for higher throughput, obtain a higher-tier license |
docker compose command not found | Old Docker (v1) | Upgrade to Docker with the v2 plugin |
| Port 18000 occupied | Port conflict | Change the compose port mapping (e.g. 18000:8000; in-container 8000 stays unchanged) |
| Memory/VRAM keeps growing | Abnormal load or extremely long text | Limit single-request text length; check for oversized concurrent batches |
| Accuracy anomaly after upgrade | Rare; possibly a version mismatch | Roll back to the previous version first, then contact Siipulse support |
License renewal
Renew the license before it expires to avoid scan interruptions:
- Host hardware unchanged → you can reuse the original machine code and request a new
license.jsonfrom the issuer; - Host hardware changed → re-collect the machine code
bash databrain-ops.sh fingerprint, then request it; - Load the new license and restart to take effect:
bash databrain-ops.sh activate license.json(= replacelicense/license.json+ restart + verify); - After expiry you enter a grace period (per authorization type); once the grace period ends, scanning returns
403; after renewal it recovers immediately — no image reinstall needed.
Upgrading the application image (02 "Upgrade / Rollback") does not touch the license volume, so upgrade ≠ renewal; the two are independent.
6. Reading the logs
The startup log shows the following key signals in order (exact format subject to actual output):
- Hardware detection completes and the effective tier is determined (e.g.
profile=gpu-large, i.e. "GPU-large"); - The engine finishes loading and reports the concurrency cap for that tier;
- The HTTP service starts listening on the port (8000 in-container, 18000 published on the host).
7. Deploying on Kubernetes and other orchestration platforms
The delivery is provided by default as Docker Compose. To deploy on Kubernetes or similar platforms:
- Import the image into your registry (
docker loadthendocker tag+ push, or use the offline package directly); - Run as a Deployment with one replica (or more for horizontal scale-out, as needed);
- Point readiness/liveness probes at
GET /v1/health, with the ready conditionready:true; - In GPU mode, declare GPU resources the platform way (e.g. K8s
nvidia.com/gpu).
For official Kubernetes manifests or a Helm Chart, contact Siipulse.
8. Getting support
When troubleshooting, prepare the following information to help locate issues quickly:
- DataBrain version number (the TAG of
docker images siipulse-databrain, orbash databrain-ops.sh version); - Full startup logs (
docker compose logs databrain); - Hardware configuration (CPU/memory, GPU model and VRAM, operating system);
- Reproduction steps and request/response (please redact before sending; do not send real PII).
08Frequently Asked Questions (FAQ)
No. DataBrain runs fully offline. All scanning and judgment happens inside the customer's server — it does not connect to Docker Hub or any external source, makes no outbound network request, and reports no statistics. From input to output, data stays within the customer's network. See 06-Security & Privacy.
The AI recognition model is based on multilingual pre-training and covers 100+ languages, including Chinese, English, Japanese, Korean, Hindi, and many European languages.
Yes. CPU mode is fully functional and identical to GPU in accuracy — only speed differs, roughly 1/19–1/33 of GPU. It suits functional validation, low-frequency scanning, or GPU-less environments. For latency-sensitive production scenarios, GPU is recommended (VRAM ≥ 8 GB, system memory ≥ 16 GB). See 01-Hardware & Software Requirements.
needs_review mean? Is it a failed detection?No. needs_review=true means the value has been detected as some type of sensitive data, but the system is under-confident about the type assignment (e.g. a national ID and a passport number have similar formats and are neck-and-neck); manual review is recommended. It is still a valid prediction and is counted in accuracy. In production you decide per business policy whether to trust it directly or route it to review.
min_confidence to?- 0.5 (default, balanced tier): returns only results likely to be real sensitive data; fits the vast majority of production scenarios;
- 0.0 (high-recall tier): returns all candidates with confidence scores, for audit / inventory "better safe than sorry" scenarios, letting the caller filter downstream by confidence.
/v1/text/scan caps a single text at 100,000 characters; anything beyond is segmented on the caller side.
Each result carries start/end (character positions in the original text); you can directly replace the corresponding span with *** or use them for highlighting.
Use the value-level HTTP endpoint POST /v1/values/scan (batch classification, ≤1000 per batch). Pass each column's values with the column name as label_hint. See section 2 of 03-API Integration Guide.
Use the delivered new-version siipulse application package and run bash siipulse-rel_databrain_<new-version>.sp. The package self-extracts, auto-reuses the on-host BaseImage, and completes load / health-gate / rollback-on-failure. No network needed. See 02-Installation & Startup.
No. Configuration is packaged with the image; at startup the runtime config is generated automatically for the current hardware, with no manual intervention. Business data is not inside the container, so upgrades are unaffected.
If a type is not in the 04-PII Type Catalog, contact Siipulse to evaluate adding it. DataBrain has a self-learning capability of "discover a new type → auto-generate samples → retrain the model"; extending to new types is one of its design capabilities.
No. The container bundles its entire runtime dependency set (AI inference framework, GPU runtime, etc.), fully isolated from the host's native environment. In GPU mode the only shared component is the GPU driver.
Run bash databrain-ops.sh verify (one command: health check + sample scan with email/phone, confirming end-to-end usability). You can also curl http://127.0.0.1:18000/v1/health or bash databrain-ops.sh health at any time (run from the install directory).
Docker Compose can scale to multiple instances + a front load balancer; on Kubernetes or similar platforms, use a multi-replica Deployment with a readiness probe pointed at /v1/health. Note that each replica independently loads the AI model (consuming VRAM), so plan replica count by hardware capacity.