DataBrain

DataBrain User Manual

An AI-native sensitive-data classification engine — given any text or structured column, it automatically discovers the sensitive data (PII) within and outputs confidence-scored classification. This manual covers everything from installation and API integration to operations and troubleshooting.

32 PII types Fully offline Data never leaves your network

How to use this manual: for first-time integration, start from "Quick Start" at the top of 02 · Installation & Startup to run a scan end-to-end in 5 minutes; for production deployment see the same chapter; to integrate the engine into your business system see 03 · API Integration Guide. The contents on the left let you jump anywhere.

REQUIREMENTS

01Hardware & Software Requirements

DataBrain ships as a single Docker image. On startup it auto-detects the hardware and selects the appropriate tier — no manual tuning required.

1. Base prerequisites

ComponentRequirement
Operating systemLinux x86_64 (Ubuntu 20.04+ / CentOS 8+ / RHEL / Rocky / Alma)
Docker Engine + compose v2 pluginInstalled (the command is docker compose, not the legacy docker-compose)
curlInstalled

GPU mode additionally requires: NVIDIA GPU driver (kernel module) + NVIDIA Container Toolkit. CPU mode can skip this.

The image bundles the entire runtime (Python / torch / transformers / CUDA runtime, etc.), so no CUDA Toolkit install is needed; after docker load it connects to no external network. The full locked dependency list is in requirements.lock inside the delivery package.

Docker install example (only if Docker is not yet installed on the host)

Ubuntu / Debian:

sudo apt-get update && sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update && sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo usermod -aG docker $USER   # log out and back in to take effect

CentOS / RHEL / Rocky / Alma 8 (CentOS 8 is EOL; needs vault redirection + runc conflict resolution + SELinux relaxation):

sudo sed -i -e 's|^mirrorlist=|#mirrorlist=|' -e 's|^#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|' /etc/yum.repos.d/CentOS-*.repo
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf install -y --allowerasing docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo setenforce 0 || true; sudo sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
sudo usermod -aG docker $USER; sudo systemctl enable --now docker

GPU: after installing Docker, install nvidia-container-toolkit (Ubuntu apt / RHEL dnf), then sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker.

Verify: docker --version && docker compose version.

ModeGPUMemoryCPUDisk
GPU production (recommended)NVIDIA, VRAM ≥ 8 GB≥ 16 GB≥ 4 cores≥ 15 GB
CPUNone≥ 16 GB (8 minimum)≥ 4 cores≥ 15 GB

On startup the tier is selected automatically by VRAM: ≥ 8 GB → GPU-large; has GPU but < 8 GB → GPU-small; no GPU → CPU. All three tiers have identical accuracy — only speed differs (CPU is roughly 1/19–1/33 of GPU). The active tier is shown in the log as profile=….

WSL2 / VMs without SMBIOS: the GPU generally cannot be passed through (automatic fallback to CPU), suitable for CPU functional validation / CI; the machine fingerprint (the hardware ID reported as the *machine code* in logs and CLI output) is stable — zero drift across upgrades, reboots, and container rebuilds.

3. Ports

PortPurpose
18000Inference API (/v1/text/scan, /v1/health); port 8000 inside the container
18001Management console WebUI (HTTPS, self-signed certificate — see 02 §Management console); port 8001 inside the container (independent from inference)

To change a port: at install time DATABRAIN_PORT=<port> bash <databrain>.sp, or edit the ports in docker-compose.prod.yml.

INSTALL

02Installation & Startup

The .sp file is a self-extracting installer: bash <file>.sp is a single command that performs loading, CPU/GPU detection, startup, health gating, and sample verification.

Two packages (the actual filenames in your delivery may differ; test builds may be siipulse-beta_*):

  • siipulse-rel_baseimage_1.0.sp (~2.9 GB): the shared Docker dependency base, installed once per machine, reused by all versions / products.
  • siipulse-rel_databrain_<ver>.sp (~0.7 GB): the application package, re-issued every release. The version number = the image tag = the trailing digits of the filename.

Quick start

# 0) One-time host preparation (needs sudo; creates the install directory + opens firewall 18000/18001)
sudo bash siipulse-rel_databrain_<ver>.sp --prepare-host

# 1) Install (enter the install directory created in step 0 — its path is printed by --prepare-host; place both .sp files there; the app package auto-installs BaseImage first, then starts the service)
cd <install-dir>
bash siipulse-rel_databrain_<ver>.sp

# 2) Activate the license (first install prints the machine code; send it to the vendor to obtain license.json)
bash databrain-ops.sh fingerprint
bash databrain-ops.sh activate license.json

# 3) Confirm readiness + first scan
bash databrain-ops.sh status
curl -X POST http://127.0.0.1:18000/v1/text/scan -H 'Content-Type: application/json' \
  -d '{"text":"Contact John Doe, email john.doe@google.com, phone +1-415-555-0142."}'

Multilingual: DataBrain handles Chinese input the same way —

curl -X POST http://127.0.0.1:18000/v1/text/scan -H 'Content-Type: application/json' \
  -d '{"text":"联系人张伟,邮箱 zhangwei@google.com,手机 +86 13800138000。"}'

If the license is not activated on first install, the installer reports "Awaiting license activation" (exit code 0, not a failure) and prints the machine code — take it to step 2; after activation the service becomes ready in ~60s automatically, no reinstall needed.

Optional environment variables: DATABRAIN_PORT, DATABRAIN_FORCE_CPU=1, DATABRAIN_FORCE_GPU=1 (the latter two are mutually exclusive). One-command smoke test: databrain-ops.sh verify.

Upgrade / rollback

bash siipulse-rel_databrain_<new-version>.sp  # installer path: auto-reuses the on-host BaseImage; auto-rolls back on failure
bash databrain-ops.sh upgrade <new-version>   # safe in-place upgrade when the image is already loaded (multi-step guard chain, auto-rollback on any failure)
bash databrain-ops.sh revert                  # undo the most recent upgrade

The license lives in the license/ volume and is never touched by an upgrade, so no re-activation is needed.

Upgrade vs first install: an upgrade preserves the license volume and existing install state (keeping the deployment rollback-safe); a first install initializes this state from scratch.

⚠️ Do not bypass the installer: never work around the install/upgrade flow by hand-editing installation state files — this breaks the rollback and consistency guarantees and can leave the deployment unable to roll back or falsely reporting success. Always use databrain-ops.sh upgrade <ver> or bash <new-version>.sp (validation and automatic rollback are guaranteed by the installer).

License (offline authorization)

DataBrain uses offline authorization. Scanning endpoints (/v1/text/scan, /v1/values/scan) require an activated license first, otherwise they return 403 license_required; /v1/health and /v1/license/status are always open.

Activation (commands are in "Quick start" step 2 above): collect the machine code from this host → send it to Siipulse in exchange for license.json → load and activate it.

The license file is named license.json, but its content is a single-line authorization text issued by the vendor (not a JSON object); do not hand-edit or fabricate it — hand it to activate as-is; malformed files are rejected.

  • The license is bound to this machine; upgrades, reboots, and container rebuilds on the same machine require no re-activation.
  • When replacing the host or migrating to another machine, you must re-collect the machine code on the new machine and re-issue the license.
  • Verify: curl http://127.0.0.1:18000/v1/license/status should return valid:true.

Management console (port 18001, HTTPS)

After the container starts, the console is served at https://<host>:18001/ on the same host, with default credentials admin/admin (you are prompted to change the password on first login). It shares the same license as the inference API. The console provides:

  • Dashboard: run overview and key metrics.
  • System management: system status / license import / .sp version management / restart.
  • Self-learning: new-type discovery and corpus-bundle management.
  • Playground: live test scans.
  • Audit: detection-log review / human labeling / statistics and pagination.
  • Troubleshooting: diagnostic bundling.
  • Top-bar health alerts: persistent status banner.

HTTPS notes (enabled by default since 2026-08-21)

  • Certificate mechanism: the console is served via native TLS with a self-signed certificate generated automatically on first start (ECDSA P-256, 10-year validity, SAN covers the host IP + localhost, determined by DATABRAIN_TLS_HOST_IP written to .env by the installer).
  • First-visit warning: private-IP hosts cannot obtain a public CA signature, so the browser shows "Your connection is not private" — click Advanced → Proceed to continue.
  • Certificate persistence: the certificate lives in the console-data/ volume; container upgrades/rebuilds keep it, no re-trusting needed.
  • Eliminating the warning: import console-data/tls/console.crt (in-container /app/webui/tls/console.crt) into the OS / enterprise certificate trust store.
  • Falling back to HTTP: set DATABRAIN_CONSOLE_TLS=0 in .env (local debugging only).
  • Public deployments: still recommended to front nginx to terminate TLS (with a proper domain certificate) + an IP allow-list.

Operations commands (databrain-ops.sh)

start | stop | restart | status | logs -f | health | license | verify | fingerprint | activate <file> | upgrade <ver> | rollback <ver> | revert | clear | deps | diagnose | version

Usable from any directory: add the install directory to your PATH (export PATH=<install-dir>:$PATH). For more troubleshooting see 07-Operations & Troubleshooting; for response field semantics see 03-API Integration Guide.

Common issues

SymptomResolution
/v1/text/scan returns 403 license_requiredLicense not activated; follow Quick start step 2
license/status valid:false (fingerprint_mismatch)Host hardware changed, or moved to another machine → re-run fingerprint and re-issue; expired → renew; revoked → contact the vendor
Reports base image siipulse-baseimage:1.0 is not loadedPut the BaseImage .sp and the app package in the same directory and re-run the app package
Container exits immediately with No space left on deviceUse the delivered compose (container shared memory is already configured correctly); do not run bare docker run
Health check ready:falsewarming = model loading (wait 30–70s); license_required = not activated
Needs GPU but runs CPUHost missing nvidia-container-toolkit, or DATABRAIN_FORCE_CPU=1 was set during reinstall
Port 18000 / 18001 already in useDATABRAIN_PORT=<port> only changes the inference port; the console port 18001 must be changed manually in the compose ports

ops-agent (optional · WebUI remote upgrade / restart)

Once installed, the console's "System Management" page can trigger restarts and .sp version upgrades remotely over a local secure channel (the service is unavailable for ~30–90s during container rebuild, with automatic rollback on failure). Once per machine (the app must already be installed):

sudo bash siipulse-rel_databrain_<ver>.sp --install-ops-agent
API

03API Integration Guide

DataBrain exposes its sensitive-data discovery capability over HTTP. Choose the integration path based on the shape of your data:

Data shapeRecommended pathEntry point
Free text / documents (email bodies, support transcripts, contract clauses, logs…)Document-level POST /v1/text/scanDiscover entities within "one text"
Structured columns (database fields, table columns, CSV column values…)Value-level POST /v1/values/scanClassify a "batch of values"

/v1/text/scan and /v1/values/scan share the same recognition engine and produce identical decisions. The only difference is the data source: /v1/text/scan locates entities within a text and returns character offsets; /v1/values/scan judges a batch of independent values one by one and returns confidence.

Every decision carries decision-provenance fields (method / role, always returned) and optional decision evidence (evidence, returned on request), making each decision traceable and auditable. Field semantics are in section 3.


1. Document-level scan POST /v1/text/scan

For unstructured free text. The engine locates all sensitive entities within a text and returns their type, character offsets, confidence, and decision provenance.

1. Health check

GET /v1/health
{"status":"ok","ready":true}
FieldMeaning
status"ok" service ready / "warming" model loading
readytrue means the AI model has finished loading and the service is callable

2. Sensitive-data scan

POST /v1/text/scan
Content-Type: application/json

{
  "text": "Text to scan (1–100,000 characters)",
  "min_confidence": 0.5,
  "return_evidence": false
}

Request parameters

FieldTypeRequiredDefaultDescription
textstringyesText to scan, length 1–100,000 characters (returns 422 if exceeded)
min_confidencenumberno0.5Emission threshold, range 0–1; see guidance below
return_evidenceboolnofalseWhether to include decision evidence in results (see section 3)

min_confidence guidance

  • 0.5 (default, balanced tier): returns only results likely to be real sensitive data; fits mainstream production scenarios.
  • 0.0 (high-recall tier): returns all candidates, each with a confidence score, prioritizing recall and letting downstream filter by confidence; for audit, data inventory, and other "better safe than sorry" scenarios.

Response

{
  "results": [
    {
      "value": "zhangwei@example.com",
      "pii_type": "EMAIL",
      "start": 11,
      "end": 31,
      "confidence": 0.99,
      "needs_review": false,
      "method": "truth_table",
      "role": "reference"
    }
  ]
}
FieldTypeDescription
valuestringThe matched original text span
pii_typestringSensitive-data type (see 04-PII Type Catalog, e.g. EMAIL/PHONE/ID_CARD)
start / endnumberCharacter start/end offsets in the original text; usable for highlighting and redaction
confidencenumberConfidence 0–1, calibrated as "the true probability of being that type"
needs_reviewbooltrue means the value was detected but the type assignment is under-confident (e.g. two types are neck-and-neck); manual or rule-based review is recommended
methodstringDecision path (always returned); see section 3
rolestringSemantic role (always returned); see section 3
evidenceobjectDecision evidence, returned only when return_evidence=true

A needs_review result is still an emitted, valid prediction (already counted in accuracy statistics), not a "failed detection" — it only flags that the item warrants review. In production you may trust it directly or route it to a review flow, depending on business tolerance.

Error codes

HTTP statusMeaning
200Success
403License not activated (error: license_required); diagnostic endpoints are unaffected
422Invalid request body (e.g. text empty, over 100,000 characters, min_confidence out of range)
503 / 504Service not ready or timed out (check ready)

2. Value-level batch classification POST /v1/values/scan

For scanning structured column values such as database tables, CSV / Excel columns, or data-lake fields. Operates on a "batch of values", judging each value independently and returning confidence.

POST /v1/values/scan
Content-Type: application/json

{
  "values": [
    {"value": "zhangwei@example.com", "value_id": "r1", "label_hint": "email"},
    {"value": "110101199003077334", "value_id": "r2", "label_hint": "id_card"},
    {"value": "13800138000", "value_id": "r3", "container_path": "customer.phone"}
  ],
  "return_evidence": false
}

Request parameters

FieldTypeRequiredDefaultDescription
valuesarrayyesList of values to classify, 1–1000 items
values[].valuestringyesA single value, length 1–10000 characters
values[].value_idstringnoauto-generatedCaller-side correlation ID, echoed back as-is; if omitted, generated as customer:request:index
values[].label_hintstringnolast segment of container_pathColumn / field-name hint; significantly improves accuracy (see section 4); an explicit value takes precedence over container_path
values[].container_pathstringnoColumn path, e.g. users.email; when label_hint is not supplied the system auto-derives the hint from its last segment (see section 4)
values[].surrounding_textstringno""Text surrounding the value (≤ 2000 characters; should contain an occurrence of the value itself — otherwise only the context channel is inert, other signals are unaffected); improves accuracy for context-dependent types
return_evidenceboolnofalseWhether to include decision evidence in results (see section 3)

The value-level endpoint returns all decisions (including values judged "non-sensitive", whose pii_type is null), each with a confidence score; acceptance is up to the caller, which filters by confidence (there is no min_confidence emission threshold at value level, unlike document level).

Optional request header X-DataBrain-Consent (data-reflow consent statement): it only affects the local detection log and self-learning capture (semantics in 06-Security & Privacy §9) and does not change decisions.

Response

{
  "results": [
    {"value_id": "r1", "value": "zhangwei@example.com", "pii_type": "EMAIL",   "confidence": 0.99, "needs_review": false, "is_mock": false, "method": "truth_table", "role": "subject"},
    {"value_id": "r2", "value": "110101199003077334",    "pii_type": "ID_CARD", "confidence": 0.92, "needs_review": false, "is_mock": false, "method": "truth_table", "role": "identifier"}
  ]
}
FieldTypeDescription
value_idstringEchoed input ID, for alignment
valuestringOriginal value
pii_typestring | nullJudged sensitive type (see 04-PII Type Catalog); null means judged non-sensitive
confidencenumberConfidence 0–1, calibrated as "the true probability of being that type"
needs_reviewbooltrue recommends manual review
is_mockboolWhether the value looks like sample / test fake data (e.g. 000-00-0000)
methodstringDecision path (always returned); see section 3
rolestringSemantic role (always returned); see section 3
evidenceobjectDecision evidence, returned only when return_evidence=true

Error codes

HTTP statusMeaning
200Success
403License not activated (error: license_required); diagnostic endpoints are unaffected
422Invalid request body (e.g. missing values, empty list, over 1000 items, a single value too long)
503 / 504Service not ready or timed out (check ready)

3. Decision-provenance fields (method / role / evidence)

Every DataBrain decision carries provenance information, making the decision process transparent, verifiable, and auditable — not a black-box output.

method — decision path

Identifies which engine path produced this result, reflecting the signal strength and confidence tier behind the decision:

ValueMeaning
truth_tableHigh-confidence direct: judged by the engine's combined assessment, with sufficient confidence — accept directly
needs_reviewBorderline / low-confidence: detected as this type but below the direct-emit threshold; manual or rule-based review recommended (the needs_review field is true in this case)
mock_filterSample-data filter: judged as suspected sample / test fake data
non_sensitive_filterPre-filter exclusion: judged clearly non-sensitive by rule pre-filtering and never entered the main recognition flow (only non-sensitive results of /v1/values/scan may show this)

The vast majority of high-confidence results take the truth_table path; results whose confidence is borderline or in a blind spot (two types neck-and-neck, sparse borderline samples) take the needs_review path. The engine performs fully local inference (it calls no external model and makes no outbound network request).

Closed set: the method value set (the four values above) is part of the external contract — when engine evolution adds a value it will be registered in this table with the release. Downstream consumers should not match values outside the table (treat unmatched as unknown).

role — semantic role

Inferred from the column / field name (label_hint), indicating the value's semantic role in the business, to assist downstream policy decisions:

ValueMeaning
subjectThe value is itself the sensitive subject (e.g. column named email / phone / passport)
identifierAn identifying / reference key (e.g. column named id / account_no / ref)
referenceDefault reference role (no label_hint, or the column name carries no clear semantics)

Document-level /v1/text/scan has no column context, so role is always reference; at value level /v1/values/scan, role is determined by the supplied label_hint. For results judged non-sensitive (pii_type is null), role is likewise null.

evidence — decision evidence (on request)

evidence is not returned by default; it is attached only when return_evidence=true is requested. It carries the objective evidence supporting the decision, for example:

Evidence keyMeaning
provider / provider_confRecognized brand / issuer (e.g. bank-card brand) and its confidence
country / subtype keysRecognized country / region / subtype
ns_reject_reasonReason for rejection when judged "non-sensitive"

evidence can carry a lot of detail, so it is designed as opt-in: keep the default false in normal integration to keep responses lean; enable return_evidence=true for audit, forensics, and policy-tuning scenarios that need the full rationale.


4. label_hint notes (value-level only)

label_hint is the column / field-name hint supplied in the request (e.g. email, id_card, passport). It is a strong clue that assists the engine's decision and notably improves accuracy on pure-digit, format-similar identifiers (e.g. national ID vs passport vs social-security number).

  • How to supply: set the label_hint field on each object in values[];
  • When label_hint is not supplied, the system auto-derives it from the last segment of container_path (e.g. customer.phonephone, users.emailemail) — so as long as you put the real column / field name into container_path, you get the same boost as an explicit label_hint, with no duplicate entry;
  • It only takes effect when the column name matches a known type word (e.g. phone / email / ssn / id_card / passport / dob); meaningless names (e.g. col_0, value1) are safely ignored and do not affect the decision;
  • An explicit label_hint always takes precedence; if neither is supplied, the engine still judges normally;
  • Production tip: database field names and CSV headers are naturally high-quality column-name hints — pass them in via container_path or label_hint.

5. Complete call examples

curl

English sample:

curl -X POST http://127.0.0.1:18000/v1/text/scan \
  -H 'Content-Type: application/json' \
  -d '{"text":"Order #A12345, customer john.doe@google.com, phone +1-415-555-0142.","min_confidence":0.5}'

Chinese sample (same engine, multilingual):

curl -X POST http://127.0.0.1:18000/v1/text/scan \
  -H 'Content-Type: application/json' \
  -d '{"text":"订单 #A12345,客户 john.doe@google.com,电话 +1-415-555-0142。","min_confidence":0.5}'

Python (HTTP)

import requests

resp = requests.post(
    "http://127.0.0.1:18000/v1/text/scan",
    json={"text": "Customer john.doe@google.com, phone +1-415-555-0142.", "min_confidence": 0.5},
    timeout=30,
)
for m in resp.json()["results"]:
    print(m["pii_type"], m["value"], m["confidence"], m["method"])

Java

// Uses JDK 11+ HttpClient
import java.net.http.*;
import java.net.URI;

var body = "{\"text\":\"Customer john.doe@google.com, phone +1-415-555-0142.\",\"min_confidence\":0.5}";
var req = HttpRequest.newBuilder()
    .uri(URI.create("http://127.0.0.1:18000/v1/text/scan"))
    .header("Content-Type", "application/json")
    .POST(HttpRequest.BodyPublishers.ofString(body))
    .build();
var resp = HttpClient.newHttpClient().send(req, HttpResponse.BodyHandlers.ofString());
System.out.println(resp.body());

C# (.NET)

using var http = new HttpClient();
var body = """{"text":"Customer john.doe@google.com, phone +1-415-555-0142.","min_confidence":0.5}""";
var content = new StringContent(body, System.Text.Encoding.UTF8, "application/json");
var resp = await http.PostAsync("http://127.0.0.1:18000/v1/text/scan", content);
Console.WriteLine(await resp.Content.ReadAsStringAsync());

Go

body := []byte(`{"text":"Customer john.doe@google.com, phone +1-415-555-0142.","min_confidence":0.5}`)
resp, err := http.Post("http://127.0.0.1:18000/v1/text/scan",
    "application/json", bytes.NewReader(body))
// ...read resp.Body

Value-level batch (/v1/values/scan)

import requests

resp = requests.post(
    "http://127.0.0.1:18000/v1/values/scan",
    json={"values": [
        {"value": "john.doe@google.com", "label_hint": "email"},
        {"value": "123-45-6789",          "label_hint": "ssn"},
    ]},
    timeout=30,
)
for r in resp.json()["results"]:
    print(r["value_id"], r["pii_type"], r["confidence"], r["method"], r["role"])

With decision evidence (return_evidence=true)

curl -X POST http://127.0.0.1:18000/v1/values/scan \
  -H 'Content-Type: application/json' \
  -d '{"values":[{"value":"4111 1111 1111 1111","label_hint":"credit_card"}],"return_evidence":true}'

More runnable examples are in the examples/ directory.


6. Integration best practices

  1. Startup dependency check: probe /v1/health before calling, and only send requests once ready:true, to avoid 503 during cold start.
  2. Timeout: a single /v1/text/scan client timeout should be ≥ 30 seconds (the first request includes model warm-up); in steady state a single text is usually millisecond-level.
  3. Large batches of text: one HTTP request per text; for huge document volumes, concurrency on the caller side is recommended (the server-side concurrency cap depends on the hardware tier, see 05-Performance & Capacity); for very large batches of structured data use value-level /v1/values/scan (call in batches of ≤1000 per request).
  4. Handling needs_review: set policy by business — risk / compliance scenarios should do manual or secondary rule-based review; general scenarios can trust the type label directly.
  5. Use provenance: method lets you track decision-path distribution and the share of borderline cases; enable return_evidence when you need the full rationale, for audit, forensics, and policy tuning.
  6. Idempotency: the same input always yields the same output — feel free to retry, cache, and build comparison baselines.
  7. Offset usage: start/end can be used directly for front-end highlighting and redaction (e.g. replace the [start,end) span with ***).
PII TYPES

04PII Type Catalog

DataBrain supports 32 types of sensitive data, grouped into four categories by sensitivity and detection reliability. The pii_type field returned by the system takes its values from the "Type" column in the tables below.

Category overview

CategoryMeaning
I-A Strong identifiersHigh-sensitivity identifiers with mathematical validation
I-B IdentifiersIdentifiers with a clear format or validation rule
II-A Common PIIContext-dependent common personal information
II-B Intrinsic attributesIntrinsic attributes of a person (name, demographics, occupation)

I-A / I-B mostly carry "mathematical / format validation" and are the most reliably decided; II-A / II-B rely more on AI context understanding.


I-A · Strong identifiers (4 types, with mathematical validation)

TypeMeaningExample
CREDIT_CARDBank card number4111 1111 1111 1111
IBANInternational bank account numberDE89 3704 0044 0532 0130 00
SECRETKeys / access credentials (cloud-vendor API keys, access tokens, private keys, etc.)AKIA… / ghp_… / sk_live_…
AADHAARIndian national ID (Aadhaar)12 digits

I-B · Identifiers (16 types, with format / validation)

TypeMeaningExample
ID_CARDIdentity card / national ID card110101199003077334
TAX_IDTax / VAT numberDE123456789
SOCIAL_SECURITY_NUMBERSocial security number123-45-6789
PASSPORTPassport number
SWIFT_BICBank identifier code (SWIFT/BIC)DEUTDEFF500
IP_ADDRESSIP address (IPv4/IPv6)192.168.1.1
MAC_ADDRESSNetwork interface physical address00:1A:2B:3C:4D:5E
IMEIMobile equipment identity35-209900-176148-1
VEHICLE_IDVehicle identification number (VIN)17 chars
CRYPTO_ADDRESSCryptocurrency wallet address1A1zP1eP… / 0x…
UPIIndian unified payment addressname@oksbi
IFSCIndian bank branch codeSBIN0001234
CN_LICENSE_PLATEMainland-China motor-vehicle license plate京A12345
CN_TRAVEL_PASSMainland Travel Permit for Hong Kong/Macau Residents (Home Return Permit)H12345678
CN_SECURITIES_ACCOUNTChina securities account number (A/B shares / funds)A123456789
GEO_LOCATIONGPS latitude/longitude coordinate pair (location trace)39.9042, 116.4074

II-A · Common PII (9 types, context-dependent)

TypeMeaningExample
EMAILEmail addressjohn@example.com
PHONEPhone / fax+86 138 0013 8000
ADDRESSPostal address123 Main St, Springfield / 北京市朝阳区…
BANK_ACCOUNTBank account number
DRIVER_LICENSEDriver's license number
DATE_OF_BIRTHDate of birth1990-03-07
URLWeb URLhttps://example.com
PASSWORDPassword
USERNAMEUsername / account namejohn_doe

II-B · Intrinsic attributes (3 types, AI semantic recognition)

TypeMeaningExample
NAMEPerson nameJohn Smith / 张伟
DEMOGRAPHICDemographic attributes (gender, age, height, eye color, etc.)Male, 32 / 男,32 岁
EMPLOYMENTEmployment info (job title, employer)Product Manager @ Acme

Coverage breadth

  • Languages: the AI recognition model is based on multilingual pre-training and covers 100+ languages (Chinese, English, Japanese, Korean, Hindi, many European languages, etc.).
  • Regions: national IDs cover 30+ country formats and tax IDs cover 20+ country formats (per the built-in type definitions), including mainland-China national ID / license plate / travel permit / securities account, Indian Aadhaar/PAN/UPI/IFSC, Japan My Number, various European/American IDs, etc.
  • Regulatory alignment: the types above cover the sensitive-data categories regulated by mainstream data-protection laws (see the brochure's "Regulatory alignment" section).

If a type you need is not in the tables above, contact Siipulse to evaluate adding it — DataBrain has a self-learning capability of "discover a new type → generate samples → retrain" (see the brochure's "Three-loop self-learning" section).

PERFORMANCE

05Performance & Capacity

This chapter gives measured reference performance and hardware-sizing guidance, to help estimate throughput and capacity.

All numbers are reference measurements (test hardware: NVIDIA RTX 5070 GPU, single machine). Actual performance depends on hardware model, text length, and load characteristics.


1. Latency & throughput (reference)

Document-level (HTTP /v1/text/scan, single text)

MetricGPU tier (host direct)In-container (GPU tier)
Avg latency per text~8.6 ms/text~9.8 ms/text
Single-stream throughput~116 texts/sec~100 texts/sec

Value-level (batch classification benchmark)

Hardware tierPer-value latencyThroughput
GPU tier~1.6 ms/value~640 values/sec
CPU tier~29.5 ms/value~34 values/sec

CPU and GPU are identical in accuracy; the only difference is speed — the CPU tier is roughly 1/19–1/33 of the GPU tier, with the bottleneck on AI inference compute.


2. Concurrency & throughput notes

  • Per-tier concurrency caps: CPU tier 4 / GPU-small 8 / GPU-large 16;
  • Multiple requests can overlap concurrently; effective throughput rises with concurrency until GPU compute saturates;
  • Callers are advised to do client-side concurrency control according to the hardware tier's cap, to avoid queuing under overload.

3. Sizing guidance

ScenarioRecommended tierNotes
Production, latency/throughput sensitiveGPU-large (≥ 8 GB VRAM)Best price/performance
Medium traffic, limited VRAMGPU-smallAuto-applied
Functional validation / low frequency / no GPUCPU tierUsable; speed ~1/19–1/33 of GPU
Massive structured-data batch scan (e.g. full-DB redaction)GPU-large + value-level /v1/values/scanBatch value classification, high throughput

4. Capacity estimation example

Estimated with GPU-large, document-level single-stream ~100 texts/sec:

Daily volumeSingle-machine time (serial)Notes
1 million texts~2.5 hoursSingle-stream serial; concurrency shortens this significantly
10 million texts~25 hoursSharding + horizontal scale-out recommended

Longer text and denser PII slightly increase per-text cost. Value-level batch-scan throughput is notably higher than document-level (no text-segmentation overhead).


5. VRAM & memory footprint

  • AI model: ~0.7 GB (multilingual sensitive-data recognition model, delivered encrypted inside the image, loaded into VRAM at startup);
  • Dependency layer: ~2.7 GB (AI runtime stack, disk footprint, not resident in VRAM);
  • Runtime peak: VRAM ≥ 8 GB and system memory ≥ 16 GB recommended;
  • Disk: image ~3.4 GB; recommend reserving ≥ 15 GB (including 2-version rollback headroom). The detection log rotates daily with a default 180-day retention and is protected by a low-disk-space guard (auto-stops writing below 2 GB free; see 06-Security & Privacy).
SECURITY

06Security & Privacy

This chapter describes DataBrain's security design across deployment topology, data handling, supply chain, and offline authorization.


1. No data egress (offline deployment)

  • Fully offline operation: the image is loaded from an offline package; at runtime it does not connect to Docker Hub, does not connect to any external source, and makes no outbound network request.
  • Data never leaves the customer network: all sensitive-data scanning and judgment happens entirely inside the customer's own server/container; results are returned only to the caller.
  • No telemetry, no callbacks: the system neither collects nor reports any scanned content or statistics to Siipulse or any third party.

That is, from the moment data enters DataBrain to the moment the result is produced, it stays within the customer's controllable environment.

2. Data persistence posture

  • Scanning is real-time compute by default: input text is recognized in memory and the result emitted, with no on-disk persistence of the scanned content itself; operational logs (startup, health, errors) do not log PII.
  • Detection log (default-on): the decision stream is written to disk for the console audit page (review and human labeling); values in records are redacted by default (only a few leading/trailing characters kept), rotated daily with a default 180-day retention and automatic cleanup on expiry.
  • Low-disk-space guard: if free disk space drops below 2 GB, detection-log writing pauses automatically (keeping the classification service alive); it resumes once space recovers to about 3 GB; the stopped state is surfaced via /v1/health and console alerts.
  • Data-reflow exception (default-on, see §9): when a request carries valid consent, the system retains an encrypted sample of the PII on the customer's local disk (for human review + offline model retraining, AES-256-GCM encrypted), bounded by a dual gate + retention period.

Ordinary scans (no consent header) leave no plaintext on disk — detection-log values are redacted. Two exceptions exist: console-submitted text enters encrypted self-learning capture by default (see §4), and data reflow is a local encrypted retention the customer explicitly enables via consent (see §9). Neither leaves the network.

3. Container security

  • Non-root: the container runs as a dedicated low-privilege user databrain, not root;
  • Dependency isolation: the container bundles its entire runtime dependency set (AI inference framework, GPU runtime, etc.), isolated from the host's native environment, with no mutual interference;
  • Only shared resource: in GPU mode the host and container share only the GPU driver (kernel module); no other system component is shared.

4. Data control rights

ConcernNotes
Data storage locationOrdinary scans store no plaintext; the detection log is stored on-disk redacted (local); data reflow retains an encrypted corpus on the customer's host (output/corpus, no egress)
Data retention periodDetection log defaults to 180 days (daily rotation with automatic cleanup, configurable); data-reflow corpus bundles default to a 90-day retention (bundle_retention_days, configurable); labelable detection-log records from the console (submission text retained as encrypted samples for labeling) live in a separate console-playground bucket (see below)
Console Playground capture (default-on)Console-submitted text enters encrypted self-learning capture (console-playground bucket) for audit-page labeling; opt-out and export are described below
Access controlContainer / host network and permission controls; data-reflow review endpoints require a license + reviewer token + reviewer header (see §9)
Data destructionData reflow supports Art17 withdrawal (/v1/human-review/withdraw, soft delete); stopping / deleting the container clears the runtime state
Response-field minimizationDecision evidence is not returned by default (opt-in via return_evidence=true only); the response contains only the decision and provenance signals (method/role) by default, following data-minimization

Console Playground capture: text submitted by operators in the console Playground enters the self-learning capture pipeline by default (console-playground customer bucket, encrypted retention), feeding the audit page for human labeling corrections. Per-request opt-out: check "do not record" when submitting (request body no_audit: true; the request is then not captured). Channel-wide off: deployment environment variable DATABRAIN_CONSOLE_CONSENT=false (default true). The bucket's retention cleanup and export are explicit ops actions (GET /v1/learning/export-corpus?customer=console-playground); it does not appear in the console's default corpus view.

5. Input limits (abuse prevention)

  • /v1/text/scan caps a single text at 100,000 characters (exceeding returns 422), preventing oversized-payload memory abuse;
  • min_confidence lets the caller control the emission threshold, tuning between "high precision" and "high recall" per scenario.

6. Compliance posture

DataBrain's detection capability covers the sensitive-data types regulated by mainstream data-protection laws (GDPR, CCPA, China PIPL, HIPAA, PCI-DSS, SOC 2, etc.), with type-level mappings established.

Note: DataBrain provides a "sensitive-data discovery" capability that assists with the data discovery and inventory step of compliance requirements. Whether you are "compliant" depends on the overall compliance system (processes, people, controls, etc.); DataBrain is one component, not a compliance certification in itself. See the brochure's "Regulatory alignment" section.

7. Upgrade & supply-chain security

  • Upgrade packages (.sp) are offline incremental packages verified before loading with ECDSA signature validation + payload SHA-256 integrity comparison (the signature is bound to the payload hash, preventing "keep the signature, swap the content"); packages uploaded through the console's version-management page go through the same verification chain;
  • The upgrade script has built-in path-traversal / absolute-path defenses and version-format validation, rejecting tampered packages;
  • Versions are always pinned to an explicit tag (no :latest), for audit and precise rollback.

8. Offline license authorization

DataBrain uses fully offline authorization: all authorization checks happen locally on the customer side, with no outbound network request, no callback to Siipulse, and no online attestation, deployable in strict air-gapped environments.

  • Host binding: the license is bound to the machine — bound to the host, not the container, so container rebuilds on the same machine do not affect authorization; each deployment instance must activate its own license. Replacing the host or migrating to another machine requires re-activation (see 02 "License (Offline Authorization)").
  • Gate scope: /v1/text/scan, /v1/values/scan require a valid license; /v1/health, /v1/license/status are always open for diagnostics.
  • Validity & grace period: a license has an expiry date, followed by a grace period (per authorization type); after the grace period the scan endpoints return 403, while the health / status endpoints stay open; renew before expiry — no reinstall needed.

For the activation flow see 02 "License (Offline Authorization)"; for authorization status and renewal see 07-Operations & Troubleshooting.

9. Data reflow (self-learning data collection)

Data reflow makes DataBrain more accurate over time: it samples "needs-review" verdicts from production traffic → human annotation → export of an encrypted corpus → hand-off to offline model retraining. Sampled data stays encrypted on the customer's own host and never leaves the network.

Not enabling it: no changes needed

Without the request headers below, the scan APIs behave exactly as usual, with zero collection. Collection is guarded fail-closed by two gates (an encryption key auto-generated at deployment + a per-request consent header); if either is missing, nothing is collected at all — the capability ships pre-installed and is activated per request only via the headers.

Enabling collection: add one header to scan requests

X-DataBrain-Consent: {"ts":"<YYYY-MM-DD>","crown_pii":true}
  • This header only affects local sampling and detection logs; it never changes detection results
  • crown_pii: true consents to collecting the PII original values in that request; omitting the header = no collection for that request

Review annotation and corpus management are performed by operators in the console (/self-learning and the review UI); no caller involvement is required. Data subjects exercising erasure rights should contact the operator.

OPERATIONS

07Operations & Troubleshooting

One-click self-check (online diagnostics, preferred)

The console's Diagnostics page (host port 18001 → Diagnostics) provides a one-click self-check that runs 9 checkpoints (in 6 groups) across deployment, startup, and runtime — each with a status light: 🟢 green = normal / 🟡 yellow = abnormal but core functions unaffected / 🔴 red = core functions impaired / ⚪ grey = not applicable (e.g. dependencies that cannot be assessed while the inference service is unreachable).

  • Automatic: runs once after startup (first install / upgrade / manual restart); retries during model loading until a meaningful result converges.
  • Manual: click the "Self-check" button at any time.
  • Check groups: service status (inference service / ops agent) → licensing (license, distinguishing expiring / grace-period / pending-activation states) → model assets (integrity of the recognition-model components) → core function (built-in sample-detection verification, consistent with the pre-release verification criteria) → pipeline health (audit persistence / audit index) → data persistence (storage writes). Resource capacity (memory / disk / GPU), quality gates, drift, and p95 latency are on the dashboard and health-alerting pages.
  • Remediation guidance: every yellow/red light carries handling advice and a jump link (e.g. license activation, audit page, restart from system administration); the raw data is expandable.
  • Proactive alerting: when the overall result is red/yellow, the console's top alert bar shows "Self-check found issues" and can be pushed out via webhook (if DATABRAIN_ALERT_WEBHOOK_URL is configured).
  • Result retention: output/selfcheck/ (history included), exported together with the diagnostic bundle; on a fresh install without an activated license, a yellow "pending activation" light is the expected state.

Diagnostic bundle (offline fallback)

When the container or the console fails to start and you need to send diagnostic information to support, run on the host, from the install directory:

bash databrain-ops.sh diagnose
  • Output: <install-dir>/diagnostics/diag_<timestamp>.zip (send to the vendor).
  • Collected: version / machine code / health / license status / container process logs (docker compose logs) / docker info / nvidia-smi, all best-effort — even if the container is down or docker is missing, a usable partial diagnostic bundle is still produced.
  • Redaction: automatically strips sensitive environment-variable values; the license file and key material of any kind are never bundled.
  • When the container is running normally, it is preferable to use the console's Troubleshooting page (host port 18001) for one-click bundling — its content is more complete (includes audit samples / metric series / model version), with the same redaction.

After unzipping you can run python analyze.py (shipped inside the zip) for 6 quick checks (license/health/GPU/error logs/need_review ratio/type confidence); exit code 0 = OK, 1 = warnings present.

1. Daily operations commands

The databrain-ops.sh commands below and throughout this chapter are run from the install directory (or add the install directory to your PATH first — see the tip below).

CommandDescription
bash databrain-ops.sh statusSingle-screen overview: container status + ready + license + profile + ports
bash databrain-ops.sh logsRecent logs (non-following by default, --tail 200)
bash databrain-ops.sh logs -fLive-follow logs (Ctrl+C to exit)
bash databrain-ops.sh restartRestart
bash databrain-ops.sh stopStop
bash databrain-ops.sh startStart (auto-detects CPU/GPU, uses the current version)
bash databrain-ops.sh healthHealth check
bash databrain-ops.sh licenseAuthorization status
bash databrain-ops.sh verifySample-scan smoke test (includes EMAIL/PHONE)
bash databrain-ops.sh versionApp tag + base tag + CLI version
bash databrain-ops.sh depsSelf-checks base-image third-party components (BOM diff)
bash databrain-ops.sh fingerprintCollect the machine code (also writes machine-code.txt)
bash databrain-ops.sh activate license.jsonLoad license + restart + verify (use after renewal/re-issue)
bash databrain-ops.sh diagnoseOffline fallback: bundle version/health/license/logs/docker-info/nvidia-smi into ./diagnostics/ (use when the container won't start, see "Diagnostic bundle")
bash databrain-ops.sh rollback <old-version>Switch to a specified old version (version switching & cleanup: see §4)
bash databrain-ops.sh revertUndo the most recent upgrade (back to the pre-upgrade version)
bash databrain-ops.sh clear [--yes]DANGER: clean from scratch (keeps license/+.sp/base)

Skip the long prefix: after a one-time echo 'export PATH="<install-dir>:$PATH"' >> ~/.bashrc && source ~/.bashrc, all the above can be abbreviated to databrain-ops.sh … (for multi-user sharing, write it into a system-wide profile file).

2. Monitoring essentials

MetricHow to checkWhat to watch
Service readyGET /v1/healthreadyShould be true
Authorization statusGET /v1/license/statusvalidShould be true; watch days_remaining, grace_active; renew before expiry
Container healthdocker ps (STATUS column healthy)Docker auto-probes every 30s
Detection-log disk spaceGET /v1/healthaudit block / console top-bar alertBelow 2 GB free the detection log stops writing (audit_disk_stopped alert); it resumes after disk cleanup brings free space above ~3 GB
GPU utilizationnvidia-smi (GPU mode)VRAM and utilization, to spot overload
Resource usagedocker stats databrainAbnormal CPU/memory
Active tierstartup log profile=...Confirm it is running the intended GPU tier

Docker's built-in health check treats ready:true as healthy: it takes 3 consecutive probe failures (30s interval) to mark unhealthy, with a 60-second grace period after start.

3. Version management

  • Always pin an explicit TAG (e.g. TAG=1.0.1); do not use :latest, for audit and precise rollback;
  • Keep the latest 2 application versions (current + previous) for instant rollback;
  • Never manually delete the dependency layer (base) — it is large but changes rarely (about once a year), and it is key to rollback; only manually clean up "orphan" dependency layers that no application version references;
  • Upgrades are handled end-to-end (signature verification → load → health gate → rollback on failure) by the application siipulse package (bash siipulse-rel_databrain_<new-version>.sp), reusing the on-host BaseImage; .sp packages uploaded through the console's version-management page go through the same signature and integrity verification (see 06-Security & Privacy §7).

4. Rollback / revert / cleanup / dependency self-check

4.1 Roll back to a specific version rollback

# Switch back to any old version still in the local image store (dependency layer + old app image still present)
bash databrain-ops.sh rollback <old-version>

A rollback completes within seconds (a tag switch + restart + ~30–70 s model reload).

4.2 Undo the most recent upgrade revert

# No need to remember the version: returns to the pre-upgrade version (auto-recorded by the installer)
bash databrain-ops.sh revert

Difference between revert and rollback: rollback <ver> requires an explicit version number and can switch to any old version; revert takes no argument and specifically "undoes the last upgrade", relying on the previous version recorded by the installer. Neither touches the license volume.

4.3 Clean from scratch clear (dangerous)

For extreme troubleshooting or before migrating to a new machine, you can fully clean this installation back to a "only delivery packages left" state:

bash databrain-ops.sh clear        # interactive confirm (type YES)
bash databrain-ops.sh clear --yes  # skip confirm (for scripts / automation)

clear will: stop & remove containers → delete all siipulse-databrain:* application images → delete compose and the installation state files. Kept: license/ (the authorization volume, sparing re-activation), *.sp delivery packages, siipulse-baseimage:1.0 (the base, sparing a re-load). After cleanup, reinstall with bash siipulse-rel_databrain_<version>.sp.

Note: keeping the license and base is intentional — after cleanup a reinstall needs no re-activation flow and no 2.9 GB base re-load. To also delete the base, manually docker rmi siipulse-baseimage:1.0.

4.4 Dependency self-check deps

Confirm whether the current base image contains all runtime dependencies (the dependency list is in requirements.lock inside the delivery package):

bash databrain-ops.sh deps

It checks the installed version of each key component (torch / transformers / fastapi, etc.) inside the base image, marking missing ones as MISS. Use it at delivery acceptance or when troubleshooting to answer "is the base complete?".

5. Troubleshooting table

SymptomPossible causeResolution
/v1/health license_required (ready:false)License not activated; AI model not loadedActivate the license per 02 "License (offline authorization)"
/v1/health warming and stuck for a long timeModel load failed (insufficient VRAM/memory, corrupt image)Check logs docker compose logs databrain; confirm VRAM ≥8GB, memory ≥16GB
Audit page stops receiving new records + console audit_disk_stopped alertFree disk below 2 GB; detection-log write-stop protectionClean up disk (the detection log rotates daily — old files can be removed); writing resumes once free space recovers to ~3 GB, no restart needed
Exits immediately on start (Exited)Config / port / permission issueCheck logs; check whether port 18000 is occupied, whether the compose file is complete
Needs GPU but shows profile=cpunvidia-container-toolkit not installed, or DATABRAIN_FORCE_CPU=1 was setConfirm the host has the toolkit installed (nvidia-smi works in-container); check install-time environment variables
Intermittent 503/504Cold-start window or momentary overloadProbe health before calling; client timeout ≥30s; control concurrency per tier
422 returnedInvalid request bodyCheck text is non-empty and ≤100,000 chars, min_confidence within 0–1
Scan returns 403 license_requiredNot activated / license invalidActivate per 02 "License (Offline Authorization)"; check reason in /v1/license/status
license/status valid:false (expired)License expiredRenew & re-issue (reuse the original machine code if the host is unchanged); still usable within the grace period
license/status valid:false (fingerprint_mismatch)Host hardware changedRe-collect the machine code bash databrain-ops.sh fingerprint, then re-issue
license/status valid:false (revoked)License revokedContact the issuer to re-issue or update the authorization
license/status valid:false (missing_asset_key / empty_asset_key / malformed_asset_key)The license file was mis-issued, or a license from another product was usedContact the vendor to verify and re-issue
Actual tier below hardware capabilityLimited by the license tierThis is normal; for higher throughput, obtain a higher-tier license
docker compose command not foundOld Docker (v1)Upgrade to Docker with the v2 plugin
Port 18000 occupiedPort conflictChange the compose port mapping (e.g. 18000:8000; in-container 8000 stays unchanged)
Memory/VRAM keeps growingAbnormal load or extremely long textLimit single-request text length; check for oversized concurrent batches
Accuracy anomaly after upgradeRare; possibly a version mismatchRoll back to the previous version first, then contact Siipulse support

License renewal

Renew the license before it expires to avoid scan interruptions:

  1. Host hardware unchanged → you can reuse the original machine code and request a new license.json from the issuer;
  2. Host hardware changed → re-collect the machine code bash databrain-ops.sh fingerprint, then request it;
  3. Load the new license and restart to take effect: bash databrain-ops.sh activate license.json (= replace license/license.json + restart + verify);
  4. After expiry you enter a grace period (per authorization type); once the grace period ends, scanning returns 403; after renewal it recovers immediately — no image reinstall needed.

Upgrading the application image (02 "Upgrade / Rollback") does not touch the license volume, so upgrade ≠ renewal; the two are independent.

6. Reading the logs

The startup log shows the following key signals in order (exact format subject to actual output):

  • Hardware detection completes and the effective tier is determined (e.g. profile=gpu-large, i.e. "GPU-large");
  • The engine finishes loading and reports the concurrency cap for that tier;
  • The HTTP service starts listening on the port (8000 in-container, 18000 published on the host).

7. Deploying on Kubernetes and other orchestration platforms

The delivery is provided by default as Docker Compose. To deploy on Kubernetes or similar platforms:

  • Import the image into your registry (docker load then docker tag + push, or use the offline package directly);
  • Run as a Deployment with one replica (or more for horizontal scale-out, as needed);
  • Point readiness/liveness probes at GET /v1/health, with the ready condition ready:true;
  • In GPU mode, declare GPU resources the platform way (e.g. K8s nvidia.com/gpu).

For official Kubernetes manifests or a Helm Chart, contact Siipulse.

8. Getting support

When troubleshooting, prepare the following information to help locate issues quickly:

  1. DataBrain version number (the TAG of docker images siipulse-databrain, or bash databrain-ops.sh version);
  2. Full startup logs (docker compose logs databrain);
  3. Hardware configuration (CPU/memory, GPU model and VRAM, operating system);
  4. Reproduction steps and request/response (please redact before sending; do not send real PII).
FAQ

08Frequently Asked Questions (FAQ)

Q1Will my data be sent externally?

No. DataBrain runs fully offline. All scanning and judgment happens inside the customer's server — it does not connect to Docker Hub or any external source, makes no outbound network request, and reports no statistics. From input to output, data stays within the customer's network. See 06-Security & Privacy.

Q2Which languages are supported?

The AI recognition model is based on multilingual pre-training and covers 100+ languages, including Chinese, English, Japanese, Korean, Hindi, and many European languages.

Q3Can I use it without a GPU?

Yes. CPU mode is fully functional and identical to GPU in accuracy — only speed differs, roughly 1/19–1/33 of GPU. It suits functional validation, low-frequency scanning, or GPU-less environments. For latency-sensitive production scenarios, GPU is recommended (VRAM ≥ 8 GB, system memory ≥ 16 GB). See 01-Hardware & Software Requirements.

Q4What does needs_review mean? Is it a failed detection?

No. needs_review=true means the value has been detected as some type of sensitive data, but the system is under-confident about the type assignment (e.g. a national ID and a passport number have similar formats and are neck-and-neck); manual review is recommended. It is still a valid prediction and is counted in accuracy. In production you decide per business policy whether to trust it directly or route it to review.

Q5What should I set min_confidence to?
  • 0.5 (default, balanced tier): returns only results likely to be real sensitive data; fits the vast majority of production scenarios;
  • 0.0 (high-recall tier): returns all candidates with confidence scores, for audit / inventory "better safe than sorry" scenarios, letting the caller filter downstream by confidence.
Q6How long a text can a single request handle?

/v1/text/scan caps a single text at 100,000 characters; anything beyond is segmented on the caller side.

Q7How do I use the results for redaction / masking?

Each result carries start/end (character positions in the original text); you can directly replace the corresponding span with *** or use them for highlighting.

Q8How do I scan a whole database table?

Use the value-level HTTP endpoint POST /v1/values/scan (batch classification, ≤1000 per batch). Pass each column's values with the column name as label_hint. See section 2 of 03-API Integration Guide.

Q9How do I upgrade in an offline environment?

Use the delivered new-version siipulse application package and run bash siipulse-rel_databrain_<new-version>.sp. The package self-extracts, auto-reuses the on-host BaseImage, and completes load / health-gate / rollback-on-failure. No network needed. See 02-Installation & Startup.

Q10Will an upgrade lose my configuration?

No. Configuration is packaged with the image; at startup the runtime config is generated automatically for the current hardware, with no manual intervention. Business data is not inside the container, so upgrades are unaffected.

Q11Can it recognize my custom, special sensitive-data types?

If a type is not in the 04-PII Type Catalog, contact Siipulse to evaluate adding it. DataBrain has a self-learning capability of "discover a new type → auto-generate samples → retrain the model"; extending to new types is one of its design capabilities.

Q12Will the container's dependencies conflict with the host's?

No. The container bundles its entire runtime dependency set (AI inference framework, GPU runtime, etc.), fully isolated from the host's native environment. In GPU mode the only shared component is the GPU driver.

Q13How do I confirm the service is truly healthy and recognizes correctly?

Run bash databrain-ops.sh verify (one command: health check + sample scan with email/phone, confirming end-to-end usability). You can also curl http://127.0.0.1:18000/v1/health or bash databrain-ops.sh health at any time (run from the install directory).

Q14How do I do multiple replicas / high availability?

Docker Compose can scale to multiple instances + a front load balancer; on Kubernetes or similar platforms, use a multi-replica Deployment with a readiness probe pointed at /v1/health. Note that each replica independently loads the AI model (consuming VRAM), so plan replica count by hardware capacity.