Skip to main content

System Architecture

1. High-level architecture​

CDPG is a cloud-native data-exchange platform implemented as independently deployable Go services behind one API gateway. The Control Plane owns identity-linked governance, discovery, policies, commerce, and platform administration; the Data Plane ingests and serves resource content; the Agentic Plane lets governed AI agents exercise revocable delegated authority. dx-common-go/platform supplies the shared runtime contract, while PostgreSQL/PostGIS, Elasticsearch, Redis, RabbitMQ, S3-compatible storage, Keycloak, OpenFGA, and the planned OPA policy engine provide specialized infrastructure. The primary non-functional requirement is security-preserving multi-organization operation: identity, authorization, failure, cache, event, and recovery paths default to denial rather than broadening access.

Architecture status: the service fleet and primary business paths are Partially implemented. The complete intended design appears here; Current State and Target State is authoritative for operational status.

2. Diagram​

system_architecture · complete Go platform context
People, applications, agents, and external providers enter through identity and gateway boundaries. Control Plane services govern identity, catalogues, policies, commerce, and platform records. Data Plane services serve NGSI-LD, OGC, file, and subscription data. Agentic Plane services govern delegated execution. Shared infrastructure provides stores, events, and observability. Cards show implementation status.external actors & dependenciespublic trust boundarycontrol planedata planeagentic plane · in developmentshared infrastructure & platform operationsOIDC / HTTPScallbacks / providersevents + carried decisionsPeople & applicationsprovider · consumer · adminAgent clientsowner · approverExternal systemsdata · payment · email · llmIdentity Providerusers · workloads · agentsAPI Gateway · PEPauthenticate · authorize · routeProvider boundariesverified callbacks / sourcesUser & Orgidentity-domain recordsPARTIALCataloguemetadata · discoveryPARTIALPolicy · PAPgrants · lifecycleIMPLEMENTEDAuthz · PDPOpenFGA + planned OPAPARTIALMarketplaceproducts · entitlementPARTIALPlatform servicessupporting domainsIMPLEMENTEDNGSI-LDentity · temporal · searchPARTIALOGC APIsfeatures · tiles · processesPARTIALFile Exchangeobjects · processingPARTIALSubscriptionsdefinitions · deliveryIMPLEMENTEDEnforcement at executionverify carried decision · filter · mask · meter · deliverTARGET CONTRACT PLANNEDAgent RegistryIN DEVELOPMENTAgent RuntimeIN DEVELOPMENTMCP GatewayIN DEVELOPMENTPostgreSQL/PostGIS · Search · Redis · RabbitMQ · S3 · Telemetry
actors & external systemscontrol, identity & decisionsdata access & deliveryagentic governanceevents & shared infrastructure

System boundary​

Inside the boundary are the gateway, Go service fleet, shared platform SDK, service-owned data stores, and platform-operated infrastructure. Outside it are end users, client applications, data providers and source systems, payment providers, email delivery, LLM providers, and federated participants. External systems are never trusted merely because they can reach a service; every boundary has a protocol-specific authentication and authorization control.

Trust boundaries​

  1. Public edge: HTTPS reaches only the load balancer/WAF and dx-gateway-go. Payment callbacks use provider-specific signature verification at their dedicated endpoint.
  2. Identity domain: Keycloak authenticates users, applications, agents, and service workloads and publishes verification keys. It does not decide resource access.
  3. Private service network: each call presents a destination-bound workload token. Receiving services allowlist callers and independently decide which callers may assert an end-user subject.
  4. Authorization domain: at target state, dx-authz-go combines OpenFGA relationship results with OPA contextual policy, delegation state, and trust rules. OpenFGA evaluation exists now; OPA composition is Planned. Policy administration remains in dx-acl-go.
  5. Data domain: data services verify the carried decision and enforce row, field, quota, and ownership obligations against their own stores.
  6. Agentic domain: agent runtime, tool gateway, token vault, and approvals form a stricter boundary. Model output is untrusted and cannot directly call a platform API.
  7. Federation boundary: when enabled, cross-operator traffic adds mutual TLS, trust-list validation, envelope integrity/confidentiality, and a versioned federation contract.

3. Components​

ComponentTechResponsibility
API Gateway (dx-gateway-go, host :8000)Go, HTTP reverse proxyValidates external credentials, resolves operation policy, obtains edge authorization, enforces edge-known obligations, attaches trusted internal context, rate-limits, and routes.
Identity ProviderKeycloak, OIDC/OAuth 2.0Authenticates human, application, agent, and workload principals; issues signed tokens; publishes JWKS; performs token exchange.
Authorization Service (dx-authz-go, host :8090)Go, OpenFGA, planned OPAActs as the composite PDP and emits final decisions, reasons, revisions, obligations, and attestations.
Policy Service (dx-acl-go, host :8100)Go, PostgreSQL, RabbitMQValidates, stores, versions, expires, and revokes access grants; publishes durable grant and delegation facts.
User Service (dx-user-go, host :8190)Go, Keycloak, PostgreSQLOwns profiles, organizations, membership, delegations, application credentials, and membership events.
Catalogue (dx-catalogue-go, host :8120)Go, Elasticsearch, JSON-LDOwns resource metadata, discovery, ownership facts, supported access types, and lifecycle state.
Marketplace (dx-marketplace-go, host :8110)Go, PostgreSQL, payment adapterOwns products, merchants, orders, payments, invoices, and purchase-to-entitlement orchestration.
Supporting servicesGo, PostgreSQL/RabbitMQRegistry resolves endpoints; credits meters balances; audit stores activity; notifications dispatch email; subscriptions manage streaming registrations.
NGSI-LD Data Plane (dx-dataplane-rs-go, host :8260)Go, ElasticsearchServes entity, temporal, geo, aggregation, bulk, and CSV query operations.
OGC Data Plane (dx-dataplane-ogc-go, host :8200)Go, PostgreSQL/PostGISServes OGC Features, Tiles, Coverages, Processes/jobs, Records-lite, CQL2, and CRS operations.
File Exchange (dx-files-connect-api-go, host :3000)Go, PostgreSQL, Redis, RabbitMQ, S3Owns file metadata, multipart transfers, presigned access, processing jobs, cleanup, and object ownership checks.
Community Layer (dx-community-layer-go, host :5000)Go, PostgreSQL, Redis, S3Owns discussions, challenges, submissions, attachments, moderation, and community workflows.
Agent Registry (:8270)Go, PostgreSQL, Keycloak adminOwns agent records, versioned templates, credential bindings, lifecycle, delegations facade, and kill switch.
MCP Gateway (:8280)Go, MCP, RedisExposes the governed tool catalogue and enforces schema, scope, resource, risk, approval, credential, and sole-egress controls.
Agent Runtime (:8290)Go, PostgreSQL, Redis, SSE, LLM adaptersOwns sessions, transcripts, guardrails, planning loop, token exchange, tool orchestration, and streaming responses.
Shared Go platformdx-common-go/platformSupplies bootstrap, configuration, HTTP/gRPC, identity, persistence, events, cache, health, observability, error, and resilience conventions.
InfrastructurePostgreSQL/PostGIS, Elasticsearch, Redis, RabbitMQ, S3/MinIOProvides service-owned persistence, spatial/search indexes, hot state, durable messaging, and object storage.
GitOps and observabilityKubernetes, Helm, ArgoCD, External Secrets, Prometheus/OpenTelemetryReconciles deployment state, injects secrets, isolates networks, and exposes operational signals.

4. Data flow​

  1. A user signs in with Keycloak and receives a signed user token; an application or workload receives a credential appropriate to its principal type.
  2. The client sends an HTTPS request to dx-gateway-go, which matches the exact operation and validates the declared authentication mode.
  3. The gateway builds normalized principal, actor, workload, organization, operation, resource, and request context and calls the composite authorization contract when the operation requires it.
  4. At target state, dx-authz-go asks OpenFGA for relationship facts, evaluates contextual and attribute rules in OPA, applies delegation/trust state, and returns a default-deny decision with reasons, obligations, revisions, expiry, and a decision ID. The OPA and obligation stages are Planned.
  5. The gateway enforces edge-known controls and forwards the request with a destination-bound workload credential and an integrity-protected decision artefact. If the authoritative resource is body- or query-derived, the application service completes the PEP step before invoking the data operation.
  6. The target service verifies caller workload, permitted subject assertion, operation/resource binding, and decision validity. Data services translate typed obligations into parameterized PostGIS/Elasticsearch/SQL queries, field shaping, and quota consumption.
  7. The service reads or mutates only its owned store, returns the standard response or standards-native representation, and emits metrics, traces, logs, and an audit event.
  8. For asynchronous changes, the domain transaction appends an outbox event. RabbitMQ delivers the versioned fact to idempotent consumers; dead letters, replay, and reconciliation recover failures.

The most important alternate path is failure: missing identity, wrong audience, unknown caller, PDP timeout, OPA undefined/error, OpenFGA error, stale or invalid decision, unsupported obligation, ownership mismatch, or unavailable quota state all deny. A retry is allowed only when it cannot duplicate a side effect or broaden access.

Event-driven integrations​

event_integrations · durable facts and projections
Policy, membership, agent, business, data, and file producers write transactional outboxes. RabbitMQ delivers versioned events to authorization projections, audit, notifications, subscriptions, and workers with idempotency, retry, dead-lettering, and reconciliation.authoritative producersdurable deliveryprojectors & workersdomain change + outboxversioned eventbounded retryreconcile projections with sourcesPolicy / membershipgrant · org · groupAgent lifecycledelegate · suspend · approveBusiness servicesorders · credits · resourcesData / filesingest · job · deliveryOutboxessame transactionRabbitMQconfirm · routeDLQinspect · repair · replayAuthorization projectionsOpenFGA · OPA data · invalidationAudit and notificationsappend · template · dispatchSubscriptions and data consumersfan-out · checkpoint · deliverBackground workersfile jobs · leases · idempotency
versioned eventprojection / governanceagent lifecyclenotification / processing

The event bus carries facts, not commands disguised as events. The producing service remains the source of truth; consumers build projections or trigger bounded side effects. See Messaging Backbone and Audit Events.

5. Design decisions & trade-offs​

  • One public gateway and per-service workload identity — centralizes edge policy while preserving accountability between internal callers. It costs identity-provider configuration and token caching per destination.
  • OpenFGA plus OPA behind one composite PDP — separates graph reachability from contextual policy without exposing engine-specific contracts to services. It adds policy-bundle operations and a multi-engine failure surface that must fail closed.
  • Central decisions, distributed enforcement — one service explains why access is allowed; the component with data context applies filtering, masking, and quotas. It requires a versioned, integrity-protected decision format and obligation conformance tests.
  • Service-owned data and events — independent evolution and failure containment in exchange for more operational components and explicit projection consistency.
  • gRPC internally, HTTP/REST externally — typed internal contracts and familiar public APIs in exchange for protobuf governance and two transports to operate.
  • Optional Agentic and federated capabilities — core exchange operation does not depend on them. Optionality requires explicit feature gates, isolated failure domains, and status labels that prevent design-only components appearing operational.

6. Best practices​

Scalability​

  • Keep gateway, stateless control APIs, and authorization API pods horizontally scalable; isolate projector and worker concurrency with durable leases.
  • Partition Elasticsearch/PostGIS and object storage by service-owned resource identity; introduce replicas and aliases only from measured load.
  • Scale RabbitMQ consumers by idempotent event ID and exclusive outbox claims; never scale a process-local singleton horizontally.

Security​

  • Require issuer, audience, expiry, token ID, caller allowlist, and subject-asserter checks on internal credentials; rotate through identity-provider key rollover.
  • Keep operation policy versioned with OpenAPI, fail startup on invalid manifests, and deny unknown operations or policy profiles.
  • Scope every object or row lookup by owner and organization in the datastore query and make forbidden and absent resources non-enumerating.
  • Keep LLMs, prompts, tool output, and policy author input outside the trusted computing base for authorization.

Observability​

  • Propagate request, trace, event, grant, decision, delegation, agent, session, and approval IDs through every relevant hop.
  • Record authorization engine revisions, cache result, reason codes, obligations selected/enforced, and failure class without logging sensitive policy data.
  • Alert on PDP error/timeout rates, policy projection lag, outbox/DLQ depth, missing event consumers, decision-attestation rejection, agent kill-switch latency, and organization-isolation failures.