Overview
Domain: Fleet data synchronization, aggregation, and search
Platform: The read-oriented data service within the Fleet Operations Platform
Requirements
- Receive approved source changes through events and service integrations.
- Normalize and synchronize operational records from multiple sources.
- Build read-oriented data for aggregation and search.
- Provide REST and gRPC lookup responses to related platform services.
- Track integration and resynchronization state at a generalized level.
Key Challenges
- Normalize contracts that arrive from different source systems.
- Keep persisted read data and search projections aligned.
- Separate background synchronization from interactive lookup paths.
- Make eventual consistency and resynchronization behavior observable.
C4 Model
C1
System Context
People and approved external systems interacting with the Fleet Operations Platform.
C1 architecture summary
Elements
- Fleet Operations Staff (person)
- Administrator / Dispatcher (person)
- Fleet Operations Platform (software system)
- Approved Operational Data Sources (external system)
Relationships
- Fleet Operations Staff use the Fleet Operations Platform for workflow, lookup, and reporting.
- Administrators and dispatchers use the Fleet Operations Platform for planning, resources, and coordination.
- The Fleet Operations Platform exchanges approved REST or gRPC requests, responses, and events with Approved Operational Data Sources.
C2
Container View
The service within the generalized Fleet Operations Platform container topology.
C2 architecture summary
Current highlighted service: Fleet Data Intelligence Hub.
Elements
- Operations Client
- Administration Client
- Fleet Operations Core service
- Fleet Administration & Dispatch service
- Fleet Data Intelligence Hub service
- Kafka event broker
- Operations PostgreSQL
- Operational Redis cache
- Administration PostgreSQL
- Administration Redis cache
- Intelligence PostgreSQL
- Elasticsearch search index
Relationships
- The Operations Client calls Fleet Operations Core through REST or gRPC.
- The Administration Client calls Fleet Administration & Dispatch through REST or gRPC.
- Fleet Administration & Dispatch supplies plans, resources, and configuration to Fleet Operations Core through REST or gRPC.
- Fleet Operations Core requests lookup and search data from Fleet Data Intelligence Hub through REST or gRPC.
- Fleet Administration & Dispatch requests aggregated lookup data from Fleet Data Intelligence Hub through REST or gRPC.
- Fleet Operations Core publishes operational events to Kafka.
- Fleet Administration & Dispatch publishes reference and coordination events to Kafka.
- Kafka delivers synchronization events to Fleet Data Intelligence Hub.
- Fleet Operations Core uses Operations PostgreSQL and the Operational Redis cache.
- Fleet Administration & Dispatch uses Administration PostgreSQL and the Administration Redis cache.
- Fleet Data Intelligence Hub uses Intelligence PostgreSQL and the Elasticsearch search index.
C3
Component View
Generalized internal components and their primary responsibilities.
C3 architecture summary
Elements
- Kafka
- Kafka Consumers
- Approved Source Systems
- Integration Adapters
- Synchronization Workers
- Normalization and Mapping
- Data Repositories
- Search Adapter
- Integration State Tracking
- REST and gRPC Lookup API
- Query and Aggregation Services
- PostgreSQL
- Elasticsearch
Relationships
- Kafka delivers events to Kafka Consumers.
- Approved Source Systems call Integration Adapters through REST or gRPC.
- Kafka Consumers and Integration Adapters supply Synchronization Workers.
- Synchronization Workers delegate to Normalization and Mapping.
- Normalization and Mapping updates Data Repositories, the Search Adapter, and Integration State Tracking.
- The REST and gRPC Lookup API delegates requests to Query and Aggregation Services.
- Query and Aggregation Services use Data Repositories and the Search Adapter.
- Data Repositories access PostgreSQL.
- The Search Adapter accesses Elasticsearch.
Main Flow
- Kafka consumers or approved integration adapters receive source changes.
- Synchronization workers validate, map, and normalize the records.
- The service persists operational read data and updates search projections.
- REST or gRPC interfaces accept lookup and aggregation requests.
- Query services select PostgreSQL or search-backed access for the request.
- A normalized response returns to Operations Core or Administration and Dispatch.
My Contributions
- Developed synchronization workers for approved multi-source data flows.
- Built REST lookup and aggregation APIs for supported read use cases.
- Worked on Elasticsearch-backed query and data-processing paths.
- Contributed to Kafka consumer flows and integration-state handling.
- Integrated related services through REST and gRPC boundaries.
Tech Stack
Java 17Spring BootRESTgRPCKafkaElasticsearchPostgreSQLOpenFeignDocker
Reliability & Security
- Separate background synchronization from interactive read paths.
- Treat search indexes as read-oriented projections.
- Keep resynchronization state bounded and observable.
- Exclude source identifiers and private records from public descriptions.
Trade-offs / Design Decisions
- Asynchronous synchronization reduces source coupling but introduces eventual consistency.
- Search projections improve flexible lookup but add index lifecycle work.
- Multiple integration styles improve fit while increasing contract-management overhead.
Outcome / Impact
- Provided a normalized lookup boundary across approved operational data sources.
- Separated synchronization processing from interactive search and aggregation concerns.
Lessons Learned
- Treat synchronized data as an explicit read model with known freshness.
- Make integration and resynchronization state observable.
- Keep search projections rebuildable from controlled data sources.
Related Platform Services
Service 01 / Operational CoreFleet Operations CoreThe central backend service for fleet workflows, interactive lookup, operational statistics, and document generation.Service 02 / AdministrationFleet Administration & DispatchA backend service for planning, resources, devices, configuration, and coordination across fleet operations.