> For the complete documentation index, see [llms.txt](https://docs.openg2p.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openg2p.org/platform/platform-services/connector/technical-architecture.md).
# Technical Architecture
Components of the Connector, how data and job messages move between them, and what the connector keeps in its database.
## Components
Connector Service architecture
| Component | Responsibility |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| Connector Admin UI | Web application where operators create pipelines, adjust configuration, and monitor activity. |
| Service API | Receives pushed payloads after partner key validation, serves the Admin UI, and exposes pipeline and monitoring endpoints. |
| Fetch worker | Reads from sources that have to be asked, and writes what it collects to the database. |
| Push worker | Reads records waiting in the database, signs each batch with the connector key, and sends it to the registry. |
| Job scheduler | Starts both workers at their configured intervals. |
| Connector database | Holds pipeline configuration, received payloads, records and their state, and run history. |
| Task queue | Holds short job messages such as "fetch this pipeline now" or "send the next batch". A worker takes the next job whenever it is free. |
Everything that receives data writes to the connector database: the Service API for pushed payloads, and the fetch worker for anything it collects. The database is the single place where data waits, which is what allows receiving and sending to run at their own speeds.
## The database and the task queue
The database and the task queue hold different things, and it is worth being precise about the difference. The database holds the data itself: payloads, records, and their state. The task queue holds nothing but instructions, each one a few words telling a worker to go and do a piece of work. A job message says which pipeline to act on; the worker that picks it up reads the actual data from the database.
The queue exists so that the scheduler does not have to know anything about the workers. It posts a job and moves on. Whichever worker instance is free takes it, which is what makes it possible to run several of them at once, restart one without losing work, and add more when volume grows.
## Background workers
Two workers do the background work. Both are started by the job scheduler and take their jobs from the task queue, and they are deliberately kept separate.
| Worker | Started by | What it does |
| ------------ | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- |
| Fetch worker | The scheduler, at each pipeline's interval | Reads from a configured source and writes what it collects into the database as records. It never contacts the registry. |
| Push worker | The scheduler, at its own interval | Takes records waiting in the database, groups them into a batch, and sends the batch to the registry. It never contacts a source. |
Keeping them apart matters in both directions. The fetch worker keeps collecting on its interval whether or not the registry is reachable, so a registry outage delays delivery but never causes a source to be missed. The push worker sets the pace of delivery on its own terms, so a large import arriving at once does not turn into a burst of traffic at the registry.
Neither worker is called directly. When a pipeline's fetch interval comes round, the scheduler posts a job to the task queue saying that pipeline is due; a free fetch worker takes the job, reads the pipeline's configuration from the database, and does the work. The push worker is started the same way. Because both take their work from the same queue, either can be scaled on its own: a deployment pulling from many sources runs more fetch workers, one with a delivery backlog runs more push workers, and nothing else has to change.
## What the connector stores
| Held | Purpose |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------- |
| Pipeline configuration | Source, transport, authentication, mapping, sender, target register, intervals, timeouts, cursor, and batch size. |
| Received payloads | The unaltered copy of everything received, kept for tracing and replay. |
| Records | The individual records a payload was broken into, each with its state, attempt count, and outcome. |
| Run history | What was attempted, when, and how it ended, including the registry correlation identifier. |
| Duplicate keys | Keys of records already processed, so the same record is not sent twice. |
| Fetch positions | The point each fetched pipeline resumes from. |
| Dead letter entries | Records that exhausted their attempts, held with their error for review and replay. |
Storage of received payloads is configurable, since payloads may carry personal data and retention is a decision for the deployment rather than for the connector.
{% hint style="info" %}
The diagrams on these pages are maintained as draw\.io sources in `.gitbook/assets` (`connector-service-architecture.drawio`, `connector-payload-lifecycle.drawio`, `connector-fetch-run.drawio`) alongside their PNG exports.
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.openg2p.org/platform/platform-services/connector/technical-architecture.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
