Reporting Framework
Last updated
Was this helpful?
Last updated
Was this helpful?
The Reporting Framework is an infrastructure to make database data available in searchable indexes for efficient slicing and dicing of data and real-time monitoring with visualization and reports. The framework utilizes open-source components to achieve this data pipeline and visualization capabilities.
Postgres WAL contains events on the database, specifically, changes to any table fields. These events are read by Debezium and pushed to Kafka Topics. Using a connector the fields are further pushed into OpenSearch indexes. These indexes may be searched by APIs or consumed by OpenSearch Dashboards for visualization.
The salient features of the framework are the following:
Searchable indexes in OpenSearch
No load on main DB esp when dataset is large and SQL queries get complex
Real-time slicing and dicing of data with searchable indexes
OpenSearch Dashboards
Ease of creation of dashboards and reports
Authentication and authorisation using Keycloak
Authorized access to dashboards
Single sign-on with Keycloak
PII data not available for search*
Privacy protection
Clone the https://github.com/openg2p/openg2p-reporting repository, and navigate to scripts folder.
Run:
Import Sample Dashboards from the dashboards folder into OpenSearch Dashboards through UI.