Bringing Real-Time PI System Data into Power BI in 5 Seconds

Case Study: PI Integrator for Business Analytics — Kafka Confluent — Power BI

Background

A gas processing facility runs PI System as its primary historian. Process data is fully available to the operations team through PI Vision — but management and the business analytics team work in Power BI, and that is where the problem started.

The initial approach was a common one: PI Integrator for Business Analytics published an asset view to SQL Server, and Power BI ran a scheduled refresh every 15–30 minutes. That was adequate for daily reporting, but not for operational monitoring. When flow rate dropped or a compressor tripped, the dashboard only reflected it several minutes later. Every new data consumer also meant additional query load on the database — and ultimately on the PI Data Archive.

The requirement was clear: a Power BI dashboard refreshing every 5 seconds, without opening direct access from the IT network into the OT zone.

Solution Architecture

The data path was designed as a one-way flow from OT to IT, with Kafka as the streaming backbone in the middle:

An important point: PI Integrator only opens an outbound connection to the Kafka brokers (TLS, SASL). No inbound port is opened toward the OT zone, so the IEC 62443 segregation principle stays intact.

Key Configuration

1. PI Integrator — Asset View The view is built on top of an AF Element Template rather than a raw tag list, so adding a new asset does not require reconfiguration. The publication is set to streaming mode with a 5-second time-based interval and interpolated values, producing a clean time series that is unaffected by exception and compression settings on the PI side.

2. Publish Target — Kafka One view maps to one topic. The message key uses the AF Element path, so all data for a given asset always lands in the same partition and its time ordering is preserved. Messages are published in Avro and registered in Confluent Schema Registry with BACKWARD compatibility, so adding an attribute does not break existing consumers.

3. Topic Design Partitions are split per process unit, retention is set to 24 hours (enough to replay after a short outage), and compression is enabled to reduce WAN bandwidth.

4. Consumer to Power BI ksqlDB applies a 5-second tumbling window to take the latest value per asset, and Kafka Connect HTTP Sink issues one POST every 5 seconds to the Power BI push dataset REST API. At this cadence, quota consumption stays well below the Power BI limit (120 requests per minute per dataset), leaving headroom for additional datasets.

5. On the Power BI Side A hybrid streaming dataset (push + streaming) is used so the real-time tiles stay live while a short history is retained for trending over the last few hours.

Latency Budget

Why Route Through Kafka?

The question that comes up most often: why not go directly from PI Integrator to Power BI?

• Decoupling. PI Integrator publishes once. Power BI, the data lake, and analytics models all read the same topic without adding load to the PI Data Archive.
• Buffering and replay. If the WAN link or the Power BI service is disrupted, data remains in the topic and can be re-consumed once service is restored — with no need to re-publish from the historian.
• Backpressure. A slow consumer does not drag down the source system.
• Schema governance. Data structure changes are controlled through Schema Registry rather than manual coordination between teams.

Kafka here is not merely a pipe — it is the integration point that lets the next use case (soft sensors, anomaly detection, production reporting) be added without touching anything on the OT side.

Results

• Dashboard refresh dropped from 15 minutes to 5 seconds.
• Query load on the PI Data Archive decreased, since a single publication now serves many consumers.
• The same topic was later reused for downstream analytics use cases with no changes in the OT zone.
• No new inbound port was opened into the control network.

Implementation Notes

A few things worth settling early:

1. Be selective about tags. Not every value needs a 5-second cadence. Define the operational KPIs that genuinely require low latency; the rest remain more efficient over a daily batch path.
2. A streaming dataset is not a historian replacement. History retention in a Power BI push dataset is limited. Long-term trending should still come from PI or a lakehouse.
3. Time synchronisation. Consistent NTP and UTC timestamps end to end. A few seconds of drift is very visible on a 5-second dashboard.
4. Licensing. Check up front: PI Integrator for Business Analytics licensing, Confluent capacity, and the Power BI tier (Pro/PPU/Premium) required for automatic page refresh.
5. Test the failure scenarios. Simulate broker restarts, WAN loss, and PI Integrator restarts to confirm the replay mechanism behaves as expected.

Closing

Real-time data that stops at the control room is only valuable to the operator. With a PI Integrator for Business Analytics — Kafka Confluent — Power BI architecture, that same data can reach the management desk in seconds, without compromising OT/IT segregation and without adding load to the historian.

What gets built is not just a single dashboard, but a data integration foundation ready to serve the analytics use cases that follow.