One control room for every ClickHouse ingestion pipe

Build me an ingestion control room for every ClickPipe running into ClickHouse Cloud. Today I have to open each database service separately to find out whether its pipelines are alive, and I want one screen that answers that for the whole organization and lets me act on what I find.

The main view is a single table of every ClickPipe across every service. To build it, use ClickHouse Cloud List Organizations to resolve the organization, then List Services to enumerate the services, then List ClickPipes for each service. ClickPipes are nested under a service, so there is no single call that returns them all. Each row should show the pipe name, the service it belongs to, the source type (Kafka, Kinesis, object storage and so on), the destination table, the current state, and the scaling settings such as replica or worker count. Include when the pipe last changed state so recent activity is visible at a glance.

Pace the sweep. A ClickHouse Cloud key is limited to ten requests every ten seconds, and a large organization with many services will blow through that if the handler fans out all at once. Stagger the per-service calls inside a server handler, keep the whole sweep under the limit, back off and retry when the API returns a rate-limit error, and show the user a progress indicator plus a count of services swept so a slow load never looks like a hang. If one service fails to return its pipes, show the rest of the table and mark that service as errored rather than failing the whole view.

Give the table filters for stopped pipes, failing pipes, and recently changed pipes, plus free-text search on pipe name, service and destination table. The saved default filter is "anything not running", so the app opens showing only pipes that are not in a healthy running state. Let the user change the default and remember their choice per user so it survives a reload.

Selecting a row opens a detail panel. Load the full pipe configuration with Get ClickPipe, and alongside it load the health of the parent service with Get Service Metrics so I can see whether the pipe is broken or the database underneath it is struggling. Show the source and destination configuration, field mappings, current scaling, and the recent state history if the response carries it.

From the detail panel I should be able to fix things. Use Update ClickPipe State to start, stop or resync the pipe. Use Update ClickPipe Scaling to adjust replica and worker counts for Kafka, Kinesis and object-storage pipes, and hide or disable that control for source types where scaling does not apply. Use Update ClickPipe to rename a pipe or correct its field mappings, source and destination settings, presenting the current values in an editable form rather than a blank one. After any action, refresh just that pipe and update its row in the table without re-sweeping the whole organization.

Resync must always ask for confirmation before it runs, because it reprocesses data from the source. The confirmation should name the pipe and the service and say plainly that data will be reprocessed. Start and stop can run without a confirmation step.

Add a "Raise a bug" action that creates a Linear issue with Linear Create Issue, prefilled from the selected pipe: the pipe name and service in the title, and a description carrying the source type, destination table, current state, scaling settings and anything visible in the service metrics at that moment. Let me pick the Linear team (populate the picker with Linear List Teams), edit the title and description before sending, and optionally set an assignee and priority. Show a link to the created issue afterwards, and link it back from the pipe row so I can see a bug was already filed.

This is a working surface for an on-call data engineer, not an alerting tool. Prioritize a dense, scannable table, clear state colouring, and fast action feedback with the result of every write reflected in the UI. Errors from ClickHouse Cloud, including permission errors when the key is read-only rather than admin, should surface as readable messages next to the control that failed, and read-only keys should show the action buttons disabled rather than hidden.