Skip to content
ClickHouse Docs
49.6kGet Started

Discover ClickPipe source schema

Beta
POST/v1/organizations/{organizationId}/services/{serviceId}/clickpipes/schemaDiscovery
API playground

This endpoint is in beta. API contract is stable, and no breaking changes are expected in the future.

Infers the schema (field names and ClickHouse data types) of a ClickPipe source without creating a pipe. Supported for Kafka, Kinesis, Pub/Sub, and object storage sources. Object storage inference runs on the destination service, which must be running.

Permission

The API key must have the control-plane:service:manage-clickpipes permission.

Authorizations

Path parameters

  • organizationIdstringrequired

    ID of the organization that owns the service.

    format: uuid
  • serviceIdstringrequired

    ID of the service to run schema discovery against.

    format: uuid

Request bodyJSON

  • sourceobjectrequired
    4 properties
    • kafkaoptionalobject
      15 properties
      • typeoptionalkafkaorredpandaormskorgcmkorconfluentorwarpstream+2 more

        Type of the Kafka source.

      • formatoptionalJSONEachRoworAvroorAvroConfluentorProtobuf

        Format of the Kafka source.

      • brokersoptionalstring

        Brokers of the Kafka source.

      • topicsoptionalstring

        One or more Kafka topics as a comma-separated string. All topics must have the same schema and are ingested into the same destination table by a single ClickPipe.

        Example: "topic1,topic2"
      • consumerGroupoptionalstring | null

        Consumer group of the Kafka source. If not provided "clickpipes-<>" will be used.

        Example: "my-clickpipe-consumer-group"
      • authenticationoptionalPLAINorSCRAM-SHA-256orSCRAM-SHA-512orIAM_ROLEorIAM_USERorMUTUAL_TLS+1 more

        Authentication method of the Kafka source. SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources. Supported authentication methods: kafka: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS, msk: SCRAM-SHA-512, IAM_ROLE, IAM_USER, MUTUAL_TLS, gcmk: PLAIN, MUTUAL_TLS, SERVICE_ACCOUNT_WORKLOAD_IDENTITY, confluent: PLAIN, MUTUAL_TLS, warpstream: PLAIN, azureeventhub: PLAIN, redpanda: SCRAM-SHA-256, SCRAM-SHA-512, MUTUAL_TLS, dokafka: SCRAM-SHA-256, MUTUAL_TLS

      • iamRoleoptionalstring | null

        IAM role for the Kafka source. Use with IAM role authentication. Read more in ClickPipes documentation: https://clickhouse.com/docs/en/integrations/clickpipes/kafka#iam

        Example: "arn:aws:iam::123456789012:role/MyRole"
      • offsetoptionalClickPipeKafkaOffsetornull
        2 variants

        One of the following:

        • ClickPipeKafkaOffset
          View ClickPipeKafkaOffset schema →
          2 properties
          • strategyoptionalfrom_beginningorfrom_latestorfrom_timestamp

            Offset strategy.

          • timestampoptionalstring | null

            A minute precision UTC timestamp to start from. Required for "from_timestamp" strategy.

            Example: "2021-01-01T00:00"
        • null
      • schemaRegistryoptionalClickPipeMutateKafkaSchemaRegistryornull
        2 variants

        One of the following:

      • caCertificateoptionalstring | null

        PEM encoded CA certificates to validate the broker's certificate.

      • reversePrivateEndpointIdsoptionalarray ofstring

        Reverse private endpoint UUIDs used for a secure private connection to the Kafka source.

      • exactlyOnceoptionalboolean | null

        Enable exactly-once delivery. Guarantees every Kafka record is inserted exactly once across restarts and rebalances. Can only be set at pipe creation.

      • tombstoneModeoptionaldeleteorsoft_delete

        How Kafka tombstone records are handled. Set to "delete" to delete the matching destination row; this requires exactly-once delivery. Set to "soft_delete" to write a row with the _is_deleted virtual column set to true; exactly-once delivery is not required. Can only be set at pipe creation.

        Example: "soft_delete"
      • credentialsoptionalPLAINorMskIamUserorAzureEventHuborMutualTLS

        Credentials for Kafka source. Choose one that is supported by the authentication method.

        4 variants

        One of the following:

        • PLAIN
          View PLAIN schema →
          2 properties
          • usernameoptionalstring

            Database username.

            Example: "postgres_user"
          • passwordoptionalstring

            Database password.

            format: password
            Example: "your_secure_password"
        • MskIamUser
          View MskIamUser schema →
          2 properties
          • accessKeyIdoptionalstring

            IAM access key ID.

          • secretKeyoptionalstring

            IAM secret key.

        • AzureEventHub
          View AzureEventHub schema →
          1 properties
          • connectionStringoptionalstring

            Connection string for Azure EventHub source.

        • MutualTLS
          View MutualTLS schema →
          2 properties
          • certificateoptionalstring

            PEM encoded client certificate for mTLS authentication.

          • privateKeyoptionalstring

            PEM encoded client private key for mTLS authentication.

            format: password
      • protobufSchemaoptionalstring

        Base64-encoded .proto source or serialized FileDescriptorSet. Supported only with Protobuf format and cannot be combined with schemaRegistry.

        maxLength: 1048576, minLength: 1
        Example: "c3ludGF4ID0gInByb3RvMyI7IG1lc3NhZ2UgRXZlbnQge30="
    • kinesisoptionalobject
      11 properties
      • formatoptionalJSONEachRoworAvroorAvroConfluentorProtobuf

        Format of the Kinesis stream.

      • streamNameoptionalstring

        Name of the Kinesis stream.

        Example: "my-stream"
      • regionoptionalstring

        AWS region of the Kinesis stream.

        Example: "us-east-1"
      • useEnhancedFanOutoptionalboolean | null

        Use enhanced fan-out for the Kinesis stream.

      • iteratorTypeoptionalTRIM_HORIZONorLATESTorAT_TIMESTAMP

        Type of iterator to use when reading from the Kinesis stream. If AT_TIMESTAMP is used, the timestamp field must be provided.

      • timestampoptionalinteger | null

        UNIX timestamp to start reading from the Kinesis stream. Required if iteratorType is AT_TIMESTAMP.

        Example: 1615766400
      • authenticationoptionalIAM_ROLEorIAM_USER

        Authentication method to use with the Kinesis stream.

      • iamRoleoptionalstring | null

        IAM role to use for authentication. Required if IAM_ROLE is used.

        Example: "arn:aws:iam::123456789012:role/MyRole"
      • schemaRegistryoptionalClickPipeKinesisSchemaRegistryornull
        2 variants

        One of the following:

        • ClickPipeKinesisSchemaRegistry
          View ClickPipeKinesisSchemaRegistry schema →
          4 properties
          • typegluerequired

            Type of the schema registry. Kinesis ClickPipes support the AWS Glue Schema Registry, which authenticates with IAM instead of credentials.

          • glueRegionstringrequired

            AWS region of the Glue Schema Registry.

            Example: "us-east-1"
          • glueRegistryNamestringrequired

            Name of the Glue Schema Registry.

            Example: "my-registry"
          • glueRoleArnoptionalstring | null

            IAM role to assume for Glue Schema Registry access. Defaults to the IAM identity of the Kinesis source.

            Example: "arn:aws:iam::123456789012:role/MyGlueRegistryRole"
        • null
      • accessKeyoptionalMskIamUserornull
        2 variants

        One of the following:

      • protobufSchemaoptionalstring

        Base64-encoded .proto source or serialized FileDescriptorSet. Required with Protobuf format unless a schema registry is configured, and not supported with other formats.

        maxLength: 1048576, minLength: 1
        Example: "c3ludGF4ID0gInByb3RvMyI7IG1lc3NhZ2UgRXZlbnQge30="
    • pubsuboptionalClickPipePostPubSubServiceAccountSourceorClickPipePostPubSubWorkloadIdentitySource
      2 variants

      One of the following:

      • ClickPipePostPubSubServiceAccountSource
        View ClickPipePostPubSubServiceAccountSource schema →
        10 properties
        • formatJSONEachRoworAvroorProtobufrequired

          Format of messages in the Pub/Sub topic. GCP Pub/Sub ClickPipes are in limited preview — contact support to enable this feature for your organization.

          Example: "JSONEachRow"
        • projectIdstringrequired

          GCP project ID that owns the Pub/Sub topic.

          Example: "my-gcp-project"
        • topicstringrequired

          Pub/Sub topic name (not the fully-qualified path).

          Example: "my-topic"
        • authenticationSERVICE_ACCOUNTrequired

          Authenticate with a GCP service account JSON key.

          Example: "SERVICE_ACCOUNT"
        • seekTypelatestorearliestortimestamprequired

          Starting position strategy for consuming the subscription. The seekTimestamp companion is required only when seekType is "timestamp"; setting it for a mismatched seek type is rejected.

          Example: "earliest"
        • serviceAccountKeyobjectrequired
          1 properties
          • serviceAccountFilestringrequired

            Google Cloud service account JSON key file content, base64 encoded.

        • seekTimestampoptionalstring | null

          RFC 3339 / ISO 8601 timestamp to seek to. Required when seekType is "timestamp"; must be omitted otherwise.

          format: date-time
          Example: "2026-04-10T12:00:00Z"
        • filteroptionalstring | null

          Optional Pub/Sub subscription filter expression (CEL). Maximum 256 characters.

          maxLength: 256
        • enableOrderingoptionalboolean | null

          Whether to enable ordered delivery of messages (requires messages to be published with ordering keys).

        • ackDeadlineoptionalinteger | null

          Acknowledgement deadline for messages, in seconds. Must be between 10 and 600.

          maximum: 600, minimum: 10
      • ClickPipePostPubSubWorkloadIdentitySource
        View ClickPipePostPubSubWorkloadIdentitySource schema →
        9 properties
        • formatJSONEachRoworAvroorProtobufrequired

          Format of messages in the Pub/Sub topic. GCP Pub/Sub ClickPipes are in limited preview — contact support to enable this feature for your organization.

          Example: "JSONEachRow"
        • projectIdstringrequired

          GCP project ID that owns the Pub/Sub topic.

          Example: "my-gcp-project"
        • topicstringrequired

          Pub/Sub topic name (not the fully-qualified path).

          Example: "my-topic"
        • authenticationSERVICE_ACCOUNT_WORKLOAD_IDENTITYrequired

          SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources.

          Example: "SERVICE_ACCOUNT_WORKLOAD_IDENTITY"
        • seekTypelatestorearliestortimestamprequired

          Starting position strategy for consuming the subscription. The seekTimestamp companion is required only when seekType is "timestamp"; setting it for a mismatched seek type is rejected.

          Example: "earliest"
        • seekTimestampoptionalstring | null

          RFC 3339 / ISO 8601 timestamp to seek to. Required when seekType is "timestamp"; must be omitted otherwise.

          format: date-time
          Example: "2026-04-10T12:00:00Z"
        • filteroptionalstring | null

          Optional Pub/Sub subscription filter expression (CEL). Maximum 256 characters.

          maxLength: 256
        • enableOrderingoptionalboolean | null

          Whether to enable ordered delivery of messages (requires messages to be published with ordering keys).

        • ackDeadlineoptionalinteger | null

          Acknowledgement deadline for messages, in seconds. Must be between 10 and 600.

          maximum: 600, minimum: 10
    • objectStorageoptionalobject
      16 properties
      • typeoptionals3orgcsordospacesorazureblobstorageorcloudflarer2orovhobjectstorage

        Type of the ObjectStorage source.

      • formatoptionalJSONEachRoworJSONAsObjectorCSVorCSVWithNamesorTabSeparatedorTabSeparatedWithNames+2 more

        Format of the files.

      • urloptionalstring

        Provide a path to the file(s) you want to ingest. You can specify multiple files using bash-like wildcards. For more information, see the documentation on using wildcards in path: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations

        Example: "https://datasets-documentation.s3.eu-west-3.amazonaws.com/http/**.ndjson.gz"
      • delimiteroptionalstring | null

        Delimiter used in the files.

        Example: ","
      • compressionoptionalnoneorgziporgzorbrotliorbrorxz+3 more

        Compression algorithm used for the files.

        Example: "auto"
      • isContinuousoptionalboolean | null

        If set to true, the pipe will continuously read new files from the source. If set to false, the pipe will read the files only once. New files have to be uploaded lexically order.

      • queueUrloptionalstring | null

        Queue URL for event-based continuous ingestion. For S3, provide an SQS queue URL. For GCS, provide a Pub/Sub subscription (e.g. projects/{project}/subscriptions/{name}). When provided, files are ingested based on event notifications rather than lexicographical order. Only applicable when isContinuous is true and authentication is not public.

        Example: "https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue"
      • skipInitialLoadoptionalboolean | null

        If set to true, skips the initial load and only ingests files delivered by queue notifications. Only applicable when queueUrl is provided.

      • startAfteroptionalstring | null

        Skip all files up to and including this object key during the initial load. Cannot be provided when skipInitialLoad is true.

        Example: "events/2026-06-01/"
      • authenticationoptionalIAM_ROLEorIAM_USERorCONNECTION_STRINGorSERVICE_ACCOUNTorSERVICE_ACCOUNT_WORKLOAD_IDENTITY

        Authentication method. IAM_USER is for S3, GCS, and DigitalOcean Spaces. IAM_ROLE is for S3 only. SERVICE_ACCOUNT is for GCS only. For GCS, SERVICE_ACCOUNT_WORKLOAD_IDENTITY is in Private Preview. ClickPipes uses the GCP service account returned in gcpWorkloadIdentity.principal by the operation with operationId clickPipesServiceContextGet; grant it access to the source resources. CONNECTION_STRING is for Azure Blob Storage. PUBLIC uses no authentication.

      • iamRoleoptionalstring | null

        IAM role to be used with IAM role authentication. Read more in ClickPipes documentation: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#authentication

        Example: "arn:aws:iam::123456789012:role/MyRole"
      • connectionStringoptionalstring | null

        Connection string for Azure Blob Storage authentication. Required when authentication is CONNECTION_STRING.

        Example: "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net"
      • pathoptionalstring | null

        Path to the file(s) within the Azure container. Used for Azure Blob Storage sources. You can specify multiple files using bash-like wildcards. For more information, see the documentation on using wildcards in path: https://clickhouse.com/docs/en/integrations/clickpipes/object-storage#limitations

        Example: "data/logs/*.json"
      • azureContainerNameoptionalstring | null

        Container name for Azure Blob Storage. Required when type is azureblobstorage.

        Example: "mycontainer"
      • accessKeyoptionalMskIamUserornull
        2 variants

        One of the following:

      • serviceAccountKeyoptionalstring | null

        Base64-encoded GCP service account JSON key. Required when authentication is SERVICE_ACCOUNT.

Response

JSON

200

Successful response

JSON
  • statusoptionalnumber

    HTTP status code.

    Example: 200
  • requestIdoptionalstring

    Unique id assigned to every request. UUIDv4

    format: uuid
  • resultoptionalobject
    2 properties
    • fieldsoptionalarray ofobject

      Inferred schema fields with their ClickHouse data types.

      3 properties
      • nameoptionalstring

        Name of the inferred field.

        Example: "user_id"
      • typeoptionalstring

        Inferred ClickHouse data type of the field.

        Example: "Int64"
      • optionaloptionalboolean | null

        Whether the field is optional (nullable) in the source.

    • metaoptionalClickPipeSchemaDiscoveryMetaornull
      2 variants

      One of the following:

Report an issue
Navigation