Today we are excited to announce the beta launch of Kafka support on RapidAPI Marketplace and RapidAPI Enterprise Hub. RapidAPI is the first API platform to allow discovery of Kafka clusters and topics, viewing of schemas, and testing/consuming of records from a browser. Development teams can now discover all available Kafka clusters / instances in the same place where they find, test, and connect to microservices, REST, SOAP, and GraphQL APIs.
Kafka is now supported as another “API type” on RapidAPI Marketplace and RapidAPI Enterprise Hub
Making Kafka Services Discoverable
Apache Kafka is a distributed streaming platform that allows building real-time streaming pipelines and applications. Kafka is increasingly used as the message broker in event-driven architectures with asynchronous microservices. Kafka clients allow you to write distributed applications and microservices that read, write, and process streams of events. Kafka has been adopted in more than 80% of Fortune 100 companies (source), as well as many of RapidAPI’s enterprise customers.
The rapid adoption and growth of Kafka services has created challenges similar to those faced with building software using REST APIs: discovery, understanding formats, and testing services. Specifically developers and technology teams struggle with:
- Which Kafka clusters are available, what topics have been defined on those, and what is the purpose of each topic?
- How to connect to those topics and understand their configuration, and how many partitions are set up?
- What are the formats and schemas for the key and value?
- How to easily test and debug for the Kafka consumer (or producer) without an existing producer (or vice versa)?
To solve these challenges, we are excited to announce that we are adding support for Kafka as another “API type” on RapidAPI Marketplace and RapidAPI Enterprise Hub. At RapidAPI our mission is to empower developers to create transformative software applications – thus we see Kafka services as another important “API type” for development teams. With Kafka APIs on RapidAPI, API providers can make Kafka topics and metadata discoverable, and developers can discover Kafka topics, explore topic schemas and configuration, as well as consume and produce records from a browser.
Getting Started with Kafka APIs on RapidAPI
You can now publish Kafka APIs to RapidAPI Marketplace and RapidAPI Enterprise Hub by specifying one or more bootstrap server host:port pairs, authentication information (if needed), and schema registry information. Just like REST and GraphQL APIs on RapidAPI, Kafka APIs allow you to add markdown Docs, Spotlights, and Tutorials. See our public documentation on publishing Kafka APIs here for more information.
API providers can now publish Kafka topics, schemas and configurations to RapidAPI Marketplace and RapidAPI Enterprise Hub using the new Kafka API type.
You can also now search for Kafka APIs just like you can search for other APIs on RapidAPI Marketplace and RapidAPI Enterprise Hub. For Kafka APIs that you have access to, you can search and view available topics. For each topic, you can view configuration, key and value schemas (AVRO, Protobuf, and JSON Schema are supported). You can view records before or after a specific date and search for keywords within records. If enabled by the API provider, you can also produce records to topics, or even force a specific partition. See our public documentation on consuming Kafka APIs here for more details.
RapidAPI provides a playground for API consumers and providers to search and view available clusters and topics, as well as view configuration, key and value schemas.
Open Source Contribution
This release also included a contribution to the open source community. As part of the development process, the team used a public open source library to communicate with the Confluent schema registry. This library originally supported serialization in the Apache Avro and Confluent wire formats. Our team extended the use of the library to support more formats – we added JSON and protocol buffer support. The improvements by the RapidAPI team have been published by the maintainer of the library, ensuring that the broader Kafka community benefits from these efforts.
During this beta launch, we are excited to learn about your experience and get your feedback as we continue our journey in supporting Kafka APIs. RapidAPI support for Kafka APIs will reach general availability by Q3 of 2021. You can check out a demo API for Kafka on RapidAPI here. For enterprise customers, support for Kafka was introduced as part of our latest version of RapidAPI Enterprise Hub – see release notes and this blog to learn about other features included in this update.