PHP message queuing Kafka uses

Install Kafka service

Go directly to kafka website and download the latest



Unzip, enter directory

tar -zxvf kafka_2.13-2.5.0.tgz
cd kafka_2.13-2.5.0


Start Kafka service

Starting a single node Zookeeper instance using the script in the installation package

bin/ -daemon config/


Using kafka server- Start kafka service


bin/ config/


Create topic

bin/ --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test


Check the topic list to see if it is created successfully

bin/ --list --zookeeper localhost:2181
$ test


Producers, sending messages

bin/ --broker-list --topic test


The service aspect is almost here. Next is php.

Install PHP extension

The installation of rdkafka depends on librdkafka, so install librdkafka first

git clone
cd librdkafka
make && make install


Install PHP rdkafka extension

git clone
cd php-rdkafka
./configure --with-php-config=/usr/local/Cellar/php@7.2/7.2.24/bin/php-config  ## Here, fill in the path according to your own situation
make && make install


Add PHP ini


Restart PHP FPM, and you should see the extension.

Using Kafka

Create a producer class

class KafkaProducer
    public static $brokerList = '';
    public static function send($message, $topic)
        self::producer($message, $topic);
    public static function producer($message, $topic = 'test')
        $conf = new \RdKafka\Conf();
        $conf->set('', self::$brokerList);
        $producer = new \RdKafka\Producer($conf);
        $topic = $producer->newTopic($topic);
        $topic->produce(RD_KAFKA_PARTITION_UA, 0, json_encode($message));
        $result = $producer->flush(10000);
        if (RD_KAFKA_RESP_ERR_NO_ERROR !== $result) {
            throw new \RuntimeException('Was unable to flush, messages might be lost!');



Create a consumption class

class KafkaConsumer
    public static $brokerList = '';
      public static function consumer()
        $conf = new \RdKafka\Conf();
        $conf->set('', 'test');
        $rk = new \RdKafka\Consumer($conf);
        $topicConf = new \RdKafka\TopicConf();
        $topicConf->set('', 100);
        $topicConf->set('', 'broker');
        $topicConf->set('auto.offset.reset', 'smallest');
        $topic = $rk->newTopic('test', $topicConf);
        $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
        while (true) {
            $message = $topic->consume(0, 120*10000);
            switch ($message->err) {
                case RD_KAFKA_RESP_ERR_NO_ERROR:
                case RD_KAFKA_RESP_ERR__PARTITION_EOF:
       echo "No more messages; will wait for more\n";
                case RD_KAFKA_RESP_ERR__TIMED_OUT:
                    echo "Timed out\n";
                    throw new \Exception($message->errstr(), $message->err);


Problem summary

1, No Java runtime present, requesting install

Because kafka needs java environment support, install java environment. You can go to javase-jdk14-downloads and choose your own version to download and install

2. Create topic: Replication factor: 1 larger than available brokers: 0

That is to say, there is no effective brokers available. You need to make sure your kafka is activated

Tags: PHP kafka Zookeeper git

Posted on Tue, 19 May 2020 07:36:32 -0700 by dmeade