knamp-content-transmitter

Crawl HTML from a server (triggered by Events) and produce it to Apache Kafka

Usage no npm install needed!

<script type="module">
  import knampContentTransmitter from 'https://cdn.skypack.dev/knamp-content-transmitter';
</script>

README

Content Transmitter

Crawl HTML Pages (triggered by Kafka events) and produces selected content to Kafka topic.

Usage

Install via yarn

yarn install knamp-content-transmitter

Then configure it and use it

import ContentTransmitter from "knamp-content-transmitter";

(async () => {

  const processor = await ContentTransmitter({
    clientName: "transmitter-client",
    consumeFrom: "transmitter-consume",
    groupId: "transmitter-group",
    produceTo: "generator-consume",
  });

  processor.on("error", (error) => {
    console.error(error);
  });
})();

Uses

  • Sinek, consuming and producing messages to and from Apache Kafka

License

This project is under MIT.