AWS Kinesis python으로 다뤄보기

Python-kinesis

Python에서 Kinesis Datastream에 연동하여 데이터를 주고/받는 과정

Kinesis로 데이터를 전달할 때, byte혹은 bytearray로 전달 해야 함 (encode('utf-8')로 전달 해 주어야 함)

1. Requirements

   • Install boto3, kinesis

     pip3 install -U pip
     python3 -m pip install boto3 kinesis

2. Setting Credential and config

   아래 두개의 파일을 생성한다.

   credentials, config 두개의 파일을 ~/.aws에 위치시킨다

   ~/.aws/

   • credentials —> setting access_key, sercret_key

     [default]
     aws_access_key_id=
     aws_secret_access_key=
     

   • config —> setting region, type

     [default]
     region=region_info
     output=json
     

3. Test setting

   • test

     import boto3
     from kinesis.producer import KinesisProducer
     from kinesis.consumer import KinesisConsumer
      
     consumer = KinesisConsumer(stream_name=env.KINESIS_STREAM_NAME)
     for message in consumer:
         print("############")
         print(message)
         print("############")

   • test consumer

     from kinesis.consumer import KinesisConsumer
     from kinesis.state import DynamoDB
      
     consumer = KinesisConsumer(stream_name='my-stream', state=DynamoDB(table_name='my-kinesis-state'))
     for message in consumer:
         print ("Received message: {0}".format(message))

   • test producer

     from kinesis.producer import KinesisProducer
      
     producer = KinesisProducer(stream_name='my-stream')
     producer.put('Hello World from Python')

---

REFERENCE

• python-kinesis document (Link)
• python-kinesis document github (Link)
• aws boto3/kinesis configure documentation (Link)
• python-kinesis & python-s3 example Blod(Link)
• aws boto3 data stream document(Link)
• boto3 - how to credential —> credential process (Link)