For more information, see Monitoring with Amazon CloudWatch Logs. If you enable data transformation with Lambda, Firehose can log any Lambda invocation and data delivery errors to Amazon CloudWatch Logs so that you can view the specific error logs if Lambda invocation or data delivery fails. You can specify an extra prefix to be added in front of the “YYYY/MM/DD/HH” UTC time prefix generated by Firehose. If you use data transformation with Lambda, you can enable source record backup, and Amazon Kinesis Data Firehose will deliver the un-transformed incoming data to a separate S3 bucket. You can configure a new Lambda function using one of the Lambda blueprints we provide or choose an existing Lambda function. Q: What is data transformation with Lambda?įirehose can invoke an Amazon Lambda function to transform incoming data before delivering it to destinations. For more information about KMS, see Amazon Key Management Service. While creating your delivery stream, you can choose to encrypt your data with an Amazon Key Management Service (KMS) key that you own. All log events from CloudWatch Logs are already compressed in gzip format, so you should keep Firehose’s compression configuration as uncompressed to avoid double-compression.Īmazon Kinesis Data Firehose allows you to encrypt your data after it’s delivered to your Amazon S3 bucket. You can use CloudWatch Logs subscription feature to stream data from CloudWatch Logs to Kinesis Data Firehose. Q: How does compression work when I use the CloudWatch Logs subscription feature? The service currently supports GZIP, ZIP, and SNAPPY compression formats. Only GZIP is supported if the data is further loaded to Amazon Redshift. For more information, see Creating a Delivery Stream.Īmazon Kinesis Data Firehose allows you to compress your data before delivering it to Amazon S3. You can create an Amazon Kinesis Data Firehose delivery stream through the Firehose Console or the CreateDeliveryStream operation. Q: What are the limits of Amazon Kinesis Data Firehose?įor information about limits, see Amazon Kinesis Data Firehose Limits in the developer guide. For example, a source can be a logging server running on Amazon EC2 instances, an application running on mobile devices, a sensor on an IoT device, or a Kinesis stream.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |