libZSservicesZSamazonka-kinesis-firehoseZSamazonka-kinesis-firehose
Copyright(c) 2013-2021 Brendan Hay
LicenseMozilla Public License, v. 2.0.
MaintainerBrendan Hay <brendan.g.hay+amazonka@gmail.com>
Stabilityauto-generated
Portabilitynon-portable (GHC extensions)
Safe HaskellNone

Amazonka.Firehose.Types.OrcSerDe

Description

 
Synopsis

Documentation

data OrcSerDe Source #

A serializer to use for converting data to the ORC format before storing it in Amazon S3. For more information, see Apache ORC.

See: newOrcSerDe smart constructor.

Constructors

OrcSerDe' 

Fields

  • bloomFilterFalsePositiveProbability :: Maybe Double

    The Bloom filter false positive probability (FPP). The lower the FPP, the bigger the Bloom filter. The default value is 0.05, the minimum is 0, and the maximum is 1.

  • dictionaryKeyThreshold :: Maybe Double

    Represents the fraction of the total number of non-null rows. To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.

  • enablePadding :: Maybe Bool

    Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is false.

  • compression :: Maybe OrcCompression

    The compression code to use over data blocks. The default is SNAPPY.

  • bloomFilterColumns :: Maybe [Text]

    The column names for which you want Kinesis Data Firehose to create bloom filters. The default is null.

  • rowIndexStride :: Maybe Natural

    The number of rows between index entries. The default is 10,000 and the minimum is 1,000.

  • formatVersion :: Maybe OrcFormatVersion

    The version of the file to write. The possible values are V0_11 and V0_12. The default is V0_12.

  • blockSizeBytes :: Maybe Natural

    The Hadoop Distributed File System (HDFS) block size. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is 256 MiB and the minimum is 64 MiB. Kinesis Data Firehose uses this value for padding calculations.

  • stripeSizeBytes :: Maybe Natural

    The number of bytes in each stripe. The default is 64 MiB and the minimum is 8 MiB.

  • paddingTolerance :: Maybe Double

    A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size. The default value is 0.05, which means 5 percent of stripe size.

    For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

    Kinesis Data Firehose ignores this parameter when OrcSerDe$EnablePadding is false.

Instances

Instances details
Eq OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

Methods

(==) :: OrcSerDe -> OrcSerDe -> Bool #

(/=) :: OrcSerDe -> OrcSerDe -> Bool #

Read OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

Methods

readsPrec :: Int -> ReadS OrcSerDe #

readList :: ReadS [OrcSerDe] #

readPrec :: ReadPrec OrcSerDe #

readListPrec :: ReadPrec [OrcSerDe] #

Show OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

Methods

showsPrec :: Int -> OrcSerDe -> ShowS #

show :: OrcSerDe -> String #

showList :: [OrcSerDe] -> ShowS #

Generic OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

Associated Types

type Rep OrcSerDe :: Type -> Type #

Methods

from :: OrcSerDe -> Rep OrcSerDe x #

to :: Rep OrcSerDe x -> OrcSerDe #

NFData OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

Methods

rnf :: OrcSerDe -> () #

Hashable OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

Methods

hashWithSalt :: Int -> OrcSerDe -> Int #

hash :: OrcSerDe -> Int #

ToJSON OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

Methods

toJSON :: OrcSerDe -> Value #

toEncoding :: OrcSerDe -> Encoding #

toJSONList :: [OrcSerDe] -> Value #

toEncodingList :: [OrcSerDe] -> Encoding #

FromJSON OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

Methods

parseJSON :: Value -> Parser OrcSerDe #

parseJSONList :: Value -> Parser [OrcSerDe] #

type Rep OrcSerDe Source # 
Instance details

Defined in Amazonka.Firehose.Types.OrcSerDe

type Rep OrcSerDe = D1 ('MetaData "OrcSerDe" "Amazonka.Firehose.Types.OrcSerDe" "libZSservicesZSamazonka-kinesis-firehoseZSamazonka-kinesis-firehose" 'False) (C1 ('MetaCons "OrcSerDe'" 'PrefixI 'True) (((S1 ('MetaSel ('Just "bloomFilterFalsePositiveProbability") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe Double)) :*: S1 ('MetaSel ('Just "dictionaryKeyThreshold") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe Double))) :*: (S1 ('MetaSel ('Just "enablePadding") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe Bool)) :*: (S1 ('MetaSel ('Just "compression") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe OrcCompression)) :*: S1 ('MetaSel ('Just "bloomFilterColumns") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe [Text]))))) :*: ((S1 ('MetaSel ('Just "rowIndexStride") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe Natural)) :*: S1 ('MetaSel ('Just "formatVersion") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe OrcFormatVersion))) :*: (S1 ('MetaSel ('Just "blockSizeBytes") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe Natural)) :*: (S1 ('MetaSel ('Just "stripeSizeBytes") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe Natural)) :*: S1 ('MetaSel ('Just "paddingTolerance") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedStrict) (Rec0 (Maybe Double)))))))

newOrcSerDe :: OrcSerDe Source #

Create a value of OrcSerDe with all optional fields omitted.

Use generic-lens or optics to modify other optional fields.

The following record fields are available, with the corresponding lenses provided for backwards compatibility:

$sel:bloomFilterFalsePositiveProbability:OrcSerDe', orcSerDe_bloomFilterFalsePositiveProbability - The Bloom filter false positive probability (FPP). The lower the FPP, the bigger the Bloom filter. The default value is 0.05, the minimum is 0, and the maximum is 1.

$sel:dictionaryKeyThreshold:OrcSerDe', orcSerDe_dictionaryKeyThreshold - Represents the fraction of the total number of non-null rows. To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.

$sel:enablePadding:OrcSerDe', orcSerDe_enablePadding - Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is false.

$sel:compression:OrcSerDe', orcSerDe_compression - The compression code to use over data blocks. The default is SNAPPY.

$sel:bloomFilterColumns:OrcSerDe', orcSerDe_bloomFilterColumns - The column names for which you want Kinesis Data Firehose to create bloom filters. The default is null.

$sel:rowIndexStride:OrcSerDe', orcSerDe_rowIndexStride - The number of rows between index entries. The default is 10,000 and the minimum is 1,000.

$sel:formatVersion:OrcSerDe', orcSerDe_formatVersion - The version of the file to write. The possible values are V0_11 and V0_12. The default is V0_12.

$sel:blockSizeBytes:OrcSerDe', orcSerDe_blockSizeBytes - The Hadoop Distributed File System (HDFS) block size. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is 256 MiB and the minimum is 64 MiB. Kinesis Data Firehose uses this value for padding calculations.

$sel:stripeSizeBytes:OrcSerDe', orcSerDe_stripeSizeBytes - The number of bytes in each stripe. The default is 64 MiB and the minimum is 8 MiB.

$sel:paddingTolerance:OrcSerDe', orcSerDe_paddingTolerance - A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size. The default value is 0.05, which means 5 percent of stripe size.

For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

Kinesis Data Firehose ignores this parameter when OrcSerDe$EnablePadding is false.

orcSerDe_bloomFilterFalsePositiveProbability :: Lens' OrcSerDe (Maybe Double) Source #

The Bloom filter false positive probability (FPP). The lower the FPP, the bigger the Bloom filter. The default value is 0.05, the minimum is 0, and the maximum is 1.

orcSerDe_dictionaryKeyThreshold :: Lens' OrcSerDe (Maybe Double) Source #

Represents the fraction of the total number of non-null rows. To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.

orcSerDe_enablePadding :: Lens' OrcSerDe (Maybe Bool) Source #

Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is false.

orcSerDe_compression :: Lens' OrcSerDe (Maybe OrcCompression) Source #

The compression code to use over data blocks. The default is SNAPPY.

orcSerDe_bloomFilterColumns :: Lens' OrcSerDe (Maybe [Text]) Source #

The column names for which you want Kinesis Data Firehose to create bloom filters. The default is null.

orcSerDe_rowIndexStride :: Lens' OrcSerDe (Maybe Natural) Source #

The number of rows between index entries. The default is 10,000 and the minimum is 1,000.

orcSerDe_formatVersion :: Lens' OrcSerDe (Maybe OrcFormatVersion) Source #

The version of the file to write. The possible values are V0_11 and V0_12. The default is V0_12.

orcSerDe_blockSizeBytes :: Lens' OrcSerDe (Maybe Natural) Source #

The Hadoop Distributed File System (HDFS) block size. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is 256 MiB and the minimum is 64 MiB. Kinesis Data Firehose uses this value for padding calculations.

orcSerDe_stripeSizeBytes :: Lens' OrcSerDe (Maybe Natural) Source #

The number of bytes in each stripe. The default is 64 MiB and the minimum is 8 MiB.

orcSerDe_paddingTolerance :: Lens' OrcSerDe (Maybe Double) Source #

A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size. The default value is 0.05, which means 5 percent of stripe size.

For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

Kinesis Data Firehose ignores this parameter when OrcSerDe$EnablePadding is false.