FUNDAMENTALS OF COMPUTER

Detailed explanation-1: -The default size of a block in HDFS is 128 MB (Hadoop 2. x) and 64 MB (Hadoop 1. x) which is much larger as compared to the Linux system where the block size is 4KB. The reason of having this huge block size is to minimize the cost of seek and reduce the meta data information generated per block.

Detailed explanation-2: -The default block size is 1024 bytes for file systems smaller than 1 TB, and 8192 bytes for file systems 1 TB or larger. Choose a block size based on the type of application being run. For example, if there are many small files, a 1 KB block size may save space.

Detailed explanation-3: -The size of the data block in HDFS is 64 MB by default, which can be configured manually. In general, the data blocks of size 128MB is used in the industry.

Detailed explanation-4: -The default size of the HDFS data block is 128 MB. If blocks are small, there will be too many blocks in Hadoop HDFS and thus too much metadata to store. Managing such a huge number of blocks and metadata will create overhead and lead to traffic in a network.

Detailed explanation-5: -This configuration controls the size of these input blocks. By default, it is 128 MB (128000000 bytes). Setting this value in the notebook with spark. conf.