HDFS articles

Recovery in HDFS

Recovery process in HDFS helps to achieve fault tolerance. It concerns as well worker pipeline as blocks.

Continue Reading β†’

HDFS on disk explained

Among all previous posts we could learn a lot about HDFS transaction logs, operations on closed files and so on. Thanks to that we can take a look now on data structure of NameNode and DataNode.

Continue Reading β†’

Checkpoint in HDFS

HDSF is not an exception in the Big Data world and as other actors, it also uses checkpoints.

Continue Reading β†’

Handling small files in HDFS

HDFS is not well suited tool to store a lot of small files. Even if that's true, some methods exist to handle small files better.

Continue Reading β†’

Snapshot in HDFS

Implementing snapshots in distributed file systems is not a simple job. It must take into account different aspects, such as file deletion or content changes, and keep file system consistent among them.

Continue Reading β†’

Cache in HDFS

Hadoop 2.3.0 brought an in-memory caching layer to HDFS. Even if this is quite old feature (released in 02/2014), it's always beneficial to know it.

Continue Reading β†’

Append and truncate in HDFS

Making an immutable distributed file system is easier than building a mutable one. HDFS, even if initially was destined to not changing data, supports mutability through 2 operations: append and truncate.

Continue Reading β†’

FSImage in HDFS

Edit log would be useless without its complementary structure called FSImage.

Continue Reading β†’

Edit log in HDFS

HDFS stores everything that happens on transaction log files. They're used during checkpoint and file system recovery. So, they take quite important place in HDFS architecture.

Continue Reading β†’

States in HDFS

Replica and blocks are HDFS entities having more than 1 state in their whole lifecycle. Being aware of these states helps a lot to understand what happens when a new file is written.

Continue Reading β†’

Replication in HDFS

HDFS is a fault-tolerant and resilient distributed file system. It wouldn't be possible without, among others, blocks replication.

Continue Reading β†’

File operations in HDFS

Previous article presented theoretical information about HDFS files. This post deepens this topic.

Continue Reading β†’

Files in HDFS

Files in HDFS are different from files from local file system. They're fault-tolerant, can be stored in different abstractions and are based on quite big blocks comparing to blocks in local file system.

Continue Reading β†’

Introduction to HDFS

HDFS is one of most popular distributed file systems in our days. It changes from other older distributed file systems thanks to its reliability.

Continue Reading β†’