How EMRFS and HDFS Can Optimize Big Data Processing on Amazon EMR

|

Amazon EMR (Elastic MapReduce) is a managed big-data platform that simplifies running frameworks like Apache Hadoop and Spark at scale. Choosing the right storage approach—EMRFS vs HDFS—is central to optimizing performance, cost, and flexibility. Here’s how each file system helps—and when a hybrid approach shines.

What Is HDFS on Amazon EMR?

Hadoop Distributed File System (HDFS) is a distributed, scalable, and fault-tolerant file system that stores data across cluster nodes. It ensures redundancy through replication, low latency by virtue of data-local compute, and strong performance for disk-heavy or iterative workloads docs.aws.amazon.com+1.

On EMR, HDFS uses ephemeral EC2 instance storage. It’s ideal for:

  • Caching intermediate results between MapReduce or Spark steps

  • Handling disk I/O–intensive workloads or iterative read patterns

  • Low-latency access requirements docs.aws.amazon.com+1

However, HDFS storage is ephemeral: once the cluster terminates, the data disappears docs.aws.amazon.comaws.github.io. It also comes with cost overhead due to data replication (2–3× storage) and the need for provisioning adequate EBS storage aws.github.io.

What Is EMRFS?

EMRFS is the Amazon EMR File System—an implementation of the Hadoop file interface that redirects reads and writes to Amazon S3. It allows Hadoop/Spark jobs running on EMR to treat S3 as if it were a file system—with important added capabilities: strong read-after-write consistency, optional encryption, and IAM-based access control docs.aws.amazon.com+2docs.aws.amazon.com+2.

Advantages of EMRFS include:

  • Persistent, decoupled storage—data outlives the cluster; storage and compute scale independently docs.aws.amazon.comaws.github.io.

  • Cost efficiency: S3 storage ($0.023/GB-month) is far cheaper than replicated HDFS on EBS ($0.10/GB-month, with 3× replication = 12× cost) aws.github.io.

  • Durability, scalability, integration with AWS data ecosystem.

Drawbacks include potential higher latency vs HDFS, and semantics differences (e.g. rename/list behavior).

EMRFS vs HDFS: A Side-by-Side Comparison

Feature / Use Case HDFS (ephemeral) EMRFS (S3-based)
Storage Location On-cluster (EC2 storage) Amazon S3
Persistence Temporary (cluster-bound) Persistent after cluster ends
Latency / Throughput Low latency, high throughput Higher latency, network-bound
Cost Higher (EBS + replication) Lower (S3 pay-only storage)
Scalability Limited by cluster size Virtually unlimited via S3
Best For Iterative, disk-intensive jobs One-off reads/writes, archival
Data Access Local data access Across clusters and jobs

When to Use HDFS vs EMRFS — Use-Case-Based Guidance

Use HDFS when:

  • Your job involves iterative algorithms (e.g., machine learning cycles, multi-stage Spark transforms).

  • You require low-latency or random I/O to datasets repeatedly.

  • You’re optimizing for local performance and in-cluster cache reuse docs.aws.amazon.com.

  • You need to cache or stage intermediate output fast during job flows docs.aws.amazon.com.

Use EMRFS when:

  • Data needs to be persistent across cluster runs or shared across teams/jobs.

  • You plan to use transient clusters—spin up, run job, shut down—for cost efficiency.

  • You want scalable, durable storage with lower cost and no capacity planning.

  • Data latency is acceptable for one-pass workloads like ETL, batch reads, or archival writes.

Hybrid Approach:

A common best practice is:

  1. Store source data on S3 (via EMRFS).

  2. Load into HDFS (or let Spark cache) for compute-intensive steps.

  3. Write final output back to S3 via EMRFS.

This balances cost, performance, and persistence.

Advanced Optimization: Committers & Direct Write

EMRFS S3-Optimized Committer

EMRFS includes a S3-optimized committer (available by default in EMR 5.20.0+, and enhanced in 6.4+) that improves performance by avoiding S3 list/rename operations during job commit phases. This substantially accelerates write workloads and reduces API overhead docs.aws.amazon.com.

Spark’s S3A Committers

Beyond EMRFS, Spark (via hadoop-aws) supports S3A committers:

  • DirectoryCommitter: splits data writes into temporary directories and consolidates on commit.

  • MagicS3GuardCommitter: adds metadata consistency via S3Guard.

  • PartitionedCommitter: optimized for partitioned data structures (e.g. Parquet, Hive) Medium.

Configuration example (PySpark):

spark = SparkSession.builder \
.config("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", "2") \
.config("spark.hadoop.fs.s3a.committer.name", "magic") \
.config("spark.hadoop.fs.s3a.bucket.all.committer.magic.enabled", "true") \
.getOrCreate()

These stabilize writes and improve throughput when writing to S3.

EMRFS Direct Write

With EMR 6.1.0+, EMRFS direct write allows Spark and Hadoop jobs to write directly to S3 (during shuffle/spill), bypassing HDFS staging—streamlining performance and resource use Medium.

Configuration Examples

1. EMRFS S3-Optimized Committer (Spark on EMR)

# Example spark-submit enabling optimized committer
spark-submit \
--conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 \
--conf spark.hadoop.fs.s3a.committer.name=directory \
--conf spark.hadoop.fs.s3a.committer.staging.conflict.mode=overwrite \
your_spark_job.py

Notes: ensure EMR version is >= 5.19 (enabled by default in 5.20+), formats supported (6.4+ includes Parquet, ORC, CSV, JSON) docs.aws.amazon.com.

2. Hybrid Processing Workflow (PySpark)

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName(“HybridWorkflow”).getOrCreate()

# Load from S3 (EMRFS)
df = spark.read.parquet(“s3://my-bucket/input-data/”)

# Persist in memory or write to HDFS for faster iteration
df.cache()
result = df.groupBy(“key”).count()

# Write result back directly to S3 using optimized committer
result.write \
.format(“parquet”) \
.option(“compression”, “snappy”) \
.option(“fs.s3a.committer.name”, “directory”) \
.mode(“overwrite”) \
.save(“s3://my-bucket/output-data/”)

3. HDFS for Temporary Caching

# Load into HDFS
hdfs dfs -copyFromLocal local_data.csv /tmp/data.csv# Process with Spark from HDFS
spark-submit –class MyJob \
–master yarn \
myapp.jar hdfs:///tmp/data.csv hdfs:///tmp/output/# Move output to S3
hdfs dfs -copyToLocal /tmp/output/*.parquet .
aws s3 cp ./output/ s3://my-bucket/final-output/ –recursive

Or better, use S3DistCp for efficient parallel copy between HDFS and S3 netapp.com.

Best Practices to Optimize EMR Processing

  • Compress and use columnar formats (Parquet, ORC) for faster reads and lower storage/network usage aws.github.io.

  • Avoid small files; aim for larger file sizes (~128 MB+) to reduce S3 LIST overhead aws.github.io.

  • Partition S3 data to limit scan scope and accelerate queries aws.github.io.

  • Use S3-optimized committers to reduce write overhead and increase speed docs.aws.amazon.comMedium.

  • Combine HDFS and EMRFS when needing fast intermediate compute and persistent storage.

Conclusion

Choosing between HDFS and EMRFS on Amazon EMR is not about picking the superior—often, the optimal solution is a thoughtful blend.

  • Use HDFS when you need high-speed, low-latency, and I/O-intensive intermediate processing, keeping data ephemeral within the cluster.

  • Use EMRFS (S3) for durable, scalable storage that lives beyond the cluster—especially suitable for one-pass workloads, batch jobs, and cross-cluster data sharing.

  • Leverage EMRFS S3-Optimized Committers, S3A Committers, and direct write to significantly accelerate and stabilize data writes to S3.

  • Embrace file-format best practices—compression, partitioning, file sizing—to reduce cost and improve query performance.

By aligning storage choices with workload characteristics and using tooling smartly, you achieve fast, resilient, cost-efficient big-data pipelines on Amazon EMR.