Exam Professional-Data-Engineer Topic 4 Question 215 Discussion
Actual exam question for Google's Professional-Data-Engineer exam
Question #: 215
Topic #: 4
Question #: 215
Topic #: 4
You've migrated a Hadoop job from an on-prem cluster to dataproc and GCS. Your Spark job is a complicated analytical workload that consists of many shuffing operations and initial data are parquet files (on average 200-400 MB size each). You see some degradation in performance after the migration to Dataproc, so you'd like to optimize for it. You need to keep in mind that your organization is very cost-sensitive, so you'd like to continue using Dataproc on preemptibles (with 2 non-preemptible workers only) for this workload.
What should you do?
What should you do?
Suggested Answer: C Vote an answer
To optimize the performance of a complex Spark job on Dataproc that heavily relies on shuffling operations, and given the cost constraints of using preemptible VMs, switching from HDDs to SSDs and using HDFS as an intermediate storage layer can significantly improve performance. Here's why option C is the best choice:
Performance of SSDs:
SSDs provide much faster read and write speeds compared to HDDs, which is crucial for performance-intensive operations like shuffling in Spark jobs.
Using SSDs can reduce I/O bottlenecks during the shuffle phase of your Spark job, improving overall job performance.
Intermediate Storage with HDFS:
Copying data from Google Cloud Storage (GCS) to HDFS for intermediate storage can reduce latency compared to reading directly from GCS.
HDFS provides better locality and faster data access within the Dataproc cluster, which can significantly improve the efficiency of shuffling and other I/O operations.
Cost Considerations:
Although SSDs are more expensive than HDDs, the performance improvement for shuffle-heavy workloads can justify the cost, especially if the improved performance reduces the overall runtime and thereby the cost of using preemptible VMs.
Using preemptible VMs with SSDs for this workload balances the cost and performance trade-offs effectively.
Performance of SSDs:
SSDs provide much faster read and write speeds compared to HDDs, which is crucial for performance-intensive operations like shuffling in Spark jobs.
Using SSDs can reduce I/O bottlenecks during the shuffle phase of your Spark job, improving overall job performance.
Intermediate Storage with HDFS:
Copying data from Google Cloud Storage (GCS) to HDFS for intermediate storage can reduce latency compared to reading directly from GCS.
HDFS provides better locality and faster data access within the Dataproc cluster, which can significantly improve the efficiency of shuffling and other I/O operations.
Cost Considerations:
Although SSDs are more expensive than HDDs, the performance improvement for shuffle-heavy workloads can justify the cost, especially if the improved performance reduces the overall runtime and thereby the cost of using preemptible VMs.
Using preemptible VMs with SSDs for this workload balances the cost and performance trade-offs effectively.
by Wendell at Feb 08, 2026, 07:20 AM
0
0
0
10
Comments
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
Report Comment
Commenting
You can sign-up / login (it's free).