Exam CDP-3002 Topic 1 Question 172 Discussion

The spark.sql.autoBroadcastJoinThreshold configuration parameter in Spark specifies the maximum size (in bytes) of a table that can be broadcast to all worker nodes for a join. Setting this value to -1 effectively disables the broadcast join optimization, meaning that Spark will not attempt to broadcast any table regardless of its size. As a result, all joins will use the shuffle join mechanism, which can be less efficient for joining small tables. Options B, C, and D misinterpret the effect of setting the parameter to 1 ; it does not set an unlimited threshold, automatically adjust the threshold, or increase it to improve performance, but rather disables the broadcast join feature altogether.