Provide Databricks Databricks-Certified-Professional-Data-Engineer Practice Test Engine for Preparation [Q40-Q62]

Provide Databricks Databricks-Certified-Professional-Data-Engineer Practice Test Engine for Preparation

Detailed New Databricks-Certified-Professional-Data-Engineer Exam Questions for Concept Clearance

NEW QUESTION # 40
In order to use Unity catalog features, which of the following steps needs to be taken on man-aged/external tables in the Databricks workspace?

A. Copy data from workspace to unity catalog
B. Migrate/upgrade objects in workspace managed/external tables/view to unity catalog
C. Enable unity catalog feature in workspace settings
D. Upgrade to DBR version 15.0
E. Upgrade workspace to Unity catalog

Answer: B

Explanation:
Explanation
Upgrade tables and views to Unity Catalog - Azure Databricks | Microsoft Docs Managed table: Upgrade a managed to Unity Catalog External table: Upgrade an external table to Unity Catalog

NEW QUESTION # 41
An external object storage container has been mounted to the location/mnt/finance_eda_bucket.
The following logic was executed to create a database for the finance team:

After the database was successfully created and permissions configured, a member of the finance team runs the following code:

If all users on the finance team are members of thefinancegroup, which statement describes how thetx_sales table will be created?

A. A logical table will persist the physical plan to the Hive Metastore in the Databricks control plane.
B. A logical table will persist the query plan to the Hive Metastore in the Databricks control plane.
C. An external table will be created in the storage container mounted to /mnt/finance eda bucket.
D. A managed table will be created in the DBFS root storage container.
E. An managed table will be created in the storage container mounted to /mnt/finance eda bucket.

Answer: E

Explanation:
https://docs.databricks.com/en/lakehouse/data-objects.html

NEW QUESTION # 42
A junior data engineer on your team has implemented the following code block.

The view new_events contains a batch of records with the same schema as the events Delta table. The event_id field serves as a unique key for this table.
When this query is executed, what will happen with new records that have the same event_id as an existing record?

A. They are inserted.
B. They are ignored.
C. They are deleted.
D. They are merged.
E. They are updated.

Answer: B

Explanation:
This is the correct answer because it describes what will happen with new records that have the same event_id as an existing record when the query is executed. The query uses the INSERT INTO command to append new records from the view new_events to the table events. However, the INSERT INTO command does not check for duplicate values in the primary key column (event_id) and does not perform any update or delete operations on existing records. Therefore, if there are new records that have the same event_id as an existing record, they will be ignored and not inserted into the table events. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Append data using INSERT INTO" section.
"If none of the WHEN MATCHED conditions evaluate to true for a source and target row pair that matches the merge_condition, then the target row is left unchanged."
https://docs.databricks.com/en/sql/language-manual/delta-merge-into.html#:~:text=If%20none%20of%20the%20

NEW QUESTION # 43
A Delta Lake table in the Lakehouse named customer_parsams is used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.
Immediately after each update succeeds, the data engineer team would like to determine the difference between the new version and the previous of the table.
Given the current implementation, which method can be used?

A. Execute DESCRIBE HISTORY customer_churn_params to obtain the full operation metrics for the update, including a log of all records that have been added or modified.
B. Parse the Delta Lake transaction log to identify all newly written data files.
C. Parse the Spark event logs to identify those rows that were updated, inserted, or deleted.
D. Execute a query to calculate the difference between the new version and the previous version using Delta Lake's built-in versioning and time travel functionality.

Answer: D

Explanation:
Delta Lake provides built-in versioning and time travel capabilities, allowing users to query previous snapshots of a table. This feature is particularly useful for understanding changes between different versions of the table. In this scenario, where the table is overwritten nightly, you can use Delta Lake's time travel feature to execute a query comparing the latest version of the table (the current state) with its previous version. This approach effectively identifies the differences (such as new, updated, or deleted records) between the two versions. The other options do not provide a straightforward or efficient way to directly compare different versions of a Delta Lake table.
Reference:
Delta Lake Documentation on Time Travel: Delta Time Travel
Delta Lake Versioning: Delta Lake Versioning Guide

NEW QUESTION # 44
Which of the following is correct for the global temporary view?

A. global temporary views can be still accessed even if the cluster is restarted
B. global temporary views can be accessed across many clusters
C. global temporary views cannot be accessed once the notebook is detached and attached
D. global temporary views can be still accessed even if the notebook is detached and at-tached
E. global temporary views are created in a database called temp database

Answer: D

Explanation:
Explanation
The answer is global temporary views can be still accessed even if the notebook is detached and attached There are two types of temporary views that can be created Local and Global
* A local temporary view is only available with a spark session, so another notebook in the same cluster can not access it. if a notebook is detached and reattached local temporary view is lost.
* A global temporary view is available to all the notebooks in the cluster, even if the notebook is detached and reattached it can still be accessible but if a cluster is restarted the global temporary view is lost.

NEW QUESTION # 45
You were asked to setup a new all-purpose cluster, but the cluster is unable to start which of the following steps do you need to take to identify the root cause of the issue and the reason why the cluster was unable to start?

A. Check the cluster driver logs
B. Storage account
C. Workspace logs
D. Data plane
E. Check the cluster event logs
(Correct)

Answer: E

Explanation:
Explanation
Cluster event logs are very useful, to identify issues pertaining to cluster availability. Cluster may not start due to resource limitations or issues with the cloud providers.
Some of the common issues include a subnet for compute VM reaching its limits or exceeding the subscription or cloud account CPU quota limit.
Here is an example where the cluster did not start due to subscription reaching the quota limit on a certain type of cpu cores for a VM type.
Graphical user interface, text, application, email Description automatically generated

Click on event logs
Graphical user interface, text, application, email Description automatically generated

Click on the message to see the detailed error message on why the cluster did not start.
Graphical user interface, text, application, email Description automatically generated

NEW QUESTION # 46
When you drop an external DELTA table using the SQL Command DROP TABLE table_name, how does it impact metadata(delta log, history), and data stored in the storage?

A. Drops table from metastore, metadata(delta log, history)but keeps the data in storage
B. Drops table from metastore and data in storage but keeps metadata(delta log, history)
C. Drops table from metastore, but keeps metadata(delta log, history)and data in storage
D. Drops table from metastore, data but keeps metadata(delta log, history) in storage
E. Drops table from metastore, metadata(delta log, history)and data in storage

Answer: C

Explanation:
Explanation
The answer is Drops table from metastore, but keeps metadata and data in storage.
When an external table is dropped, only the table definition is dropped from metastore everything including data and metadata(Delta transaction log, time travel history) remains in the storage. Delta log is considered as part of metadata because if you drop a column in a delta table(managed or external) the column is not physically removed from the parquet files rather it is recorded in the delta log. The delta log becomes a key metadata layer for a Delta table to work.
Please see the below image to compare the external delta table and managed delta table and how they differ in how they are created and what happens if you drop the table.
Diagram Description automatically generated

NEW QUESTION # 47
A Delta Lake table representing metadata about content from user has the following schema:
user_id LONG, post_text STRING, post_id STRING, longitude FLOAT, latitude FLOAT, post_time TIMESTAMP, date DATE Based on the above schema, which column is a good candidate for partitioning the Delta Table?

A. Post_time
B. Date
C. User_id
D. Post_id

Answer: B

Explanation:
Partitioning a Delta Lake table improves query performance by organizing data into partitions based on the values of a column. In the given schema, the date column is a good candidate for partitioning for several reasons:
Time-Based Queries: If queries frequently filter or group by date, partitioning by the date column can significantly improve performance by limiting the amount of data scanned.
Granularity: The date column likely has a granularity that leads to a reasonable number of partitions (not too many and not too few). This balance is important for optimizing both read and write performance.
Data Skew: Other columns like post_id or user_id might lead to uneven partition sizes (data skew), which can negatively impact performance.
Partitioning by post_time could also be considered, but typically date is preferred due to its more manageable granularity.
Reference:
Delta Lake Documentation on Table Partitioning: Optimizing Layout with Partitioning

NEW QUESTION # 48
you are currently working on creating a spark stream process to read and write in for a one-time micro batch, and also rewrite the existing target table, fill in the blanks to complete the below command sucesfully.
1.spark.table("source_table")
2..writeStream
3..option("____", "dbfs:/location/silver")
4..outputMode("____")
5..trigger(Once=____)
6..table("target_table")

A. checkpointlocation, True, complete
B. checkpointlocation, True, overwrite
C. checkpointlocation, complete, True
D. targetlocation, overwrite, True
E. checkpointlocation, overwrite, True

Answer: C

NEW QUESTION # 49
A production cluster has 3 executor nodes and uses the same virtual machine type for the driver and executor.
When evaluating the Ganglia Metrics for this cluster, which indicator would signal a bottleneck caused by code executing on the driver?

A. Overall cluster CPU utilization is around 25%
B. Total Disk Space remains constant
C. Network I/O never spikes
D. The five Minute Load Average remains consistent/flat
E. Bytes Received never exceeds 80 million bytes per second

Answer: A

Explanation:
Explanation
This is the correct answer because it indicates a bottleneck caused by code executing on the driver. A bottleneck is a situation where the performance or capacity of a system is limited by a single component or resource. A bottleneck can cause slow execution, high latency, or low throughput. A production cluster has 3 executor nodes and uses the same virtual machine type for the driver and executor. When evaluating the Ganglia Metrics for this cluster, one can look for indicators that show how the cluster resources are being utilized, such as CPU, memory, disk, or network. If the overall cluster CPU utilization is around 25%, it means that only one out of the four nodes (driver + 3 executors) is using its full CPU capacity, while the other three nodes are idle or underutilized. This suggests that the code executing on the driver is taking too long or consuming too much CPU resources, preventing the executors from receiving tasks or data to process. This can happen when the code has driver-side operations that are not parallelized or distributed, such as collecting large amounts of data to the driver, performing complex calculations on the driver, or using non-Spark libraries on the driver. Verified References: [Databricks Certified Data Engineer Professional], under "Spark Core" section; Databricks Documentation, under "View cluster status and event logs - Ganglia metrics" section; Databricks Documentation, under "Avoid collecting large RDDs" section.

NEW QUESTION # 50
You would like to build a spark streaming process to read from a Kafka queue and write to a Delta table every
15 minutes, what is the correct trigger option

A. trigger(processingTime = 15)
B. trigger(15)
C. trigger("15 minutes")
D. trigger(processingTime = "15 Minutes")
E. trigger(process "15 minutes")

Answer: D

Explanation:
Explanation
The answer is trigger(processingTime = "15 Minutes")
Triggers:
*Unspecified
This is the default. This is equivalent to using processingTime="500ms"
*Fixed interval micro-batches .trigger(processingTime="2 minutes")
The query will be executed in micro-batches and kicked off at the user-specified intervals
*One-time micro-batch .trigger(once=True)
The query will execute a single micro-batch to process all the available data and then stop on its own
*One-time micro-batch.trigger .trigger(availableNow=True) -- New feature a better version of (once=True) Databricks supports trigger(availableNow=True) in Databricks Runtime 10.2 and above for Delta Lake and Auto Loader sources. This functionality combines the batch processing approach of trigger once with the ability to configure batch size, resulting in multiple parallelized batches that give greater control for right-sizing batches and the resultant files.

NEW QUESTION # 51
What is the purpose of gold layer in Multi hop architecture?

A. Optimized query performance for business-critical data
B. Preserves grain of original data, without any aggregations
C. Eliminate duplicate records
D. Data quality checks and schema enforcement
E. Optimizes ETL throughput and analytic query performance

Answer: A

Explanation:
Explanation
Medallion Architecture - Databricks
Gold Layer:
1. Powers Ml applications, reporting, dashboards, ad hoc analytics
2. Refined views of data, typically with aggregations
3. Reduces strain on production systems
4. Optimizes query performance for business-critical data
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.

NEW QUESTION # 52
A Delta Lake table was created with the below query:

Consider the following query:
DROP TABLE prod.sales_by_store -
If this statement is executed by a workspace admin, which result will occur?

A. The table will be removed from the catalog and the data will be deleted.
B. Nothing will occur until a COMMIT command is executed.
C. The table will be removed from the catalog but the data will remain in storage.
D. An error will occur because Delta Lake prevents the deletion of production data.
E. Data will be marked as deleted but still recoverable with Time Travel.

Answer: A

Explanation:
When a table is dropped in Delta Lake, the table is removed from the catalog and the data is deleted. This is because Delta Lake is a transactional storage layer that provides ACID guarantees. When a table is dropped, the transaction log is updated to reflect the deletion of the table and the data is deleted from the underlying storage. References:
* https://docs.databricks.com/delta/quick-start.html#drop-a-table
* https://docs.databricks.com/delta/delta-batch.html#drop-table

NEW QUESTION # 53
A DLT pipeline includes the following streaming tables:
Raw_lot ingest raw device measurement data from a heart rate tracking device.
Bgm_stats incrementally computes user statistics based on BPM measurements from raw_lot.
How can the data engineer configure this pipeline to be able to retain manually deleted or updated records in the raw_iot table while recomputing the downstream table when a pipeline update is run?

A. Set the SkipChangeCommits flag to true raw_lot
B. Set the pipelines, reset, allowed property to false on raw_iot
C. Set the skipChangeCommits flag to true on bpm_stats
D. Set the pipelines, reset, allowed property to false on bpm_stats

Answer: B

Explanation:
In Databricks Lakehouse, to retain manually deleted or updated records in the raw_iot table while recomputing downstream tables when a pipeline update is run, the property pipelines.reset.allowed should be set to false. This property prevents the system from resetting the state of the table, which includes the removal of the history of changes, during a pipeline update. By keeping this property as false, any changes to the raw_iot table, including manual deletes or updates, are retained, and recomputation of downstream tables, such as bpm_stats, can occur with the full history of data changes intact.
Reference:
Databricks documentation on DLT pipelines: https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-overview.html

NEW QUESTION # 54
You are looking to process the data based on two variables, one to check if the department is supply chain or check if process flag is set to True

A. if department == "supply chain" | process == TRUE:
B. if department == "supply chain" | if process == TRUE:
C. if department == "supply chain" or process = TRUE:
D. if department == "supply chain" or process:
E. if department = "supply chain" | process:

Answer: D

NEW QUESTION # 55
Incorporating unit tests into a PySpark application requires upfront attention to the design of your jobs, or a potentially significant refactoring of existing code.
Which statement describes a main benefit that offset this additional effort?

A. Improves the quality of your data
B. Troubleshooting is easier since all steps are isolated and tested individually
C. Ensures that all steps interact correctly to achieve the desired end result
D. Validates a complete use case of your application
E. Yields faster deployment and execution times

Answer: A

NEW QUESTION # 56
A production cluster has 3 executor nodes and uses the same virtual machine type for the driver and executor.
When evaluating the Ganglia Metrics for this cluster, which indicator would signal a bottleneck caused by code executing on the driver?

A. Overall cluster CPU utilization is around 25%
B. Total Disk Space remains constant
C. Network I/O never spikes
D. The five Minute Load Average remains consistent/flat
E. Bytes Received never exceeds 80 million bytes per second

Answer: A

NEW QUESTION # 57
Question-26. There are 5000 different color balls, out of which 1200 are pink color. What is the maximum
likelihood estimate for the proportion of "pink" items in the test set of color balls?

A. .48
B. 24 0
C. 4.8
D. .24
E. 2.4

Answer: D

Explanation:
Explanation
Given no additional information, the MLE for the probability of an item in the test set is exactly its frequency
in the training set. The method of maximum likelihood corresponds to many well-known estimation methods
in statistics. For example, one may be interested in the heights of adult female penguins, but be unable to
measure the height of every single penguin in a population due to cost or time constraints. Assuming that the
heights are normally (Gaussian) distributed with some unknown mean and variance, the mean and variance
can be estimated with MLE while only knowing the heights of some sample of the overall population. MLE
would accomplish this by taking the mean and variance as parameters and finding particular parametric values
that make the observed results the most probable (given the model).
In general, for a fixed set of data and underlying statistical model the method of maximum likelihood selects
the set of values of the model parameters that maximizes the likelihood function. Intuitively, this maximizes
the "agreement" of the selected model with the observed data, and for discrete random variables it indeed
maximizes the probability of the observed data under the resulting distribution. Maximum-likelihood
estimation gives a unified approach to estimation, which is well-defined in the case of the normal distribution
and many other problems. However in some complicated problems, difficulties do occur: in such problems,
maximum-likelihood estimators are unsuitable or do not exist.

NEW QUESTION # 58
A junior data engineer on your team has implemented the following code block.

The viewnew_eventscontains a batch of records with the same schema as theeventsDelta table.
Theevent_idfield serves as a unique key for this table.
When this query is executed, what will happen with new records that have the sameevent_idas an existing record?

A. They are inserted.
B. They are ignored.
C. They are deleted.
D. They are merged.
E. They are updated.

Answer: B

Explanation:
This is the correct answer because it describes what will happen with new records that have the same event_id as an existing record when the query is executed. The query uses the INSERT INTO command to append new records from the view new_events to the table events. However, the INSERT INTO command does not check for duplicate values in the primary key column (event_id) and does not perform any update or delete operations on existing records. Therefore, if there are new records that have the same event_id as an existing record, they will be ignored and not inserted into the table events. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Append data using INSERT INTO" section.
"If none of the WHEN MATCHED conditions evaluate to true for a source and target row pair that matches the merge_condition, then the target row is left unchanged."https://docs.databricks.com/en/sql/language-manual/delta-merge-into.html#:~:text=If%20none%20o

NEW QUESTION # 59
A Databricks job has been configured with 3 tasks, each of which is a Databricks notebook. Task A does not depend on other tasks. Tasks B and C run in parallel, with each having a serial dependency on task A.
If tasks A and B complete successfully but task C fails during a scheduled run, which statement describes the resulting state?

A. Because all tasks are managed as a dependency graph, no changes will be committed to the Lakehouse until ail tasks have successfully been completed.
B. All logic expressed in the notebook associated with tasks A and B will have been successfully completed; some operations in task C may have completed successfully.
C. All logic expressed in the notebook associated with task A will have been successfully completed; tasks B and C will not commit any changes because of stage failure.
D. Unless all tasks complete successfully, no changes will be committed to the Lakehouse; because task C failed, all commits will be rolled back automatically.
E. All logic expressed in the notebook associated with tasks A and B will have been successfully completed; any changes made in task C will be rolled back due to task failure.

Answer: B

Explanation:
Explanation
The query uses the CREATE TABLE USING DELTA syntax to create a Delta Lake table from an existing Parquet file stored in DBFS. The query also uses the LOCATION keyword to specify the path to the Parquet file as /mnt/finance_eda_bucket/tx_sales.parquet. By using the LOCATION keyword, the query creates an external table, which is a table that is stored outside of the default warehouse directory and whose metadata is not managed by Databricks. An external table can be created from an existing directory in a cloud storage system, such as DBFS or S3, that contains data files in a supported format, such as Parquet or CSV.
The resulting state after running the second command is that an external table will be created in the storage container mounted to /mnt/finance_eda_bucket with the new name prod.sales_by_store. The command will not change any data or move any files in the storage container; it will only update the table reference in the metastore and create a new Delta transaction log for the renamed table. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "ALTER TABLE RENAME TO" section; Databricks Documentation, under "Create an external table" section.

NEW QUESTION # 60
You are working on a process to query the table based on batch date, and batch date is an input parameter and expected to change every time the program runs, what is the best way to we can parameterize the query to run without manually changing the batch date?

A. There is no way we can combine python variable and spark code
B. Store the batch date in the spark configuration and use a spark data frame to filter the data based on the spark configuration.
C. Manually edit code every time to change the batch date
D. Create a dynamic view that can calculate the batch date automatically and use the view to query the data
E. Create a notebook parameter for batch date and assign the value to a python variable and use a spark data frame to filter the data based on the python variable

Answer: E

Explanation:
Explanation
The answer is, Create a notebook parameter for batch date and assign the value to a python variable and use a spark data frame to filter the data based on the python variable

NEW QUESTION # 61
A data engineer needs to capture pipeline settings from an existing in the workspace, and use them to create and version a JSON file to create a new pipeline.
Which command should the data engineer enter in a web terminal configured with the Databricks CLI?

A. Use list pipelines to get the specs for all pipelines; get the pipeline spec from the return results parse and use this to create a pipeline
B. Use the get command to capture the settings for the existing pipeline; remove the pipeline_id and rename the pipeline; use this in a create command
C. Use the alone command to create a copy of an existing pipeline; use the get JSON command to get the pipeline definition; save this to git
D. Stop the existing pipeline; use the returned settings in a reset command

Answer: B

Explanation:
The Databricks CLI provides a way to automate interactions with Databricks services. When dealing with pipelines, you can use the databricks pipelines get --pipeline-id command to capture the settings of an existing pipeline in JSON format. This JSON can then be modified by removing the pipeline_id to prevent conflicts and renaming the pipeline to create a new pipeline. The modified JSON file can then be used with the databricks pipelines create command to create a new pipeline with those settings.
Reference:
Databricks Documentation on CLI for Pipelines: Databricks CLI - Pipelines

NEW QUESTION # 62
......

Databricks-Certified-Professional-Data-Engineer 2024 Training With 122 QA's: https://www.freecram.com/Databricks-certification/Databricks-Certified-Professional-Data-Engineer-exam-dumps.html

Databricks-Certified-Professional-Data-Engineer Exam Preparation Material with New Databricks-Certified-Professional-Data-Engineer Dumps Questions.: https://drive.google.com/open?id=1inlhtwx-7fP-0O7anpOxDKBRytKF2BP6

Go To Databricks-Certified-Professional-Data-Engineer Questions

0 Happy Clients

0 Shares

0 Demo Downloads

10 Years in Business

Provide Databricks Databricks-Certified-Professional-Data-Engineer Practice Test Engine for Preparation [Q40-Q62]

Related Articles