![]() Number of RowsĪd-hoc schema: You should be able to read a maximum of 5 million rows (~5.6 GB data on disk) of non-XDM (ad-hoc) data in less than 14 minutes. Adding additional rows may result in errors. XDM ExperienceEvent schema: You should be able to read a maximum of 2 million rows (~6.1 GB data on disk) of XDM data in less than 22 minutes. For more information on the efficiency of each mode, see the PySpark or Scala data limit tables below. For PySpark and Scala notebooks, batch mode should be used when 5 million rows of data or more is being read.Interactive is made for fast results whereas batch mode is for large datasets. ![]() When reading datasets with PySpark and Scala notebooks, you have the option to use interactive mode or batch mode to read the dataset. When to use batch mode vs interactive mode This data also varied in size starting from one thousand (1K) rows ranging up-to one billion (1B) rows. The ad-hoc schema data was pre-processed using Query Service Create Table as Select (CTAS). Note that for the PySpark and Spark metrics, a date span of 10 days was used for the XDM data. The ExperienceEvent schema data used varied in size starting from one thousand (1K) rows ranging up-to one billion (1B) rows. For PySpark and Scala, a databricks cluster configured at 64GB RAM, 8 cores, 2 DBU with a maximum of 4 workers was used for the benchmarks outlined below. ![]() The following information defines the max amount of data that can be read, what type of data was used, and the estimated timeframe reading the data takes.įor Python and R, a notebook server configured at 40GB RAM was used for the benchmarks. Try switching to “batch” mode to resolve this error. For PySpark and Scala notebooks if you are receiving an error with the reason “Remote RPC client disassociated.” This typically means the driver or an executor is running out of memory. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |