Dilorom Abdullah – Medium

Dilorom Abdullah

Pinned

Published in
DataDarvish

Unit Testing in Data Engineering: Python, PySpark, and GitHub CI Workflow

Learn how to implement unit tests for Python and PySpark, automate testing with CI, and boost data pipeline reliability.

Mar 5

Unit Testing in Data Engineering: Python, PySpark, and GitHub CI Workflow

Mar 5

How I Saved Money, Time, and Stress by Optimizing Databricks the Right Way

I recently wrapped up a Databricks cost optimization project where we slashed unnecessary spend, improved workload performance, and freed…

Mar 28

How I Saved Money, Time, and Stress by Optimizing Databricks the Right Way

Mar 28

Published in
DataDarvish

Databricks Cost Optimization: Practical Tips for Performance and Savings

In a recent Databricks cost optimization project I led, I achieved significant results: reducing unnecessary compute spend, improving…

Mar 28

Databricks Cost Optimization: Practical Tips for Performance and Savings

Mar 28

Published in
DataDarvish

Mastering Budget Policies in Databricks

Learn how to set Databricks budget policies to control costs, monitor usage, and prevent overruns — ensuring efficient and optimized spend.

Feb 16

Mastering Budget Policies in Databricks

Feb 16

How to cherry-pick on GitHub

Imagine you have some changes in the dev branch. And you need one or few of the changes go to the main branch, and it can’t wait until the…

Jul 5, 2024

How to cherry-pick on GitHub

Jul 5, 2024

Databricks-SDK for Python

I needed to run Databricks CLI in a workspace via a notebook. But this is what I got:

Apr 27, 2024

Databricks-SDK for Python

Apr 27, 2024

How I automated uploading local files to AWS S3 bucket

I have a lot of files in my local machine that need to be uploaded to AWS S3 bucket. The files are stored in subdirectories within…

Mar 24, 2024

How I automated uploading local files to AWS S3 bucket

Mar 24, 2024

Resolving Access Issues in Databricks DBFS with Unity Catalog

Are you experiencing sudden issues accessing Databricks DBFS? Unsure about the difference between path patterns like /dbfs/ and dbfs:/…

Feb 27, 2024

Resolving Access Issues in Databricks DBFS with Unity Catalog

Feb 27, 2024

Managing System Schemas in Databricks: Enabling, Access Control, and Dashboard Creation

If you have recently enabled Unity Catalog in your account, you should see a system catalog along with other default catalogs included with…

Feb 21, 2024

Managing System Schemas in Databricks: Enabling, Access Control, and Dashboard Creation

Feb 21, 2024

Creating a PySpark DataFrame with Timestamp Column for a Given Range of Dates: Two Methods

This article explains two ways one can write a PySpark DataFrame with timestamp column for a given range of time.

Feb 7, 2023

Creating a PySpark DataFrame with Timestamp Column for a Given Range of Dates: Two Methods

Feb 7, 2023

Dilorom Abdullah

Dilorom Abdullah

Dilorom is a Senior Data Engineer. Follow me on Linkedin: https://www.linkedin.com/in/diloromabdullah/ Follow my blog at: https://medium.com/datadarvish

Following

Help
Status
About
Careers
Press
Blog
Privacy
Rules
Terms
Text to speech