The following article depicts performance optimization on Azure Data Lake.It is Assumed that people reading this have a general idea on ADL, U-SQL and general querying.Before we move on let’s understand key terms in determining a query performance: Analytic UnitStages in a jobJob Read More …
Category: Azure
Using Azure Data Lake to copy data from CSV file to a SQL database
Great, but what is Azure Data Lake? Azure Data Lake includes all the capabilities required to make it easy for developers, data scientists, and analysts to store data of any size, shape, and speed, and do all types of processing and analytics Read More …
U-SQL (Intro)
U-SQL is a language that unifies the benefits of SQL with the expressive power of your own code to process all data at any scale. U-SQL’s scalable distributed query capability enables you to efficiently analyze data in the store and Read More …
Including File Properties and Metadata in a U-SQL Script
U-SQL adds support for computed file property columns on EXTRACT statement. Sometimes customers would like to get information about the files that they process, such as the full URI path or information about size, creation or modification dates. Likewise customers would Read More …
U-SQL Table
Azure Data Lake Analytics (U-SQL) originates from the world of Big Data, in which data is processed in a scale-out manner by using multiple nodes. These nodes can access the data in several formats, from flat files to U-SQL tables. Read More …
Data Factory Ingestion Framework: Part 1 – Schema Loader
Use Case A business wants to utilize cloud technology to enable data science and augment data warehousing by staging and prepping data in a data lake. There are multiple different systems we want to pull from, both in terms of Read More …
T-SQL TO U-SQL DATA TYPE CONVERSION
When working with code generated solutions we often need to convert datasets from SQL Server (T-SQL) data types to Azure Data Lake Analytics (U-SQL) data types. As you probably know U-SQL has a hybrid syntax of T-SQL and C# which Read More …
Part 9: Extending U-SQL
There are 5 kinds of User-Defined entities in U-SQL User-Defined Functions (UDFs)User-Defined Types (UDTs)User-Defined Aggregators (UDAggs)User-defined Operators (UDOs)User-Defined Appliers All of them are defined by .NET code. C# is not required. Any .NET language will work. User-Defined Functions User defined functions are normal static methods on a .NET Class. Read More …
Part 8: Set operations and Joins
Set operations are a way of merging rowsets together based on set theoretic operations such as union (UNION). intersection (INTERSECT), complement (EXCEPT). Sample data Let’s define two RowSets: @a and @b. Notice that both RowSets have duplicate rows. UNION UNION combines two rowsets. UNION Read More …
Part 7: Window Functions
Window functions were introduced to the ISO/ANSI SQL Standard in 2003. U-SQL adopts a subset of window functions as defined by the ANSI SQL Standard. Window functions are used to do computation within sets of rows called windows. Windows are defined Read More …