Let’s set up a few prerequisites first.

1. Setting the Power BI Tenant

Note: If you don’t have a Power BI account with administrative rights then use this guide to create one.

Login to your Power BI account with administrative privileges and in Admin Portal make sure you have enabled access to XMLA endpoints for your tenant in Power BI Admin Portal.

Because we will call the XMLA endpoint as a Power BI User with Premium Per User capacity, go to the Premium Per User section and allow read-only access to XMLA endpoints in Dataset workload settings.

For the Power BI Premium per Capacity settings check this video from 6:48 because there it needs to be set too.

Then go to your workspaces and choose the one you will connect to.

Copy the Workspace connection because we will use it later in the C# Console app.

Note: XMLA endpoints are supported only for workspaces hosted on Premium capacity. If you will try to connect to a non-premium workspace you will get this exception:

Microsoft.AnalysisServices.ConnectionException: 
Initial catalog property is required in order to connect to Power BI Pro workspaces.

You can test the proper configuration using the SQL Server Management Studio.

Fill in your Power BI User account credentials (same as for powerbi.com) once prompted. You will see a list of available databases after successful authentication.

2. Downloading Tabular Object Model using C# console application

Because our Power BI XMLA endpoint is ready for connection now, we will test it from a simple C# console application. You can build your own or just download the solution targeting .NET.Core 5.0.

• PowerBIXMLA

Two NuGet packages are required to connect to Analysis Services running under the hood of Power BI Service.

The main method:

using System;
using Microsoft.AnalysisServices;

namespace PowerBIXMLA
{
    class Program
    {
        static void Main(string[] args)
        {

            Server server = new Server();

            // Connect as Power BI User
            string workspaceConnection = "powerbi://api.powerbi.com/v1.0/myorg/YOUR_WORSPACE_NAME";
            string userId = "";
            string password = "";
            string connectStringUser = $"DataSource={workspaceConnection};User ID={userId};Password={password};";


            // Connect as Service Principal
            /*
            string workspaceConnection = "powerbi://api.powerbi.com/v1.0/myorg/YOUR_WORSPACE_NAME";
            string tenantId = "";
            string appId = "";
            string appSecret = "";
            string connectStringServicePrincipal = $"DataSource={workspaceConnection};User ID=app:{appId}@{tenantId};Password={appSecret};";
            */

            // Connect
            server.Connect(connectStringUser);

            // List Databases (Datasets - tabular models) and extract TOM (Tabular Object Model)
            foreach (Database database in server.Databases)
            {
                Console.WriteLine(database.Name);

                string tmslTable = JsonSerializer.SerializeDatabase(database);

                Console.WriteLine(tmslTable);
            }
        }
    }
}

You need to modify the section with Power BI User credentials:

string workspaceConnection = "powerbi://api.powerbi.com/v1.0/myorg/YOUR_WORSPACE_NAME"; 
string userId = ""; 
string password = "";

• worskspaceConnection: this is the Workspace Connection value we have copied above (powerbi://api.powerbi.com/v1.0/myorg/Premium WS)
• userId: Power BI User account name (username@domain.xx). The same you are using to sign in to Power BI Service.
• password: Your Power BI User password.

We can list all server Databases once connected and use the SerializeDatabase() method to get the Tabular Object Model.

Rebuild the application and run int. In case of success you will see:

Because we have called the SerializeDatabase() method the whole Database is serialized to JSON. Database-level properties are marked with blue. The Tabular Object Model is marked with green.

If you would like to serialize just the Tabular Object Model you can call SerializaeObject() method inside the Tabular namespace.

Microsoft.AnalysisServices.Tabular.JsonSerializer.SerializeObject(database.Model);

To do more with the main Tabular Object Model just reference the Tabular namespace and process the object like any other strong-typed objects.

using Microsoft.AnalysisServices.Tabular;
...

Model model = database.Model;

foreach (Table t in model.Tables)
{
    Console.WriteLine(t.Name);
}

Finally, it’s important to understand that datasets uploaded to Power BI Service from Power BI *.pbix files are in fact hosted as standard Analysis Service Tabular Models in the Power BI Service. That means that you can use the same approach when connecting to the on-prem Tabular Analysis Services instance. Only the authentication process will be different. The XMLA itself is fully compatible.

References

• Dataset connectivity with the XMLA endpoint
• Tabular Object Model (TOM)
• Programing Datasets using the Tabular Object Model

In this step-by-step tutorial, I will do the following:

1. Configure an Azure Application.
2. Test the Application for successful connection to Azure AD.
3. Configure Power BI Tenant to allow Service Principals to read the new Admin API.
4. Connect to the Power BI tenant and print a list of Workspaces to check that everything is configured properly.

Prerequisites:

• Active Azure Portal subscription. You can create one for free.
• Power BI Service tenant and account with administrative privileges. You can use this simple tutorial to configure it.
• Downloaded and running Power BI API Connector that we will be used for testing of the whole solution.

Let’s start the challenge!

1. Configure an Azure Application

In the main Azure Portal menu navigate to the Azure Active Directory item and select App registration.

Start New registration of an Application.

On the main registration screen please choose the application name. In our case, we will call the application SPW API Reader APP. You can keep default values for the rest of the form.

Press the Register button and wait for the deployment to finish. The Overview page will show us two important values identifying our Application which will be used later.

• Application (client ID) is the unique identification of the Application.
• Directory (tenant) ID is the unique identification of our tenant (organization).

Both these IDs are needed for Azure Active Directory access via the OAuth protocol. Because they are like “user” names we also must create a password. Navigate to the Certificates & secrets in the menu and press the New client secret button.

Add a secret name and choose the expiration that fits your needs. Press Add button and a new Client Secret will be created.

You must copy the Value of the secret now! On the next page refresh, the Copy icon will disappear and the secret value can’t be copied again. You must delete the secret then and create a new one.

Because the Power BI Admin API allows only the Service Principal to be an Azure AD Group, we will create a new Group and assign our Application to it. Go back to Azure Active Directory and navigate to Groups.

Press the New group button.

Create a new Group. Choose Security Group type and some meaningful Group name that you can later search for it easy.

Once the Group is created navigate to it and click the Group name.

Press the Add members button.

In the search box try to search for the name of our Application and then select the proper item filtered that it will move to the Selected items section.

Double-check that our Application is a member of the Group now.

You have successfully:

• created new Azure Application
• obtained unique Application ID, Tenant ID, and Application Secret values
• created a new Azure Active Directory group and assigned the Application to be a member of it

2. Test the Application for successful connection to Azure AD.

It’s time to test that the Application can successfully connect to Azure Active Directory and obtain a valid security token. This is the must-have step before we can try to access the Power BI Admin API itself.

To do this please download the Power BI API Connector utility and check the post for details on how it works. Try to connect and obtain the security token. You can continue this tutorial once done.

Note: You can also try to test the Power BI Admin API connection using the utility but it will fail with Unauthorized because our Application doesn’t have access to the Power BI Admin API now.

3. Configure Power BI Tenant to allow Service Principals to read the new Admin API.

Please login to the Power BI Service with an account with administrative privileges. If you will use an account without tenant-level administrative rights the Tenant settings section in the Admin portal is hidden at all. You can create a new Power BI Administrator or assign an existing user to the Power BI administrator role following this post.

Once done navigate in the Power BI menu to the Admin portal section and then select Tenant settings.

Scroll down and find the Admin API settings section. Press the Enabled button and then choose the Specific security groups option. Search for the Azure Active Directory Group we have created above where our Application is a member of it. Press the Apply button.

Security settings will be applied soon. It can take up to 15 minutes but usually, it takes not more than 2 minutes for the API is accessible.

That’s all from the configuration point of view.

4. Connect to the Power BI tenant and print a list of Workspaces.

It’s time to test that our Application can really access the Power BI Admin API now. We will use the Power BI API Connector utility again and print a list of Tenant available Workspaces. To do this please create at least one Workspace if you have a brand new Power BI account because the API method GetGroupsAsAdmin() used for testing doesn’t list the default My workspace item.

Navigate to Workspaces and press the Create a workspace button.

Type Workspace name and Save it.

New Workspace should be visible in the list of available Workspaces.

Run the Power BI API Connector utility again and after you will allow it to run the Power BI Admin API test it should connect successfully to your Power BI tenant, call the API and print a list of all available Workspace.

As you can see our testing Workspace name is listed. There is also “Samples WS” Workspace listed to demonstrate that it’s really admin access listing all tenant Workspaces as Service Principal and not only Workspaces of the given Power BI account as if the API is accessed in the authenticated user access mode.

That’s it! You are done and you can continue building your own application to do something great with all the data available via the Power BI Admin API.

References

• Power BI Admin REST API
• https://docs.microsoft.com/en-us/power-bi/admin/read-only-apis-service-principal-authentication
• https://docs.microsoft.com/en-us/power-bi/developer/automation/overview-of-power-bi-rest-api
• https://powerbi.microsoft.com/en-us/blog/announcing-new-admin-apis-and-service-principal-authentication-to-make-for-better-tenant-metadata-scanning
• https://docs.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal

Application is created in .NET.Core 5.0 and you can download it as:

• PowerBIApiConnector_Executable – executable files in *.zip package for direct use
• PowerBIApiConnector_Solution – Visual Studio 2019 Project

How to use it?

Extract the executable package files to a directory. You have two options how to launch the application:

A) Setup parameters at the application launch:

PowerBIApiConnector.exe {Application (client) ID} {Directory (tenant) ID} {Client Secret}

B) Execute it without parameters. Then you will be prompted for input values:

Input parameters are:

• Application (client) ID: This is an Azure Application ID
• Directory (tenant) ID: Your organization (tenant) ID
• Client Secret: Application secret value
  

You can get proper values for these parameters in Azure Portal: Navigate to your Application in Azure Active Directory and copy the first two parameters from the Application Overview page:

For the Client Secret navigate as in the image below and copy the Value or create a new secret:

When you launch the application and authentication to Azure AD will succeed, you will see an authentication token printed to the console window:

Same time you will be prompted if you would like to test the Power BI Admin Read-Only API access. You can skip this step in case you would like to test the Azure AD authentication only.

If you will input “Y” the GetGroupsAsAdmin() API call will be executed and when the application has properly configured access to the Power BI Admin API available tenant workspaces will be listed by name:

Solution content

Two dependency packages are required (will be automatically downloaded via NuGet packages manager):

Main method code:

using Microsoft.Identity.Client;
using Microsoft.PowerBI.Api;
using Microsoft.Rest;
using System.Linq;
using System;

namespace PowerBIApiConnector
{
    class Program
    {

        static PowerBIClient powerBIClient = null;
        static string accesToken = null;
        static IConfidentialClientApplication app;

        // Authentication variables
        private static string clientId;
        private static string tenantId;
        private static string clientSecret;

        // Power BI API paths
        private const string resource = "https://analysis.windows.net/powerbi/api";
        private const string ApiUrl = "https://api.powerbi.com";

        private static string authority;

        public static void Main(string[] args)
        {
            // Capture users input via prompt
            if (args.Count() == 0)
            {
                Console.WriteLine("Application (client) ID:");
                clientId = Console.ReadLine();

                Console.WriteLine("Directory (tenant) ID:");
                tenantId = Console.ReadLine();

                Console.WriteLine("Client Secret:");
                clientSecret = Console.ReadLine();
            }
            // Capture users input as launch parameters
            else if (args.Count() == 3)
            {
                clientId = args[0];
                tenantId = args[1];
                clientSecret = args[2];

                Console.WriteLine("------------------------");
                Console.WriteLine($"Application (client) ID: {clientId}");
                Console.WriteLine($"Directory (tenant) ID: {tenantId}");
                Console.WriteLine($"Client Secret: {clientSecret}");
                Console.WriteLine("------------------------");
            }
            else
            {
                Console.WriteLine("Mssing valid input parameters! Type: {Application (client) ID} {Directory (tenant) ID} {Client Secret} ");
            }

            // Set main API path for authentication
            authority = $"https://login.microsoftonline.com/{tenantId}";

            // Create authorization app
            try
            {
                app = ConfidentialClientApplicationBuilder
                    .Create(clientId)
                    .WithClientSecret(clientSecret)
                    .WithAuthority(new Uri(authority))
                    .Build();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                Environment.Exit(1);
            }

            string[] scopes = new string[] { $"{resource}/.default" };
            AuthenticationResult result = null;

            // Authenticate and obtain permissions scope
            try
            {
                result = app.AcquireTokenForClient(scopes).ExecuteAsync().Result;
            }
            catch (MsalUiRequiredException ex)
            {
                // Insufficient permissions.
                Console.WriteLine(ex.Message);
                Environment.Exit(1);
            }
            catch (MsalServiceException ex) when (ex.Message.Contains("AADSTS70011"))
            {
                // Invalid application scope.
                Console.WriteLine(ex.Message);
                Environment.Exit(1);
            }

            // Print token
            Console.WriteLine($"App-Only Access Token:\n{result.AccessToken}\n");
            
            accesToken = result.AccessToken;
            var tokenCredentials = new TokenCredentials(result.AccessToken, "Bearer");
            
            // Test PowerBI Admin API
            Console.WriteLine("Test Power BI API (List workspaces as service principal via Read-Only Admin Api?) Type Y/N:");

            if ("y" == Console.ReadLine().ToLower())
            {
                Console.WriteLine("------------------------");
                Console.WriteLine("Connecting to Power BI API...");

                try
                {
                    // Create PowerBIClient instance
                    powerBIClient = new PowerBIClient(new Uri(ApiUrl), tokenCredentials);

                    // List workspace names with the Admin only method (GetGroupsAsAdmin)
                    var workspaceNames = powerBIClient.Groups.GetGroupsAsAdmin(100, filter: "type eq 'Workspace'").Value.Select(x => x.Name);

                    //Print output
                    Console.WriteLine("Workspaces found:");

                    foreach (string workspaceName in workspaceNames)
                    {
                        Console.WriteLine("  => " + workspaceName);
                    }
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.Message);
                    Environment.Exit(1);
                }
            }

            Console.WriteLine("Done!");

        }
    }
}

References

For additional references, I will recommend this excellent video from Marco Russo and the official Microsoft guide.

1. Create a new user in Azure Active Directory

Log in to the Azure Portal, navigate to the Azure Active Directory item and click on Users.

Press the + New user button.

Create a new user. For this tutorial, we will call him Power BI Admin and powerbiadmin@sqlpowered.com account name (email address) will be used.
Don’t forget to click the Show Password box and note the password. We will need it in the next step!
When completed, press the create button. You should see the newly created user in the list of Azure AD users.

2. Sign in to the Power BI

Go to the powerbi.microsoft.com and sign it with the credentials created above (user name (email) and password).

Because the password set in Azure AD is only temporal and must be changed the first time the user is log in to any Microsoft service, insert the current password in the Current password box and then insert a different new password which should be used by the user permanently and sign in.

Now the password was changed and you will be notified that an account with the same name already exists and you need to sign in again with the new password set in the previous step. This is just a not very clear request to sign in again with the new password.

On the next screen you can opt-out from some Microsoft marketing materials. Set is base on your preferences and press the Start button.

There is an option to invite other people to the Power BI Service. Just Skip it for now.

You should be successfully logged in to the Power BI Service Portal now.

3. Review the Admin portal setting

Press the circle in the top right corner and navigate to the Admin portal section. You can ignore the Power BI Pro offer for now. It doesn’t have any impact on the Power BI Administrator setup process.

What you will see is just this limited list. There is zero option to change anything on the tenant level.

To allow more settings available here we must leave the Power BI Service portal and move again to Azure Portal to configure administrator access.

4. Assign user to the Power BI administrator role

Go back to the Azure Portal, navigate to the Azure Active Directory, and list of users please select the one created above.

In the user’s detail click on Assigned roles, then + Add assignments and the filter for the Power BI administrator role. Select it and press the Add button.

Our user is now assigned to the Power BI administrator role.

5. Review the Admin portal section in the Power BI Portal

When you will go back to the Power BI Portal and refresh the page you may see the full list of available administrative options and configure a lot of setting for the organizational Power BI tenant.
Note: If the simple page refresh doesn’t work you must sign out and sign in again. This will refresh your permissions list.

Alternative path: Office 365

Users can be assigned to the Power B administrator role also in Office 365 admin center. Sign in to the Office 365 Admin Portal, navigate to Users => Active user and select the user you will promote to Power Bi administrator. In the Roles section click the Manage roles link and then select the Power BI administrator role from the list and assign it to the user.

Let’s go to Microsoft Forms and build such a simple form as in the picture:

Go to Power Automate and search for “Store Microsoft Forms responses in a SQL Server Table” template:

Click on the template icon and on the next screen there is a simple description of the task. Below are two + buttons to set connections to your Microsoft Form you will receive responses from and to SQL Server database and table where received responses should be persisted.

Setup the Microsoft Forms connection first. It’s easy: just click the + button and it will connect you to the Microsoft forms store which belongs to your account you have signed in. The connection will be marked with a green OK symbol.

Then press the + for the SQL Server connection and fill in the form. In case it’s Azure SQL Database the connection will fail for the first time saying that the client with the displayed IP address can not connect to the database. This means that the connection was blocked by the Azure SQL Server firewall and you should take the address and create a new firewall rule for it.

Once both connections are set properly you can continue in the wizard.

Prepare the table where form responses will be stored. First there columns in the table are special columns for the ID of the responses, who submitted the response, and when. The remaining columns are matching to form questions one by one. Copy the CREATE table code and run it in the target database you have connected above.

CREATE TABLE [dbo].[SampleFormAnswers](
	[ResponseId] [INT] NOT NULL,
	[SubmittedBy] [NVARCHAR](200) NULL,
	[SubmittedDtm] [NVARCHAR](50) NULL,
	[Question1_Choice] [NVARCHAR](500) NULL,
	[Question2_Text] [NVARCHAR](500) NULL,
	[Question3_Rating] [NVARCHAR](500) NULL
)
GO

Go back to Power Automate and continue the wizard. It’s quite intuitive. Select your sample form in the first section. “Get response details” section keep as it was prefilled automatically. The interesting part is coming in the “Insert row (V2)” task where you will configure the mapping between form response values and database table columns. Select the Server, database, and the table you have created in the previous step. Power Automate will automatically list all table columns and display them in the form.

Connect all columns to response properties as in the picture. If you will click inside the column box search box will appear on the right side of it and you can search for the proper form fields in it:

Once completed, press the Save button in the wizzard and that’s it.

It’s time to test if it works. Navigate back to your form and press the Preview button in the top panel. This will allow you to fill in the form. Fill it as in the picture to have some interesting variants of data in the table later:

Submit the form and you should see the approval message:

Execute SQL query against your database to see if the form response arrived there:

SELECT * FROM [dbo].[SampleFormAnswers]
GO

In case you have properly configured everything the response looks like as in the picture (with a different ID maybe). You can see more execution details if you will click on the flow in the Power Automate administration:

In case of the failure you will see the failed execution in the “28-day run history” section and you can observe error details. Errors are enough descriptive to fix them quickly. What is really amazing is the Duration: form response arrived in the database nearly realtime.

Few points to consider:

• See the Power Automate pricing. It’s comparable to Zappier or other services but with the advantage that there is no such 15 minutes synchronization limit.
• You can use STRING_AGG() function for multivalue answers parsing.
• It’s a question if the Power Automate IP address you need to grant access in the Aure SQL Server firewall won’t change in time? I’m not sure about that.
• SubmittedDtm column is easier to have a string. When it’s DATETIME the flow will fail with a casting error.
• When you will do some changes in the table it may be required to drop the last task in the flow (insert row) and add it again to refresh the columns list.

One option we have is to use nic-ipconfig CLI. But I will show you another option which works without scripting. We will exchange NIC (Network Interfaces) between VMs and this will also switch IP addresses associated with them.

We will play with two virtual machines: int-worker and int-worker2. Our goal is to take the current IP address of the int-worker VM and assign it to int-worker2 and the opposite.

1. Check our VMs status first

Both VMs are running now.

Go to Networking and check settings for both VMs.

We can see that:

– int-worker VM is attached to NIC dev-worker399 with IP address starting 52.166.
– int-worker2 VM is attached to NIC dev-worker2167 with IP address starting 40.91.

2. We must stop both VMs before we can detach/attach NIC from it. VMs must have status Stopped (deallocated).

3. Because we can’t have VM without at least one NIC attached to it we must create another new NIC as spare. Search for “Network interface” in Azure Portal and add a new one. It must be in the same resource group and virtual network as our VMs.

We have to call this NIC “dummy-nic” that we can search for it easily and is clear to everyone it was created just for maintenance purposes.

4. Go to the Networking configuration section of the int-worker VM and attach the newly created dummy-nic to it.

5. Because the VM has 2 NICs assigned now, one of them can be detached. Detach NIC dev-worker399.

6. Go to the same setting screen of int-worker2 VM and attach the dev-worker399 NIC to it. It should appear in the selector because the NIC is currently not assigned to any VM and can be used for attaching it to a new one.

7. Next, you detach the original NIC from int-worker2 VM and continue the same path to assign it to int-worker VM and detach dummy-nic from it.

8. Check the new IP address of VMs. As you can see int-worker2 VM has the IP address assigned which was originally assigned to int-worker VM.

That’s it we are done. If you would like to know more about NICs, you can follow this article. You can also use CLI to script that for automation.

Creating a masking rule

1. Navigate to your database in Azure Portal and choose the Dynamic Data Masking item from the Security menu:

2. Click Add mask button to create a new masking rule. I’m using AdwentureWorkLT as a sample database for this tutorial. There is table SalesLT.Customer table with column EmailAddress which I would like to mask to hide email addresses from low privileged users.

There are few predefined masking functions and one of them is created exactly for simple masking of email addresses. Please choose it in the field Masking field format:

3. Save the rule and you will see it listed in the Masking rules overview:

We have successfully created a masking rule to hide email addresses and we will test it now.

Testing masking rule

Connect to Azure SQL Database for which we have created masking rule and run following script from SSMS, Azure Data Studio, or Query Editor in Azure Portal:

-- run as admin user
SELECT CustomerID, EmailAddress FROM SalesLT.Customer
GO

-- create user with low permissions
CREATE USER LowPermUser WITHOUT LOGIN
GO

-- grant SELECT to LowPermUser
GRANT SELECT ON SalesLT.Customer TO LowPermUser
GO

-- execute as LowPermUser
EXECUTE AS USER = 'LowPermUser'

SELECT CustomerID, EmailAddress FROM SalesLT.Customer
GO

Result set 1:

Result set 2:

The query will return two result sets you can see above. How that works?

1. The first result set is coming from a simple select to source table which we have executed against the database under the admin account which has by default all data unmasked.

2. We have created low privileged database user LowPermUser and granted only SELECT permission on our sample table.

3. We switched execution context to the LowPermUser and executed again SELECT query which returned Result set 2 containing email addresses masked with the predefined Email masking function.

That’s it. Easy.

if you would like to review existing masking rules configured for the database then it can be done with this simple query:

SELECT 
	s.name SchemaName, t.name TableName, c.name ColumnName, 
	c.is_masked, c.masking_function
FROM sys.masked_columns c
	INNER JOIN sys.tables t ON t.object_id = c.object_id 
	INNER JOIN sys.schemas s ON s.schema_id = t.schema_id
GO

As a next step, I will recommend reviewing some security concerns related to Dynamic Data Masking.

The only one prerequisite you will need is configured Azure SQL Server. There is no need to have any database created because TDE is configured at server level and at database level you can only switch it on or off.

To use our own encryption keys we need to have Azure Key Vault preconfigured. If you haven’t done it till now, follow a few simple steps listed below.

Configure Azure Key Vault

Open Azure Portal and search for Key Vault item in available services. Click on the Create button and complete the form using your own Resource group name:

Press the Review + Create button and wait a few seconds before the deployment is completed.

Create encryption key

When the Key vault was successfully created we can create a new key that will be later used by for encryption by TDE service. Select for Keys item in the Settings section and Genera new one:

Name of the key is up to you. For other items, we will use default settings.

Press the Create button and verify that the creation of the key was completed:

Configure Transparent data encryption

Navigate to your SQL Server instance and in the Security group and click on Transparent data encryption item. As you can see using service-managed keys is the default option.

Because we will use for the encryption our own keys, change the setting to Yes and when prompted pickup you Azure Key Vault and select proper encryption Key:

Finally, press the Save button.

If saving your new configuration has failed with following error, continue with next step:

Using Azure Key Vault together with TDE requires the Key Vault to be properly configured. Soft delete functionality needs to be enabled to protect you against full data loss in case of accidental deletion of encryption keys. There is no such configuration option in Azure Portal and we will need to open Cloud Shell and configure the required setting using Powershell:

($resource = Get-AzResource -ResourceId (Get-AzKeyVault -VaultName "ContosoVault").ResourceId).Properties | Add-Member -MemberType "NoteProperty" -Name "enableSoftDelete" -Value "true"

Set-AzResource -resourceid $resource.ResourceId -Properties $resource.Properties

Console output:

You can navigate back to Key Vault detail to check is the configuration succeed:

Let’s go back to TDE configuration screen and try again:

The operation completed successfully this time and our TDE is using our own keys stored in Azure Key Vault instead of service-managed keys.

Azure Data Lake consists of two important parts working together:

1. Azure Data Lake Storage – massively scalable, secure data lake functionality built on Azure Blob Storage.
2. Azure Data Lake Analytics – main data lake analytics service using Azure Data Lake Storage to read data from and process them

Because Azure Data Lake Analytics service depends on Azure Data Lake Storage, we will first create that one and then continue with Data Lake Analytics.

Creating Azure Data Lake Storage

1. Open Azure Portal and search for Azure Data Lake Storage. You can select Gen1. Gen2 will be covered in another post.

2. Click Add button to initiate the New Data Lake Storage Gen 1 window:

3. Fill all mandatory fields base on your preferred values.

You should setup:

• Name which will form later also URL of your storage. Follow naming rules.
• If you have more Subscriptions select the proper one.
• Resource group means where your Data Lake Storage belongs. If you don’t have one, create new one.
• The Location should be selected geographically closer to your current location.

Keep other settings as they are and press the Create button.

4. You should see a notification that your request was registered and is being processed. Once it’s completed then press the Refresh button and you should see your new Data Lake Storage Gen1 object in the list:

With this step, we have completed the preparation of the storage account and we can continue with the creation of the Data Lake Analytics service.

Configure Azure Data Lake Analytics

1. Go back to Portal and search for Data Lake Analytics.

2. Press the Add button to create a new Data Lake Analytics instance.

3. Fill in all required fields.

You can do it the same way as you did for storage configuration.

Important is to select your previously created Data Lake Storage account to be used with this instance of Data Lake Analytics.

After the form is completed and validated press the Create button. You should see a notification about processing your request.

4. Once processing is completed press the Refresh button and you should see your new instance of Data Lake Analytics services listed below.

With all these steps we have successfully completed the first part of our tutorial: Azure Data Lake Storage account and Azure Data Lake Analytics service were created and are ready to be used.

Using Azure Data Lake

In the second part of this tutorial, we will test the basic scenario of how to use configured services to process data sample data.

First of all open this link to download a sample CSV file of 500000 sales records.zip from cool page http://eforexcel.com/wp

Unzip the file locally and continue with the following steps:

Upload sample data to Azure Data Lake Storage

1. Navigate to the All resources icon in the left panel in Portal and from the list click on the Data Lake Storage Gen 1 resource we have created before. After it will open there is a Data explorer button you should click on.

2. Create a new folder called Samples where we will upload our sample data file.

3. After the folder is ready double click it to open and then press the Upload button and select the file 500000 Sales Records.csv you have downloaded and unzipped before.

Uploading the file can take some time based on your connection speed. You can observe the progress and when the task is completed notification will popup.

When the file was successfully uploaded navigate back to the Samples folder and double click file name.

It will open the file and previews its content. But at a first look, the text isn’t divided into columns. You can fix it in a minute: just click the Format button and select semicolon as column divider and mark the first row as containing column names. Azure will format file content as you can see in the picture:

At this stage, we have successfully prepared our sample data in Azure Data Lake Storage and is time to do some basic data analysis over the data set.

Create a new Data Lake Analytics job and query our data with U-SQL

1. Navigate to the All resources menu item in the left panel of Azure Portal and select the Data Lake Analytics services we have created in previous steps. Once the window with its detail popup, press the button New job to create your first data analytics job.

In the box below is a sample U-SQL script. You only need to copy/paste it to New job window like in the following picture. An explanation of the script can be found there. In a few words, we will load the content of the file with a text extractor, summarize the total number of Units Sold by Region and save the output to another text file in the same folder.

@sales = 
    EXTRACT Region string, 
            Country string,
            [Item Type] string,
            [Sales Channel] string,
            [Order Priority] string,
            [Order Date] DateTime,
            [Order ID] int,
            [Ship Date] DateTime,
            [Units Sold] int,
            [Unit Price] decimal,
            [Unit Cost] decimal,
            [Total Revenue] decimal,
            [Total Cost] decimal,
            [Total Profit] decimal
    FROM "/Samples/500000%20Sales%20Records.csv"
    USING Extractors.Text(delimiter: ';', skipFirstNRows: 1);

@agg = 
  SELECT
    Region,
    SUM([Units Sold]) AS UnitsSold
  FROM @sales
  GROUP BY Region;

OUTPUT @agg
    TO "/Samples/UnitsSoldByRegion_output.csv"
    USING Outputters.Csv(outputHeader : true);

2. Paste the script and enter the name of the job: UnitsSoldByRegion. Press Submit button and the job will be created and immediately queued for processing.

There are three statuses of job processing: Preparation, Queued, and Running. You can observe the progress graphically in the right window where is processing plan available:

After the job has been completed you can click on the last box in the processing steps chart. This will navigate you to the results file.

Press the Data button and observe file content:

This is exactly what we were trying to achieve: Total number of Units Sold by Region. Columns are named exactly as we have specified in the U-SQL script.

If you will navigate back to Azure Data Lake Storage you will see our output file there.

That’s all. You have successfully configured your first Azure Data Lake Storage, uploaded some data into it, aggregated values using Azure Data Lake U-SQL language, and stored results in a text file.

There are basic steps listed to accomplish this task:

1. Create a new SQL Server database that we will later backup to Azure Blob Storage. Name the database  CompanyData.

2. Then open the Backup data dialog as usual. Instead of Disk choose URL option in Back up to section.

3. In the next dialog press the button New container because we haven’t created it before in Storage Account. If you already have one then you can list for it.

4. Next press Sign In in the next screen and sing in into your Azure subscription where your Storage Account was created.

5. After you have signed in choose the right subscription, Storage Account, Blob Container, and press the button Create Credential. This will create the Shared Acess Signature (SAS).

6. When you have pressed OK in the previous step you are back in Select Backup Destination screen where you can see your Storage Account container successfully attached. Press the OK button.

7. You can see that everything is ready for backup now. Press OK and the backup process will start like in backup to Disk scenario.

The Error occurred icon in the picture will occur if you will try to press the Script button. Hopefully, this will be corrected soon in the nearest SSMS release. For now, we will capture the background query via Profiler:

BACKUP DATABASE [CompanyData] TO  
URL = N'https://sqlpoweredbackups.blob.core.windows.net/sqlbackups/companydata_backup_2019_05_25_081859.bak' 
WITH NOFORMAT, NOINIT,
NAME = N'CompanyData-Full Database Backup', 
NOSKIP, NOREWIND, NOUNLOAD,  
STATS = 10
GO

8. You can use Azure Storage Explorer to check that our backup file exists in the target  Azure Blob Storage container.

That’s it. Nothing complicated and very similar do Disk backup type.

You would like to where are your authentication credentials stored? In the BACKUP statement itself we have captured with Profiler there is no such reference. Your credentials were used to create the Shared Access Signature key that is located in the Credentials folder under Security parent for later use.