Let’s set up a few prerequisites first.

1. Setting the Power BI Tenant

Note: If you don’t have a Power BI account with administrative rights then use this guide to create one.

Login to your Power BI account with administrative privileges and in Admin Portal make sure you have enabled access to XMLA endpoints for your tenant in Power BI Admin Portal.

Because we will call the XMLA endpoint as a Power BI User with Premium Per User capacity, go to the Premium Per User section and allow read-only access to XMLA endpoints in Dataset workload settings.

For the Power BI Premium per Capacity settings check this video from 6:48 because there it needs to be set too.

Then go to your workspaces and choose the one you will connect to.

Copy the Workspace connection because we will use it later in the C# Console app.

Note: XMLA endpoints are supported only for workspaces hosted on Premium capacity. If you will try to connect to a non-premium workspace you will get this exception:

Microsoft.AnalysisServices.ConnectionException: 
Initial catalog property is required in order to connect to Power BI Pro workspaces.

You can test the proper configuration using the SQL Server Management Studio.

Fill in your Power BI User account credentials (same as for powerbi.com) once prompted. You will see a list of available databases after successful authentication.

2. Downloading Tabular Object Model using C# console application

Because our Power BI XMLA endpoint is ready for connection now, we will test it from a simple C# console application. You can build your own or just download the solution targeting .NET.Core 5.0.

• PowerBIXMLA

Two NuGet packages are required to connect to Analysis Services running under the hood of Power BI Service.

The main method:

using System;
using Microsoft.AnalysisServices;

namespace PowerBIXMLA
{
    class Program
    {
        static void Main(string[] args)
        {

            Server server = new Server();

            // Connect as Power BI User
            string workspaceConnection = "powerbi://api.powerbi.com/v1.0/myorg/YOUR_WORSPACE_NAME";
            string userId = "";
            string password = "";
            string connectStringUser = $"DataSource={workspaceConnection};User ID={userId};Password={password};";


            // Connect as Service Principal
            /*
            string workspaceConnection = "powerbi://api.powerbi.com/v1.0/myorg/YOUR_WORSPACE_NAME";
            string tenantId = "";
            string appId = "";
            string appSecret = "";
            string connectStringServicePrincipal = $"DataSource={workspaceConnection};User ID=app:{appId}@{tenantId};Password={appSecret};";
            */

            // Connect
            server.Connect(connectStringUser);

            // List Databases (Datasets - tabular models) and extract TOM (Tabular Object Model)
            foreach (Database database in server.Databases)
            {
                Console.WriteLine(database.Name);

                string tmslTable = JsonSerializer.SerializeDatabase(database);

                Console.WriteLine(tmslTable);
            }
        }
    }
}

You need to modify the section with Power BI User credentials:

string workspaceConnection = "powerbi://api.powerbi.com/v1.0/myorg/YOUR_WORSPACE_NAME"; 
string userId = ""; 
string password = "";

• worskspaceConnection: this is the Workspace Connection value we have copied above (powerbi://api.powerbi.com/v1.0/myorg/Premium WS)
• userId: Power BI User account name (username@domain.xx). The same you are using to sign in to Power BI Service.
• password: Your Power BI User password.

We can list all server Databases once connected and use the SerializeDatabase() method to get the Tabular Object Model.

Rebuild the application and run int. In case of success you will see:

Because we have called the SerializeDatabase() method the whole Database is serialized to JSON. Database-level properties are marked with blue. The Tabular Object Model is marked with green.

If you would like to serialize just the Tabular Object Model you can call SerializaeObject() method inside the Tabular namespace.

Microsoft.AnalysisServices.Tabular.JsonSerializer.SerializeObject(database.Model);

To do more with the main Tabular Object Model just reference the Tabular namespace and process the object like any other strong-typed objects.

using Microsoft.AnalysisServices.Tabular;
...

Model model = database.Model;

foreach (Table t in model.Tables)
{
    Console.WriteLine(t.Name);
}

Finally, it’s important to understand that datasets uploaded to Power BI Service from Power BI *.pbix files are in fact hosted as standard Analysis Service Tabular Models in the Power BI Service. That means that you can use the same approach when connecting to the on-prem Tabular Analysis Services instance. Only the authentication process will be different. The XMLA itself is fully compatible.

References

• Dataset connectivity with the XMLA endpoint
• Tabular Object Model (TOM)
• Programing Datasets using the Tabular Object Model

In this step-by-step tutorial, I will do the following:

1. Configure an Azure Application.
2. Test the Application for successful connection to Azure AD.
3. Configure Power BI Tenant to allow Service Principals to read the new Admin API.
4. Connect to the Power BI tenant and print a list of Workspaces to check that everything is configured properly.

Prerequisites:

• Active Azure Portal subscription. You can create one for free.
• Power BI Service tenant and account with administrative privileges. You can use this simple tutorial to configure it.
• Downloaded and running Power BI API Connector that we will be used for testing of the whole solution.

Let’s start the challenge!

1. Configure an Azure Application

In the main Azure Portal menu navigate to the Azure Active Directory item and select App registration.

Start New registration of an Application.

On the main registration screen please choose the application name. In our case, we will call the application SPW API Reader APP. You can keep default values for the rest of the form.

Press the Register button and wait for the deployment to finish. The Overview page will show us two important values identifying our Application which will be used later.

• Application (client ID) is the unique identification of the Application.
• Directory (tenant) ID is the unique identification of our tenant (organization).

Both these IDs are needed for Azure Active Directory access via the OAuth protocol. Because they are like “user” names we also must create a password. Navigate to the Certificates & secrets in the menu and press the New client secret button.

Add a secret name and choose the expiration that fits your needs. Press Add button and a new Client Secret will be created.

You must copy the Value of the secret now! On the next page refresh, the Copy icon will disappear and the secret value can’t be copied again. You must delete the secret then and create a new one.

Because the Power BI Admin API allows only the Service Principal to be an Azure AD Group, we will create a new Group and assign our Application to it. Go back to Azure Active Directory and navigate to Groups.

Press the New group button.

Create a new Group. Choose Security Group type and some meaningful Group name that you can later search for it easy.

Once the Group is created navigate to it and click the Group name.

Press the Add members button.

In the search box try to search for the name of our Application and then select the proper item filtered that it will move to the Selected items section.

Double-check that our Application is a member of the Group now.

You have successfully:

• created new Azure Application
• obtained unique Application ID, Tenant ID, and Application Secret values
• created a new Azure Active Directory group and assigned the Application to be a member of it

2. Test the Application for successful connection to Azure AD.

It’s time to test that the Application can successfully connect to Azure Active Directory and obtain a valid security token. This is the must-have step before we can try to access the Power BI Admin API itself.

To do this please download the Power BI API Connector utility and check the post for details on how it works. Try to connect and obtain the security token. You can continue this tutorial once done.

Note: You can also try to test the Power BI Admin API connection using the utility but it will fail with Unauthorized because our Application doesn’t have access to the Power BI Admin API now.

3. Configure Power BI Tenant to allow Service Principals to read the new Admin API.

Please login to the Power BI Service with an account with administrative privileges. If you will use an account without tenant-level administrative rights the Tenant settings section in the Admin portal is hidden at all. You can create a new Power BI Administrator or assign an existing user to the Power BI administrator role following this post.

Once done navigate in the Power BI menu to the Admin portal section and then select Tenant settings.

Scroll down and find the Admin API settings section. Press the Enabled button and then choose the Specific security groups option. Search for the Azure Active Directory Group we have created above where our Application is a member of it. Press the Apply button.

Security settings will be applied soon. It can take up to 15 minutes but usually, it takes not more than 2 minutes for the API is accessible.

That’s all from the configuration point of view.

4. Connect to the Power BI tenant and print a list of Workspaces.

It’s time to test that our Application can really access the Power BI Admin API now. We will use the Power BI API Connector utility again and print a list of Tenant available Workspaces. To do this please create at least one Workspace if you have a brand new Power BI account because the API method GetGroupsAsAdmin() used for testing doesn’t list the default My workspace item.

Navigate to Workspaces and press the Create a workspace button.

Type Workspace name and Save it.

New Workspace should be visible in the list of available Workspaces.

Run the Power BI API Connector utility again and after you will allow it to run the Power BI Admin API test it should connect successfully to your Power BI tenant, call the API and print a list of all available Workspace.

As you can see our testing Workspace name is listed. There is also “Samples WS” Workspace listed to demonstrate that it’s really admin access listing all tenant Workspaces as Service Principal and not only Workspaces of the given Power BI account as if the API is accessed in the authenticated user access mode.

That’s it! You are done and you can continue building your own application to do something great with all the data available via the Power BI Admin API.

References

• Power BI Admin REST API
• https://docs.microsoft.com/en-us/power-bi/admin/read-only-apis-service-principal-authentication
• https://docs.microsoft.com/en-us/power-bi/developer/automation/overview-of-power-bi-rest-api
• https://powerbi.microsoft.com/en-us/blog/announcing-new-admin-apis-and-service-principal-authentication-to-make-for-better-tenant-metadata-scanning
• https://docs.microsoft.com/en-us/power-bi/developer/embedded/embed-service-principal

Application is created in .NET.Core 5.0 and you can download it as:

• PowerBIApiConnector_Executable – executable files in *.zip package for direct use
• PowerBIApiConnector_Solution – Visual Studio 2019 Project

How to use it?

Extract the executable package files to a directory. You have two options how to launch the application:

A) Setup parameters at the application launch:

PowerBIApiConnector.exe {Application (client) ID} {Directory (tenant) ID} {Client Secret}

B) Execute it without parameters. Then you will be prompted for input values:

Input parameters are:

• Application (client) ID: This is an Azure Application ID
• Directory (tenant) ID: Your organization (tenant) ID
• Client Secret: Application secret value
  

You can get proper values for these parameters in Azure Portal: Navigate to your Application in Azure Active Directory and copy the first two parameters from the Application Overview page:

For the Client Secret navigate as in the image below and copy the Value or create a new secret:

When you launch the application and authentication to Azure AD will succeed, you will see an authentication token printed to the console window:

Same time you will be prompted if you would like to test the Power BI Admin Read-Only API access. You can skip this step in case you would like to test the Azure AD authentication only.

If you will input “Y” the GetGroupsAsAdmin() API call will be executed and when the application has properly configured access to the Power BI Admin API available tenant workspaces will be listed by name:

Solution content

Two dependency packages are required (will be automatically downloaded via NuGet packages manager):

Main method code:

using Microsoft.Identity.Client;
using Microsoft.PowerBI.Api;
using Microsoft.Rest;
using System.Linq;
using System;

namespace PowerBIApiConnector
{
    class Program
    {

        static PowerBIClient powerBIClient = null;
        static string accesToken = null;
        static IConfidentialClientApplication app;

        // Authentication variables
        private static string clientId;
        private static string tenantId;
        private static string clientSecret;

        // Power BI API paths
        private const string resource = "https://analysis.windows.net/powerbi/api";
        private const string ApiUrl = "https://api.powerbi.com";

        private static string authority;

        public static void Main(string[] args)
        {
            // Capture users input via prompt
            if (args.Count() == 0)
            {
                Console.WriteLine("Application (client) ID:");
                clientId = Console.ReadLine();

                Console.WriteLine("Directory (tenant) ID:");
                tenantId = Console.ReadLine();

                Console.WriteLine("Client Secret:");
                clientSecret = Console.ReadLine();
            }
            // Capture users input as launch parameters
            else if (args.Count() == 3)
            {
                clientId = args[0];
                tenantId = args[1];
                clientSecret = args[2];

                Console.WriteLine("------------------------");
                Console.WriteLine($"Application (client) ID: {clientId}");
                Console.WriteLine($"Directory (tenant) ID: {tenantId}");
                Console.WriteLine($"Client Secret: {clientSecret}");
                Console.WriteLine("------------------------");
            }
            else
            {
                Console.WriteLine("Mssing valid input parameters! Type: {Application (client) ID} {Directory (tenant) ID} {Client Secret} ");
            }

            // Set main API path for authentication
            authority = $"https://login.microsoftonline.com/{tenantId}";

            // Create authorization app
            try
            {
                app = ConfidentialClientApplicationBuilder
                    .Create(clientId)
                    .WithClientSecret(clientSecret)
                    .WithAuthority(new Uri(authority))
                    .Build();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                Environment.Exit(1);
            }

            string[] scopes = new string[] { $"{resource}/.default" };
            AuthenticationResult result = null;

            // Authenticate and obtain permissions scope
            try
            {
                result = app.AcquireTokenForClient(scopes).ExecuteAsync().Result;
            }
            catch (MsalUiRequiredException ex)
            {
                // Insufficient permissions.
                Console.WriteLine(ex.Message);
                Environment.Exit(1);
            }
            catch (MsalServiceException ex) when (ex.Message.Contains("AADSTS70011"))
            {
                // Invalid application scope.
                Console.WriteLine(ex.Message);
                Environment.Exit(1);
            }

            // Print token
            Console.WriteLine($"App-Only Access Token:\n{result.AccessToken}\n");
            
            accesToken = result.AccessToken;
            var tokenCredentials = new TokenCredentials(result.AccessToken, "Bearer");
            
            // Test PowerBI Admin API
            Console.WriteLine("Test Power BI API (List workspaces as service principal via Read-Only Admin Api?) Type Y/N:");

            if ("y" == Console.ReadLine().ToLower())
            {
                Console.WriteLine("------------------------");
                Console.WriteLine("Connecting to Power BI API...");

                try
                {
                    // Create PowerBIClient instance
                    powerBIClient = new PowerBIClient(new Uri(ApiUrl), tokenCredentials);

                    // List workspace names with the Admin only method (GetGroupsAsAdmin)
                    var workspaceNames = powerBIClient.Groups.GetGroupsAsAdmin(100, filter: "type eq 'Workspace'").Value.Select(x => x.Name);

                    //Print output
                    Console.WriteLine("Workspaces found:");

                    foreach (string workspaceName in workspaceNames)
                    {
                        Console.WriteLine("  => " + workspaceName);
                    }
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex.Message);
                    Environment.Exit(1);
                }
            }

            Console.WriteLine("Done!");

        }
    }
}

References

For additional references, I will recommend this excellent video from Marco Russo and the official Microsoft guide.

The original requirement was to extract metadata from more than five hundred Power BI reports deployed in on-premises Power BI Report Server. It’s not an issue to download all these reports to the local folder via a custom tool (I will publish one soon) or using the Report Server REST API. But this API doesn’t expose metadata contained inside the Power BI report file (*.pbix). And opposite there is nothing directly inside the Power BI Desktop tool to export metadata in some human-readable or further processing suitable format. A little bit more (means not a full metadata-export solution) can be realized with the Power BI Cloud Service API but this isn’t an option for on-premises running Power BI Report Server.

Let’s concentrate on the *.pbix file itself and keep the API option to another post.

If I’m talking about Power BI “metadata” I mean everything that is hidden inside the report: data sources, tables, columns, measures, relations, visuals used, etc. “External” metadata like where the report is sitting on the Report Server, how often it is used, and who can access it that’s easy to collect directly from the Report Server. But users, especially data analysts, DWH developers, and metadata solution specialists are more interested to see what is referenced from the DWH inside the Power BI report, how the whole data lineage looks like, how measures are calculated and relations used.

There basically two ways how to get this metadata:

• via opening the *.pbix file in Power BI and connecting to the underlying SSAS service hosting an instance of the Power BI model
• extracting the *.pbix file content and processing various files (*.json) from it.

I don’t want to describe the SSAS option there because we have realized this will require a significant amount of time to automate it somehow for hundreds of reports. It may also need dedicated hardware to process report changes in time. But still, it’s a valid solution and you can view more on it there:

• Connect to Power BI Desktop Model from Excel and SSMS
• Document your Power BI model with DAX Studio

Power BI files (*.pbix, *.pbit)

The *.pbix file is the main file of the Power BI report. It contains all metadata and also all data imported inside the model using the Import option for source queries. *.pbit is the Power BI Template file. It has in general the same content regarding metadata but it doesn’t contain any data imported from data sources like the *.pbix file. Both files are in fact *.zip files with other files packed inside. So rename them with *.zip extension and review the content.

It looks like that both files have the same content. But if you will review it twice there is a significant difference in how the DataModel is stored. Inside the *.pbix file is the model stored using Xpress9 compression to make the file much smaller because it contains data too. Inside the *.pbit template file it is an extracted DataModelShema in JSON format instead of it.

The Xpress9 compression is my point of view the main issue of why it’s so complicated to get the metadata from the Power BI. I did several attempts to extract the file playing with a lot of stuff from C++ libraries thru Python scripts, 7z, ending inside UWP Windows. Compression namespace but without success. All these tools can work with the original Express only compression algorithm which is a part of the Windows encryption API for a long time. But I really didn’t found anything to extract the new Xpress9 algorithm.

What does it mean? Simple answer: Without the ability to decompress the Xpress9 DataModel file only two options are in-game on how to get the Power BI metadata from the file:

• via connection to running Power BI SSAS instance (as described above)
• or via the option to save Power BI Template in the Power BI Desktop application and processing the *.pbit file.

Saving Power BI files as a template looks to be a simple solution at first look. But let’s go back to the initial requirement: process metadata for more than five hundred Power BI reports deployed to Report Server or even tens of files stored somewhere locally. There is no option to call Power BI from the command line to do this conversion in the background or something like that. It really means that every single Power bI report must be opened in Power BI Desktop and manually resaved as a Power BI Template file.

There is one solution in this video. We can use Power Automate Desktop to capture the manual conversion from *.pbix to *.pbit and then replay it automatically for other *.pbix file in the folder. This solution may be suitable for the initial conversion of multiple *.pbix files to templates but it may be hard to implement it on regular basis to the customer’s infrastructure. It’s still a workaround solution only.

The *.pbit file content

What is inside the Power BI Report file is pretty well described in this blog post.

The DataMashup file, containing Query Editor metadata, may be extracted with 7z:

7z x DataMashup -o".\DataMashupDir\" -aoa

It’s reporting some warnings but looks like all files are extracted properly:

For working with the DataModelSchema JSON file inside the *.pbit file there is one great tool available: https://tabulareditor.com. You can use it to open the *.pbit file directly and review its metadata:

It’s doing a great job in parsing the DataModelSchema JSON and it is open source, so you can get inspired in the source a lot on what is inside the JSON.

For the live connection to the running Power BI model, DAX Studio is the ultimate solution.

Conclusion

Without the Xpress9 issue, it will be easy to have some simple solution to download *.pbix files from dbo.Catalog table in Report Server, convert it to ZIP, extract DataModel JSON schema, parse it, and upload it to metadata catalog. Conversion to *.pbit makes it cumbersome and workarounds are complicated.

Further reading:

• R library to get Power BI metadata (using the SSAS connection solution but with interesting info about DataModel file)

Starting 2019 is should be much easier to answer the question because Microsoft has announced regular release scheduling for Power BI Report Server: Starting in 2019, we will be shipping a new release of Power BI Report Server in January, May and September each year.

This should help us properly to plan for that, inform our users in advance and schedule updates for Power BI Desktop for Report Server to be updated on time.

In the latest major update released in December 2019, it brings a huge list of new features like pages, visuals and bookmarks metadata available, visual comparer, visualization tree, the reverse dependency tree, DMV execution page, and dependant calculations for tables and columns with many more features.

The most liked feature I’m using is the option to perform an in-depth review of how data are spread in Power BI *.pbix file before I will distribute them to the end-user. Usually, there is a lot of ballast from the development process consuming the memory and affecting performance. Using Power bI Helper I can easily identify such cases and remove obsolete data from the file and compact it for delivery and performance. The file compaction ratio can be significant. Check out more on radacad.com blog.

Pomocí zde popsaného postupu je možné zakázat zobrazování Sign-up formuláře při spuštění Power BI Desktop na úrovni registru, tedy rámci podnikového prostředí hromadně pomocí distribuce příslušné Policy.

Při instalaci Power BI Desktop na WINDOWS 10 uvedený klíč v registru zcela chybí, proto je přikládám zde:

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Power BI Desktop] “ShowLeadGenDialog”=dword:00000000

Stačí jej vložit do souboru s příponou *.reg a spustit.

Po aplikaci se Power BI Desktop spustí rovnou takto:

PowerBI služby a nástroje můžeme rozdělit do tří velkých skupin:

• www.powerbi.com – cloudová služba, umožňující jak datovou analýzu, tak publikování reportů z desktopových a mobilních aplikací, sdílení dat a spoustu dalších funkcionality, které se typicky ke cloudovým službám vážou.
• Power BI Desktop – plnohodnotná desktopová aplikace pro snadné vytváření a publikování graficky pokročilých reportů. Obsahuje následující moduly:
  • Power Query
    • Pokročilý nástroj k extrakci dat z velkého množství podporovaných datových zdrojů a jejich transformaci pomocí jazyka M
  • Power Pivot
    • In-memory nástroj pro datové modelování, práci s relacemi, agregacemi a výrazy.
  • Power View
    • Interaktivní nástroj pro vizualizaci dat se širokou podporou různých vizuálních prvků a jejich vzájemnou interakci.
• Power BI doplňky pro Excel
  • Excel je stále primárním nástrojem pro datovou analýzu a doplňky, které je možné v něm aktivovat nebo jsou jeho nativní součástí od verze 2016, jej vybavují velmi podobnou funkcionalitou, jakou nabízí Power BI Dektop, a přinášejí i některé pokročilé funkce, které pro změnu v Power BI Desktop dostupné nejsou.
  • Power Pivot
    • Umožňuje přímo v Excelu vytvářet komplexní datové modely s relacemi, hierarchiemi, pokročilými agregacemi a výrazy.
    • Data v modelu jsou zpracovávána přímo v paměti (xVelocity, Vertipaq engine) a můžeme dotazovat řádově větší objemy dat mnohem efektivněji, než jsou-li data uložena v listech
  • Power Query
    • Samostatný plugin pro vytváření pokročilých dotazů z nejrůznějších datových zdrojů – orientován na ETL proces.
    • Od verze Excel 2016 již není nutné instalovat plugin samostatně a Power Query je přímo součástí záložky Data.
    • Podporuje široké množství datových zdrojů od SQL Serveru, Analytických služeb, Sharepoint, XML až po Facebook nebo Google Analytics.
  • Power View
    • Nástroj pro vytváření graficky působivých a především interaktivních reportů nad různými datovými zdroji.
  • Power Map
    • Nástroj pro rychlé a jednoduché zobrazování dat v mapách a mapových grafech podle geografických údajů z datového zdroje.
    • V Office 2016 přejmenováno na 3D Maps (ale stále dostupné jako COM doplněk PowerMap)
• Power BI Report Server
  • Speciální edice Microsoft Reporting Services, která umožňuje zobrazovat soubory vytvořené pomocí Power BI Desktop v on-premise prostředí stejně jako klasických reporty SSRS (paginated reports).
  • Pro tvorbu reportů vyžaduje speciální edici Power BI Desktop for Report Server.
• Power BI Mobile Apps
  • Mobilní aplikace pro zobrazování Power BI sestav na platformách Android, Apple či Windows Mobile

V praxi se můžeme setkat ještě s produktem pod označením “Power BI for Office 365”, jehož podpora ale byla od 31.12.2015 omezena a od 31.3.2016 byl definitivně označen za zastaralý. Aktivním předplatitelům byla nabídnuta možnost přejít na Power BI Pro službu a podrobnosti najdeme v článku Migrating to the new Power BI experience.

Cena

Pokud se bavíme o ceně za Power BI, máme na mysli především její cloudovou verzi a řešení pro on-premise(Power BI Report Server) . Power BI Desktop je zdarma, své lokální reporty tedy můžete vytvářet bez omezení.

Cloudová služba Power BI existuje ve Free a placené Pro nebo Premium verzi.  Ve free verzi si vyzkoušíte vše podstatné, především prohlížení datových sestav na webu. Hlavními důvody, které by nás mohly vést k pořízení placené verze, jsou především pokročilé moduly pro import dat z interních datových zdrojů (On Premise prostředí – Personal Data Gateway), emailové subscribce nebo sdílení sestav s ostatními uživateli.

Power BI Premium je určeno především pro podnikové prostřední a integraci s on-premise řešeními.

Detalní srovnání Free, Pro a Premium najdete zde.

Aktuální cena Pro verze je $9.99 za uživatele a měsíc, což lze považovat za relativně rozumné nastavení i pro českého uživatele.

Pokud si budete chtít Power BI vyzkoušet, je třeba pamatovat na jednu podstatnou skutečnost: pro registraci není možné využít emailové adresy z volně dostupných poštovních služeb (*.hotmail, *.gmail, atd. a bohužel ani řadu českých freemailových služeb), ale je nutné použít pracovní emailovou adresu.