After some searching and investigating detail about the server we captured two core informations:

1. The database is set for read-only access
2. The server was upgraded last week to never version (higher service pack).

This combination is the root cause of the database to be inaccessible: because it was set read-only during the instance upgrade, setup wasn’t able to upgrade the database too. The error above appears at every single action to recover now.

All these steps has failed:

1. Set the database for read_write access:

USE master;  
GO  
ALTER DATABASE [DB1] SET READ_WRITE;  
GO

2. Detach the database

3. Backup the database

4. Check if files aren’t locked at the file system level (read-only attribute)

In the end, we have finished with dropping the database and restoring it from the previous backup which led to successful database version upgrade.

It’s really one the best practices that before any patching or upgrade of SQL Server instance all read-only database should be set back to read-write mode.

There is the query executed by SSMS in the background:

exec sp_executesql N'SELECT
v.name AS [Name],
SCHEMA_NAME(v.schema_id) AS [Schema],
''Server[@Name='' + quotename(CAST(
        serverproperty(N''Servername'')
       AS sysname),'''''''') + '']'' + ''/Database[@Name='' + quotename(db_name(),'''''''') + '']'' + ''/View[@Name='' + quotename(v.name,'''''''') 
	   + '' and @Schema='' + quotename(SCHEMA_NAME(v.schema_id),'''''''') + '']'' AS [Urn],
v.create_date AS [CreateDate],
ISNULL(sv.name, N'''') AS [Owner],
CAST(CASE WHEN ISNULL(smv.definition, ssmv.definition) IS NULL THEN 1 ELSE 0 END AS bit) AS [IsEncrypted],
CAST(ISNULL(OBJECTPROPERTYEX(v.object_id, N''IsSchemaBound''),0) AS bit) AS [IsSchemaBound],
CAST(CASE idx.index_id WHEN 1 THEN 1 ELSE 0 END AS bit) AS [HasClusteredIndex]
FROM
sys.all_views AS v
LEFT OUTER JOIN sys.database_principals AS sv ON sv.principal_id = ISNULL(v.principal_id, (OBJECTPROPERTY(v.object_id, ''OwnerId'')))
LEFT OUTER JOIN sys.sql_modules AS smv ON smv.object_id = v.object_id
LEFT OUTER JOIN sys.system_sql_modules AS ssmv ON ssmv.object_id = v.object_id
LEFT OUTER JOIN sys.indexes AS idx ON idx.object_id = v.object_id and (idx.index_id < @_msparam_0)
WHERE
(v.type = @_msparam_1)and(CAST(
 case 
    when v.is_ms_shipped = 1 then 1
    when (
        select 
            major_id 
        from 
            sys.extended_properties 
        where 
            major_id = v.object_id and 
            minor_id = 0 and 
            class = 1 and 
            name = N''microsoft_database_tools_support'') 
        is not null then 1
    else 0
end          
             AS bit)=@_msparam_2)
ORDER BY
[Schema] ASC,[Name] ASC',N'@_msparam_0 nvarchar(4000),@_msparam_1 nvarchar(4000),@_msparam_2 nvarchar(4000)',
@_msparam_0=N'2',@_msparam_1=N'V',@_msparam_2=N'0'

If we will try to execute this query directly from SSMS we will see this error message:

After deeper investigation I have found following single statement is causing the exception:

SELECT OBJECTPROPERTY(v.object_id, 'OwnerId')
FROM sys.all_views AS v

When calling the OBJECTPROPERTY function with ‘OwnerId’ as a parameter exception is fired and connection closed. Thankfully this is happening only when querying cloned database and not standard one.

I thought that it can be somehow related to security and SIDs matching in cloned database.

Error was reported to Microsoft Azure Feedback.

This error occurred after about one hour of applying snapshot to subscribers database and caused restart of the whole process.

Investigation of error details in distribution database tables was lost time. First after monitoring of the full traffic against the subscriber database we have discovered that our error was caused by this statement:

EXEC(N'
CREATE UNIQUE NONCLUSTERED INDEX [UX_RecordCategory_Record_ID_Category_ID] ON [ts].[RecordCategory]([Record_ID],[Category_ID]) 
WITH (DATA_COMPRESSION = PAGE ) 
WHERE ([Record_ID] IS NOT NULL AND [Category_ID] IS NOT NULL)')

What’s wrong there? It’s maybe not visible at the first look but yes, there is serious syntax error: WITH (DATA_COMPRESSION = PAGE ) should be after the WHERE first:

EXEC(
N'CREATE UNIQUE NONCLUSTERED INDEX [UX_RecordCategory_Record_ID_Category_ID] ON [ts].[RecordCategory]([Record_ID],[Category_ID]) 
WHERE ([Record_ID] IS NOT NULL AND [Category_ID] IS NOT NULL) 
WITH (DATA_COMPRESSION = PAGE ) 
')

This invalid statement was hidden in one of *.idx files in snapshot folder. *.idx files are executed as last during snapshot application so this is the reason why we were waiting for this error more than one hour (and 60 GB of data inserted to subscriber DB).

The general message for the whole audience is that you simply cannot generate valid snapshot in case that your database contains filtered indexes with data compression at the same time. MS guys should do better job when testing new features:)

You can check your database for this kind of indexes with this query:

SELECT *
FROM sys.[indexes] [i]
INNER JOIN sys.[partitions] [p] ON [p].[object_id] = [i].[object_id] AND [p].[index_id] = [i].[index_id]
WHERE [i].[has_filter] = 1 AND [p].[data_compression_desc] <> 'NONE'

I have created Microsoft Collaborate item for this.

You can see following error message:

There are few options how to solve this issue: https://github.com/Microsoft/DataConnectors/issues/2

In our case it really helped to update SSDT using Visual Studion Tools => Extensions and Updates:

After update was installed and Visual Studio restarted everything works fine.

Create sample table with one column of BIT datatype called SEND:

CREATE TABLE [dbo].[TestTable](
	[SEND] [bit] NOT NULL
) ON [PRIMARY]
GO

Now go to SSMS and open data editor:

Then edit column value to insert True:

You can see that an error has occurred. It is not saying to much until you check SQL statements sent from SSMS to SQL engine:

exec sp_executesql N'INSERT TOP (200) INTO TestTable(SEND) VALUES (@SEND)',N'@SEND bit',@SEND=1
GO

exec sp_executesql N'SELECT TOP (200) SEND FROM TestTable WHERE (''SEND'' = @Param1)',N'@Param1 bit',@Param1=1
GO

The first statement has completed successfully. But the second one has failed because SEND is an keyword and it is treated as string value and cast to BIT data type has failed. Looks like the issue is that apostrophe is used instead of square brackets.

If we will try the same but instead of SEND we will use column name which isn’t a keyword, i.e. [AAA], following statement is generated and executed without an error:

exec sp_executesql N'SELECT TOP (200) AAA FROM TestTable WHERE (AAA = @Param1)',N'@Param1 bit',@Param1=1
GO

This example is another plus point for the best practice to eliminated key/reserved words in development.

In our case the error was generated from a code similar to this pattern:

DECLARE @TempTable TABLE (...)

INSERT INTO @TempTable
    SELECT * FROM ....

INSERT INTO @TempTable
    select *
    from ( select distinct * from @TempTable ) tt
        cross join dbo.TableA a
    where 
        a.Col1 >=  1

The issue is that data are inserted into @TempTable at the same time when they are retrieved. Writing into this table is causing the reallocation of data pages which should first be read and at the time of reading they aren’t located at the expected place.

A simple and intuitive solution is code refactoring: Data are inserted into #Distinct temporary table first and after that used to prepare the final data to be inserted into @TempTable. This way we can prevent error to be fired.

DECLARE @TempTable TABLE (...)

INSERT INTO @TempTable
    SELECT * FROM ....

SELECT DISTINCT * 
    INTO #Distinct
FROM @TempTable

INSERT INTO @TempTable
    SELECT *
    FROM #Distinct tt
        cross join dbo.TableA a
    WHERE 
        a.Col1 >=  1
GO

No one of various suggestions found on the internet helped in our environment (security, etc.) After serious investigation, we found that the issue is presented only with specific values set to sp_send_mail procedure parameters. Following combinations of values is  generating the error:

EXEC msdb.dbo.sp_send_dbmail
	@profile_name = 'DEFAULT_DBMAIL_PROFILE',
	@recipients = 'test@test.com',
	@subject = 'Test',
	@body = 'Test',
	@query = 'SELECT 1', 
	@query_result_header = 0,
	@query_no_truncate = 1,
	@query_result_no_padding = 0

As most important looks like the combination of @query_result_header and @query_no_truncated. After doing this change error disappeared. Following is working:

EXEC msdb.dbo.sp_send_dbmail
	@profile_name = 'DEFAULT_DBMAIL_PROFILE',
	@recipients = 'test@test.com',
	@subject = 'Test',
	@body = 'Test',
	@query = 'SELECT 1', 
	@query_result_header = 1,
	@query_no_truncate = 1,
	@query_result_no_padding = 0

We are generating HTML content to the email body and after changing the @query_result_header parameter from 0 to 1 all the logic is working without any change and the email body is delivered as expected.

V našem konkrétním příkladě se chyba objevovala v kódu, který využíval následující logiku:

DECLARE @TempTable TABLE (...)

INSERT INTO @TempTable
    SELECT * FROM ....

INSERT INTO @TempTable
    select *
    from ( select distinct * from @TempTable ) tt
        cross join dbo.TableA a
    where 
        a.Col1 >=  1

Problematickou operací je zde vkládání do @TempTable ve stejném okamžiku, kdy se z tabulky čte. Zápis do tabulky způsobí realokaci stránek, které mají být teprve přečteny a v okamžiku čtení se již nenacházejí na svém místě.

Řešením je intuitivní přepsání kódu tak, že data se nejprve vloží do dočasné tabulky #Distinct a tato tabulka je následně využita k získání dat pro vložení do @TempTable. Tím se předejde realokaci stránek a výše zmíněné chybě.

DECLARE @TempTable TABLE (...)

INSERT INTO @TempTable
    SELECT * FROM ....

SELECT DISTINCT * 
    INTO #Distinct
FROM @TempTable

INSERT INTO @TempTable
    SELECT *
    FROM #Distinct tt
        cross join dbo.TableA a
    WHERE 
        a.Col1 >=  1