How to store each SharePoint Site Collection in its own Database – Part 2

As I mentioned in Part 1, I decided to store each site collection in its own database for better managability on both of our SharePoint farms. This post is Part 2 and hopes to answer the question: “What happens when someone deletes a site collection?” In a normal sharepoint environment that only has a few content databases, this is not a big deal. But in our environment it’s a much bigger deal because each site collection has its own database and that database name includes the site collection url. Our web service from Part 1 helps make sure we don’t put a site into a database that is left over from a deleted site collection, but it’s our event receivers that do the real clean up.

We have two event receivers for this. One for SiteDeleting (Before the site collection is deleted) and the other for SiteDeleted (After the site collection is deleted). SiteDeleting will check if the site collection being deleted is in a database which will be empty after the site collection is deleted. If this is the case, then we set the database status in sharepoint to offline (disabled). SiteDeleted will get a list of all content databases that are empty and remove them from SharePoint. It also sets the database to single user and then drops it.

Below is the code or click this link to download:

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.SharePoint;
using Microsoft.SharePoint.Administration;
using System.Data;
using System.Data.SqlClient;
using Microsoft.SharePoint.Utilities;
using System.Web;

public class DeleteDatabaseOnSiteDeleted : SPWebEventReceiver
{
    public override void SiteDeleting(SPWebEventProperties properties)
    {
        base.SiteDeleting(properties);
        using (SPSite site = properties.Web.Site)
        {
            //Before the site collection is deleted
            //set status to disabled if the currently being deleted site collection
            //is the last one in this database
            if (site.ContentDatabase.CurrentSiteCount == 1)
            {
                site.ContentDatabase.Status = SPObjectStatus.Disabled;
                site.ContentDatabase.Update();
            }
        }
    }

    public override void SiteDeleted(SPWebEventProperties properties)
    {
        base.SiteDeleted(properties);

        Uri url = new Uri(properties.FullUrl);
        string currUrl = properties.FullUrl;
        string basepath = currUrl.Substring(0, currUrl.IndexOf(url.Host) + url.Host.Length);

        SPWebApplication webApp = SPWebApplication.Lookup(new Uri(basepath));
        List<SPContentDatabase> emtpyDbs = new List<SPContentDatabase>();

        //After the site collection is deleted find all databases that 
        //don't have any sites in them
        foreach (SPContentDatabase database in webApp.ContentDatabases)
        {
            if (database.CurrentSiteCount <= 0)
            {
                emtpyDbs.Add(database);
            }
        }

        foreach (SPContentDatabase database in emtpyDbs)
        {
            RemoveAndDeleteDb(database);
        }
    }

    private void RemoveAndDeleteDb(SPContentDatabase db)
    {
        string dbName = db.Name;

        if (db.CurrentSiteCount != 0) return;

        //Remove database from SharePoint
        db.WebApplication.ContentDatabases.Delete(db.Id);

        //Get database server name
        SPWebService service = SPFarm.Local.Services.GetValue<SPWebService>();
        SPDatabaseServiceInstance defaultDatabaseInstance = service.DefaultDatabaseInstance;
        string databaseServer = SPHttpUtility.NoEncode(defaultDatabaseInstance.NormalizedDataSource);

        //Build connection string
        string connectionString = "Data Source=" + databaseServer + ";Initial Catalog=Master;User ID=uid;Password=pwd;";

        SqlConnection connection = new SqlConnection();
        connection.ConnectionString = connectionString;
        string sql;

        //Force database to close other connections
        sql = "ALTER DATABASE [" + dbName + "] SET SINGLE_USER WITH ROLLBACK IMMEDIATE";
        SqlCommand command1 = new SqlCommand(sql, connection);
        command1.CommandType = CommandType.Text;

        //Remove database from sql server
        sql = "DROP DATABASE [" + dbName + "]";
        SqlCommand command2 = new SqlCommand(sql, connection);
        command2.CommandType = CommandType.Text;

        using (connection)
        {
            connection.Open();
            command1.ExecuteScalar();
            command2.ExecuteScalar();
        }
    }
}

As with Part 1, the main issue with this implementation is concurrency. You probably shouldn’t be creating and deleting a site collection at the same time. Since the chances in our environment for this are very slim, we’ve never had any issues. Also, since the database is dropped, any changes made since the last backup are lost. It might be a better solution to call a stored procedure during SiteDeleted which backs up the database first, and then sets it to single user and drops it.

Of course for this solution to work, you’ll need to write a feature that adds the eventreceivers to all existing site collections and any new site collections. I haven’t included that code here because it’s pretty straightforward, but if anyone is interested, leave a comment and I’ll see if I can get it posted.

————————————
Update 11/06/2009: I’ve created a Part 3 which includes the feature code.

How to store each SharePoint Site Collection in its own Database – Part 1

We’ve been experiencing mediocre performance with our SharePoint implementation at work. I think it’s just grown too large for our current infrastructure. I felt like I had done everything I could with code optimization and since we had some support hours left over for the year we decided to ask a Microsoft consultant to come by for a few days. Daniel Painter was the consultant that helped us out. While going through our environment and looking at our custom code I had to explain what it did and why it was there. One of the pieces really intrigued him and he asked me to blog about it so here it is.
When I inherited our intranet SPS 2003 and WSS V2 environment in 2005 it had one content database that was 130 GB in size. Backups would take a really long time and we were concerned not only with disaster recovery but with accidental deletion (there wasn’t a recycle bin in SP2003). Microsoft recommended not having any content databases over 50 GB in size. So my strategy was to have each site collection in its own database. This would really help out the backups, we would be within recommendations for database size, and recovery of deleted data would be easier.
I decided that the best way to do this is to have a SharePoint Site Request Form which is an ASP.NET web application that would call custom built web services in SharePoint. The web service would do the following:
  1. Create a database name which is valid and has the url of the site collection it contains as part of the name
  2. Call a stored procedure in the master db on the sql server which does the following
    • Creates a database with data and log files in the correct locations
    • Sets appropriate file growth and other parameters
    • Performs a backup so that the log backup’s won’t fail at night
  3. Set all databases in SharePoint that have available space in them to offline (disabled). A database which is offline (disabled) means that it won’t accept any new site collections in it but SharePoint will still serve any content that is in that database.
  4. Add the newly created database to the web application in SharePoint
    • We set the warning level to 0 and maximum to 1
  5. Create a new site collection. Because our newly created db is the only db that is online and has space in it (1 space) then our newly created site collection will be put in our newly created db.
  6. Set all databases in SharePoint that have available space in them to online
Doing this resolved our issues for any new site collections (migrating the existing site collections into their own databases took some time but was a pretty straightforward, backup, delete, then restore process) as well as had a few more benefits, including now we can easily tell which site collection is in which database because of the database naming. We can also make sure the data and log files are in their correct location (different drives).
When we first implemented this, we had issues when a new site collection was created our backups would fail at night. This was because it was trying to do a log backup first and then the full backup, but since a full backup didn’t exist yet, the backup would fail that night but succeed the next night assuming no new site collections were created. We fixed the backups failing issue by adding to our stored procedure to backup the db and log right after the db was created.
Below is the code or click these links to download: StoredProcedure and WebService
Stored Procedure:
USE [master]
GO

SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

CREATE PROCEDURE [dbo].[usp_create_sharepoint_database2] (
@databaseName varchar(1000)
)
AS

declare @dataPath varchar(1000)
declare @logPath varchar(1000)
declare @name1 varchar(1000)
declare @name2 varchar(1000)
declare @sql varchar(8000)

set @dataPath = 'd:mssql90data'
set @logPath = 'f:mssql90log'

set @sql = 'create DATABASE ' + @databaseName + '
ON ( NAME = ' + @databaseName + ',
	FILENAME = ''' + @dataPath + @databaseName + '.mdf'',
	SIZE = 3MB, MAXSIZE = UNLIMITED, FILEGROWTH = 250MB )
LOG ON
( NAME = ' + @databaseName + '_log,
   FILENAME = ''' + @logPath + @databaseName + '.ldf'',
   SIZE = 1MB,
   MAXSIZE = UNLIMITED,
   FILEGROWTH = 100MB )
COLLATE Latin1_General_CI_AS_KS_WS'
EXEC (@sql)

select @name1='e:mssql90backup' + @databaseName +  '.BAK', @name2 = @databaseName + '_C'
    BACKUP DATABASE @databaseName TO DISK = @name1 WITH INIT, NOUNLOAD, NAME = @name2, NOSKIP, STATS = 10, NOFORMAT
    select @name1='E:mssql90backup' + @databaseName + '_' + 'LOG' + '.BAK', @name2 = @databaseName + '_L'
    BACKUP LOG @databaseName TO DISK = @name1 WITH INIT, NOUNLOAD, NAME = @name2, NOSKIP, STATS = 10, NOFORMAT
Web Service:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint;
using System.Text.RegularExpressions;
using Microsoft.SharePoint.Administration;
using Microsoft.SharePoint.Utilities;
using System.Web;
using System.Data.SqlClient;
using System.Data;
using System.Web.Services;

[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
[System.ComponentModel.ToolboxItem(false)]
// To allow this Web Service to be called from script, using ASP.NET AJAX, uncomment the following line. 
// [System.Web.Script.Services.ScriptService]
public class SiteCreationWebService : WebService
{
    [WebMethod]
    public void CreatePortalSiteCollectionAndDatabase(string basepath, string serverRelativeUrl, string userName, string displayName, string emailAddress, string template, string quota, string title, string description, string portalUrl, string portalName)
    {
        //Make sure web doesn't already exist
        bool webExists = false;
        try
        {
            using (SPSite site = new SPSite(basepath + serverRelativeUrl))
            {
                using (SPWeb web = site.OpenWeb())
                {
                    webExists = web.Exists;
                }
            }
        }
        catch { }
        if (webExists)
        {
            throw new ApplicationException("This Site already Exists");
        }

        //Construct database name to include the url and only use allowed characters
        //Also append some random numbers at the end to pretty much guarantee a unique name
        string WebURL = serverRelativeUrl.Replace("/", "_");
        string databaseName = "wss__" + WebURL + "__" + System.Guid.NewGuid().ToString().Substring(0, 8);
        databaseName = Regex.Replace(databaseName, "[^A-Za-z0-9_]", string.Empty);

        //Make sure the site creation doesn't fail if the template is not specified
        if (string.IsNullOrEmpty(template))
        {
            template = null;
        }

        SPWebApplication webApp = SPWebApplication.Lookup(new Uri(basepath));

        //Take all databases offline if they have space to ensure the created site is put in
        //the database we create
        foreach (SPContentDatabase database in webApp.ContentDatabases)
        {
            if (database.CurrentSiteCount < database.MaximumSiteCount)
            {
                database.Status = SPObjectStatus.Disabled;
                database.Update();
            }
        }

        //Get the default database server's name
        SPWebService service = SPFarm.Local.Services.GetValue();
        SPDatabaseServiceInstance defaultDatabaseInstance = service.DefaultDatabaseInstance;
        string databaseServer = SPHttpUtility.NoEncode(defaultDatabaseInstance.NormalizedDataSource);

        //Build the connection string
        string connectionString = "Data Source=" + databaseServer + ";Initial Catalog=Master;User ID=uid;Password=pwd";

        //Build the sql statement
        SqlConnection connection = new SqlConnection();
        connection.ConnectionString = connectionString;

        SqlCommand command = new SqlCommand("usp_create_sharepoint_database", connection);
        command.CommandType = CommandType.StoredProcedure;
        command.Parameters.AddWithValue("@databaseName", databaseName);

        //Create the database
        using (connection)
        {
            connection.Open();
            command.ExecuteScalar();
        }

        //Add our newly created database to SharePoint
        webApp.ContentDatabases.Add(databaseServer,
                                    databaseName,
                                    webApp.WebService.DefaultDatabaseUsername,
                                    webApp.WebService.DefaultDatabasePassword,
                                    0,
                                    1,
                                    0);

        //HACK:  Needed when creating a site collection from a web service
        HttpContext.Current.Items["FormDigestValidated"] = true;

        //Create site collection
        using (SPSite site = webApp.Sites.Add(basepath + serverRelativeUrl,
                                              title,
                                              description,
                                              1033,
                                              template,
                                              userName,
                                              displayName,
                                              emailAddress))
        {
            //Add portal connection if available
            if (false == string.IsNullOrEmpty(portalName) &&
                false == string.IsNullOrEmpty(portalUrl))
            {
                site.PortalUrl = portalUrl;
                site.PortalName = portalName;
            }

            //Set quota if available
            if (false == string.IsNullOrEmpty(quota))
            {
                SPQuotaTemplateCollection quotas = webApp.WebService.QuotaTemplates;
                site.Quota = quotas[quota];
            }

            //Take databases back online
            foreach (SPContentDatabase database in webApp.ContentDatabases)
            {
                if (database.Status != SPObjectStatus.Online &&
                    database.CurrentSiteCount != 0)
                {
                    database.Status = SPObjectStatus.Online;
                    database.Update();
                }
            }
        }
    }
}

There are some issues with this implementation. The first one is concurrency. You can’t really create two site collections at the exact same time with this method. If you do, then you might have issues where you get a site collection in a wrong database or one site collection creation fails, etc. The other issue is what happens when someone deletes their site collection? That means there will be a database with 1 space available for a site collection but can’t really be used because it’s name is for the deleted site collection. Step #3 and #6 of the web service partly takes care of this. The full solution is in Part 2.