Mixing Mobile Friendly Forms based auth with NTLM Windows auth in SharePoint

One of the complaints I hear often is when users on mobile devices, mainly iOS, attempt to login to SharePoint.  Typically they are presented with a popup box asking for their username and password but there isn’t an option to save the password.  This is because our SharePoint web applications are using NTLM windows authentication because this is the most compatible with desktop PCs and provides the most functionality in SharePoint.  Apple is decidedly “nerfing” their safari browser to force more apps to be sold which makes them more money.  Part of this is them not providing an option to save passwords for NTLM authenticated sites and thus requiring apps to be made to support browsing SharePoint.

I’ve been working on this problem for a while.  Because of BI and some other features we couldn’t move away from Windows authentication for SharePoint.  So last year a coworker and I worked on creating our own browser app in iOS which would save user’s credentials and allow access to our SharePoint sites.  The problem with this approach is now we are developing a commodity, a web browser, which could end up taking a lot of time and effort for not much benefit.  After a long year last year working on many projects I got some breathing room at the beginning of this year and my brain decided to work on this problem and I came up with an idea.

Since I know SharePoint takes a windows principal and converts it into a windows claim, what if I could inject myself right BEFORE it converts the principal into a claim and replace the principal with a user of my choice.  I reflectored the code in Microsoft.SharePoint.IdentityModel.SPWindowsClaimsAuthenticationHttpModule and found the right spot and after building my own httpmodule and configuring it to run before SharePoint’s (see web.config example below), I was able to change the currently logged in user to be one of my choice.  This was great because this meant that I could potentially create a forms based experience for mobile devices and then tell SharePoint to use that user instead of thinking we weren’t authenticated.  The reason forms based authentication is needed is because iOS supports remembering passwords for forms based authentication.

SharePoint web.config

<system.webServer>
...
 <modules runAllManagedModulesForAllRequests="true">
 ...
 <add name="WindowsFormLogin" preCondition="integratedMode" type="SharePoint.HttpModules.WindowsFormLogin, SharePoint.HttpModules, Version=1.0.0.0, Culture=neutral, PublicKeyToken=***********" />
 <add name="SPWindowsClaimsAuthentication" type="Microsoft.SharePoint.IdentityModel.SPWindowsClaimsAuthenticationHttpModule, Microsoft.SharePoint.IdentityModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
 ...
 </modules>
...
</system.webServer>

Notice in the httpmodule below that I am checking to see if the useragent of the device is a mobile device on line 21.  Then I am checking if the status being returned is a 401 (unauthorized) on line 39.  Originally those were the only rules I had in place before I redirected to my forms login page.  But then I ran into some issues with the SharePlus app, which uses the same useragent as safari, trying to use webservices and requesting files and it didn’t understand our new authentication.  So i placed some other checks in there to determine whether it was a request from a browser or from an app.  Apps never have referrers set, so I check that first (line 59), and then I have a few other checks in place including looking at the querystring for web=1 (line 69) which is what SP uses to open files in the office web apps (office online).  So if i think this request is coming from a mobile device and it’s not an app, then I redirect to my special login page (line 85), otherwise I add some headers (line 90) which may not be needed but potentially apps such as office might be interested in.

HttpModule – PreSendRequestHeaders

public class WindowsFormLogin : IHttpModule
{
	public void Init(HttpApplication context)
	{
		if (context == null)
		{
			return;
		}
		context.AuthenticateRequest += context_AuthenticateRequest;
		context.PreSendRequestHeaders += context_PreSendRequestHeaders;
	}

	void context_PreSendRequestHeaders(object sender, EventArgs e)
	{
		HttpApplication httpApplication = sender as HttpApplication;
		HttpContext context = httpApplication.Context;

		if (context == null) return;
		if (context.Request == null) return;
		if (context.Request.Browser == null) return;
		if (false == context.Request.Browser.IsMobileDevice) return;
		

		if (context.Request.Cookies[Constants.MOBILE_LOGIN_EXCLUDE_COOKIE_NAME] != null)
		{
			string cookieValue = context.Request.Cookies[Constants.MOBILE_LOGIN_EXCLUDE_COOKIE_NAME].Value;
			bool exclude;
			if (false == bool.TryParse(cookieValue, out exclude))
			{
				exclude = false;
			}

			if (exclude) return;
		}

		string userAgent = context.Request.ServerVariables["HTTP_USER_AGENT"];
		if (userAgent.Contains("AppleWebKit") &&
		    false == userAgent.Contains("Safari")) return;

		int status = context.Response.StatusCode;
		switch (status)
		{
			case 401:
				if (IsPageRequest(context))
		                    {
		                        HandlePageRequest(context);
		                    }
		                    else
		                    {
		                        HandleNonPageRequest(context);
		                    }
				
				break;
		}
	}

private bool IsPageRequest(HttpContext context)
        {
            if (context.Request.Url == null) return false;
            if (string.IsNullOrEmpty(context.Request.Url.AbsolutePath)) return false;

            if (context.Request.UrlReferrer != null &&
                false == string.IsNullOrEmpty(context.Request.UrlReferrer.ToString()))
            {
                return true;
            }

            if (context.Request.Url.AbsolutePath.ToLower().EndsWith(".aspx")) return true;
            if (context.Request.Url.AbsolutePath.Contains("/_layouts/")) return true;
            if (context.Request.Url.AbsolutePath.Contains("/_")) return false;

            if (context.Request.QueryString["web"] != null &&
                context.Request.QueryString["web"] == "1")
            {
                return true;
            }

            try
            {
                if (SPContext.Current.Web.ServerRelativeUrl.ToLower() == context.Request.Url.AbsolutePath.ToLower()) return true;
            }
            catch { }

            return false;
        }

        private void HandlePageRequest(HttpContext context)
        {
            context.Response.Redirect("/_mobilelogin/Home/Login?ReturnUrl=" + HttpUtility.UrlEncode(context.Request.Url.ToString()));
        }

        private void HandleNonPageRequest(HttpContext context)
        {
            if (context.Response.Headers["X-Forms_Based_Auth_Required"] == null)
            {
                context.Response.Headers.Add("X-Forms_Based_Auth_Required", "/_mobilelogin/Home/Login?ReturnUrl=" + HttpUtility.UrlEncode(context.Request.Url.ToString()));
            }
            if (context.Response.Headers["X-Forms_Based_Auth_Return_Url"] == null)
            {
                Uri relativeUri;
                Uri.TryCreate("/" + SPUtility.ContextLayoutsFolder + "/error.aspx", UriKind.Relative, out relativeUri);
                string absoluteUri2 = new Uri(SPAlternateUrl.ContextUri, relativeUri).AbsoluteUri;
                context.Response.Headers.Add("X-Forms_Based_Auth_Return_Url", absoluteUri2);
            }

            if (context.Response.Headers["X-MSDAVEXT_Error"] == null)
            {
                string value = "917656; " + HttpUtility.HtmlAttributeEncode(SPResource.GetString("FormsAuthenticationNotBrowser", new object[0]));
                context.Response.Headers.Add("X-MSDAVEXT_Error", value);
            }
        }


}

I decided to create a ASP.NET MVC web application for my login page.  For things to work as expected, this needs to be hosted in your SharePoint farm.  I decided to add it as a virtual directory called _mobilelogin in IIS under my SharePoint web applications.  Because I need a windows principal object later, I need to store the person’s username and password somewhere I can get to it later.  There are a few different ways of handling this, each with their own pros and cons but to keep things simple for this blog post, I will use the FormsAuthenticationTicket (line 70) that normal ASP.NET Forms based authentication uses to store both the userid and password.  This ticket gets encrypted (line 78) and stored as a cookie in the browser of the mobile device (line 84).  Since the password is sensitive, it will be doubly encrypted because it gets encrypted once (line 75) and then put into the ticket which gets encrypted again.

MVC Model and Controller

public class LoginModel
{
	public string UserName { get; set; }
	public string Password { get; set; }
	public bool CookieSet { get; set; }
	public bool RememberMe { get; set; }
}
	
public class HomeController : Controller
{
	[HttpGet]
	[OutputCache(NoStore = true, Duration = 0, VaryByParam = "None")] 
	public ActionResult Login()
	{
		LoginModel mdl = new LoginModel();
		mdl.RememberMe = true;
		try
		{
			if (Request.Cookies[Constants.MOBILE_LOGIN_COOKIE_NAME] != null)
			{

				string cookieValue = Request.Cookies[Constants.MOBILE_LOGIN_COOKIE_NAME].Value;

				if (false == string.IsNullOrEmpty(cookieValue))
				{
					FormsAuthenticationTicket ticket = FormsAuthentication.Decrypt(cookieValue);
					if (ticket.Expired)
					{
						Utility.LogoutHelper(Response);
					}
					else
					{
						mdl.UserName = ticket.Name;
						mdl.CookieSet = true;
					}
				}
			}
		}
		catch (Exception e)
		{
			ViewBag.ErrorMsg = e.Message;
		}

		return View(mdl);
	}

	[HttpPost]
	[OutputCache(NoStore = true, Duration = 0, VaryByParam = "None")] 
	public ActionResult Login(LoginModel mdl)
	{
		try
		{
			if (false == string.IsNullOrEmpty(mdl.UserName) &&
				false == string.IsNullOrEmpty(mdl.Password))
			{
				string username = mdl.UserName;
				if (false == Utility.IsAnEmailAddress(username) &&
					string.IsNullOrEmpty(Utility.ExtractDomain(username)))
				{
					username = ConfigurationManager.AppSettings["DefaultDomain"] + "\\" + username;
				}
				WindowsIdentity identity = Utility.Logon(username, mdl.Password);

				DateTime expiration = DateTime.Now.AddMinutes(Constants.MOBILE_LOGIN_TTL_SHORT_MIN);
				if (mdl.RememberMe)
				{
					expiration = DateTime.Now.AddDays(Constants.MOBILE_LOGIN_TTL_LONG_DAYS);
				}

				FormsAuthenticationTicket ticket = new FormsAuthenticationTicket(1,
					username,
					DateTime.Now,
					expiration,
					mdl.RememberMe,
					Utility.Protect(mdl.Password, Constants.MOBILE_LOGIN_SALT),
					FormsAuthentication.FormsCookiePath);

				HttpCookie cookie = new HttpCookie(Constants.MOBILE_LOGIN_COOKIE_NAME, FormsAuthentication.Encrypt(ticket));
				if (mdl.RememberMe)
				{
					cookie.Expires = ticket.Expiration;
				}
				Utility.SetAttributes(cookie);
				Response.Cookies.Add(cookie);

				HttpCookie cookieExclude = new HttpCookie(Constants.MOBILE_LOGIN_EXCLUDE_COOKIE_NAME, false.ToString());
				Utility.SetAttributes(cookieExclude);
				cookieExclude.Expires = DateTime.Now.AddDays(-1);
				Response.Cookies.Add(cookieExclude);
				
				Redirect();
			}
			else
			{
				ViewBag.ErrorMsg = "Please fill in both the UserName and Password fields.";
			}
		}
		catch (Exception e)
		{
			ViewBag.ErrorMsg = e.Message;
		}

		return View(mdl);
	}

	public ActionResult Logout()
	{
		Utility.LogoutHelper(Response);

		return View();
	}

	[OutputCache(NoStore = true, Duration = 0, VaryByParam = "None")] 
	public ActionResult Exclude()
	{
		HttpCookie cookieExclude = new HttpCookie(Constants.MOBILE_LOGIN_EXCLUDE_COOKIE_NAME, true.ToString());
		Utility.SetAttributes(cookieExclude);
		Response.Cookies.Add(cookieExclude);

		Utility.LogoutHelper(Response);
		Redirect();

		return View();
	}

	private void Redirect()
	{
		string returnUrl = "/";
		if (false == string.IsNullOrEmpty(Request.QueryString["ReturnUrl"]))
		{
			returnUrl = HttpUtility.UrlDecode(Request.QueryString["ReturnUrl"]);
		}

		Response.Redirect(returnUrl);
	}
}

MVC View

@model LoginModel

@{
    ViewBag.Title = "Login";
}


<div class="container">
    <div class="row">
        <div class="col-sm-12">
            @using (Html.BeginForm())
            {
                <div class="form-horizontal" style="margin-top:90px;width:90%">
                    <div class="form-group">
                        <div class="col-md-offset-3 col-sm-9">
                            <h4>This is a special login page just for mobile devices.  If you feel you reached this page in error, @Html.ActionLink("click here to restore normal login functionality", "Exclude", "Home").</h4>
                        </div>
                    </div>
                    <div class="form-group">
                        <div class="col-md-offset-3 col-sm-9">
                            @if (false == string.IsNullOrEmpty(ViewBag.ErrorMsg))
                            {
                                <p class="error">@ViewBag.ErrorMsg</p>
                            }
                        </div>
                    </div>
                    @if (Model.CookieSet)
                    {
                        <div class="form-group">
                            <div class="col-md-offset-3 col-sm-9">
                                <h4>
                                    You are currently logged in as: @Model.UserName <br />
                                    @Html.ActionLink("Click here to logout", "Logout", "Home", new { ReturnUrl = Request.QueryString["ReturnUrl"] }, new { } ).
                                </h4>
                            </div>
                        </div>
                        <div class="form-group">
                            <div class="col-md-offset-3 col-sm-9">
                                <h4>
                                    @if (string.IsNullOrEmpty(Request.QueryString["ReturnUrl"]))
                                    {
                                        <a href="/">Return to site</a>
                                    }else{
                                        <a href="@HttpUtility.UrlDecode(Request.QueryString["ReturnUrl"])">Return to site</a>
                                    }
                                </h4>
                            </div>
                        </div>
                    }
                    else
                    {
                    <div class="form-group">
                        @Html.Label("User Name", new { @class = "col-sm-3 control-label" })
                        <div class="col-md-9">
                            @Html.TextBoxFor(m => m.UserName, new { @class = "form-control" })
                            @Html.ValidationMessageFor(m => m.UserName, null, new { @class = "help-inline" })
                        </div>
                    </div>
                    <div class="form-group">
                        @Html.LabelFor(m => m.Password, new { @class = "col-sm-3 control-label" })
                        <div class="col-md-9">
                            @Html.PasswordFor(m => m.Password, new { @class = "form-control" })
                            @Html.ValidationMessageFor(m => m.Password, null, new { @class = "help-inline" })
                        </div>
                    </div>
                    <div class="form-group">
                        <div class="col-md-offset-3 col-sm-9">
                            <input type="submit" value="Log in" class="btn btn-default" />
                            @Html.Label("Remember Me")
                            @Html.CheckBoxFor(m => m.RememberMe, new { @checked = "checked" })
                        </div>
                    </div>
                    <div class="form-group">
                        <div class="col-md-offset-3 col-sm-9">
                            <h4>If you have previously saved your password in your browser, select the username field above and then select autofill password at the top of the onscreen keyboard.</h4>
                        </div>
                    </div>
                    }
                    <br />
                </div>
            }
        </div>
    </div>
</div>

After we set the cookie in our MVC application, we need to redirect back to where we came from.  Then the HttpModule from earlier kicks in but on a different event: AuthenticateRequest.  Notice on line 26 where our Ticket gets unencrypted and I check to make sure it is still valid and not expired.  After that I get a WindowsIdentity object by logging in as the user with their username and password (line 33).  I then create a WindowsPrincipal object from the identity and assign it to the user property of the HttpContext (line 36).  This HttpContext.User property is what the SPWindowsClaimsAuthentication HttpModule will look at when creating the Windows Claim which SharePoint then uses everywhere.  I also hijack the signout page on line 38 to redirect to our forms logout page when we are logged in using our forms method.

HttpModule – AuthenticateRequest

public class WindowsFormLogin : IHttpModule
{
	public void Init(HttpApplication context)
	{
		if (context == null)
		{
			return;
		}
		context.AuthenticateRequest += context_AuthenticateRequest;
		context.PreSendRequestHeaders += context_PreSendRequestHeaders;
	}

	void context_AuthenticateRequest(object sender, EventArgs e)
	{
		HttpApplication httpApplication = sender as HttpApplication;
		HttpContext context = httpApplication.Context;

		try
		{
			if (context.Request.Cookies[Constants.MOBILE_LOGIN_COOKIE_NAME] == null) return;

			string cookieValue = context.Request.Cookies[Constants.MOBILE_LOGIN_COOKIE_NAME].Value;

			if (false == string.IsNullOrEmpty(cookieValue))
			{
				FormsAuthenticationTicket ticket = FormsAuthentication.Decrypt(cookieValue);
				if (ticket.Expired)
				{
					throw new ApplicationException("Mobile Login Expired");
				}

				string user = ticket.Name;
				WindowsIdentity identity = Utility.Logon(user, Utility.Unprotect(ticket.UserData, Constants.MOBILE_LOGIN_SALT));

				context.Request.ServerVariables["LOGON_USER"] = user;
				context.User = new WindowsPrincipal(identity);

				int index = context.Request.RawUrl.ToLower().IndexOf("/_layouts/15/signout.aspx");
				if (index >= 0)
				{
					string returnUrl = "/";
					if (index > 0)
					{
						returnUrl = context.Request.RawUrl.ToLower().Substring(0, index);
					}
					if (context.Request.UrlReferrer != null &&
						false == string.IsNullOrEmpty(context.Request.UrlReferrer.ToString()))
					{
						returnUrl = context.Request.UrlReferrer.ToString();
					}
					context.Response.Redirect("/_mobilelogin/Home/Logout?ReturnUrl=" + HttpUtility.UrlEncode(returnUrl));
				}
			}
		}
		catch (Exception ex)
		{
			//MobileLogin Login Error
			Utility.LogoutHelper(context);
		}
	}
}

I have used a few helper methods in my code above so below is this code which should enable you to put this solution together in your own environment.  Also, the MVC login app pool service account and your SharePoint app pool service account need special permissions on the server in order for them to login as another user on line 61.  This permission is called “Act as part of the operating system” which can be found in user rights assignment in local security policies.

Utility Class

public class Constants{
	public const string MOBILE_LOGIN_COOKIE_NAME = "MobileLogin";
	public const string MOBILE_LOGIN_EXCLUDE_COOKIE_NAME = "MobileLoginExcluded";
	public const int MOBILE_LOGIN_TTL_LONG_DAYS = 90;
	public const int MOBILE_LOGIN_TTL_SHORT_MIN = 90;
	public const string MOBILE_LOGIN_SALT = "TEST 123";
}

public class Utility
{
	public static string Protect(string text, params string[] purpose)
	{
		if (string.IsNullOrEmpty(text))
			return null;

		byte[] stream = Encoding.UTF8.GetBytes(text);
		byte[] encodedValue = MachineKey.Protect(stream, purpose);
		return HttpServerUtility.UrlTokenEncode(encodedValue);
	}
	public static string Unprotect(string text, params string[] purpose)
	{
		if (string.IsNullOrEmpty(text))
			return null;

		byte[] stream = HttpServerUtility.UrlTokenDecode(text);
		byte[] decodedValue = MachineKey.Unprotect(stream, purpose);
		return Encoding.UTF8.GetString(decodedValue);
	}

	public static WindowsIdentity Logon(string username, string password)
	{
		string domain = Utility.ExtractDomain(username);
		username = Utility.ExtractName(username);

		IntPtr handle = new IntPtr(0);
		handle = IntPtr.Zero;

		const int LOGON32_LOGON_NETWORK = 3;
		const int LOGON32_PROVIDER_DEFAULT = 0;

		// attempt to authenticate domain user account
		bool logonSucceeded = LogonUser(username, domain, password, LOGON32_LOGON_NETWORK, LOGON32_PROVIDER_DEFAULT, ref handle);

		if (!logonSucceeded)
		{
			// if the logon failed, get the error code and throw an exception
			int errorCode = Marshal.GetLastWin32Error();
			throw new Exception("User logon failed. Error Number: " + errorCode);
		}

		// if logon succeeds, create a WindowsIdentity instance
		WindowsIdentity winIdentity = new WindowsIdentity(handle, "NTLM");

		// close the open handle to the authenticated account
		CloseHandle(handle);

		return winIdentity;
	}

	[DllImport("advapi32.dll", SetLastError = true)]
	private static extern bool LogonUser(string lpszUsername,
										string lpszDomain,
										string lpszPassword,
										int dwLogonType,
										int dwLogonProvider,
										ref IntPtr phToken);

	[DllImport("kernel32.dll", CharSet = CharSet.Auto)]
	private static extern bool CloseHandle(IntPtr handle);
	
	public static bool IsAnEmailAddress(string text)
	{
		if (text == null) return false;
		if (text.Length <= 0) return false;
		if (text.IndexOf(" ") >= 0) return false;

		string patternLenient = @"\w+([-+.]\w+)*@\w+([-.]\w+)*\.\w+([-.]\w+)*";
		Regex reLenient = new Regex(patternLenient);
		string patternStrict = @"^(([^<>()[\]\\.,;:\s@\""]+"
			+ @"(\.[^<>()[\]\\.,;:\s@\""]+)*)|(\"".+\""))@"
			+ @"((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"
			+ @"\.[0-9]{1,3}\])|(([a-zA-Z\-0-9]+\.)+"
			+ @"[a-zA-Z]{2,}))$";
		Regex reStrict = new Regex(patternStrict);
		return reStrict.IsMatch(text);
	}

	public static string ExtractDomain(string UserName)
	{
		if (UserName.IndexOf(Convert.ToChar("\\")) > 0)
		{
			string[] fullDomainUserName = UserName.Split(new char[] { '\\' }, 2);
			return fullDomainUserName[0];
		}
		else
		{
			return string.Empty;
		}
	}

	public static string ExtractName(string name)
	{
		if (name.IndexOf(Convert.ToChar("\\")) > 0)
		{
			string[] fullDomainName = name.Split(new char[] { '\\' }, 2);
			return fullDomainName[fullDomainName.Length - 1];
		}
		else
		{
			return name;
		}
	}
	
	public static void SetAttributes(HttpCookie cookie, HttpContext context)
	{
		cookie.HttpOnly = true;
		string domain = GetRootCookieDomain(GetBasePath(context));
		if (false == string.IsNullOrEmpty(domain) &&
			domain.Contains('.'))
		{
			cookie.Domain = domain;
		}
	}

	public static void LogoutHelper(HttpContext context)
	{
		try
		{
			HttpCookie cookie = new HttpCookie(Constants.MOBILE_LOGIN_COOKIE_NAME, string.Empty);
			cookie.Expires = DateTime.Now.AddDays(-1);
			SetAttributes(cookie, context);

			context.Response.Cookies.Add(cookie);
		}
		catch (Exception err)
		{
			//MobileLogin Logout Cookie Error
		}
	}
	
	public static string GetRootCookieDomain(string url)
	{
		url = GetBasePathFromUrl(url);
		if (string.IsNullOrEmpty(url)) return url;
		Uri uri = new Uri(url);
		string host = uri.Host;
		if (false == host.Contains('.')) return uri.Host;

		string[] hostParts = host.Split('.');
		string domain = "." + hostParts[hostParts.Length - 2] + "." + hostParts[hostParts.Length - 1];

		return domain;
	}
	
	public static string GetBasePathFromUrl(string url)
	{
		string newUrl = string.Empty;
		if (url.Contains("://"))
		{
			newUrl = url.Substring(0, url.IndexOf("://") + 3);
			url = url.Substring(url.IndexOf("://") + 3, url.Length - url.IndexOf("://") - 3);
			if (url.Contains("/"))
			{
				newUrl += url.Substring(0, url.IndexOf("/"));
			}
			else
			{
				newUrl += url;
			}
		}
		return newUrl;
	}
	
	public static string GetBasePath(HttpContext context)
	{
		string basepath = "";
		if (context.Request.ServerVariables["HTTPS"].ToString().ToUpper().Equals("ON"))
		{
			basepath = "https://";
		}
		else
		{
			basepath = "http://";
		}
		basepath += context.Request.ServerVariables["SERVER_NAME"].ToString();
		if (!context.Request.ServerVariables["SERVER_PORT"].ToString().Equals("80") &&
			!context.Request.ServerVariables["SERVER_PORT"].ToString().Equals("443"))
		{
			basepath += ":" + context.Request.ServerVariables["SERVER_PORT"].ToString();
		}
		return basepath;
	}
}

Edit 05/18/2016:

Added a useragent check before the 401 status check to help with apps like the SharePoint iOS app and SharePlus which already know how to connect to SharePoint using NTLM.

SharePoint Cross Site Word Document Properties

Recently I had a project where I was using SharePoint columns as document property fields in MS Word.  This is pretty straight forward when you have one library and some files but when you have a central repository of master files that are then pushed out to different sites across your farm it makes it a little more interesting.  We built our content types and columns in Visual Studio which were deployed on all sites and thus had the same contenttypeid’s and column guids.  One thing I noticed was that in the master library the document field would work correctly but once the file was copied to another site the document field would no longer work.  I was unable to find anything on the web about this scenario which is why I’m posting my findings here.

First thing I did was to open the copied file in one of the non-master sites and fix the field.  I just deleted the old one and re-added it and everything was fine.  Next I renamed both the fixed document and the master document to have a .zip extension.  This enables me to extract the contents of the word file so I could examine the differences.  What I found was that in the word\document.xml file where the document fields were inserted they had different GUIDs for the ns3 namespace attribute (see on line 21).  Also check out the xpath statement on line 23 and then notice how the namespace (ns3) for our document field (ProjectName) is the namespace where we have different GUIDs between the two documents.

<w:sdt>
	<w:sdtPr>
		<w:rPr>
			<w:b/>
			<w:smallCaps/>
			<w:snapToGrid w:val="0"/>
			<w:sz w:val="18"/>
		</w:rPr>
		<w:alias w:val="Project Name"/>
		<w:tag w:val="ProjectName"/>
		<w:id w:val="-1780325992"/>
		<w:lock w:val="sdtContentLocked"/>
		<w:placeholder>
			<w:docPart w:val="1BC5893A50C54E06B3DBC32132B17E14"/>
		</w:placeholder>
		<w:showingPlcHdr/>
		<w:dataBinding 
			w:prefixMappings="xmlns:ns0='http://schemas.microsoft.com/office/2006/metadata/properties' 
			xmlns:ns1='http://www.w3.org/2001/XMLSchema-instance' 
			xmlns:ns2='http://schemas.microsoft.com/office/infopath/2007/PartnerControls' 
			xmlns:ns3='b5fc2735-42dd-40f0-a521-255c535f6692' 
			xmlns:ns4='cb6c538f-4b29-4910-832a-74df7c62e784' " 
			w:xpath="/ns0:properties[1]/documentManagement[1]/ns3:ProjectName[1]" 
			w:storeItemID="{59580BC9-FAD5-4EB7-BF24-59B00B2B217C}"/>
		<w:text/>
	</w:sdtPr>
	<w:sdtEndPr/>
	<w:sdtContent>
		<w:r w:rsidR="00556E42" w:rsidRPr="00920EC6">
			<w:rPr>
				<w:rStyle w:val="PlaceholderText"/>
			</w:rPr>
			<w:t>[Project Name]</w:t>
		</w:r>
	</w:sdtContent>
</w:sdt>

So following a hunch I looked up the SPWeb.ID and sure enough, it was the same GUID as ns3.  So, when copying documents from one site to another, the document field xml data from SharePoint must be tied to the GUID of the site it came from.  I was able to verify this by looking at the customXml\item1.xml file (it’s always makes things a lot easier to be able to view your xml data when working with xpath queries) which showed all of my document fields coming from SharePoint and they all had a namespace of the SPWeb.ID GUID.

Since the namespace will be changing every time I copy the file I decided to update the xpath query to ignore the namespace.  Changing it from:

w:xpath="/ns0:properties[1]/documentManagement[1]/ns3:ProjectName[1]"

to:

w:xpath="/ns0:properties[1]/documentManagement[1]/*[local-name() = 'ProjectName'][1]"

This xpath trick will query only based on the name of the element and exclude any namespaces.  Sure enough, once I updated the file in my master site and did another copy down to a non-master site, the field updated correctly in word and everything was happy.  You can update the document.xml file using any text editor you’d like, and then just zip all of your files back up, and then rename the file to be a .docx and it should open correctly in word.

Unable to Manually Start Workflows

Recently, after we finished migrating our large external farm from SharePoint 2010 to SharePoint 2013, I started hearing complaints that some user’s could not manually start workflows.  They would either get a 403 Forbidden or get the “Sorry, you don’t have access to this page” message.

2014-09-16_8-57-02

The stack trace on the back end produced the following:

Microsoft.SharePoint.Utilities.SPUtility.HandleAccessDenied(HttpContext context) at Microsoft.SharePoint.Utilities.SPUtility.HandleAccessDenied(Exception ex) at Microsoft.SharePoint.SPSecurableObject.CheckPermissions(SPBasePermissions permissionMask) at Microsoft.SharePoint.WorkflowServices.StoreSubscriptionService.EnumerateSubscriptionsByEventSource(Guid eventSourceId) at Microsoft.SharePoint.WorkflowServices.ApplicationPages.WorkflowPage.ConstructStartArray() at Microsoft.SharePoint.WorkflowServices.ApplicationPages.WorkflowPage.OnLoad(EventArgs e)

Figuring out the problem just took a little investigation.  I recently used this as a Demo for a SharePoint session I did recently at my local SharePoint Saturday.  From that session I have a video of the issue and how I determined the cause.  Skip to about 2:15 for the start of the issue and investigation.  Click here to see the video

So basically the new SharePoint 2013 Workflow Manager is checking for Contribute access at the Site (SPWeb) scope when determining if someone should see the list of workflows for a list (see first line of the method in the code below).

// Microsoft.SharePoint.WorkflowServices.StoreSubscriptionService
public override WorkflowSubscriptionCollection EnumerateSubscriptionsByEventSource(Guid eventSourceId)
{
	this.context.Web.CheckPermissions(SPBasePermissions.EditListItems);
	WorkflowStore workflowStore = new WorkflowStore(this.context.Web);
	eventSourceId = StoreSubscriptionService.ConvertToGuidToken(eventSourceId, this.context.Web);
	WorkflowFile[] files = workflowStore.QueryWithGuid("0x0100AA27A923036E459D9EF0D18BBD0B9587", StoragePublishState.Unchanged, "WSEventSourceGUID", eventSourceId);
	return this.ConvertToWorkflowSubscriptionCollection(files);
}

So in our scenario, our users had read access to the site but contribute access on the list.  This should allow them to start the workflow and it did in SP 2010.  I currently have a Design Change Request (DCR) open with MS regarding this issue so hopefully it will be fixed soon.  Once I hear more, I will update this post.  In the meantime there seems to be two workarounds.

Workaround 1:  Earlier in the code I determined that if you DON’T setup and connect a SharePoint 2013 Workflow Manager to the farm, then this code will never run and thus it will work just like it did in SP 2010.  Of course the issue with this workaround is you can’t have any SP 2013 workflows.

Workaround 2:  Basically give everyone that needs to manually start workflows contribute access to the site, and then break inheritance to all lists and libraries where the user DOESN’T need contribute access and remove their contribute access.

We ended up implementing the second workaround above which sucks for my users.  I am hoping this is fixed soon as I have also seen other people complaining about it (http://sharepoint.stackexchange.com/questions/115311/manually-start-sharepoint-2010-workflow-in-sharepoint-2013-farm/)

Speaking at SharePoint Saturday Charlotte 2014

I am speaking at the Charlotte SharePoint Saturday 2014 event this Saturday.  Below is my abstract and slide deck if you are interested.  If you are a reader of my blog and are going to the event, please stop by and say hello.

Session Title:  Unwrapping the Black Box: Advanced SharePoint Troubleshooting and Forensics

Session Topic:  SharePoint can often feel like a black box, and with little to no knowledge of internal workings, troubleshooting can be daunting. In this session, I remind you that SharePoint is nothing more than ASP.NET Web Forms and apply some of the same troubleshooting techniques to get you out of the dark. I will also delve into some SharePoint specific tools and demonstrate actual errors and troubleshooting where the resolution requires reading Microsoft SharePoint code.

Tools/Techniques

  • Internet Explorer F12 Developer Tools
  • Fiddler2
  • Developer Dashboard
  • ULS Viewer
  • SharePoint Manager
  • Reflector/ILSpy
  • SQL Profiler

Click here for the presentation including embedded videos of demos and scenarios.

SharePoint 2013 conflicts with custom site definition

I was updating one of our custom Site Definitions from SharePoint 2010 to SharePoint 2013 recently and everything was going good until I tried to create a site from the updated definition.  I kept getting the error:

Microsoft.SharePoint.SPException: The template you have 
chosen is invalid or cannot be found.

Searching on the internet told me the most common cause of this error was a conflicting ID for the template.  I was pretty sure this couldn’t be the case since we followed the instructions here:  http://msdn.microsoft.com/en-us/library/office/ms454677(v=office.14).aspx which stated:

Change the ID attribute of the Template element to a 
value of 10000 or more. This ensures that your ID will 
not conflict with future site definitions produced by 
Microsoft. If there are other custom site definitions 
on your target farm, make sure that each one has a 
unique ID.

and we had given our template an ID of 10000.  I had previously updated another custom site definition of ours with an ID of 10001 without any issues.

So I decided to give a quick search in the SharePointRoot\template\1033\XML folder for anything that contains ID=”10000″.  Sure enough I found two site definitions, mine and a new one that comes with SharePoint 2013 called the Academic Library.

I found this article:  http://social.technet.microsoft.com/wiki/contents/articles/20149.sharepoint-2013-default-site-templates.aspx which discusses the new site definitions that come with SharePoint 2013.  You can see it listed there as:

ID Name Title
10000 DOCMARKETPLACESITE#0 Academic Library
The Academic Library template provides a rich view and consumption experience for published content and management. Authors populate metadata and apply rules at the time of publishing, such as description, licensing, and optional rights management.(IRM). Visitors of the site can search or browse published titles and add authorized selections to their collection to consume, subject to the rights and rules applied by the author. The site provides an IRM-capable document library, a publishing mechanism for authors to publish documents, detailed views for each document, a check-out mechanism, and related search capabilities.

So it would seem that if you followed the recommendations for SharePoint 2010 and used an ID of 10000 for your custom site definition, then when you try to go to SharePoint 2013, your site definition won’t work.

This is an issue because there is not a way to change the template ID of a site after it’s been created through the API.  Fortunately, I have found 2 ways to get around this.  One of them I haven’t fully run to ground and verified that it works but the concept seems valid and the other isn’t supported by Microsoft.

Option 1:

This option I got the idea from this blog:  http://iknowsharepoint2007.blogspot.com/2010/02/changing-sharepoint-site-definition.html.  The basic idea is you export your site without using compression, change some xml configuration files so they use your new ID, and then import it back into your SharePoint environment.  You might not even have to change any xml files since that post was for SP 2007.  If someone tries this method, please comment below, otherwise if I find time to give it a try, i’ll update with my findings here.

Option 2:

This involved editing data directly in the SharePoint content database.  This is unsupported by Microsoft but worked for me during my testing.  Go to the AllWebs table in the content database for the site in question.  You should be able to find your site listed by scanning the FullUrl column.  Once found, write a SQL update statement which updates the WebTemplate column to the new site definition ID.  This was an instant fix for me as I was able to immediately start the upgrade to SharePoint 2013 for that site collection.

It sucks that Microsoft told us they would never use Site Definitions ID of 10000 or more and then they go back on their word with SharePoint 2013.  Anyways, I hope this helps someone else out trying to complete their migration to SharePoint 2013.  If anyone finds a better way to fix the issue, please post in the comments below.  Thanks.