Today I’m flying to London with some of my Mavention colleagues to visit the SharePoint Best Practices Conference 2011. I think it will be an exciting 3 days full of SharePoint 2010. During the conference I shall try to update you through my blog.
You can also follow us at twitter: Marcel van der Lem (@marcelvanderlem), Waldek Mastykarz (@waldekm) at me, Marc Molenaar (@marhcuz).
Speakers I’m definitely going to listen to are:
Laura Rogers - Data view webpart
Mike FitzMaurice - Nintex Analytics 2010
Gary Lapointe - PowerShell
Brian Alderman - Optimizing SQL 2008
Bill English - Information Architecture
…and many more…
Tags: Best Practices, BPC11, Mavention, SP2010
Recently I have worked on the website for the Dutch Justice Department based on SharePoint 2007. Today this site is live. The URL is http://www.rechtspraak.nl
Some of my last posts where related to this site and the infrastructure.
Click image for best view.
Previous post: Search query results in 100 percent cpu time
No solution so far. I took these steps but no results:
Search query role
Assigned the search query role to the Application server, previously it was assigned to both Web Front Ends. Same result: very high cpu utilization during search queries.
Custom solutions
To exclude an error in one of the custom solutions I created a new standard web application with a standard internetsite in SharePoint with anonymous access, just like the original site. Made a new content source and did some other search settings. Same result: high cpu utilization.
Logging
Maybe the extensive logging might cause a performance problem. Diagnostics logging level was Verbose. Now set to High. Same result: high cpu utilization.
I’ll keep you informed. Please let me know if you have anything to add.
UPDATE april 7:
After thoroughly monitoring the ULS logging we pinned it down code on the search page. It seemed that Faceted Search 2.5 web part caused all the trouble. After removing this web part from the search page the cpu utilization dropped dramatically.
Tags: error, performance
I’m having, well to be precise the SharePoint farm I’m dealing with, performance issues with SharePoint 2007 search queries. Let me introduce the farm first to give you an idea:
2x WFE
CPU 4x Xeon 2 GHz E5504 (1x CPU quad core), 24 GB RAM
Hardware load balanced
OS Windows 2003 R2 SP2 x64
SP2007 untill April 2010 CU
both search query role
1x APP
CPU 1x Xeon 2 GHz E5405 (1x CPU dual core with HT), 24 GB RAM
OS Windows 2003 R2 SP2 x64
SP2007 untill April 2010 CU
indexer role
2x SQL
CPU 2x Xeon 2,4 GHz E5620 (2x CPU quad core with HT), 32 GB RAM
OS Windows 2008 x64 (no R2)
SQL 2008 R2
No cluster due to restrictions in the datacenter of provider, databases mirrored. Config and search on node 1, Content on node 2.
Only accessible by SP servers through firewall at port 1433
We did some performance tests and the results were not quite satisfying. A lot of queries resulted in long periods before results were shown and % cpu time increased dramatically.
For example:
A query with only 4 results by 20 simultanuous users results in almost 90% CPU time on one WFE
A query with only 50 results by 20 simultanuous users results in almost 100% CPU time on one WFE
I already tried to resolve this with caching but no results. Also changing registry keys (this site http://trycatch.be/blogs/gert/archive/2009/02/10/w3wp-high-memory-usage-in-sharepoint.aspx) and changing permissions (this site http://www.eggheadcafe.com/software/aspnet/29553964/wss3-search-crawl-causes-100-cpu-usage.aspx) did not do the trick.
Is there anyone with some good ideas? Please let me know.
Tags: performance
Context
Let me first tell you what the background is. I’m working at a client where they want a web application with multiple site collections each in it’s own database. Also each site collection has a managed path (explicit inclusion), some site collection names have special characters and each site collection must have an ID in the description field.
# Creating site collections. $web_application = "<web application name>" $web_application_url = "<web application url>" $db_server = "<servername>" $max_site_count = "<number>" $warning_site_count = "<number>" $owner_alias = "<domain\username>" $site_collection_template = "<template cde like STS#1>"
# Foreach loop to create site collections in array.
$projectsToCreate = @("DenBosch","Breda","Maastricht","Almelo","DenHaag","Rotterdam","Amsterdam","Alkmaar","Haarlem","Utrecht","Leeuwarden","Groningen")
$ID = 0
foreach ($project in $projectsToCreate) {
$ID += 1
Write-Host -ForegroundColor Red "--------------------------------------------------------------------"
Write-Host "Creating project" $project "( ID =" $ID ")"
$site_collection_db = "SP_Content_"+$project
Write-Host -ForegroundColor Red "--------------------------------------------------------------------"
Write-Host "Site Collectie Database = "$site_collection_db
Write-Host -ForegroundColor Red "--------------------------------------------------------------------"
# The projectname will be converted to lowercase for the managedpath.
$projectToLower = $project.ToLower()
$site_collection_url = $web_application_url + '/' + $projectToLower
Write-Host "Site Collectie URL =" $site_collection_url
Write-Host -ForegroundColor Red "--------------------------------------------------------------------"
Write-Host " "
# Creating content database for each project.
New-SPContentDatabase -Name $site_collection_db -WebApplication $web_application -MaxSiteCount $max_site_count -WarningSiteCount $warning_site_count
Write-Host -ForegroundColor Yellow "Content database" $site_collection_db "has been created."
# Creating managed path for each site collection.
New-SPManagedPath -RelativeURL $projectToLower -WebApplication $web_application -Explicit
Write-Host -ForegroundColor Yellow "Managed path" $projectToLower "has been created."
# There are some projects where the name of the site collection has to be written differently (with special characters).
# With this Switch the names can be defined. Or continue with default settings.
Switch ($project) {
"DenBosch" { $sitename = "`'s-Hertogenbosch"}
"DenHaag" { $sitename = "`'s-Gravenhage"}
default { $sitename = $project }
}
# Creating site collection with the name defined by the Switch
New-SPSite -Name $sitename -Description $ID -URL $site_collection_url -ContentDatabase $site_collection_db -OwnerAlias $owner_alias -Language 1043 -Template $site_collection_template
Write-Host -ForegroundColor Yellow "Site collection" $project "has been created with the name" $sitename
# Change status of content database to Disabled so the next site collection can be created in a new Online database.
Set-SPContentDatabase -Identity $site_collection_db -Status Disabled
Write-Host -ForegroundColor Yellow "The status of content database" $site_collection_db "is Disabled."
Write-Host -ForegroundColor Red "--------------------------------------------------------------------"
}
# Change the status of all created content databases to Online.
foreach ($project in $projectsToCreate) {
$site_collection_db = "SP_Content_"+$project
Set-SPContentDatabase -Identity $site_collection_db -Status Online
Write-Host -ForegroundColor Yellow "Status of database" $site_collection_db "is Online."
}
Tags: powershell, script, SP2010
At the moment I am installing a SharePoint 2007 farm with the Index server role at the Application server and the Query server role at the Web Front Ends. Because I did not configure anything with SSL in the first place, this causes some errors in the eventlog.
This event is 6482 (with two different descriptions):
Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared services
Event ID: 6482
Date:
Time:
User: N/A
Computer: <servername>
Description:
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (GUID).
The underlying connection was closed: An unexpected error occurred on a send.
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)
— End of inner exception stack trace —
at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)
at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(WellKnownSearchCatalogs catalog, SearchSharedApplication application)
at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)
Event Type: Error
Event Source: Office SharePoint Server
Event Category: Office Server Shared services
Event ID: 6482
Date:
Time:
User: N/A
Computer: <servername>
Description:
Application Server Administration job failed for service instance Microsoft.Office.Server.Search.Administration.SearchServiceInstance (GUID).
Unable to connect to the remote server
System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 159.46.4.74:56738
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception)
— End of inner exception stack trace —
at Microsoft.Office.Server.Search.Administration.SearchApi.RunOnServer[T](CodeToRun`1 remoteCode, CodeToRun`1 localCode, Boolean useCurrentSecurityContext, Int32 versionIn)
at Microsoft.Office.Server.Search.Administration.SearchApi..ctor(WellKnownSearchCatalogs catalog, SearchSharedApplication application)
at Microsoft.Office.Server.Search.Administration.SearchServiceInstance.Synchronize()
at Microsoft.Office.Server.Administration.ApplicationServerJob.ProvisionLocalSharedServiceInstances(Boolean isAdministrationServiceJob)
I found out that this kb article (http://support.microsoft.com/kb/962928) solved some of the issues. With the manual in this kb article and the tool SelfSSL in the IIS Resource Kit I was able to reconfigure the certificate for the Office Server Web Services site. After doing this I saw other events appear in the eventlog of the WFE server:
Event Type: Error
Event Source: crypt32
Event Category: None
Event ID: 8
Date:
Time:
User: N/A
Computer: <servername>
Description:
Failed auto update retrieval of third-party root list sequence number from: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootseq.txt> with error: The specified server cannot perform the requested operation.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
For reasons the hosting provider, where the servers reside, has done some configurations so the WFE servers can be reached from the internet but can not browse the internet themselves. So the crypt32 error appears.
After a few tries the txt file was retrieved and also a cab file. See this events:
Event Type: Information
Event Source: crypt32
Event Category: None
Event ID: 7
Date: 14-1-2011
Time: 13:38:27
User: N/A
Computer: <servername>
Description:
Successful auto update retrieval of third-party root list sequence number from: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootseq.txt>
Event Type: Information
Event Source: crypt32
Event Category: None
Event ID: 2
Date: 14-1-2011
Time: 13:38:32
User: N/A
Computer: <servername>
Description:
Successful auto update retrieval of third-party root list cab from: <http://www.download.windowsupdate.com/msdownload/update/v3/static/trustedr/en/authrootstl.cab>
So, problems are solved.
Tags: error, Infrastructure, security, SSL, tools
The EasyAssist call with the Microsoft Engineer is finished. It seems that a proxy setting caused all the trouble. We went through a lot of settings like BackConnectionHostnames, DisableLoopbackCheck, crawl rules, security settings on the databases etc.
But using the commandline tool proxycfg triggered it. See the screenshot below. The proxycfg command shows if there are proxies configured. In the client environment there is a proxy configured at the management network:
As you can see, a bypass list is not configured. Therefore the crawler wants to use this proxy setting to crawl the site. That is not possible and this results in an error. With Microsoft we reconfigured these proxy settings and included a bypass list. See below:
Now the crawler will bypass the proxy settings and is able to crawl the complete site.
By the way: the proxy is needed by WSUS to deploy updates to the servers over the management lan.
part 3: the solution
Tags: crawl, error, Infrastructure
At the moment we have a call for this at Microsoft. Their first idea was to enable DisableLoopbackCheck. DisableLoopbackCheck has already been configured (also backconnectionhostnames by the way).
Some additional info, these error/warning appear in the applicationlog when a Full crawl is started (all errors are in Dutch because I work for a government department but through the event ids I figured out the English translations):
Event Type: Error
Event Source: Windows SharePoint Services 3 Search
Event Category: Gatherer
Event ID: 2424
Date:
Time:
User: N/A
Computer: ComputerName
Description: The update cannot be started because the content sources cannot be accessed. Fix the errors and try the update again.
Context: Application SSP1, Catalog Portal_Content
…update can not be started because there is no access to content sources. Resolve the errors and try to run the update again… (free translation, don’t know the exact translation)
For event 2424 I tried: http://support.microsoft.com/kb/927012 I made the content access account also WSS search service account, this account also has db_owner (role membership) permission to the Search_<servername> database. No results so far. (changed settings back to original)
Event Type: Warning
Event Source: Windows SharePoint Services 3 Search
Event Category: Gatherer
Event ID: 2436
Date:
Time:
User: N/A
Computer: YOURMOSSSERVER
Description:
The start address ’site’ cannot be crawled.
Context: Application SSP1, Catalog Portal_Content
The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly
I think that the Catalog Portal_Content is the directory on the MOSS server where index files are stored (there is a directory Portal_Content in the path where MOSS is installed 12.0\Data\Office Server\Applications\<GUID>\Projects\Portal_Content\Indexer\CiFiles\AnchorIndex)
Is this a correct assumption? If so, should certain accounts have permission here? I see that the local group WSS_ADMIN_WPG (which holds some MOSS accounts) has FULL CONTROL on this directory.
The server is multi homed. There is a Client network and a management network. One of my colleagues figured this out. When we shutdown the management network card there was another error in the crawl log. It was something like.”Cannot connect to the repository”.
I have an EasyAssist appointment with Microsoft later today. I shall post the results.
part 2: the errors
Tags: error
Hello readers. I need some help with a very strange problem I have come up to at a customer farm.
The error that I am researching occurs when SharePoint crawls the default content source which is, in my test farm, a normal non-edited publishing portal. The error that appears in de crawl log is “The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly.”
FARM / NETWORK CONFIGURATION:
The farm consists of (for testing)
1 SP server:
OS W2K3R2 SP2 Enterprise x64 English installation,
CPU E5405 @2GHz, 24GB RAM
SP native dutch version 12.0.0.6548
Location: The Netherlands, Regional settings: Dutch, Input Language: Dutch – US International
OS updated by Windows Update with critical patches on 1/10/2011
1 DB Server:
OS W2K8R2 Enterprise x64 English installation
2x CPU E5620 @2.4GHz, 32GB RAM
SQL server 2008 R2
Location: The Netherlands, Regional settings: Dutch, Input Language: Dutch – US International
Forest functional level is 2000 and Domain functional level is 2000 mixed.
SP server and DB server are on different VLANs and separated by a Firewall. Only port 1433 is open. SP finds the DB server by Alias. This Alias is configured in SQL Server Configuration Manager which is installed at the SP server. The Alias directs to the DB server with Protocol TCP and Parameter 1433. Only by this Alias/Port 1433 the DB server can be reached.
Windows Firewalls on both servers are not enabled.
STEPS TAKEN SO FAR:
I wanted to start with a clean SP install because a lot of configuration has been done on the SP servers to make it work. All configuration has been done in my absence. Because I have to support this farm from now on I want to know every bit of it.
1. So disjoined the previous installation from the farm. Deleted all databases. Did a complete deinstallation of SP software on the SP server and rebooted.
2. Installed SP including service pack 2 (native dutch installation).
3. Start config wizard for configuring the farm.
4. Configuration in Central Admin:
4a. Services on server > WSS Search (Search and Content Access account are separate)
4b. Office Search (for this test the SP server is Search and Index server)
5. Continued with Configuring server farm’s shared services
6. Configured a Shared Services Provider (SSP1).
7. For every web application the “Security Configuration” is.
7a. authentication provider: NTLM
7b. Allow Anonymous: NO
7c. Use Secure Sockets Layer (SSL): NO
8. Made a Web application for SSP, Mysite and Portal
9. In the Portal Web App I made a Site collection with the Publishing portal template (this SP farm will host a Internet facing site)
10. Set search server at every database with CA > application management > content databases
11. Crawl results in error (error is in dutch but this is the translation): The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly
12. CU October 2010 Installed and restarted the server
13. same Crawl error
14. Language settings set the same on SP and SQL server. Location: Dutch, regional setting: Dutch. Input language: Dutch-US International
15. Install account was not a local admin at SQL server, added the account to local admin on SQL server
16. Run the command: Stsadm –o upgrade –forceupgrade > errors (install account does not have access to search db)
17. on SQL: added the install account to the search db as db_owner
18. Run the command: Stsadm –o upgrade –forceupgrade > errors (install account does not have access to another db)
19. Made the install account db_owner of all databases
20. Run the command: Stsadm –o upgrade –forceupgrade uitgevoerd. > Completed successfully.
21. same Crawl error
22. installed High prio windows updates through Windows Update on the SP server
23. Added application pool accounts to local group WSS_ADMIN_WPG at SP server
24. Added group WSS_ADMIN_WPG to IIS WAMREG Service properties. Given local launch and local activation
25. same Crawl error
26. added a Crawl rule including everything
27. same Crawl error > Crawl rule deleted
28. In IE > tools > internet options > security added the portal url to trusted sites.
29. Crawl error
30. In IIS added a hostheader, in SP added a AAM and in the hosts file added a local URL. Made a separate content source and crawled
31. Crawl error
32. Deleted hostheader, aam and line in hosts file. Also content source.
33. In hosts file disabled every line except the lines redirecting the SP urls directly to the SP server ipaddress
34. Crawl error (full and incremental crawl)
35. Standard time zone on webapplications set to GMT+1 Amsterdam
36. Crawl error
37. Search server set on content databases
38. Crawl error
39. Content Access account set secondary admin on site collection > crawl error > removed
40. Content Access account given Full access to site collection by policies > crawl error > removed
41. First site coll was publishing portal > deleted >
42. made a new sitecollection based on the teamsite template > crawl error.
I have run out of options. Do you have any suggestions? Please insert a comment or mail me. All help is welcome.
part 1: step by step
My other post about preparation for the 70-667 SP2010 exam got very positive reactions so in this post you will find a list of the resources I used to prepare for the second exam.
What you will need, according to my opinion is:
- Off course a lot of experience with SharePoint and Windows Infrastructure deployments (I have 10+ years Windows experience (I am also MCITP EA) and SharePoint 5+ years)
- Planning and architecture for SharePoint Server 2010 on Technet: http://technet.microsoft.com/nl-nl/library/cc261834(en-us).aspx (read a lot of articles and whitepapers)
- The book Microsoft SharePoint 2010 Administrator’s Companion
by Bill English, Brian Alderman & Mark Ferrarz
- SharePoint 2010 Advanced IT Professional Training: http://technet.microsoft.com/en-us/sharepoint/ff420396.aspx (I watched all vids on this site)
- Learning resources from the SharePoint 2010 Readiness site: http://www.sharepoint2010readiness.com/
