csrss.exe High I/O in combination with Terminal service

[Vincent]

Hi,

Currently I’m running into some issues which occur on each and every
windows 2003 server that I’m managing:

When someone logs in on a server the csrss.exe process starts to use up a
lot of disk I/O. It seems to have a minimum of 10 MB/s with a maximum of 80
MB/s although the average seems to be around 30 MB/s. The high I/O continues
until the user logs off. The high I/O also stops completely when (and this is
especially weird) the Terminal service client screen is minimized, but high
disk activity will resume as soon as the screen is restored.

I am aware of knowledge base article 934330, and even though the article in
question states the fix is foor the issue that crss.exe utilizes a lot of
cpu instead of high I/O, I decided to install the hotfix on a server to see
if the problems are related. Unfortunately it seems they’re not.

As I already said I'm experiencing this issue on multiple windows 2k3
server versions (web, standard, standard R2). They are all running service
pack 2 and are fully patched. The hardware differs as well. I've seen this
problem occur on Dell and HP servers so it is practically impossible that its
hardware related.

I'm at loss as to what the cause is. Does anybody have a clue?

Thanks in advance,
Vincent

[rpremuz]

Post by Vincent
Currently I’m running into some issues which occur on each and every
When someone logs in on a server the csrss.exe process starts to use up a
lot of disk I/O. It seems to have a minimum of 10 MB/s with a maximum of 80
MB/s although the average seems to be around 30 MB/s. The high I/O continues
until the user logs off. The high I/O also stops completely when (and this is
especially weird) the Terminal service client screen is minimized, but high
disk activity will resume as soon as the screen is restored.

Vincent, I don't know the solution to the problem but I can confirm
that I experience the same on two MS Windows Server 2003 SP2 (Standard
Edition) with all the latest patches (the servers don't have the same
hardware or software configuration).

Here is how I can get extremely high I/O for the csrss.exe: when I'm
connected to the server through the Remote Desktop I press and hold a
key (e.g. Shift or Alt) on the keyboard: the Process Explorer shows
that Other Bytes Delta is about 200 MB in that case. See picture on
http://www.hotshare.net/image/91284-64755441d6.html

This is really a major issue. It there any MSMVP who has seen that
before and knows the solution?

-- rpr.

[Robert Premuž]

Post by rpremuz

Post by Vincent
Currently I’m running into some issues which occur on each and every
When someone logs in on a server the csrss.exe process starts to use up a
lot of disk I/O. It seems to have a minimum of 10 MB/s with a maximum of 80
MB/s although the average seems to be around 30 MB/s. The high I/O continues
until the user logs off. The high I/O also stops completely when (and this is
especially weird) the Terminal service client screen is minimized, but high
disk activity will resume as soon as the screen is restored.

Vincent, I don't know the solution to the problem but I can confirm
that I experience the same on two MS Windows Server 2003 SP2 (Standard
Edition) with all the latest patches (the servers don't have the same
hardware or software configuration).
Here is how I can get extremely high I/O for the csrss.exe: when I'm
connected to the server through the Remote Desktop I press and hold a
key (e.g. Shift or Alt) on the keyboard: the Process Explorer shows
that Other Bytes Delta is about 200 MB in that case. See picture on
http://www.hotshare.net/image/91284-64755441d6.html
This is really a major issue. It there any MSMVP who has seen that
before and knows the solution?

I installed the 934330 hotfix received from MS Support (see
http://support.microsoft.com/kb/934330 that explains a similar problem
with csrss.exe) but it didn't solved the problem.

-- rpr.

[jolteroli]

Do the session, the "freaking csrss" belongs to, also have a process named
"ntvdm" running?

-jolt

Post by Robert Premuž

Post by rpremuz

Post by Vincent
Currently I’m running into some issues which occur on each and every
When someone logs in on a server the csrss.exe process starts to use up a
lot of disk I/O. It seems to have a minimum of 10 MB/s with a maximum of 80
MB/s although the average seems to be around 30 MB/s. The high I/O continues
until the user logs off. The high I/O also stops completely when (and this is
especially weird) the Terminal service client screen is minimized, but high
disk activity will resume as soon as the screen is restored.

Vincent, I don't know the solution to the problem but I can confirm
that I experience the same on two MS Windows Server 2003 SP2 (Standard
Edition) with all the latest patches (the servers don't have the same
hardware or software configuration).
Here is how I can get extremely high I/O for the csrss.exe: when I'm
connected to the server through the Remote Desktop I press and hold a
key (e.g. Shift or Alt) on the keyboard: the Process Explorer shows
that Other Bytes Delta is about 200 MB in that case. See picture on
http://www.hotshare.net/image/91284-64755441d6.html
This is really a major issue. It there any MSMVP who has seen that
before and knows the solution?

I installed the 934330 hotfix received from MS Support (see
http://support.microsoft.com/kb/934330 that explains a similar problem
with csrss.exe) but it didn't solved the problem.
-- rpr.

[rpremuz]

Post by jolteroli
Do the session, the "freaking csrss" belongs to, also have a process named
"ntvdm" running?

There is no ntvdm process running on the server.

-- rpr.

[jolteroli]

Not sure if this will help...

If you look in proc-explorer on the threads-tab in the csrss process, you
should find a thread causing all this IO and having a pretty high
cpu/context-switch-delta value.

If you suspend this thread does the IO drop? (Do this on the csrss of a
test-user session)

What is the "starting address" and the call [Stack] of the thread?

-jolt

Post by jolteroli
Do the session, the "freaking csrss" belongs to, also have a process named
"ntvdm" running?

There is no ntvdm process running on the server.

-- rpr.

[rpremuz]

Post by jolteroli
If you look in proc-explorer on the threads-tab in the csrss process, you
should find a thread causing all this IO and having a pretty high
cpu/context-switch-delta value.
If you suspend this thread does the IO drop? (Do this on the csrss of a
test-user session)
What is the "starting address" and the call [Stack] of the thread?

The thread of csrss.exe with the highest CSwitch Delta has the
following start address: winsrv.dll!ConServerDllInitialization+0x5866

If the thread is suspended, the I/O drops but the terminal session
screen freezes (no animation) and it becomes unresponsive to keyboard
and mouse inputs. If the thread is resumed (from the Process Explorer
running in another terminal session), the terminal session unfreezes
and high I/O problem starts again.

The call stack of the thread is the following:
0 ntoskrnl.exe+0x397ea
1 ntoskrnl.exe+0x3df9e
2 ntoskrnl.exe+0x50675
3 ntoskrnl.exe+0x40720
4 ntoskrnl.exe+0x3f326
5 win32k.sys+0x15287
6 win32k.sys+0x6fd25
7 win32k.sys+0x98a52
8 ntoskrnl.exe+0x33bef
9 ntdll.dll!KiFastSystemCallRet

The high I/O can be induced by keyboard or mouse activity.
Here is what a little testing on a server showed for the csrss.exe
corresponding to a terminal session:
-- the terminal session window restored or maximized
+ no mouse or keyboard activity -- Other Bytes Delta: 7 MB
+ mouse activity -- Other Bytes Delta: 50 - 70 MB
+ keyboard activity -- Other Bytes Delta: 150 - 200 MB
-- the terminal session window minimized -- Other Bytes Delta: 0

If the server console is used for log on, there is no high I/O for the
corresponding csrss.exe process.

-- rpr.

[a***@gmail.com]

Hi.

I have no solution, but same issue on three w2k3 enterprise R2 sp2.
I try patch from microsoft kb 934330 without success.

Some ideas?

Andrea

Post by rpremuz

Post by jolteroli
If you look in proc-explorer on the threads-tab in the csrss process, you
should find a thread causing all this IO and having a pretty high
cpu/context-switch-delta value.
If you suspend this thread does the IO drop? (Do this on the csrss of a
test-user session)
What is the "starting address" and the call [Stack] of the thread?

The thread of csrss.exe with the highest CSwitch Delta has the
following start address: winsrv.dll!ConServerDllInitialization+0x5866
If the thread is suspended, the I/O drops but the terminal session
screen freezes (no animation) and it becomes unresponsive to keyboard
and mouse inputs. If the thread is resumed (from the Process Explorer
running in another terminal session), the terminal session unfreezes
and high I/O problem starts again.
0 ntoskrnl.exe+0x397ea
1 ntoskrnl.exe+0x3df9e
2 ntoskrnl.exe+0x50675
3 ntoskrnl.exe+0x40720
4 ntoskrnl.exe+0x3f326
5 win32k.sys+0x15287
6 win32k.sys+0x6fd25
7 win32k.sys+0x98a52
8 ntoskrnl.exe+0x33bef
9 ntdll.dll!KiFastSystemCallRet
The high I/O can be induced by keyboard or mouse activity.
Here is what a little testing on a server showed for the csrss.exe
-- the terminal session window restored or maximized
  + no mouse or keyboard activity -- Other Bytes Delta: 7 MB
  + mouse activity -- Other Bytes Delta: 50 - 70 MB
  + keyboard activity -- Other Bytes Delta: 150 - 200 MB
-- the terminal session window minimized -- Other Bytes Delta: 0
If the server console is used for log on, there is no high I/O for the
corresponding csrss.exe process.
-- rpr.

[jolteroli]

This post might be inappropriate. Click to display it.

[rpr]

This post might be inappropriate. Click to display it.

[jolteroli]

Hee Rob,

Post by rpr
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session
Manager \Environment" /v _NT_SYMBOL_PATH /d
"SRV*c:\windows\symbols*http://msdl.microsoft.com/download/symbols" /f

Real men don't click, cool... :)

Post by rpr
ntoskrnl.exe!KiSwapContext+0x26
ntoskrnl.exe!KiSwapThread+0x2e5
ntoskrnl.exe!KeWaitForSingleObject+0x346
ntoskrnl.exe!KiSuspendThread+0x18
ntoskrnl.exe!KiDeliverApc+0x117
ntoskrnl.exe!KiSwapThread+0x300
ntoskrnl.exe!KeWaitForMultipleObjects+0x3d7
win32k.sys!RawInputThread+0x4e0
win32k.sys!xxxCreateSystemThreads+0x60
win32k.sys!NtUserCallOneParam+0x23
ntoskrnl.exe!KiFastCallEntry+0xfc
ntdll.dll!KiFastSystemCallRet
winsrv.dll!NtUserCallOneParam+0xc
Does this tell you anything?

Since our servers (not having this issue) doesn't fork to KiDeliverApc, I
guess there is constantly arriving keyb data (that is what RawInputThread
RIT "polls" for) and that's causing the IO. I have no clue what is causing
the keyb data...

In german we have the Systemcontrol "Eingabehilfen" with the symbol of Steve
Balmers wheelchair. Check if any enhanced services is turned on there...

ATI's stupid HotKeyPolling service comes to mind too...

May be something in this direction. For me it seems the data comes from the
keyb/driver.

Post by rpr
To me it tells nothing as I don't have the sources :-| (Almost a complete
waste of time.)

Any additional help would need a debugger, so it was a waste of time. Sorry
Rob.

-jolt

[rpr]

Post by jolteroli

Post by rpr
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session
Manager \Environment" /v _NT_SYMBOL_PATH /d
"SRV*c:\windows\symbols*http://msdl.microsoft.com/download/symbols" /f

Real men don't click, cool... :)

I also live in Unix world and hence I'm familiar with the command
line. Unfortunately MS KBAs rarely suggest using command line tools.
Fortunately we have this groups for writing our own knowledge base
articles :-)

Post by jolteroli
Since our servers (not having this issue) doesn't fork to KiDeliverApc, I
guess there is constantly arriving keyb data (that is what RawInputThread
RIT "polls" for) and that's causing the IO. I have no clue what is causing
the keyb data...
In german we have the Systemcontrol "Eingabehilfen" with the symbol of Steve
Balmers wheelchair. Check if any enhanced services is turned on there...
ATI's stupid HotKeyPolling service comes to mind too...
May be something in this direction. For me it seems the data comes from the
keyb/driver.

Any of the Accessibility Options is not enabled on the server.
Also, no ATI services.

-- rpr.

[TP]

Robert and Vincent,

What are the exact problem symptoms you are experiencing? Is
the server running slow? Are users complaining? If yes, what
lead you to the conclusion that this particular reading is the
source of the problems you are having?

I have looked at the numbers you are seeing and this must be
a bug. I see this on every 2003 server I have examined so far
(since reading your post). If these numbers were true the
servers would grind to a halt.

Thanks.

-TP

Post by Robert Premuž

Post by rpremuz

Post by Vincent
Currently I’m running into some issues which occur on each and
When someone logs in on a server the csrss.exe process starts to
use up a lot of disk I/O. It seems to have a minimum of 10 MB/s
with a maximum of 80 MB/s although the average seems to be around
30 MB/s. The high I/O continues until the user logs off. The high
I/O also stops completely when (and this is especially weird) the
Terminal service client screen is minimized, but high disk activity
will resume as soon as the screen is restored.

Vincent, I don't know the solution to the problem but I can confirm
that I experience the same on two MS Windows Server 2003 SP2
(Standard Edition) with all the latest patches (the servers don't
have the same hardware or software configuration).
Here is how I can get extremely high I/O for the csrss.exe: when I'm
connected to the server through the Remote Desktop I press and hold a
key (e.g. Shift or Alt) on the keyboard: the Process Explorer shows
that Other Bytes Delta is about 200 MB in that case. See picture on
http://www.hotshare.net/image/91284-64755441d6.html
This is really a major issue. It there any MSMVP who has seen that
before and knows the solution?

I installed the 934330 hotfix received from MS Support (see
http://support.microsoft.com/kb/934330 that explains a similar problem
with csrss.exe) but it didn't solved the problem.
-- rpr.

[rpr]

I discovered the high I/O problem in csrss.exe while investigating a
problem with the system clock on one of DCs in the domain. The
server's system clock is quite inaccurate.
I tried to solve that by reconfiguring the Windows Time Service
(w32time) using the procedures suggested at
http://technet.microsoft.com/en-us/library/cc773061.aspx
but it was not satisfactory as the Windows Time Service cannot take
into account the clock drift (as a good NTP service should).

At http://answers.google.com/answers/threadview?id=438195
you can find complains that suggest this high I/O slows down the
Windows Server 2003. Note that the question is written on 2006-01-26.

At http://forum.sysinternals.com/forum_posts.asp?TID=12894
Mark Russinovich said the following regarding this issue on
2008-03-06:
"This behavior is a known bug in the w2k3 terminal server I/O
counters."
But it was not explained any further.

Even if this bug causes no real harm it should have been solved long
ago. It certainly creates confusion and increases dissatisfaction with
MS software.

-- rpr.

Post by TP
Robert and Vincent,
What are the exact problem symptoms you are experiencing?  Is
the server running slow?  Are users complaining?  If yes, what
lead you to the conclusion that this particular reading is the
source of the problems you are having?
I have looked at the numbers you are seeing and this must be
a bug.  I see this on every 2003 server I have examined so far
(since reading your post).  If these numbers were true the
servers would grind to a halt.
Thanks.
-TP

[unknown]

Found your post while trying to understand Microsoft Office Server 2007 consumption of resources (MOSS 2007\SharePoint).

My SharePoint Central Administration Server is a W2k3 R2 SP2 Std Ed X64 server with 8GB of memory.
My IO Delta climbs at times from 64KB to 84GB (not a typo ...84GB!)this is for the CRSS.exe (Client Server Runtime Process) when I view it in the latest version of Process Explorer.

My "I\O Other Bytes" is reading 19 Trillion+, equivelant to 19 Tereabytes????. The server itself is on a 30GB and 40GB partitions so the 19Trillion Terabytes may be I\O's from reading the SQL DB's its attached to which reside on another server. But the SQL db's its talking to total less than 200GB?

dumb founded....this server is showing very few eventlog errors and was only rebooted 12 hours ago.

In Process Explorer I'm also seeing a lot (as in hundreds) of "Contentions" for different instancesof the MS Seach Component mssdmn.exe

MSSearch.exe is also racking up a significant amount or PageFaults??

Will be openign a case with MS as soon as schedule permits.

[kC C]

hi Guys just a quick question to you all suffering this issue, wheres your Page filing pointing to and whats it set to?

Hi,
Currently I’m running into some issues which occur on each and every
When someone logs in on a server the csrss.exe process starts to use up a
lot of disk I/O. It seems to have a minimum of 10 MB/s with a maximum of 80
MB/s although the average seems to be around 30 MB/s. The high I/O continues
until the user logs off. The high I/O also stops completely when (and this is
especially weird) the Terminal service client screen is minimized, but high
disk activity will resume as soon as the screen is restored.
I am aware of knowledge base article 934330, and even though the article in
question states the fix is foor the issue that crss.exe utilizes a lot of
cpu instead of high I/O, I decided to install the hotfix on a server to see
if the problems are related. Unfortunately it seems they’re not.
As I already said I'm experiencing this issue on multiple windows 2k3
server versions (web, standard, standard R2). They are all running service
pack 2 and are fully patched. The hardware differs as well. I've seen this
problem occur on Dell and HP servers so it is practically impossible that its
hardware related.
I'm at loss as to what the cause is. Does anybody have a clue?
Thanks in advance,
Vincent

Post by rpremuz
a
80
ues
s is
gh
Vincent, I don't know the solution to the problem but I can confirm
that I experience the same on two MS Windows Server 2003 SP2 (Standard
Edition) with all the latest patches (the servers don't have the same
hardware or software configuration).
Here is how I can get extremely high I/O for the csrss.exe: when I'm
connected to the server through the Remote Desktop I press and hold a
key (e.g. Shift or Alt) on the keyboard: the Process Explorer shows
that Other Bytes Delta is about 200 MB in that case. See picture on
http://www.hotshare.net/image/91284-64755441d6.html
This is really a major issue. It there any MSMVP who has seen that
before and knows the solution?
-- rpr.

Post by Robert Premuž
I installed the 934330 hotfix received from MS Support (see
http://support.microsoft.com/kb/934330 that explains a similar problem
with csrss.exe) but it didn't solved the problem.
-- rpr.

Post by jolteroli
Do the session, the "freaking csrss" belongs to, also have a process named
"ntvdm" running?
-jolt

Post by jolteroli
Not sure if this will help...
If you look in proc-explorer on the threads-tab in the csrss process, you
should find a thread causing all this IO and having a pretty high
cpu/context-switch-delta value.
If you suspend this thread does the IO drop? (Do this on the csrss of a
test-user session)
What is the "starting address" and the call [Stack] of the thread?
-jolt
There is no ntvdm process running on the server.
-- rpr.

Post by rpremuz
d
There is no ntvdm process running on the server.
-- rpr.

Post by rpremuz
The thread of csrss.exe with the highest CSwitch Delta has the
following start address: winsrv.dll!ConServerDllInitialization+0x5866
If the thread is suspended, the I/O drops but the terminal session
screen freezes (no animation) and it becomes unresponsive to keyboard
and mouse inputs. If the thread is resumed (from the Process Explorer
running in another terminal session), the terminal session unfreezes
and high I/O problem starts again.
0 ntoskrnl.exe+0x397ea
1 ntoskrnl.exe+0x3df9e
2 ntoskrnl.exe+0x50675
3 ntoskrnl.exe+0x40720
4 ntoskrnl.exe+0x3f326
5 win32k.sys+0x15287
6 win32k.sys+0x6fd25
7 win32k.sys+0x98a52
8 ntoskrnl.exe+0x33bef
9 ntdll.dll!KiFastSystemCallRet
The high I/O can be induced by keyboard or mouse activity.
Here is what a little testing on a server showed for the csrss.exe
-- the terminal session window restored or maximized
+ no mouse or keyboard activity -- Other Bytes Delta: 7 MB
+ mouse activity -- Other Bytes Delta: 50 - 70 MB
+ keyboard activity -- Other Bytes Delta: 150 - 200 MB
-- the terminal session window minimized -- Other Bytes Delta: 0
If the server console is used for log on, there is no high I/O for the
corresponding csrss.exe process.
-- rpr.

Post by a***@gmail.com
Hi.
I have no solution, but same issue on three w2k3 enterprise R2 sp2.
I try patch from microsoft kb 934330 without success.
Some ideas?
Andrea
ou

Post by jolteroli
Hey Robert,
I don't want bother you, but you've not configured symbols. If you want to
1.) Install latest windbg.
2.) Copy latest dbghelp.dll (in the windbg directory) to system32
3.) Create a directory for symbol files, e.g.: f:\symbols
_NT_SYMBOL_PATH=SRV*f:\symbols*http://msdl.microsoft.com/download/symbols
symchk c:\winnt\system32\winsrv.dll
symchk c:\winnt\system32\csrss.exe
symchk c:\winnt\system32\win32k.sys
symchk c:\winnt\system32\ntkrnpa.exe
symchk c:\winnt\system32\ntdll.dll
... and so on...
symchk /r c:\winnt\system32\*
6.) In the Process Explorer, configure the symbol path, to f:\symbols
You can put
dbgeng.dll
dbghelp.dll
SymbolCheck.dll
symchk.exe
symsrv.dll
on a USB pen. So you don't need the Debugging Tools any time.
ntkrnlpa.exe!KiSwapContext+0x2f
ntkrnlpa.exe!KiSwapThread+0x8a
ntkrnlpa.exe!KeWaitForMultipleObjects+0x284
win32k.sys!RawInputThread+0x4f3
win32k.sys!xxxCreateSystemThreads+0x60
win32k.sys!NtUserCallOneParam+0x23
ntkrnlpa.exe!KiFastCallEntry+0xfc
ntdll.dll!KiFastSystemCallRet
winsrv.dll!NtUserCallOneParam+0xc
And you can see which functions are involved in the freaking thread. I would
start and view the stack traces
Once, when the RDP window is minimized has no IO
Once, when the RDP window is maximized and produce IO
Any difference?
-jolt

Post by jolteroli
Hee Rob,
Real men don't click, cool... :)
Since our servers (not having this issue) doesn't fork to KiDeliverApc, I
guess there is constantly arriving keyb data (that is what RawInputThread
RIT "polls" for) and that's causing the IO. I have no clue what is causing
the keyb data...
In german we have the Systemcontrol "Eingabehilfen" with the symbol of Steve
Balmers wheelchair. Check if any enhanced services is turned on there...
ATI's stupid HotKeyPolling service comes to mind too...
May be something in this direction. For me it seems the data comes from the
keyb/driver.
Any additional help would need a debugger, so it was a waste of time. Sorry
Rob.
-jolt

Post by TP
Robert and Vincent,
What are the exact problem symptoms you are experiencing? Is
the server running slow? Are users complaining? If yes, what
lead you to the conclusion that this particular reading is the
source of the problems you are having?
I have looked at the numbers you are seeing and this must be
a bug. I see this on every 2003 server I have examined so far
(since reading your post). If these numbers were true the
servers would grind to a halt.
Thanks.
-TP

Post by rpr
Hi, jolteroli!
As you suggested I installed Debugging Tools for Windows 32-bit
Version
(from http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
).
To replace "%SystemRoot%\system32\dbghelp.dll", which is in use by
some running processes, with the newer version in
"%ProgramFiles%\Debugging Tools for Windows (x86)\" you need to use
one of the methods explained at http://support.microsoft.com/kb/181345
mkdir c:\windows\symbols
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager
\Environment" /v _NT_SYMBOL_PATH /d "SRV*c:\windows\symbols*http://
msdl.microsoft.com/download/symbols" /f
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\winsrv.dll"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\csrss.exe"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\ntoskrnl.exe"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\win32k.sys"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\ntdll.dll"
After restarting the server I got the following data from
When the RDP window is minimized, the I/O is normal.
The thread of csrss.exe with the highest CSwitch Delta has the
following start address: winsrv.dll!StartCreateSystemThreads
ntoskrnl.exe!KiSwapContext+0x26
ntoskrnl.exe!KiSwapThread+0x2e5
ntoskrnl.exe!KeWaitForSingleObject+0x346
ntoskrnl.exe!KiSuspendThread+0x18
ntoskrnl.exe!KiDeliverApc+0x117
ntoskrnl.exe!KiSwapThread+0x300
ntoskrnl.exe!KeWaitForMultipleObjects+0x3d7
win32k.sys!RawInputThread+0x4e0
win32k.sys!xxxCreateSystemThreads+0x60
win32k.sys!NtUserCallOneParam+0x23
ntoskrnl.exe!KiFastCallEntry+0xfc
ntdll.dll!KiFastSystemCallRet
winsrv.dll!NtUserCallOneParam+0xc
When the RDP window is restored or maximized, the I/O is very high.
But the thread of csrss.exe with the highest CSwitch Delta has the
start address and call stack as above (no difference when compared to
the normal state).
Does this tell you anything? To me it tells nothing as I don't have
the sources :-| (Almost a complete waste of time.)
-- rpr.
o
ad/symbols
uld

Post by rpr
I also live in Unix world and hence I'm familiar with the command
line. Unfortunately MS KBAs rarely suggest using command line tools.
Fortunately we have this groups for writing our own knowledge base
articles :-)
g
eve
he
Any of the Accessibility Options is not enabled on the server.
Also, no ATI services.
-- rpr.

Post by rpr
I discovered the high I/O problem in csrss.exe while investigating a
problem with the system clock on one of DCs in the domain. The
server's system clock is quite inaccurate.
I tried to solve that by reconfiguring the Windows Time Service
(w32time) using the procedures suggested at
http://technet.microsoft.com/en-us/library/cc773061.aspx
but it was not satisfactory as the Windows Time Service cannot take
into account the clock drift (as a good NTP service should).
At http://answers.google.com/answers/threadview?id=3D438195
you can find complains that suggest this high I/O slows down the
Windows Server 2003. Note that the question is written on 2006-01-26.
At http://forum.sysinternals.com/forum_posts.asp?TID=3D12894
Mark Russinovich said the following regarding this issue on
"This behavior is a known bug in the w2k3 terminal server I/O
counters."
But it was not explained any further.
Even if this bug causes no real harm it should have been solved long
ago. It certainly creates confusion and increases dissatisfaction with
MS software.
-- rpr.

Post by unknown
Found your post while trying to understand Microsoft Office Server 2007 consumption of resources (MOSS 2007\SharePoint).
My SharePoint Central Administration Server is a W2k3 R2 SP2 Std Ed X64 server with 8GB of memory.
My IO Delta climbs at times from 64KB to 84GB (not a typo ...84GB!)this is for the CRSS.exe (Client Server Runtime Process) when I view it in the latest version of Process Explorer.
My "I\O Other Bytes" is reading 19 Trillion+, equivelant to 19 Tereabytes????. The server itself is on a 30GB and 40GB partitions so the 19Trillion Terabytes may be I\O's from reading the SQL DB's its attached to which reside on another server. But the SQL db's its talking to total less than 200GB?
dumb founded....this server is showing very few eventlog errors and was only rebooted 12 hours ago.
In Process Explorer I'm also seeing a lot (as in hundreds) of "Contentions" for different instancesof the MS Seach Component mssdmn.exe
MSSearch.exe is also racking up a significant amount or PageFaults??
Will be openign a case with MS as soon as schedule permits.

[David Ludvik]

Hi Guys,
this hot-fix should fix the issue
http://support.microsoft.com/kb/956438

Hi,
Currently I’m running into some issues which occur on each and every
When someone logs in on a server the csrss.exe process starts to use up a
lot of disk I/O. It seems to have a minimum of 10 MB/s with a maximum of 80
MB/s although the average seems to be around 30 MB/s. The high I/O continues
until the user logs off. The high I/O also stops completely when (and this is
especially weird) the Terminal service client screen is minimized, but high
disk activity will resume as soon as the screen is restored.
I am aware of knowledge base article 934330, and even though the article in
question states the fix is foor the issue that crss.exe utilizes a lot of
cpu instead of high I/O, I decided to install the hotfix on a server to see
if the problems are related. Unfortunately it seems they’re not.
As I already said I'm experiencing this issue on multiple windows 2k3
server versions (web, standard, standard R2). They are all running service
pack 2 and are fully patched. The hardware differs as well. I've seen this
problem occur on Dell and HP servers so it is practically impossible that its
hardware related.
I'm at loss as to what the cause is. Does anybody have a clue?
Thanks in advance,
Vincent

Post by rpremuz
a
80
ues
s is
gh
Vincent, I don't know the solution to the problem but I can confirm
that I experience the same on two MS Windows Server 2003 SP2 (Standard
Edition) with all the latest patches (the servers don't have the same
hardware or software configuration).
Here is how I can get extremely high I/O for the csrss.exe: when I'm
connected to the server through the Remote Desktop I press and hold a
key (e.g. Shift or Alt) on the keyboard: the Process Explorer shows
that Other Bytes Delta is about 200 MB in that case. See picture on
http://www.hotshare.net/image/91284-64755441d6.html
This is really a major issue. It there any MSMVP who has seen that
before and knows the solution?
-- rpr.

Post by Robert Premuž
I installed the 934330 hotfix received from MS Support (see
http://support.microsoft.com/kb/934330 that explains a similar problem
with csrss.exe) but it didn't solved the problem.
-- rpr.

Post by jolteroli
Do the session, the "freaking csrss" belongs to, also have a process named
"ntvdm" running?
-jolt

Post by jolteroli
Not sure if this will help...
If you look in proc-explorer on the threads-tab in the csrss process, you
should find a thread causing all this IO and having a pretty high
cpu/context-switch-delta value.
If you suspend this thread does the IO drop? (Do this on the csrss of a
test-user session)
What is the "starting address" and the call [Stack] of the thread?
-jolt
There is no ntvdm process running on the server.
-- rpr.

Post by rpremuz
d
There is no ntvdm process running on the server.
-- rpr.

Post by rpremuz
The thread of csrss.exe with the highest CSwitch Delta has the
following start address: winsrv.dll!ConServerDllInitialization+0x5866
If the thread is suspended, the I/O drops but the terminal session
screen freezes (no animation) and it becomes unresponsive to keyboard
and mouse inputs. If the thread is resumed (from the Process Explorer
running in another terminal session), the terminal session unfreezes
and high I/O problem starts again.
0 ntoskrnl.exe+0x397ea
1 ntoskrnl.exe+0x3df9e
2 ntoskrnl.exe+0x50675
3 ntoskrnl.exe+0x40720
4 ntoskrnl.exe+0x3f326
5 win32k.sys+0x15287
6 win32k.sys+0x6fd25
7 win32k.sys+0x98a52
8 ntoskrnl.exe+0x33bef
9 ntdll.dll!KiFastSystemCallRet
The high I/O can be induced by keyboard or mouse activity.
Here is what a little testing on a server showed for the csrss.exe
-- the terminal session window restored or maximized
+ no mouse or keyboard activity -- Other Bytes Delta: 7 MB
+ mouse activity -- Other Bytes Delta: 50 - 70 MB
+ keyboard activity -- Other Bytes Delta: 150 - 200 MB
-- the terminal session window minimized -- Other Bytes Delta: 0
If the server console is used for log on, there is no high I/O for the
corresponding csrss.exe process.
-- rpr.

Post by a***@gmail.com
Hi.
I have no solution, but same issue on three w2k3 enterprise R2 sp2.
I try patch from microsoft kb 934330 without success.
Some ideas?
Andrea
ou

Post by jolteroli
Hey Robert,
I don't want bother you, but you've not configured symbols. If you want to
1.) Install latest windbg.
2.) Copy latest dbghelp.dll (in the windbg directory) to system32
3.) Create a directory for symbol files, e.g.: f:\symbols
_NT_SYMBOL_PATH=SRV*f:\symbols*http://msdl.microsoft.com/download/symbols
symchk c:\winnt\system32\winsrv.dll
symchk c:\winnt\system32\csrss.exe
symchk c:\winnt\system32\win32k.sys
symchk c:\winnt\system32\ntkrnpa.exe
symchk c:\winnt\system32\ntdll.dll
... and so on...
symchk /r c:\winnt\system32\*
6.) In the Process Explorer, configure the symbol path, to f:\symbols
You can put
dbgeng.dll
dbghelp.dll
SymbolCheck.dll
symchk.exe
symsrv.dll
on a USB pen. So you don't need the Debugging Tools any time.
ntkrnlpa.exe!KiSwapContext+0x2f
ntkrnlpa.exe!KiSwapThread+0x8a
ntkrnlpa.exe!KeWaitForMultipleObjects+0x284
win32k.sys!RawInputThread+0x4f3
win32k.sys!xxxCreateSystemThreads+0x60
win32k.sys!NtUserCallOneParam+0x23
ntkrnlpa.exe!KiFastCallEntry+0xfc
ntdll.dll!KiFastSystemCallRet
winsrv.dll!NtUserCallOneParam+0xc
And you can see which functions are involved in the freaking thread. I would
start and view the stack traces
Once, when the RDP window is minimized has no IO
Once, when the RDP window is maximized and produce IO
Any difference?
-jolt

Post by jolteroli
Hee Rob,
Real men don't click, cool... :)
Since our servers (not having this issue) doesn't fork to KiDeliverApc, I
guess there is constantly arriving keyb data (that is what RawInputThread
RIT "polls" for) and that's causing the IO. I have no clue what is causing
the keyb data...
In german we have the Systemcontrol "Eingabehilfen" with the symbol of Steve
Balmers wheelchair. Check if any enhanced services is turned on there...
ATI's stupid HotKeyPolling service comes to mind too...
May be something in this direction. For me it seems the data comes from the
keyb/driver.
Any additional help would need a debugger, so it was a waste of time. Sorry
Rob.
-jolt

Post by TP
Robert and Vincent,
What are the exact problem symptoms you are experiencing? Is
the server running slow? Are users complaining? If yes, what
lead you to the conclusion that this particular reading is the
source of the problems you are having?
I have looked at the numbers you are seeing and this must be
a bug. I see this on every 2003 server I have examined so far
(since reading your post). If these numbers were true the
servers would grind to a halt.
Thanks.
-TP

Post by rpr
Hi, jolteroli!
As you suggested I installed Debugging Tools for Windows 32-bit
Version
(from http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
).
To replace "%SystemRoot%\system32\dbghelp.dll", which is in use by
some running processes, with the newer version in
"%ProgramFiles%\Debugging Tools for Windows (x86)\" you need to use
one of the methods explained at http://support.microsoft.com/kb/181345
mkdir c:\windows\symbols
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager
\Environment" /v _NT_SYMBOL_PATH /d "SRV*c:\windows\symbols*http://
msdl.microsoft.com/download/symbols" /f
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\winsrv.dll"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\csrss.exe"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\ntoskrnl.exe"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\win32k.sys"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\ntdll.dll"
After restarting the server I got the following data from
When the RDP window is minimized, the I/O is normal.
The thread of csrss.exe with the highest CSwitch Delta has the
following start address: winsrv.dll!StartCreateSystemThreads
ntoskrnl.exe!KiSwapContext+0x26
ntoskrnl.exe!KiSwapThread+0x2e5
ntoskrnl.exe!KeWaitForSingleObject+0x346
ntoskrnl.exe!KiSuspendThread+0x18
ntoskrnl.exe!KiDeliverApc+0x117
ntoskrnl.exe!KiSwapThread+0x300
ntoskrnl.exe!KeWaitForMultipleObjects+0x3d7
win32k.sys!RawInputThread+0x4e0
win32k.sys!xxxCreateSystemThreads+0x60
win32k.sys!NtUserCallOneParam+0x23
ntoskrnl.exe!KiFastCallEntry+0xfc
ntdll.dll!KiFastSystemCallRet
winsrv.dll!NtUserCallOneParam+0xc
When the RDP window is restored or maximized, the I/O is very high.
But the thread of csrss.exe with the highest CSwitch Delta has the
start address and call stack as above (no difference when compared to
the normal state).
Does this tell you anything? To me it tells nothing as I don't have
the sources :-| (Almost a complete waste of time.)
-- rpr.
o
ad/symbols
uld

Post by rpr
I also live in Unix world and hence I'm familiar with the command
line. Unfortunately MS KBAs rarely suggest using command line tools.
Fortunately we have this groups for writing our own knowledge base
articles :-)
g
eve
he
Any of the Accessibility Options is not enabled on the server.
Also, no ATI services.
-- rpr.

Post by rpr
I discovered the high I/O problem in csrss.exe while investigating a
problem with the system clock on one of DCs in the domain. The
server's system clock is quite inaccurate.
I tried to solve that by reconfiguring the Windows Time Service
(w32time) using the procedures suggested at
http://technet.microsoft.com/en-us/library/cc773061.aspx
but it was not satisfactory as the Windows Time Service cannot take
into account the clock drift (as a good NTP service should).
At http://answers.google.com/answers/threadview?id=3D438195
you can find complains that suggest this high I/O slows down the
Windows Server 2003. Note that the question is written on 2006-01-26.
At http://forum.sysinternals.com/forum_posts.asp?TID=3D12894
Mark Russinovich said the following regarding this issue on
"This behavior is a known bug in the w2k3 terminal server I/O
counters."
But it was not explained any further.
Even if this bug causes no real harm it should have been solved long
ago. It certainly creates confusion and increases dissatisfaction with
MS software.
-- rpr.

Post by unknown
Found your post while trying to understand Microsoft Office Server 2007 consumption of resources (MOSS 2007\SharePoint).
My SharePoint Central Administration Server is a W2k3 R2 SP2 Std Ed X64 server with 8GB of memory.
My IO Delta climbs at times from 64KB to 84GB (not a typo ...84GB!)this is for the CRSS.exe (Client Server Runtime Process) when I view it in the latest version of Process Explorer.
My "I\O Other Bytes" is reading 19 Trillion+, equivelant to 19 Tereabytes????. The server itself is on a 30GB and 40GB partitions so the 19Trillion Terabytes may be I\O's from reading the SQL DB's its attached to which reside on another server. But the SQL db's its talking to total less than 200GB?
dumb founded....this server is showing very few eventlog errors and was only rebooted 12 hours ago.
In Process Explorer I'm also seeing a lot (as in hundreds) of "Contentions" for different instancesof the MS Seach Component mssdmn.exe
MSSearch.exe is also racking up a significant amount or PageFaults??
Will be openign a case with MS as soon as schedule permits.

Post by kC C
hi Guys just a quick question to you all suffering this issue, wheres your Page filing pointing to and whats it set to?

[r***@gmail.com]

Post by David Ludvik
Hi Guys,
this hot-fix should fix the issue
http://support.microsoft.com/kb/956438

Thanks. I can confirm that the 956438 hot-fix has fixed the issue.

-- rpr.

[David Ludvik]

Hi Guys,
This should fix the issue.

http://support.microsoft.com/kb/956438

Hi,
Currently I’m running into some issues which occur on each and every
When someone logs in on a server the csrss.exe process starts to use up a
lot of disk I/O. It seems to have a minimum of 10 MB/s with a maximum of 80
MB/s although the average seems to be around 30 MB/s. The high I/O continues
until the user logs off. The high I/O also stops completely when (and this is
especially weird) the Terminal service client screen is minimized, but high
disk activity will resume as soon as the screen is restored.
I am aware of knowledge base article 934330, and even though the article in
question states the fix is foor the issue that crss.exe utilizes a lot of
cpu instead of high I/O, I decided to install the hotfix on a server to see
if the problems are related. Unfortunately it seems they’re not.
As I already said I'm experiencing this issue on multiple windows 2k3
server versions (web, standard, standard R2). They are all running service
pack 2 and are fully patched. The hardware differs as well. I've seen this
problem occur on Dell and HP servers so it is practically impossible that its
hardware related.
I'm at loss as to what the cause is. Does anybody have a clue?
Thanks in advance,
Vincent

Post by rpremuz
a
80
ues
s is
gh
Vincent, I don't know the solution to the problem but I can confirm
that I experience the same on two MS Windows Server 2003 SP2 (Standard
Edition) with all the latest patches (the servers don't have the same
hardware or software configuration).
Here is how I can get extremely high I/O for the csrss.exe: when I'm
connected to the server through the Remote Desktop I press and hold a
key (e.g. Shift or Alt) on the keyboard: the Process Explorer shows
that Other Bytes Delta is about 200 MB in that case. See picture on
http://www.hotshare.net/image/91284-64755441d6.html
This is really a major issue. It there any MSMVP who has seen that
before and knows the solution?
-- rpr.

Post by Robert Premuž
I installed the 934330 hotfix received from MS Support (see
http://support.microsoft.com/kb/934330 that explains a similar problem
with csrss.exe) but it didn't solved the problem.
-- rpr.

Post by jolteroli
Do the session, the "freaking csrss" belongs to, also have a process named
"ntvdm" running?
-jolt

Post by jolteroli
Not sure if this will help...
If you look in proc-explorer on the threads-tab in the csrss process, you
should find a thread causing all this IO and having a pretty high
cpu/context-switch-delta value.
If you suspend this thread does the IO drop? (Do this on the csrss of a
test-user session)
What is the "starting address" and the call [Stack] of the thread?
-jolt
There is no ntvdm process running on the server.
-- rpr.

Post by rpremuz
d
There is no ntvdm process running on the server.
-- rpr.

Post by rpremuz
The thread of csrss.exe with the highest CSwitch Delta has the
following start address: winsrv.dll!ConServerDllInitialization+0x5866
If the thread is suspended, the I/O drops but the terminal session
screen freezes (no animation) and it becomes unresponsive to keyboard
and mouse inputs. If the thread is resumed (from the Process Explorer
running in another terminal session), the terminal session unfreezes
and high I/O problem starts again.
0 ntoskrnl.exe+0x397ea
1 ntoskrnl.exe+0x3df9e
2 ntoskrnl.exe+0x50675
3 ntoskrnl.exe+0x40720
4 ntoskrnl.exe+0x3f326
5 win32k.sys+0x15287
6 win32k.sys+0x6fd25
7 win32k.sys+0x98a52
8 ntoskrnl.exe+0x33bef
9 ntdll.dll!KiFastSystemCallRet
The high I/O can be induced by keyboard or mouse activity.
Here is what a little testing on a server showed for the csrss.exe
-- the terminal session window restored or maximized
+ no mouse or keyboard activity -- Other Bytes Delta: 7 MB
+ mouse activity -- Other Bytes Delta: 50 - 70 MB
+ keyboard activity -- Other Bytes Delta: 150 - 200 MB
-- the terminal session window minimized -- Other Bytes Delta: 0
If the server console is used for log on, there is no high I/O for the
corresponding csrss.exe process.
-- rpr.

Post by a***@gmail.com
Hi.
I have no solution, but same issue on three w2k3 enterprise R2 sp2.
I try patch from microsoft kb 934330 without success.
Some ideas?
Andrea
ou

Post by jolteroli
Hey Robert,
I don't want bother you, but you've not configured symbols. If you want to
1.) Install latest windbg.
2.) Copy latest dbghelp.dll (in the windbg directory) to system32
3.) Create a directory for symbol files, e.g.: f:\symbols
_NT_SYMBOL_PATH=SRV*f:\symbols*http://msdl.microsoft.com/download/symbols
symchk c:\winnt\system32\winsrv.dll
symchk c:\winnt\system32\csrss.exe
symchk c:\winnt\system32\win32k.sys
symchk c:\winnt\system32\ntkrnpa.exe
symchk c:\winnt\system32\ntdll.dll
... and so on...
symchk /r c:\winnt\system32\*
6.) In the Process Explorer, configure the symbol path, to f:\symbols
You can put
dbgeng.dll
dbghelp.dll
SymbolCheck.dll
symchk.exe
symsrv.dll
on a USB pen. So you don't need the Debugging Tools any time.
ntkrnlpa.exe!KiSwapContext+0x2f
ntkrnlpa.exe!KiSwapThread+0x8a
ntkrnlpa.exe!KeWaitForMultipleObjects+0x284
win32k.sys!RawInputThread+0x4f3
win32k.sys!xxxCreateSystemThreads+0x60
win32k.sys!NtUserCallOneParam+0x23
ntkrnlpa.exe!KiFastCallEntry+0xfc
ntdll.dll!KiFastSystemCallRet
winsrv.dll!NtUserCallOneParam+0xc
And you can see which functions are involved in the freaking thread. I would
start and view the stack traces
Once, when the RDP window is minimized has no IO
Once, when the RDP window is maximized and produce IO
Any difference?
-jolt

Post by jolteroli
Hee Rob,
Real men don't click, cool... :)
Since our servers (not having this issue) doesn't fork to KiDeliverApc, I
guess there is constantly arriving keyb data (that is what RawInputThread
RIT "polls" for) and that's causing the IO. I have no clue what is causing
the keyb data...
In german we have the Systemcontrol "Eingabehilfen" with the symbol of Steve
Balmers wheelchair. Check if any enhanced services is turned on there...
ATI's stupid HotKeyPolling service comes to mind too...
May be something in this direction. For me it seems the data comes from the
keyb/driver.
Any additional help would need a debugger, so it was a waste of time. Sorry
Rob.
-jolt

Post by TP
Robert and Vincent,
What are the exact problem symptoms you are experiencing? Is
the server running slow? Are users complaining? If yes, what
lead you to the conclusion that this particular reading is the
source of the problems you are having?
I have looked at the numbers you are seeing and this must be
a bug. I see this on every 2003 server I have examined so far
(since reading your post). If these numbers were true the
servers would grind to a halt.
Thanks.
-TP

Post by rpr
Hi, jolteroli!
As you suggested I installed Debugging Tools for Windows 32-bit
Version
(from http://www.microsoft.com/whdc/devtools/debugging/installx86.mspx
).
To replace "%SystemRoot%\system32\dbghelp.dll", which is in use by
some running processes, with the newer version in
"%ProgramFiles%\Debugging Tools for Windows (x86)\" you need to use
one of the methods explained at http://support.microsoft.com/kb/181345
mkdir c:\windows\symbols
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager
\Environment" /v _NT_SYMBOL_PATH /d "SRV*c:\windows\symbols*http://
msdl.microsoft.com/download/symbols" /f
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\winsrv.dll"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\csrss.exe"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\ntoskrnl.exe"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\win32k.sys"
"%ProgramFiles%\Debugging Tools for Windows (x86)\symchk.exe"
"%SystemRoot%\system32\ntdll.dll"
After restarting the server I got the following data from
When the RDP window is minimized, the I/O is normal.
The thread of csrss.exe with the highest CSwitch Delta has the
following start address: winsrv.dll!StartCreateSystemThreads
ntoskrnl.exe!KiSwapContext+0x26
ntoskrnl.exe!KiSwapThread+0x2e5
ntoskrnl.exe!KeWaitForSingleObject+0x346
ntoskrnl.exe!KiSuspendThread+0x18
ntoskrnl.exe!KiDeliverApc+0x117
ntoskrnl.exe!KiSwapThread+0x300
ntoskrnl.exe!KeWaitForMultipleObjects+0x3d7
win32k.sys!RawInputThread+0x4e0
win32k.sys!xxxCreateSystemThreads+0x60
win32k.sys!NtUserCallOneParam+0x23
ntoskrnl.exe!KiFastCallEntry+0xfc
ntdll.dll!KiFastSystemCallRet
winsrv.dll!NtUserCallOneParam+0xc
When the RDP window is restored or maximized, the I/O is very high.
But the thread of csrss.exe with the highest CSwitch Delta has the
start address and call stack as above (no difference when compared to
the normal state).
Does this tell you anything? To me it tells nothing as I don't have
the sources :-| (Almost a complete waste of time.)
-- rpr.
o
ad/symbols
uld

Post by rpr
I also live in Unix world and hence I'm familiar with the command
line. Unfortunately MS KBAs rarely suggest using command line tools.
Fortunately we have this groups for writing our own knowledge base
articles :-)
g
eve
he
Any of the Accessibility Options is not enabled on the server.
Also, no ATI services.
-- rpr.

Post by rpr
I discovered the high I/O problem in csrss.exe while investigating a
problem with the system clock on one of DCs in the domain. The
server's system clock is quite inaccurate.
I tried to solve that by reconfiguring the Windows Time Service
(w32time) using the procedures suggested at
http://technet.microsoft.com/en-us/library/cc773061.aspx
but it was not satisfactory as the Windows Time Service cannot take
into account the clock drift (as a good NTP service should).
At http://answers.google.com/answers/threadview?id=3D438195
you can find complains that suggest this high I/O slows down the
Windows Server 2003. Note that the question is written on 2006-01-26.
At http://forum.sysinternals.com/forum_posts.asp?TID=3D12894
Mark Russinovich said the following regarding this issue on
"This behavior is a known bug in the w2k3 terminal server I/O
counters."
But it was not explained any further.
Even if this bug causes no real harm it should have been solved long
ago. It certainly creates confusion and increases dissatisfaction with
MS software.
-- rpr.

Post by unknown
Found your post while trying to understand Microsoft Office Server 2007 consumption of resources (MOSS 2007\SharePoint).
My SharePoint Central Administration Server is a W2k3 R2 SP2 Std Ed X64 server with 8GB of memory.
My IO Delta climbs at times from 64KB to 84GB (not a typo ...84GB!)this is for the CRSS.exe (Client Server Runtime Process) when I view it in the latest version of Process Explorer.
My "I\O Other Bytes" is reading 19 Trillion+, equivelant to 19 Tereabytes????. The server itself is on a 30GB and 40GB partitions so the 19Trillion Terabytes may be I\O's from reading the SQL DB's its attached to which reside on another server. But the SQL db's its talking to total less than 200GB?
dumb founded....this server is showing very few eventlog errors and was only rebooted 12 hours ago.
In Process Explorer I'm also seeing a lot (as in hundreds) of "Contentions" for different instancesof the MS Seach Component mssdmn.exe
MSSearch.exe is also racking up a significant amount or PageFaults??
Will be openign a case with MS as soon as schedule permits.

Post by kC C
hi Guys just a quick question to you all suffering this issue, wheres your Page filing pointing to and whats it set to?

Post by David Ludvik
Hi Guys,
this hot-fix should fix the issue
http://support.microsoft.com/kb/956438