I have just posted the following 3 videos to the my Channel 9 PerfView Tutorial Series.

• Perfview Tutorial 6 - The Event Viewer Basics
• PerfView Tutorial 7 - Using the Event Viewer in ASP.NET Scenarios
• PerfView Tutorial 8 - Generating Your Own Events with EventSources 

If you are not already comforable with PerfView's event viewer, I do recommend you take a look.   The event viewer lets you see ETW data in its rawest form (as a time ordered list of events, where each event can have a event-specific payload).    It is particularly useful for just answer questions like 'what processes were running?  What DLLs were loaded?  What request URLs were serviced?    It is also VERY useful in getting you the infomration need to 'home in' on a particular request and get the time range and thread information needed to isolate the performance JUST THAT REQUEST from everything else happening on the server. 

I have finished three new videos that are about doing .NET GC Heap memory investigations using the PerfView tool and have posted them to Channel Nine.

1. Basics of .NET GC Snapshots
2. Investigating Memory Leaks: Part 1 Collecting the data
3. Investigating Memory Leaks: Part 2 Analyzing the data

The first video gives you the basics.  

• Determining whether you have a Memory problem and whether the problem is the GC heap.  
• Exactly happens when you take a heap snapshot (PerfView does sampling of large heaps). 

The second two videos describe a real life example:   How I used perfView's heap dumping capability to resolve a memory increasing issue in perfView itself.   I go through all the steps, from how the problem was first discoverd (by noticing memory growth in TaskManager) to finding a fix, and validating the the fix worked.  

You may also be interested in a background MSDN article that I helped write a few years back that is still very relevant  Memory Usage Auditing for .NET Applications.

By default when you collect data using PerfView you are collecting the right events to do a CPU time investigation.    However there are other things that can slow your program down beside using too much CPU time.  It is not uncommon to be disk bound (startup scenario are often like this), or network bound, or are waiting on other external programs.  For cases like this, a CPU investigation is inappropriate.

In fact when people want to do time investigations, they typically want to do wall clock time investigations (that is the time you can measure by observing the external behavior of the program).   Thus people really want to measure wall clock time, so why does PerfView concentrate on CPU time? 

There is a reason.   Wall clock time investigations are significantly more complex than CPU investigations.  The fundamental reason is in most applications your CPU usage TENDS to be on your 'critical path'.   That is MOST CPU time IS important to your end-to-end time.     This is because programs TEND not to have 'background' CPU activity that is 'uninteresting'.   Thus all CPU time TENDS too be important and so you can simply go after ANY CPU waste.    However for wall clock time this is RARELY true.   Most programs have threads that spend much of their time blocked waiting for work (and that time is NOT on anyone's critical path'), but these threads MIGHT wake up and perform a critical service.    Thus PARTS of SOME threads wall clock time is important but much of it is unimportant.   This is the key difference that is at the heart of what makes wall clock investigations hard.   You now have to think about 'critical paths', and you must be much more careful about segregating 'uninteresting' wall clock time' from 'critical path' wall clock time'.

PerfView has been able to collect and present data associated with wall clock for a while, but recently I added documentation about how to use it.   You can get at this documentation by clicking on the 'Wall Clock' hyperlink in PerfView's main 'Welcome page'.   You can also simply search for 'Wall Clock' in the documentation.  

I have also started to add series of videos on wall clock time investigations to the Channel 9 PerfView Videos.   The first of these is Wall Clock Time Basics.    There will shortly be several more.  

These videos use a 'BlockedTime' application.  I have included the source code for this application as well as the data files I used in these videos in the link below.   To use the attachment, simply open the ZIP file, and then drag the folder inside it to your desktop or other location.   The directory include the EXE as well as the PDB (symbols) for it, and the complete VS 2012 source code (in the src directory, the Program.cs file has all the code).   Also included are BlockedTimesync.ETL.ZIP (the run where all activity is on a single thread) and BlockedTimeParallel.ETL.ZIP (the run where the search is done in parallel).   These ETL.ZIP files do NOT need to be unzipped, simply open them in PerfView and they will implicitly be uncompressed.  

Update: There are now more videos on wall clock time analysis

1. Wall Clock Time Basics (the first which I talk about above (and uses the BlockedTimeSync.ETL.ZIP data))
2. Leveraging Tasks make sense of Parallel/Asynchronous programs this one uses the BlockedTimeParallel.ETL.ZIP data)
3. Investigating Wall Clock Response Time of ASP.NET scenarios which I talk about in this blog entry.

ASP.NET applications are one of the more common uses of .NET code today, and the PerfView tool has some special support for profiling them.  I created the video 'Wall Clock Analysis of ASP.NET Applications' as part of the PerfView tutorial video series.  You can learn more about wall clock time analysis in general (and find useful videos) by reading my previous blog post. 

In this video I take a hello-world ASP.NET application and modify it to consume both CPU time (by spinning setting a property a lot), and blocked time (by calling Sleep), the code is basically

    for(int j = 0; j < 100; j++)
    {
        System.Threading.Thread.Sleep(1);
        for(int i = 0; i < 2000; i++)
            HelloWorldLabel.Text = "Hello, world!  The time is " + DateTime.Now;
    }

The point is not to have an interesting application, but to show how PerfView can help you find hotspots of both CPU and blocked time (in this case Sleep, but it could have been a database call).

The key to this is the 'Asp.NET Thread time view'.   The main problem with doing ANY wall clock time analysis is to separate interesting time (that is time on the critical path between receiving a request and sending a response), from uninteresting time (time threads spend waiting for the next request).   This is what the ASP.NET Thread time view does.    It finds the region of any thread associated with a particular request and groups it under a 'pseudo frame' for that request.  The requests are further grouped by URL.   The result looks something like this

Which is very understandable.  It shows that there were 6 requests, all but one of which took 1.7seconds of time (and the highlighted one took 7.6 seconds) and that they were all responding to the URL 'HelloWorld.aspx'.    Thus the view nicely finds the important part of the execution (that related to the processing of a particular request), and groups them together.   You can then filter (look at only one request, or one URL, or whatever), and look at where the time went (opening the subnodes into a particular request).

The main point is that by doing this relatively simple transformation (grouping resource consumption (time) by the request that 'caused' it) we make doing the analysis very straightforward. 

So if you have an ASP.NET application you would like to optimized, take a look at the video, then download the attachment below and walk through it yourself.   To use the attachment, simply open the ZIP file, and then drag the folder inside it to your desktop or other location.   The ASPX file is the source code (open it in notepad), and the .ETL.ZIP file is the PerfView data file (you don't need to unzip it, just open it with PerfView). 

If any of you have tried to run PerfView a Surface or other WinRT device, you will find that it does not work.    Basically any program that is NOT a windows store application needs to be signed with a special key or it simply will fail to run on a ‘locked’ device (most devices are locked).     At some point PerfView might be signed in this way, but at present it is not.  

This blog is about working around this problem when you want to do a performance investigation on a Windows RT device (e.g. Surface).   Currently your options are

1. Do your investigation on an X86 machines:   This is what I recommend most of the time.   Generally speaking performance issues are not going to be THAT different on the two architectures (they will be the same issues, just more pronounced on ARM).   This is also likely to be easier logistically, so I recommend this to start with. 
2. Use a command line tool that ships with the Windows Performance toolkit called WPR (windows Performance Recorder) to collect the data.   The WPR.exe tool is signed to work on Windows RT, and can collect any ETW data that PerfView can collect.   Moreover collectint data with WPR is pretty straightforward, as you will see.

Getting WPR for Windows RT

Any easy way to get WPR is to simply download the Windows App Certification Kit.   The download is small and takes 10-20 seconds.   After installation the WPR.exe tool will live in at the path

"c:\Program Files\Windows Kits\8.0\Windows Performance ToolKit\wpr.exe"  

Note that if you wish to do further investigations on other WinRT devices you don't actually need to download/reintall the Windows App Certification Kit.   WPR.exe is XCOPY depoyable, so you could simply copy the directory to a USB drive / cloud drive, transfer to the target machine and run it from where you copied it

xcopy "c:\Program Files\Windows Kits\8.0\Windows Performance ToolKit" d:\WPR\

d:\WPR\Wpr.exe command line parameters discussed below...

 Using WPR to collect ETW data

WPR is a command line tool for collecting ETW data.   It was designed for 'flight recorder' style recording, where you keep a circular buffer going, and when interesting events happen, you take a snapshot.   It has three basic commands

1. Start - starts collecting (to a in memory circular memory log)
2. Stop - dumps the in memory log to a file, does NOT stop collection
3. Cancel - stops collection. 

When you start a WPR session you give it a 'Profile' which is a short name that defines what events you collect.   You can get a list of profiles by doing WPR -profiles however we will only talk about the 'GeneralProfile' which collect both CPU samples, Context switches (just like PerfView's 'ThreadTime' collection option), as well as all the CLR / JavaScript events to decode stacks (as well as some other stuff).  It is a fine default for most investigations.   A typical collection will do the following

• WPR -start GeneralProfile
• <Do your scenario>
• WPR -stop OUTPUTFILE.ETL
• WPR -cancel

The first command starts the logging, and the 3rd command dumps what is in the memory log and stops logging.  These commands need to be done from a administrator window, so

• type the Windows-R key combination and type 'cmd' to launch a CMD window
• open an elevated command window  (right click on the task-bar icon right click on 'Command Prompt' and then select 'Run as Administrator')
• run the commands above

Please don't forget the WPR -cancel command!   If you forget this command everything will seem to work, however WPR has left collection running.   This will slow your machine down by 10% or so but perhaps more importantly, it will use up 100s of meg of memory in its circular buffer, making apps die more frequently to free us space.   

Using ETW data to Collect PerfView-Style Data

The instructions use the 'GeneralProfile' which does not collect any .NET specific events (it does include the Thread Time events).    This is fine if your code has no JIT compiled code in it.   For Windows Store application that have been on the machine a while (so that the OS has had a chance to precompile (NGEN) your application, this data is OK.   However you have just installed or updated the application, or if you want to understand more about the .NET GC (that is you want the 'GCStats' view), or if you want the 'with Tasks' view which stitches together async operations, you need more events.  

To get this, download the DotNetRuntimeProfile.wprp file at the end of this blog entry and then change the collection start command to

• WPR -start DotNetRntimeProfile.wprp

This will tell WPR to collect most of the events that PerfView collects by default (along with the /ThreadTIme events that WPR would have normally collected anyway)   The rest of the procedure (WPR -stop OUTPUTFILE.etl and WPR -cancel) is the same as before. 

Uploading your ETW data

The output of the WPR commands will be the 'OUTPUTFILE.ETL' but it will also be a set of PDB files for precompiled (NGENed) .NET assemblies.  These NGEN pdbs are needed to properly decode .NET Framework code as well as some application code.   WPR places these PDBS in a directory called OUTPUTFILE.NGENPDBS

Thus to upload the data to a USB drive you might do  

• Xcopy OUTPUTFILE.ETL  d:\
• Xcopy OUTPUTFILE.NGENPDBS d:\OUTPUTFILE.NGENPDBS\

You can also ZIP them so that they are a single file for easier upload .   Simply open windows explorer, select the OUTPUTFILE.ETL and OUTPUTFILE.NGENPDB files, right click and select SendTo ->  Compressed (Zip) folder.   When it is done it will allow you to rename it, name it OUTPUTFILE.ETLl (it will put the ZIP suffix on automatically).   The resulting  .etl.zip file can be copied to another machine and opened with PerfView directly (PerfView will automatically unpack things into the proper places).

Once you the data (either as two pieces or as a ZIP file) to an X86 machine, you can use PerfView to open the OUTPUTFILE.ETL or OUTPUTFILE.ETL.ZIP file (it knows to find the PDBS in the OUTPUTFILE.NGENPDBS directory in either case).   You are also welcome to use the WPA tool that comes with the Windows Performance toolkit to look at the ETW data (but for that you  do have to unzip the .ETL.ZIP file).   From this point it is just like you collected the data with PerfView. 

One of the more common questions we get about the System.Diagnostics.Tracing.EventSource class is whether it can be used to log message to the

Windows Event Log (that is can be viewed with eventvwr.exe).   This EventLog support is also called 'Channel support' because that is the official name for a 'stream of events' in the Windows event log. 

The answer to whether EventSource supports the Windows Event log as a target has been 'no' but now the answer is 'yes'.    Moreover, you don't even have to wait for the next version of the .NET Framework to get it.

That is because we have now created a 'Stand alone' version of EventSource and published as the Microsoft EventSource Library NuGet Package (currently in Prerelease form, so you have to set your filtering to include prereleases to see it).    Note that the class in this package is in a different namespace Microsoft.Diagnostics.Tracing.EventSource, so it is 'side by side' with System.Diagnostics.Tracing.EventSource that comes with the V4.5 framework.   It is however intended to be 100% compatible with code using the framework version.   It is also effectively your way of getting at new features of the framework version before they are officially released (or your customers have upgraded their framework).    Switching from one to the other should be as easy as changing a 'Using' declaration at the top of the file that uses EvenSource. 

Thus Windows Event Log support is just the first installment of new features that we will be bringing to you through NuGet as a way of getting started now without having to wait for the next version of the runtime. 

To see more details about the Event Log support see the dotnet Blog entry 'Announcing the EventSource NuGet Package : Write to the Windows Event Log'.

Along with the Microsoft EventSource Library NuGet Package, there is also the Microsoft EventSource Library Samples NuGet Package.   This package makes it very easy to download some working EventSource code and see well commented examples of its use.  The best way to use this samples package is to

• Create a new Console Application in Visual Studio called 'DemoEventSource'
• Reference the Microsoft EventSource Library Samples NuGet Package from this new project by
• Right clicking on the 'References node in the Project tree
• Selecting 'Manage NuGet Packages
• Searching for 'EventSource Samples'
• Selecting the Microsoft EventSource Library Samples package
• It will download some code to a 'EventSourceSamples directory, which you look at
• You run the code by adding the following line to your main program
• EventSourceSamples.AllSamples.Run();
• Hit F5.  

Some of the samples are very similar to samples I have blogged about in the past, but the Window Event Log sample is new.   It is also more complex because there is a registration (install time) step with the operating system.   This is the price you pay for wanting your messages to get to the Windows EventLog.  

Finally, also included in both packages is a EventSource Users Guide.   This contains information similar to things I have blogged about, as well as best practices etc.  

Vance

Up until now, when I have demonstrated how to use the System.Diagnostics.Tracing.EventSource to generate events, I have been using the PerfView tool to turn on the EventSources as well as view the resulting events.   This is because PerfView was on the 'cutting edge' of what I will call 'registration free' ETW that EventSources use.   

Well I am happy to report that the latest version of Windows Performance Toolkit available for download in the Window 8.1 ADK preview. now supports EventSources as well.    When you do the download, it gives you the option of what tools to download.   You only need the 'Windows Performance Toolkit' part of the package.   This includes the 'WPR tool for collecting traces, the WPA tool for analyzing them and the XPERF tool (which is the older tool that predates WPR/WPA).    By default these tools end up in the 'C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit' directory and get put on your 'path' variable.  After installation you can simply type 'WPA or 'WPR' at the command prompt to launch the programs. 

Note that WPR/WPA and PerfView both use ETW (ETL files) as their format for storing the event information, which means roughly speaking (if you collect the right events), things collected with one tool, can be viewed by the other.   Thus you actually have several options for looking at ETW data (and EventSource data in particular)

• You an collect with PerfView and view with the Windows Performance Analyzer (WPA) (or PerfView)
• You an collect with the Windows Performance Recorder (WPR) and view with PerfView (or WPA)
• You can collect with the older XPERF tool and view with WPA or PerfView.  

Collecting EventSource data with WPR

WPR is the tool that the Windows team would like people to use for data collection going forward so we start with that.    WPR is designed to take its description of what you want to collect from a 'profile', which you can make by authoring an XML file which by convention is given the .WPRP extension.    I have authored a such a file that turns on one EventSource called 'MinimalEventSource', and placed it in a file called MimimalEventSource.wprp.   This file is both attached to this blog, but I have also included it at the end of the article as an 'appendix'.   Armed with this WPRP file using WPR to collect is very easy

• WPR -start MinimalEventSource.wprp
• <run A program with MinimalEventSource events logged in it>
• WPR -stop OUTPUTFILE.ETL

You can start several profiles in one command.  For example to get all the events in the built in 'GeneralProfile' as well as the ones in 'MinimalEventSource' you can do

• WPR -start GeneralProfile -start MinimalEventSource.wprp
• <run A program with MinimalEventSource events logged in it>
• WPR -stop OUTPUTFILE.ETL

Notice that you don't give a file name when you start a trace.  This is because it is being logged to memory in a circular buffer.   When you stop the trace this memory is processed and dumped to the ETL file you specify.   Once you have a ETL file (and possibly an NGENPDB directory), you can look at it with WPA easily

• WPA OUTPUTFILE.ETL

The EventSource events show up in the 'Generic Events' view of WPA.   Here is a screenshot. 

 Frankly this view is significantly better than PerfView Events view and has very powerful aggregation capabilities.    It is worth learning more about.    On the other hand, I do find that I spend a good minute setting up the view in WPA, and while there are profiles to help automate that, for quick, simple, view of my events I find I am faster with PerfView.    The nice thing is that you can choose whatever tool is most appropriate for you immediate analysis needs, since they both read the same data. 

Collecting PERFVIEW and viewing with WPA

PerfView definitely has some advantages over WPR for data collection.  For example to collect an eventSource with PerfView you need only do

• PerfView /OnlyProviders=*MinimalEventSource collect

Moreover PerfView is XCOPY deployable and a 5MB quick download off the Web, Couple that with not needing to create a WPRP file, automatic ZIPPing, good default settings for .NET programs, and a bunch of useful ways of stopping on PerfCounters, Exception and other events, PerfView is often the better choice for data collection.  

Unfortunately the current version of PerfView (1.4.1) will not symbolically decode EventSource Events in WPA.   The problem is that WPA relies on special WPA specific events that it injects into the ETL file when it stops a trace.   The support for this came after PerfView V1.4.1 was released, so it does not have the necessary information in the ETL file.     We will be updating PerfView to fix this soon, but in the meantime for those who can't wait, you can download a fixed copy of PerfView from my Skydrive.     This architecture of WPA does have some interesting ramifications

• It means it is the COLLECTOR not the VIEWER that matters.   Thus if you COLLECT with a fixed version of PerfView, you can VIEW on any version of WPA. 
• It also means that any traces that were collected before the fix and be 'fixed' by simply 'merging' them with a fixed PerfView:  'PerfView /zip:false merge FILE.ETL'

Collecting with XPERF

The windows performance toolkit does come with the older XPERF.exe tool for collecting data.   If you are used to this (and want to avoid creating a profile for WPR, you can use it for collecting EventSource data.   However this command does NOT have support for the symbolic names for EventSources, so you MUST use the GUID.  Thus you first have to get the GUID for your EventSource (An easy way is to simply use PerfView (on any machine to turn on the EventSource (even if there are not instances on that machine), and look in perfVIews log (button in lower right corner), and it will show you the GUID it actually used.  You can also use the 'Guid' property on your EventSource in code or in a debugger to get the Guid.   Once you have it, it is pretty straightforward, for MinimalEventSource it would be

• xperf -start MySession -on 7370a335-9b9a-5ba0-63c2-8ce045c35503
• <run your scenario>
• xperf -stop MySession -d OUTPUTFILE.ETL

 Recap: WPA/WPR supports EventSources

 So the short version of the announcement is that WPR/WPA supports EventSources.   Note that you only need to get the latest COLLECTOR, which means that if you use PerfView, you can simply get the updated version of that, or if you wish to use WPR/XPERF you download the WPT.   The event viewer is quite powerful and if you are doing more sophisticated data analysis, you should certainly invest in learning about it.

Appendix: The MinimalEventSource.wprp File

I have attached the MinimalEventSource.wprp at the end of the blog for easy download, but I have also included it here 'inline' for easy cut-and-paste.   The key part of the file is the name of the EventSource that you would like to enable, and I have highlighted it in red below.   Notice that we can use the name, not the GUID (but you can put a GUID there if you want).   The '*' before the name is important because like PerfView, it indicates that this name is not being looked up in some registration table, but is being hashed into the GUID using a standard mechanism. 

-----------------------------------------------------------------------------------------------------------------

<?xml version="1.0" encoding="utf-8"?>
<WindowsPerformanceRecorder Version="1.0">
  <Profiles>
    <EventCollector Id="EventCollector_MyEventSource" Name="MyEventSourceCollector">
      <BufferSize Value="1024" />
      <Buffers Value="200" />
    </EventCollector>
    <EventProvider Id="MinimalEventSource" Name="*MinimalEventSource" />

    <Profile Id="MyEventSource.Verbose.File" Name="MyEventSource" Description="EventSource Collection" LoggingMode="File" DetailLevel="Verbose">
      <Collectors>
        <EventCollectorId Value="EventCollector_MyEventSource">
          <EventProviders>
            <EventProviderId Value="MinimalEventSource" />
          </EventProviders>
        </EventCollectorId>
      </Collectors>
    </Profile>
    <Profile Id="MyEventSource.Verbose.Memory" Name="MyEventSource" Description="EventSource Collection" Base="MyEventSource.Verbose.File" LoggingMode="Memory" DetailLevel="Verbose" />
  </Profiles>
</WindowsPerformanceRecorder>

-----------------------------------------------------------------------------------------------------------------

I am happy to report that the TraceEvent Library Nuget Package and the TraceEvent Library Samples Nuget Package have been published at www.nuget.org.   If you look for them be sure to set the filtering to include ‘prerelease’ or you will not see them.    You can see this blog entry for the formal announcement of the library.   If you have feedback on the library, you can either contact me through nuget or leave a comment on that blog (or this one). 

 The library has a programmers guide that comes with it that tells you about the architecture of the overall package and other useful background information. 

The TraceEvent library for thosewho do not know is the library that makes PerfView possible.   The TraceEvent library makes controlling and parsing Event Tracing for Windows (ETW) logging easy to do from managed code.   For a while now we had the TraceEvent bcl.codeplex.com site, which also has a version of the code, but NuGet is significantly easier to use and we are transitioning the library there.    The code in the NuGet package is significantly newer then the codeplex site (that site is currently 9 months old), so it recommend that you go there.   The NuGet package has some breaking changes, however in the release notes you will find porting instructions.   To give you a feel, it took about 2 hours to port PerfView over, and it is probably the heaviest user of the library. 

The TraceEvent Samples are also a nice addition.   It makes it very easy to see some examples usage that you can cut and paste to get started.   The easiest way to use this samples is to create a new Console application and then install the Samples NuGet package into that project.  

As previously discussed on this blog, there is a companion library EventSource Library Nuget Package and the EventSource Library Samples Nuget Package.    EventSource as you now is the class that allows your .NET code to easily EMIT logging messages to ETW (or other places).    EventSource is part of the .NET V4.5 and you should use the framework version if you can.   However we have added features to the NuGet package that are not available (yet) in the .NET Framework Version.   Another valuable aspect of the EventSource Nuget package is that it gives you COMPILE TIME feedback if you make errors creating your EventSource.   This can avoid some frustration.   The samples package is great for learning about the API, whether you use the Framework or Nuget version.   See this blog entry for more.

Vance

P.S: Finally I should note that while you are more than welcome to build stuff with TraceEvent, if the functionality you want is closer to what PerfView already gives, you should consider simply writing a PerfView extension/ user command.   For more on this simply see the docs under help -> Extending PerfView.

It has been over 6 months since the last public release of PerfView, so I felt it was time to refresh the download center. 

For those of you who don't know about it is frankly the tool that I use to do most of my performance investigations with.   If you have a performance issue on Windows and don't know about PerfView, you should fix that... It is a very useful tool. 

This version is not dramatically different than V1.4.1, if that version is working well for you feel free to stick with it. 

• Support so that ETL data of EventSource.   Events taken with PerfView are viewable with the WPA tool.
• Can log performance counters to the ETL file using the /MonitorPerformanceCounter switch
• Support for collecing working set data every half second (MemInfo)
• PerfVIew is smaller and faster

Vance

P.S: Currently if you wish to use Windows Performance Analyzer (WPA) to view EventSource data currently there is a bug in the WPR collection tool that prevent the WPA tool from viewing the EventSource events correctly all the time.   The V1.5 version of PerfView has these fixes so that means if you collect EventSource events with PerfView (Select the MERGE checkbox but not that ZIP checkbox), you can view them in WPA.   The next release of WPA/WPR in a few months will also fix it, but that is still a few months out...

In my August blog I announced that the TraceEvent library for parsing Event Tracing for Windows (ETW) information was released as a Nuget package.

Well I have updated that package to with several months of updates and declared it a release candidate.   You can get the TraceEvent Nuget package or if you are new to the package I recommend that you create a empty project and get the TraceEvent Samples NuGet package, which shows you how to use the library with useful demos.  

Thankfully, the bug rate for the code was very small, most of the updates are various (relatively minor) feature/performance improvements.   I know that the documentation is not finished, which is one of the major items to complete before formal release. 

If you find any issues you can leave messages for me on my blog here to let me know.   

I will be formally moving it to 'Released' (stable), status soon (within a week or two),

Vance

Back in August in this blog enty, I announced that the TraceEvent Library Nuget Package and the TraceEvent Library Samples Nuget Package have been published at www.nuget.org as Prerelease software. 

Well I just updated this to be a stable version (version 1.0.5).  

For those who don't already know, TraceEvent is a library for parsing data that comes from Event Tracing for window (ETW) which include events you generate yourself with System.Diagnostics.Tracing.EventSource.    If you are in need of a logging system on windows machines you should definitely at least be looking at ETW, and TraceEvent is a great companion.  

You can see this blog entry for the formal announcement of the library. or my August blog entry for more details.     The release has been very stable over the 4 months since August, and most things have not changed.

The biggest different is that the TraceEvent Programmers Guide has been fleshed out a bit more (but still not complete).    

Vance

This is a quick announcement that I have refreshed the version of PerfView at the PerfView Download Center.

There was a last minute typo on my part, and the web site says it is version 1.7, but it really version 1.6 (PerfView's about box is the correct version).    The error was not worth correcting.

The main reason for the refresh was to fix a blocking bug (IndexOutOfRange Exception) that intermittently happens when collect a .NET Memory heap dumps.   

This refresh however also has many 'under the hood' changes (e.g. cleaning up TraceEvent for publication on Nuget, and then modifying PerfView to conform to the cleaned up code).    These changes should not make things noticeably different, but do have the potential for breaking things.   We have been using this version of PerfView internally for over a month without issue, so we have good confidence, but if you notice what seems to be a regression in functionality do let me know by replying to this blog entry. 

Vance

In a previous post, I talked about the TraceEvent NuGet Library, which allows you to read and manipulate Event Tracing for Windws (ETW).   There is a companion post about the EventSource  NuGet package which allows you to create your own ETW events (or in fact to send those events to anywhere you choose).    My blog entries have a by subject sorting that is handy if you wish to learn more. 

But I realized that I have not really shown you step-by-step how to get started with these NuGet packages, and while it is easy, I could easily believe many people are not familiar with NuGet.   So in this blog I will show you just how easy it is to get started with this using Visual Studio.   (By the way you can get Visual Studio Express as a free download here).  

So if you have any interest at all in tracing on windows here is your chance to experiment.    To give you an idea just how powerful ETW and the TraceEvent library is, the PerfView tool which you can see in action with its videos, is pretty much just a viewer of the ETW data you can get at with this library.  

You get started by creating a console application in Visual Studio.  Specifically

• Select File -> New -> Project.   This brings up a project creation dialog
• Select Templates -> Visual C#  -> Windows   in the left pane and 'Console Application in the main pane.
• If you wish to change its name, feel free to do so in the dialog boxes at the bottom, then Click OK.  

 This will make a new project with an empty 'Main' program.     Now add the TraceEvent Samples Nuget package by doing the following

• Right click on the 'References' node in the Solution Explore Pane on the right.
• Select the 'Managed NuGet Packages

This brings up the Nuget Package manager.

• Make sure that the 'All' selection under 'Online' is selected on the left.  
• Then type 'TraceEvent' in the search box in the upper right corner.  This finds all packages with 'TraceEvent in them.
• Find the Microsoft TraceEvent Library Samples package and click the 'Install' button.   This will cause VS to figure out all the other Nuget Packages you need (the samples need the TraceEvent package as well as the 'RX' packages because it shows off those capabilities as well.   It will bring up a dialog box to accept the license.    It is a standard, generous license. 
• Click the 'Accpept' button to accept the license and continue.

You have now downloaded the necessary software into your test application.   Note that this does not actually install anything in a normal sense.  It simply downloads this package and makes it part of the current project.   Your console application now has a  reference to the Microsoft.Diagnostics.Tracing.TraceEvent.dll which is the TraceEvent library.   One of the cool parts of Nuget is how easy it is to UNINSTALL as well as install.   If you go back to the Nuget package manager, select 'Installed Packages' in the left pane you will see all your installed packages (for this project only.  Every project is independent), and you can click 'Uninstall' on any of them and it will remove what you have installed.   It will NOT remove anything you modified but otherwise it does a good job putting things back they way they were.  Try it and see!

Anyway at this point our console application has downloaded and set up the references, but you application has not actually changed.   It has however popped up a readme associated with the package so your screen should look something like this

From here we are simply following the README instructions.    Things to note (also pointed out in the readme) is that all the sample code is under the 'TraceEventSamples' directory, and that there is a programmers guide that is also include that you can look at (note that in the future this may move directly to the web and not be in the package).     All we need to do now is

• Add the 'TraceEventSamples.AllSamples.Run();' line to the main program (like it says to do in the README).  

We now have a complete program.   You can run it simply by hitting F5 (GO).    There are 8 or so different samples and the 'AllSamples.Run' runs them all in sequence (with a Breakpoint between them).   The samples include

• Real Time monitoring of ETW data
• Logging ETW data to a file
• Reading ETW data from a file
• Generating and Reading your own events using EventSource and TraceEvent (EventSource comes with V4.5 of the .NET Runtime, or you can get the Nuget package for it if you need it to run on older runtimes).
• Real time monitoring using the 'Reactive Extensions (RX) to process the events.
• Getting stack traces from the ETW events you logged

Each of the samples is heavily commented with design guidance.   They are worth a read, and they will likely serve as the 'kernels' of your own logging projects.  

Note that the Object Browser and Intellisense works.  In particular if you do File -> View -> Object Browser you will see all the Microsoft.Diagnostics.Tracing.TraceEvent assembly and you can browse the classes and methods on it and read the documentation that comes with the classes.   After you have gotten your initial bearings using the sample applications and read the programmers guide, browsing this is the way to learn more. 

So there you have it.    After you have does your experimentation, you probably want to cut and paste any code you created into a new project that represents the application you REALLY wanted to build,refer to the TraceEvent NuGet Package for that application.    (you no longer need the samples).    As far as deployment goes, like all Nuget packages, TraceEvent is simply a set of DLLs that get put into the output directory along the DLLs you authored explicitly.   They are not special at this point.   Along with Microsoft.Diagnostics.Tracing.TraceEvent  there is also a couple unmanaged DLLs that does native code symbol lookup (msdia120.dll) and starting kernel mode session on pre Win8 OSs (KernelTraceControl.dll).    If you don't look up native code symbols (only the 'Stacks sample does this), and don't turn on kernel mode ETW (that is you are not using EnableKernelProvider API OR you don't care that it does not work on Win7) than you don't need these DLLs. 

Happy Eventing!   Now go write some code....

Vance

In my introduction to EventSource blog, I showed a simple example of using EventSource to log ETW information. In that example I used the PerfView tool to collect and display the information.   It is also possible to do the same using the XPERF or WPR tools (part of the WIndows Performance Toolkit) and in this blog entry I give an example.   In this entry I give some background

For the most part, an EventSource is really just another ETW provider.    The main difference is that an EventSource has support for logging its manifest INTO THE DATA STREAM, to make the resulting ETW data stream SELF-DESCRIBING, in the sense that you don't need other information (e.g. a registered manifest), to decode event payloads.    Thus while you CAN register the manifest of an EventSource, you don't have to, and most EventSources don't.  (If you are interested in registering EventSource manifest see my blog on channel support will lead you to the EventSource Samples Nuget package that has detailed samples and documentation on doing this (because channels REQUIRE that you register the manifest)). 

The fact that an EventSource EMITs this data is however not sufficient, you also need TOOLS to USE the information (instead of looking in the registration database), to complete the picture.   Both PerfView and WPA do this.  Thus these tools 'just work' on EventSources and decode their payloads as you would expect.   However decoding the payloads is not the only use of the registration database.   In the Operating system an ETW provider is identify by a GUID and normally will look up the name in the registration database and thus can convert the name to a GUID.   SInce EventSources don't register themselves, it means that you can't use an EventSource name to turn the provider on or off.     EventSource solves this by making it possible to FIND the GUID from the NAME without needing a registration database.    It does this by picking a VERY PARTICULAR GUID.   Basically what it does is take the name of the EventSource, capitalize it (to make it case insensitive) and then HASH the name in a standard way (using SHA1), and then use the HASH to form a Guid.  The details are not particularly important, the key is that knowing just the name, you can find the GUID for a EventSource, and it is VERY unlikely that two EventSources will have the same GUID.    

The EventSources also PROVIDE a mechanism to solve the name -> GUID problem but again, tools have to step up and use this.   The PerfView and WPA tools have done this (however at last check XPERF does not).    The convention that both PerfView and WPA use is that a name that is prefixed with a '*' means 'please hash the name into a GUID using EventSources's standard hashing mechanism.   This allows you use names when turning providers on and off. 

 OK, with that background, we can apply this knowledge to the XPERF/WPR scenarios. 

The XPERF CASE:

Because XPERF does not support the *Name syntax, you must use the GUID in the XPERF command line.   To do this you need the GUID for the EventSource.   You can get this either by asking your EventSource for its name programmatically (it is the Guid Property on the EventSource), or perhaps easier simply turn on the provider using PerfView with the *Name syntax and then look in PerfVIew's log for the GUID for the EventSource.   Once you have the GUID you can form the command line.

For example, I determined that for the MinimalEventSource, its GUID was 7370a335-9b9a-5ba0-63c2-8ce045c35503, thus I could turn it on using the command lines

• xperf -start MySession -on 7370a335-9b9a-5ba0-63c2-8ce045c35503
• Run your scenario
• xperf -stop MySession -d output.etl

Which turn on the provider, runs the scenario and stops the provider.  The resulting output.etl file can be viewed with WPA or PerfView.   In WPA the events are in the 'Generic Events' view and in PerfView in the 'events' view.  

 The WPR CASE:

WPR directly supports the *Name syntax, so you don't need to deal with the GUIDs.  However sadly unlike XPERF, WPR does not provide a way of adding another provider by passing a command line switch.   Instead you have to create  a WPRP file that describes the provider and its keywords/level and you refer to this WPRP on the command. line.   I have included such a file for the MinimalEventSource provider at the end of this blog.  If you download this you can run the commands

• WPR -start  MinimalEventSource.wprp
• Run your scenario
• WPR -stop output.etl 

Most of this is a recap of this blog entry, which gives more hands-on details.  

Well I am doing it again and teaching a beginning programming course to a set of high school students.    I did this 5 years ago when my son was the correct age (see my BouncingBalls post), and I started doing it again last year as part of TEALS effort.    This year as part of Prime Factor Programming Classes I will be doing an 8 week course for 9th and 10th graders every Thursdays, starting Oct 2 6:30PM for 90 min at Interlake High School in Bellevue WA.    The goal is to get them writing programs that they think are cool as quickly as possible.   I am using pygame as a GUI platform and by end of the 8 weeks I should have them making a variety of simple games (life, pong, breakout, concentration).   If enough students want it, we will continue with more advanced things in the spring.   I have always loved programming, and I am happy when I can pass that enthusiasm on to the next generation of programmers. 

But the real business of this blog entry to publish the class materials.   These materials contain links that contain step by step instructions for installing the software needed (I just walked though it, complete with screenshots), a set of 'cheat sheets for each lesson (enough that you can probably follow along), as well as at programming assignments/exercises (they are incomplete at this point, but not for long).   This is certainly useful for my students (I will send them links to them directly, but it is also nice that is 'findable' via web search), but it could also be useful for others who want a 'crash course' in python, or want to teach it to others (finding really good assignments is the key, and I have given it some thought.  

So.... Mission accomplished.   The class materials link is now discoverable via my blog, so I guess my students only have to remember my name to find their materials. 

P.S:If you happen to live nearby and have a 9th or 10th grader interested in programming there are still room for a couple more (but only a couple more) students.   Look over the class materials to decide if it is interesting, and contact the Prime Factor Programming Classes if you are interested. 

Vance

It has been a 9 months since the last public version of PerfView, so today I have updated the PerfView Download location from version 1.6 to Version 1.7.  

Here are some of the new features that I think are noteworthy.  You can look at the Help -> Release Notes for more information on detailed changes. 

• Much improved '(With Tasks) views handles async much better (will improve more in the future as well)
• Added the 'Size' menu and the 'Directory Size' menu item for figuring out where disk space is going.
• Enabled Goto Source for .NET Framework.   Will fetch the source code from Microsoft's server so you can see inside .NET framework code.
• Can turn on Network Capture to capture every network packet (See collection dialog box).  Rudimentary parsing in the 'events' view
• Net GC Allocations (as well as the original alot-ignore free) now possible for coarse sampling (the default)
• Provider Browser in collection menu for exploring ETW events on the system (can look at manifset)
• PerfView log from time of collection is storted in the etl.zip file and is available from the 'TraceLog' view.
• 'Set As Default Grouping Menu Item' lets you set your default stack viewer parameters.
• In the events' view you can select events and issue the 'Highlight in Histogram' to see where they are visually in the histogram.

Vance

Starting a couple of Defrag Tools #113 - PerfView Part 1months ago I did a series of Channel 9 Videos as part of Andrew Richards and Chad Beeder's 'Defrag tools' series.

I notice recently that these videos don't really show up in web searches when you use the 'PerfView' keyword, so I thought I would help people find them by putting some links in this blog entry.

Much of the material will be familiar to those who have watched my other PerfView videos, but some of it is new, and the descriptions below should be enough to decide if you already know the material or not. 

Anyway here they are

• Defrag Tools #113 - PerfView Part 1  - COLLECTION - Collecting ETW data with PerfView, stopping on particular events
• Defrag Tools #114 - PerfView Part 2 -  STACK VIEWER - Looking at the data (whirlwind, you should watch the PerfView Tutorial if you are serious), brief comparison to Windows Performance Analyst (WPA)
• Defrag Tools #115 - PerfView Part 3 - GC HEAP DUMPS - Taking heap snapshots to do a investigation of a memory leak or excessive GC heap memory consumption.
• Defrag Tools #116 - PerfView Part 4 - PERFVIEW for FUNCTIONAL ISSUES- Exploring the Event Viewer to find non-perf things like Processes Started, DLLs loaded, exceptions thrown (and their call stacks).  Access Violations that occurred.   etc.
• Defrag Tools #117 - PerfView Part 5 - MORE PERFVIEW for FUNCTIONAL ISSUES - More on Exceptions, Information on CLR DLL Loading problems (Fuslogvw replacement), ASP.NET Events, ASP.NET View, Events for PerfView itself.
• Defrag Tools #118 - PerfView Part 6 - ADDING YOUR OWN EVENTS WITH EVENTSOURCE - How easy it is to instrument your code with EventSource and display your events in PerfView. 
• Defrag Tools #119 - PerfView Part 7 - MOR FUNCTIONAL ISSUES - File opens, File Reads and writes, Memory usage (by process), TCP/IP (Network) traffic, ...
• Defrag Tools #125 - PerfView - Part 8 - .NET GARBAGE COLLECTOR / JIT -  The CLR GC events, the GCStats view.  Shows GCs, heap size ...  GC allocation sampling, Large object Heap issues.   .NET Just int time (JIT) compiler .   The JITStats view.   Time spent JIT.   

As some of you may know, since last August, there has been a Prelease version of EventSource 1.1 available on Nuget.   It has been stable for the last 5 months or so, so we have gone ahead and marked the 1.1.24 version as stable.

For those of you unfamiliar with EventSource, it is a logging system built into the .NET Framework.   You can learn more about it in my other blog posts about it. 

As a reminder, however, there are TWO versions of EventSource

1. System.Diagnostics.Tracing.EventSource - This is built into the .NET Framework, and you should use it whenever you can.   
2. Microsoft.Diagnsotics.Tracing.EvenceSource - This is what is implemented in the Nuget Package.   It is effectively a stand-alone version of the code built into the .Net Framework.    Its only purpose is to allow you to use new features (like the ones I am about to describe) on OLDER version so the .NET framework.    Everything in V1.1 of this package (what was just released) is available in the V4.6 version of the .NET Framework (which is the version that Windows 10 includes).   Thus if your application only needs to run on windows 10, then you don't need this package.   

Version 1.1 of the Nuget package (or 4.6 of the framework) has the following new features:

• DynamicEvents - Allows events defined 'on the fly' by without creating a 'event methods
• RichPayloads - Allows specially attributed classes and arrays as well as primitive types to be passed as a payload.
• ActivityTracking - Causes Start and Stop events to tag events between them with ID that represents all currently active activities.

I will be blogging about each of these in the near future, as each of the three new features are complex enough that some explanation and guidance is needed.     

Vance

 This is to let you know that the PerfView Download Site has been updated to version 1.8 today.   The pervious version (1.7) id about 10 months old, so this one has 10 months of updates.   Some of the highlights in that time are show below.  

The most important aspects of this version are that it supports various changes for Win 10 (if you use an older version of PerfView on Win10 (or Win10 profiles), you may encounter various deficiencies.    In addition to the win10 updates here are some of the other highlights. 

You can see the 'Help->Release notes' for more details on some of these.  

I will be bloggoing about some of the new features shortly, so stay tuned. 

Vance

 In this post we will be going over the New EventSource Activity support introduced in .NET version 4.6.   If you use EventSource in a server (concurrent) environment, you will want to read this post. 

What are Activities?

If you have been using EventSource in sever scenarios you probably have wanted what I am calling 'Activity' support.   

The basic issue is that events are great at telling you about what happens at A SINGLE INSTANT IN TIME, but almost always you care about the WHOLE FLOW of your application.     Now if your application is only doing one thing at a time, then the chronological ordering of events does a pretty good job of grouping related events together.   Thus in a single threaded app, if you have incoming HTTP request 'Start', a series of outgoing HTTP requests, and then the HTTP response, then the inference that the outgoing request are related to the incoming request is a pretty good assumption.    However if you have CONCURRENT or OVERLAPPING activities, than this logic VERY QUICKLY breaks down and interpreting your events becomes impossible (in this case, more information is not better).  

If you application was singly threaded, is I just that you have multiple threads running simultaneously, then the problem is pretty easy to fix.    You need only stamp each event with the ThreadID logging the event.  With this extra information you can simply sort the events into separate streams each of which a unique thread ID and we are back to the singly threaded case an life is good again.

Sadly live in a real server is not that simple.   The problem with ThreadIDs is

• Threads typically do work on behalf of other threads.   Typically threads 'listen' for incoming requests in some way, and then process them.  Thus any application that makes use of several of these services cannot easily be grouped together.  If several concurrent threads are all requesting work from these worker threads, again you get a tangle of event that is impossible to untangle. 
• Asynchronous programming (which is all to common now), makes this problem MUCH worse, because the concept of a Thread becomes tenuous.   Basically every time you do an I/O the next operation will happen on whatever thread is most convenient to use.   In such a world using threads as a correlation ID is hopeless.

So what we need is to come up with another concept that is very much like a thread of execution, but works in a concurrent / asynchronous world.    This is what an activity is.   More formally

An activity is a set of causally related events, linked by an common ID.

Thus an activity:

1. Starts at a particular point in time
2. Causes 0 or more other events to fire. 
3. Stops at a particular point in time (and thus has a duration which is the difference in time between the start and stop).

One more important aspect of an Activity is that the can NEST (that is one activity can cause another activity before the first completes.  Thus they can form stacks just like methods do.   In fact most typically there is a close correspondence between an activity and a particular CALL FRAME of a method in the code. 

EventSource Activities in .NET V4.6

In .NET V4.6 we have made it very easy to use activities.    But first, you have to have .NET Version 4.6

• If you are on Windows 10, you have .NET V4.6
• If not, do a 'start c:\Windows\Microsoft.NET\Framework\v4.0.30319', right click on the 'clr.dll' and look at the 'properties' of that dll.  Select the 'Details' tab and see the 'File Version'.  If it says V4.6.* you are good to go.
• In not you can update your system by going to the  .Net V4.6 download page, downloading and installing it.   This release is very compatible with previous versions so this is a 'safe' thing to do. 
• After probably about 4/2016, windows update should do this for you automatically. 
• If you have installed VS 2015 you also have .NET V4.6

Once you have .this version of .NET.  It is very easy to use Activities in EventSource.  Simply name some of your events with a 'Start' or 'Stop' suffix.   For example

Makes an EventSource 'Microsoft-Demos-Activities' that has a Request activity.   This activity has; a 'Start' and 'Stop' event associated with it.   (Note to simplify finding the pairs of events, 'Stop' events need to use an event ID that is one bigger than the corresponding 'Start' event.    Because the events with the 'Start' and 'Stop' EventSource knows that this represents an activity called 'Request' and does the following 'magic'

1. Creates a new Activity ID to represent the new activity when the 'RequestStart' event is logged
2. Attach this activity ID to EVERY EventSoruce event (even from different EventSources), that is logged from code CAUSED BY code after the Start event.
3. Until the 'RequestStop' event fires, in which case the Activity reverts to what it was before the ServiceStart.   

If you collect your events with PerfView (Version 1.8)  there is basically no change.  You can simply collect your events normally  For example for my I did

PerfView /onlyProviders=*Microsoft-Demos-Activities collect

 Now to make things interesting, I ran the code that processed requests in a Parallel.For loop that processes 20 requests concurrently.  After stopping PerfView and looking in the 'events' view for RequestStart and RequestStop events, this is what I see. 

Which shows that different threads are executing different requests and each get an activity ID (for example the first is //1/5).   Notice that the thread that finishes the request is not even the same thread that started it (for example Request(//1/6) starts on thread 10948 and stops on thread 8972).    Because PerfView can link up the starts and stops, it can also compute the duration, which it shows as the pseudo-field DURATION_MSEC, so we can see that Request(//1/6) took 2523 milliseconds. 

You will note that the 'ActivityID' has the from like //1/6, that is it looks like a path (list) of numbers.   This is precisely what it is and it is designed to capture the fact that activities can nest.   When an activity nests inside another activity, instead of simply picking another unique number it APPENDS a new child to the path of numbers.   This creates a very useful property:  All work associated with a particular activity has the same prefix. 

We can see this if we look at all the events in the 'events' view.   What we would see is many events interleaved with one another like this

Notice that the activity IDs 'jump around' as work on different requests gets executed in parallel.   However we can easily focus one particular request simply by including only those vents with a certain ActivityID prefix.

For example if we put //1/6 in PerfViews 'Text Filter' textbox, that will only show you events that have that string and you get the following display 

This is a very useful view!   It shows you all the events associated with a single request filtering out all other requests.   Notice it includes other events that are Not Starts and Stops (e.g. DebugMessage), and while it is not shown above, it would also include other EventSources (not just Microsoft-Demos-Activities).   It also shows 'sub-activities' ('like Phase1 and Phase2) and shows their nesting relationship.  For example we can tell that SecurityStart at 8206.265 is nested inside of Phase 1 because its activity ID (//1/6/1/1) is a superstring of Phases 1's activity ID (//1/6/1). 

Now we can see the value of making ActivityID paths of numbers rather than simply a arbitrary unique numbers.   It lets you express nesting and thus easily express concepts like 'everything that I caused, transitively'.    

 In fact, it is very natural to display activities as a tree, where the parent of a child node is simply the node that caused it.   PerfView can display this in the 'Any StartStopTree'  view which shows every event organized into a tree showing the activities that caused it.   For example in this trace.  

We see that the process started a 'Loop' activity //1/1, caused a ForkJoin' activity //1/1/1 , which cases a 'Request' activity //1/1/1/1, which caused a Phase2 activity //1/1/1/1/3 which caused a Security activity //1/1/1/1/3/1 which then logged a Debug message.   Thus activities look a whole lot like call stacks, and in fact you can treat them like 'coarse callstacks' in many analysis. 

 Sorry, Activity Support no available in the Nuget package (yet).

One important caveat to the Activity tracking support is that it is presently Not Available in the EventSource Nuget package.  (This is a reminder that we always recommend that users use the System.Diagnostics.Tracing.EventSource class built into the framework if you can, however we provide a 'clone' of this class in a Nuget package by renaming the class to Microsoft.Diagnositics.Tracing.EventSource, for those TEMPORARY cases where you need an EventSource  feature on a older runtime.    However in this case, Activity support required ehancements outside of EventSource itself to implement which is why the feature is not available in the Nuget package.   We may be able to 'hack' some sort of partial support, but for now you need to upgrade to V4.6 to get activity support.

Summary

We covered quite a bit of ground today, there is more to talk about, which I will do in the next few blog posts.   However for any who can't wait, I strongly suggest you read the New EventSource Activity support specification (attached at the end), which goes into a lot more detail about how it actually works, where it might produce surprising results, and how it interoperates with other Activity tracking conventions.    

But the key take-away, is that activities are INDISPENSIBLE in a server environment when you are doing many things concurrently/asynchronously.   EventSource has made this super-easy to do (simply name your events with the Start and Stop suffix), and PerfView makes it  very easy to immediately take advantage of the value that EventSource adds to your events.  

In my next blog entry, I will do more of a walk though of actual code so you can try it yourself.