How to read the last n% of a large file in C#

Recently we came across the problem of opening a large log file to read only the last few lines of the logs. This log file is a cumulative log file i.e. it had grown to be over 8GB in size. Not even Notepad++ could open it. We could probably find a efficient editor to do this job for us, but decided to read the end of the file ourselves.

Here is a C# snippet to read in the last n% of any file:

1. using System;
2. using System.Collections.Generic;
3. using System.IO;
4. using System.Linq;
5. using System.Text;
6. using System.Threading.Tasks;
7.  
8. namespace ReadLastPercentageOfFile
9. {
10.     class Program
11.     {
12.         public static void Main()
13.         {
14.             try
15.             {
16.                 Console.Write("Path to input file: ");
17.                 string inputfile = Console.ReadLine();
19.                 Console.Write("Enter the bottom percentage of input file to read: ");
20.                 long subLength;
21.                 long.TryParse(Console.ReadLine(), out subLength);
22.                
23.                 Console.Write("Path to output file: ");
24.                 string outputfile = Console.ReadLine();
26.                 using (StreamReader sr = new StreamReader(inputfile))
27.                 {
28.                     // position cursor in stream
29.                     long l = sr.BaseStream.Seek(sr.BaseStream.Length - ((sr.BaseStream.Length * subLength) / 100), SeekOrigin.Begin);
31.                     DateTime start = DateTime.Now;
32.                     using (StreamWriter output = File.CreateText(outputfile))
33.                     {
34.                         output.WriteLine("-----------------------------------------------------");
35.                         output.WriteLine("Input file: " + inputfile);
36.                         output.WriteLine("Bottom Percentage: " + subLength.ToString());
37.                         output.WriteLine("Output file: " + outputfile);
38.                         output.WriteLine("-----------------------------------------------------");
39.                         output.WriteLine();
41.                         long i = 0, cur = 0;
42.                         String line;
43.                         while ((line = sr.ReadLine()) != null)
44.                         {
45.                             output.WriteLine(line);
47.                             i = 100 - ((sr.BaseStream.Position * 100) / sr.BaseStream.Length);
48.                             if (cur != i)
49.                             {
50.                                 cur = i;
51.                                 Console.Write("{0} ", i);
52.                             }
53.                         }
54.                     }
56.                     Console.WriteLine();
57.                     Console.WriteLine("It took {0} seconds", (DateTime.Now - start).TotalSeconds);
58.                 }
59.             }
60.             catch (Exception e)
61.             {
62.                 Console.WriteLine("The file could not be read:");
63.                 Console.WriteLine(e.Message);
64.             }
65.         }
66.     }
67. }

Explaining the usage of Endeca Search Interfaces

Search interfaces allows for the control of the query-ability of a search across multiple dimension and properties in Endeca. A developer would create a search interface within the Oracle Endeca Developer Studio.

Querying the Endeca data catalog could happen in two ways:

1. The reference app through either querystring or selecting a properties and match mode through the web page provided. The URL parameter to use is Ntk. Note that Ntk is also used as normally for specifying an exact dimension or property name.
2. Code (.NET or Java):

1. Endeca.Navigation.ERecSearch mySearch = new Endeca.Navigation.ERecSearch(InternalSearchInterfaceName, terms, options);

The first parameter to the ERecSearch constructor is the name of the search interface created in the developer studio.

Once a search interface is either added or edited, the Endeca pipeline needs to run.

Using a search interface in search executes an OR on the dimensions and properties selected in an search interface.

When to use several search interfaces
In our case, a second search interface was added to accommodate for searches done by our internal staff versus external users. The aim was to expose searching capability within fields that are relevant only to internal staff and not necessarily exposed out to the external/public users.

CTTDNUG–Kitchener .NET UG Presentation material – Feb. 23, 2011

Demo projects are available for download.

T4 / Code Generation

View more presentations from @codekindler.

Getting started with Telerik Extensions for ASP.NET MVC 3–Razor (VS 2010)

Download Telerik Extensions the  from CodePlex

Download Change Set 61235, released on Nov 10 2010: http://telerikaspnetmvc.codeplex.com/releases/view/57046

Telerik project in Visual Studio 2010

Open the solution file using VS 2010 and convert to a VS 2010 solution.

Go to the property page for each individual project within this solution and target “.NET Framework 4” framework.

You can switch the target framework through the properties pane for both “Telerik.Web.MVC” and “Telerik.Web.Mvc.Examples” however you need to manually update the project files for “Telerik.Web.Mvc.JavaScriptTests” and “Telerik.Web.Mvc.Tests”.

For this, you can either use the PowerCommands for Visual Studio 2010 or manually update the .csproj file in the project folders.

Edit the Telerik.Web.Mvc.JavaScriptTests.csproj file (line 14).  Replace “v3.5” with “v4.0”.

Edit the Telerik.Web.Mvc.Tests.csproj file (line 13). Again, replace “v3.5” with “v4.0”.

Reload both projects in your open solution.

Now, edit the "Telerik.Web.Examples.csproj file as follows. Replace the selected line (104) with

<Reference Include="System.Web.Mvc, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL" />

Add an assembly reference for “System.Web.WebPages” to both “Telerik.Web.Mvc” and “Telerik.Web.Mvc.Tests”.

Change the solution configuration to “Debug MVC3” or “Release MVC3” and then build the solution.

ASP.NET MVC 3 project

Create a new ASP.NET MVC 3 project or open an existing one.

References

Make a reference to <Telerik download folder>\Source\Telerik.Web.Mvc\bin\Debug MVC3\Telerik.Web.Mvc.dll (or replace Debug MVC3 with Release MVC3 if you built the Telerik project with the release configuration.)

Web.Config

<configSections>
    <sectionGroup name="system.web.webPages.razor" type="System.Web.WebPages.Razor.Configuration.RazorWebSectionGroup, System.Web.WebPages.Razor, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35">
        <section name="host" type="System.Web.WebPages.Razor.Configuration.HostSection, System.Web.WebPages.Razor, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" />
        <section name="pages" type="System.Web.WebPages.Razor.Configuration.RazorPagesSection, System.Web.WebPages.Razor, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" requirePermission="false" />
    </sectionGroup>
</configSections>

<system.web.webPages.razor>
    <pages pageBaseType="System.Web.Mvc.WebViewPage">
        <namespaces>
            <add namespace="System.Web.Mvc" />
            <add namespace="System.Web.Mvc.Ajax" />
            <add namespace="System.Web.Mvc.Html" />
            <add namespace="System.Web.Routing" />

            <add namespace="Telerik.Web.Mvc.UI" />
        </namespaces>
    </pages>
</system.web.webPages.razor>

Content folder

Create a subfolder with the version number of the assembly.  In this case, it will the same name as the subfolder under the bin folder as follows:

<Telerik download folder>\Source\Telerik.Web.Mvc\bin\Release MVC3\Content\2010.3.1110

Copy all content to under this folder in your ASP.NET MVC 3 project and include in your project.

Scripts folder

Create a subfolder with the version number of the assembly. In this case, it will the same name as the subfolder under the bin folder as follows:

<Telerik download folder>\Source\Telerik.Web.Mvc\bin\Release MVC3\Scripts\2010.3.1110

Copy all content to under this folder in your ASP.NET MVC 3 project and include in your project.

/Views/Shared/_Layout.cshtml

Add the following to the head element:

@(
Html.Telerik().StyleSheetRegistrar()
     .DefaultGroup(group => group.Add("telerik.common.css")
                     .Add("telerik.vista.css"))
)

Add the following to bottom of the body element

@(Html.Telerik().ScriptRegistrar())

/Home/Index.cshtml

Add the following to the Home View Template, for example.

@(Html.Telerik().PanelBar()
    .Name("PanelBar")
    .HtmlAttributes(new { style = "width: 200px;" } )
    .Items(items => {
        items.Add().Text("Item 1")
         .Items(subItems => {
             subItems.Add().Text("Sub Item 1");
             subItems.Add().Text("Sub Item 2");
             subItems.Add().Text("Sub Item 3");
         });
    })
)

F5

That should be it.  You should see a page that looks like this:

Sample project download

You can download this ASP.NET MVC 3 (razor) project 

Toronto Architecture User Group – November 25 2010

I had the pleasure of speaking at the Toronto Architecture User Group on the topic of Domain Specific Development using T4.

The following is the slides that were used during this presentation.

Domain Specific Development using T4

View more presentations from Joubin Najmaie.

TechDays 2010 – LFT301, presentation resources

Resource URLs:

Patterns and Practices Guidance (T4)

Language Oriented Programming: The Next Programming Paradigm

T4 Templates: A Quick-Start Guide for ASP.NET MVC Developers

MSDN – Code generation and text templates

Text Template Glossary of terms

Text Template Directives

Channel9 resources:

MSDN Radio: code generation with T4 and Visual Studio with Oleg Sych

Advanced code generation patterns with T4 & DSL Tools

Visual Studio support tools:

Plug-in by Tangible Engineering

T4 Editor by Clarius

Microsoft TechDays 2010 – Toronto - LFT301

I’m pleased to be a speaker at this year’s Microsoft TechDays 2010 Toronto – Local Flavours.  The topic will be around code generation using T4 (Text Template Transformation Toolkit).  I will go over not only the motivations behind using T4, but reviewing its syntax and have lots of demos.

Hope you can join us at the conference.

T4 session: LFT301 - An Introduction to Code Generation with T4

Check out the sessions: http://www.techdays.ca/sessions

Speakers: http://www.techdays.ca/speakers?Venue=Toronto

I will be posting the resources regarding my topic in a separate blog posting.

Bio:

As a software architect and developer with over 10 years of experience Joubin has worked on and provided solutions to clients in North America and Europe. He is currently working on PageFragments (http://www.pagefragments.com), a content and internet marketing provider system. He is expert on code generation and web content management systems and holds an AIIM Enterprise Content Management Practitioner certificate. Joubin has contributed to open source projects such as the Code Kindler (http://codekindler.codeplex.com).

You can follow him on Twitter at @joubin or check out his blog at http://joubin.ca

Metro Toronto .NET user group - Groktalk 2010

I’m going to be presenting on the topic of Visual Studio 2010 template based code generation using T4 at the Metro Toronto .NET user group on June 15th 2010 at 6 PM.

Location: KPMG, 333 Bay Street, Suite 4600, Toronto

Register: http://mtdnug100615.eventbrite.com/

Here are all the sessions:

• Intro to T4 (VS2010 template based code generation) - Joubin Najmaie
• Getting things done with OneNote 2010 for developers - Shane Cusson
• Azure for beginners - Jeffrey Hornby
• Speeding up slow SharePoint Sites - John Dewees

You can download my previous slides presented at Toronto Code Camp 2010.

Hope you can join.