logQL is the trusted solution for extracting useful information for large log files. It’s extremely light weight and flexible to churn through gigabytes of information in minutes. With the release of 1.2 logQL provides support to operate on Excel files. The solution is:

Easy to install/maintain:
No database needed, no expensive hardware and logQL will run on any operating system supported by Java.

Extremely Flexible:
logQL employs a simple query language to generate the reports. Using this language, users can easily specify what data is needed in the report and apply filters on that data. For instance users may want to see how well a given product is selling? Then go on to check who the customers are? Is the demand increasing? Etc. Once the user has obtained the information needed, he/she can save the query to share with colleagues or to apply on a different dataset.

The resulting data can be exported back to excel to analyze it further with pivot tables/chats/etc.

Easy to Master:
The website www.logql.com provides video tutorials to master the language in under 15mins. The site also contains queries for different scenarios that users can use as a starting point

In conclusion, logQL 1.2 provides an easy, flexible and affordable solution for generating insightful reports from data in Excel files or it can be used to consolidate disparate data to analyze with pivot tables/etc.

Do check out the video demo of logQL 1.2 on excel files here.

Now, generating reports on that data needed some good knowledge of programming to transform it and get the report. This is gigabytes of data so the code had to be good. This solution was limited and slow. Adding another column to the report would involve changing a lot of code, with the risk of corrupting the output.

Another option was the database; load all the data into the DB and run SQL queries. This allowed for flexible reporting. But it also involved having a separate server with a rather huge data store. It proved to be unfeasible with gigabytes of data coming in everyday. Even with the disk space, the queries would slow down when the tables reached a certain size. To combat that, we had to remove the old data or have it in different tables.

Removing the data presented problems when managers wanted reports on different timelines. We’d take forever to load the data back in to generate one report.  Then we remove it so it doesn’t slow everything else down. Over all, a nightmare :-).

It struck me that the data in these files were all organized. Not in the most optimized manner, but consistent. I was wondering if we could run these queries directly on the files, eliminating the steps of loading in the DB and removing it after we’re done.

And that’s how it all began. I developed simple query engines to run queries directly on a file and pull out columns. Add to it the aggregation and filters… It was used in a limited way only by people I knew.

The name logQL came around only two years back and so the web-site.

If you are trying to generate reports off from huge log files, this is for you. Note that the basic version is free and there’s no limit on the file size.

Hope this comes in handy.