What is fixgrep

fixgrep is a command line utility for making FIX protocol messages more readable.

fixgrep features:

• Annotated fix tags and values.
• Horizontal (message per line) and vertical (field per line) formatting.
• Highlighting of tags and lines matching specified criteria.
• Grouping of messages by order chain.
• Indenting of repeating groups (when in vertical format)
• Coloring of message by type.
• Customized regex for extraction of FIX message from your logs.
• Sort by tags to bring more important tags to the front of the message.
• Exclusion of 'uninteresting' tags from the outputted FIX.
• Exclusion of 'uninteresting' messages from the outputted FIX
• Output to text or html format
• Customization of input/output delimiters.

Some introductory examples

Horizontal format:

Vertical format (aligned):

Vertical format (non-aligned):

Grouped by order:

How to get

Download the latest version from here
GitHub project can be found here

How to install

• You will need to install a version of java (jre or jdk), Java 8 or higher.
• Download the fixgrep zip file
• Unzip into a directory from where you wish to run fixgrep
• Add execute permissions to the fixgrep bash script. e.g. chmod u+x fixgrep
• Ensure that the directory where you unzipped fixgrep is on your path

How to get help

• Online here
• Running 'fixgrep man online' (no dashes) will open this page in your default browser.
• Running 'fixgrep man' (no dashes) will display the fixgrep man page in 'less'.
• Running 'fixgrep --man' (with dashes) will display the fixgrep man page in raw format without scrolling.
• Running 'fixgrep -?' will print a list of options.

How to run

There is a bash script file 'fixgrep' which should be your main interface to fixgrep. There are two ways to send fix data into fixgrep. The first is 'piping' e.g.
The second is specifying a file argument at the end of the fixgrep command. e.g.
Note, fixgrep does not curently support wildcards to specify multiple files.

Ways to configure fixgrep

Fixgrep can be configured via three methods.

• Command options. These are specified as POSIX style options after the fixgrep command. e.g. fixgrep --highlights 35 will highlight any MsgType tag/values. It is best to use command line options if your configuration is a once off, or is specific to the context you are running in.
• application.properties. If you wish to configure fixgrep to behave in a certain way every time you run it, then it is better to add your configuration to ~/.fixgrep/application.properties. You can install the default application.properties file at this location by running fixgrep --install.

Differences between command-line options, and properties configuration

Command-line options (usually) provide at least two different options which configure the same parameter. Usually a short single character option, and a more descriptive option. e.g. -o, --output-delimiter both configure the output delimiter. However when configuring via the properties file only one option is available to use. That is the longest command-line option, with dashes replaced with dots. e.g. to configure the output delimiter in the properties file, you would need to use output.delimiter option. e.g. output.delimiter=| Below is a table listing the command line options, and on the right the equivalent properties to use in properties file configuration.

option variations	equivalent properties key
-A,--align-vertical-columns,--align	align.vertical.columns
-D,--output-delimiter,--output-delim	output.delimiter
-G,--suppress-indent-group-repeats	suppress.indent.group.repeats
-O,--group-by-order	group.by.order
-V,--vertical-format	vertical.format
-a,--tag-annotations	tag.annotations
-d,--input-delimiter,--input-delim	input.delimiter
--download-url	download.url
-e,--exclude-tags	exclude.tags
--fix-spec-path,--fix-spec	fix.spec.path
-g,--line-regexgroup-for-fix	line.regexgroup.for.fix
-h,--highlights,--highlight	highlights
-m,--include-only-messages-of-type	include.only.messages.of.type
-n,--suppress-colors,--no-color	suppress.colors
-o,--group-by-orders-with-id	group.by.orders.with.id
--online-help-url	online.help.url
--output-format-grouped-order-header-console	output.format.grouped.order.header.console
--output-format-grouped-order-header-html	output.format.grouped.order.header.html
--output-format-horizontal-console	output.format.horizontal.console
--output-format-horizontal-html	output.format.horizontal.html
--output-format-vertical-console	output.format.vertical.console
--output-format-vertical-html	output.format.vertical.html
-q,--suppress-bold-tags-and-values	suppress.bold.tags.and.values
-r,--input-line-format	input.line.format
-s,--sort-by-tags	sort.by.tags
-t,--only-include-tags	only.include.tags
-v,--exclude-messages-of-type	exclude.messages.of.type
--vcs-home-url	vcs.home.url

Options

Output formatting

fixgrep supports two basic styles of formatting:

• Horizontal format
• Vertical format (aligned & non-aligned)

And each of these formats can be viewed in the console, or as html.
There are four different properties that can be used to define how output fix is formatted.

• output.format.horizontal.console
• output.format.horizontal.html
• output.format.vertical.console
• output.format.vertical.html

Each of these properties is a specification of how each fix message should be displayed in the corresponding display mode.
These properties are free text, and can include any of the following tokens:

token	description
${senderToTargetCompIdDirection}	will display the compIds of the current msg. e.g. ABC->DEF
${msgColor}	will 'start' formatting text using the pre-defined color for the particular FIX message type.
${msgReset}	stops formatting text using the pre-defined color for the FIX message.
${msgTypeName}	displays the FIX msg type htmlClass & sub-type, e.g. NewOrderSingle, CancelRequest, Exec.Fill
${msgFix}	displays the formatted FIX message
${n}	displays the value of that fix tag. e.g. for a NewOrderSingle ${35} would print 'D'
$n	(note, no braces), will print the captured regex group 'n' from the regex specified in the parameter input-line-format (or property input.line.format).

Below you can see the default values for these settings. They can be overridden in the application.properties file that ships with fixgrep.

output.format.horizontal.console

${msgFix}

output.format.horizontal.html

${msgFix}

output.format.vertical.console

================================================================================ ${msgColor}${msgTypeName}${colorReset} ================================================================================ ${msgFix}

output.format.vertical.html

<div class='msg-header'> ================================================================================</br> ${msgColor}${msgTypeName}${colorReset}<br/> ================================================================================ </div> ${msgFix} <br/>

Examples

Examples are being applied to the following lines of fix. I have purposefully over simplied these fix messages and stripped them down so that they might fit horizontally on a console:

Horizontal Formatting

By default fixgrep formats fix in 'horizontal format', i.e. one fix message per line

Vertical Formatting

So far we've looked at examples using horizontal formatting. Sometimes vertical formatting is preferable. Especially when looking at messages containing repeating groups such as prices.
Let's consider this single log lines as input. It contains 3 repeating prices:

Order Grouping

Order grouping allows the grouping of order messages by order. When using Order Grouping:

• Any non-order messages are discarded.
• Messages are cached until fixgrep has finished parsing the entire input, then the formatted messages are output.
• fixgrep does it's best to link messages together by clientOrderId, origClientOrderId and orderId.
• Using the following input fix:

FAQ

General

Is fixgrep free to use in a commercial environment

Yes. Fixgrep is released under the MIT licence which allows usage free-of-charge in a commercial environment.

Does fixgrep run under windows.

Yes. Fixgrep will run under any OS that has a JVM greater than or equal to Java 1.8. The most convenient way to run fixgrep is by using the fixgrep bash command. To run this under windows you will need to be running a bash prompt, e.g. cygwin. Currently there is no fixgrep.bat script written. At the moment I have no immediate intention to create one, but please feel free to submit a proposal.

Formatting

Why do I lose color formatting when piping fixgrep results to 'less'?

Indeed, some commands will not retain color formatting. For less, use the -R option

Performance

Can fixgrep be used as an alternative to grep when filtering large files.

It depends. Fixgrep will be much slower than grep, and will consume more CPU. For example running on my mac laptop, grep is roughly 5 times faster than fixgrep for searching for messages. If you don't want to wait too long and/or you are concerned about impact on CPU, best to use conventional grep commands to filter first and then pipe into fixgrep.

Can fixgrep be used as an alternative to 'more' or 'less' to view large FIX log files.

It depends. Applications such as 'more' or 'less' don't need to read an entire file during viewing. Whereas fixgrep does.
For large files (>100MB) it would be prudent to first extract the lines you wish to view (using grep or whatever), and pipe those lines into fixgrep.

How much memory does fixgrep use?

With one exception, fixgrep should not use much more than 64MB. Memory usage over time when parsing a large log file should not increase. The exception to this is when grouping by order. When grouping by order fixgrep will need to cache order messages before printing the out. This option is not enabled by default.

How much CPU does fixgrep use?

Fixgrep is a single threaded app, so at most it shouldn't use more than one core whilst it is running.

How can I reduce the impact on other existing processes whilst running fixgrep.

The regex which is run on each line to determine if the line is a fix message, and to capture the fix message is the single biggest CPU user in fixgrep. See the help for the input-line-format option for hints on how to make this fast.
Another way is to use the 'nice' command by specifying 'nice' before the 'fixgrep' script. e.g. 'nice fixgrep myfixlog.log' or 'cat myfixlog.log | nice fixgrep'
Nice takes a parameter which specifies how 'nice' you wish to be to other processes. Have a look at the nice man page for more info.

Licensing

Fixgrep is released under MIT license, a copy of which can be found here
This software is free to use/distribute/modify in both commercial and non-commercial environments.