MetaStock File Format

K

Karl Blau

Hi,

I am planning to program a tool that manages my historical data. I have eod, intraday (min) and tick data. I was looking for the best file format for my project.
It seems as if MetaStock is the best for me.

But I can't find the definition of the MetaStock files. I was on www.equis.com but without any result.
Does anyone of you know where I could find a tutorial for this format?
I will program the tool in c++. So if anyone had a library for this domain I would be pleased if could take a look on this.

Thanks

Karl
 
You can find the old CSI/Metastock format by looking on Wotsit's File Format Library (http://www.wotsit.org/). The old format only supports 255 securities per directory.

For the new metastock formats, you need to purchase the developer runtime from Equis. They have 2 packages (one for read only, the other for reading and writing to metastock files). They are expensive.
 
Take a look at MetaLib from http://trading-tools.com/ - it's unexpensive, easy to use and customer support is excellent and responsive.

On TurtleTrader's forum, a guy posted Java sources to read Metastock files; this may help if you're dedicated programming this yourself.

Hope this helps,
C++
 
I have attached a C header which we use for reading MetaStock in all three versions MASTER, EMASTER, and the newer XMASTER format for more than 255 files.

The zip file also includes a text file with conversion routines for converting the Microsoft Basic format. The data files and some of the header files include floating point data values in this very old format so you can use the conversion routines to convert them to normal floating point format.

Also remember the byte order might be reversed if you are not on an x86 processor.

I got this information from a variety of sources and unfortunately can't remember them all, so I apologize for the lack of proper attribution.

There may be problems with these formats if you wish to write MetaStock files because there are areas of the format that are unknown. They work fine for reading MetaStock data.

- Curtis

P.S. The above mentioned forum is at turtletradingsoftware.com, not turtle trader. I run that site. See: http://www.turtletradingsoftware.com/forum/viewtopic.php?t=742
 

Attachments

@inflector: I think the file you posted is exactly what I was looking for. Thanks!

I have a problem opening a MetaStock Master File.
I have a directory where a several FXX.DAT files and one file named "MASTER". I use the following code to open the Master Header:

Code: 
int Mode;    // 1, 2 or 3

ifstream in(file.c_str(), ios::in | ios_base::binary);

//--------------------------------------------------------------
//	MASTER file description (the original 255 record format:
//		floats are in Microsoft Basic format
//		strings are padded with spaces, not null terminated
//--------------------------------------------------------------
if (Mode==1)
{
	USHORT			totalFiles;		// Number of files master contains
	USHORT			lastFNumber;		// Next file number to use (highest F# used)
	CHAR			padding[49];

	in >> totalFiles >> lastFNumber >> padding;
	cout << "MASTER-Format [1]" << endl;
	cout << "totalFiles  = " << totalFiles << endl;
	cout << "lastFNumber = " << lastFNumber << endl;
	cout << "padding     = " << padding << endl;
}

//--------------------------------------------------------------
//	EMASTER file description:
//		floats are in IEEE format
//		strings are padded with nulls
//--------------------------------------------------------------
if (Mode==2)
{
	USHORT			totalFiles;		// number of files in emaster */
	USHORT			file_num;		// last (highest) file number */
	CHAR			stuff[188];

	in >> totalFiles >> file_num >> stuff;
	cout << "EMASTER-Format [2]" << endl;
	cout << "totalFiles  = " << totalFiles << endl;
	cout << "file_num    = " << file_num << endl;
	cout << "stuff       = " << stuff << endl;
}

//--------------------------------------------------------------
//	XMASTER file description (150 byte records):
//		no floats
//		strings are padded with nulls
//--------------------------------------------------------------
if (Mode==3)
{
	CHAR			char5D;			// 000 - 0x5D
	CHAR			charFE;			// 001 - 0xFE
	CHAR			xmChars[2];		// 002 - "XM"
	CHAR			pad[6];
	USHORT			totalFiles;		// 010 - Number of files master contains
	CHAR			pad2[2];
	USHORT			totalFiles2;		// 014 - Number of files master contains
	CHAR			pad3[2];	
	USHORT			lastFNumber;		// 018 - Next file number to use (highest F# used)
	CHAR			padding[130];

	in >> char5D >> charFE >> xmChars >> pad >> totalFiles;
	in >> pad2 >> totalFiles2 >> pad3 >> lastFNumber >>	padding;
	cout << "XMASTER-Format [3]" << endl;
	cout << "char5D      = " << char5D << endl;
	cout << "charFE      = " << charFE << endl;
	cout << "xmChars     = " << xmChars << endl;
	cout << "pad         = " << pad << endl;
	cout << "totalFiles  = " << totalFiles << endl;
	cout << "pad2        = " << pad2 << endl;
	cout << "totalFiles2 = " << totalFiles2 << endl;
	cout << "pad3        = " << pad3 << endl;
	cout << "lastFNumber = " << lastFNumber << endl;
	cout << "padding     = " << padding << endl;
}

in.close();

I implemented the Header of the three different formats (Mode=1, Mode=2, Mode=3). But with no one I get any output that makes sense.

Am I doing something wrong?

One more question: How can I find out in which format my masterfile is stored? Does it only depend on how the filename is (MASTER, EMASTER and XMASTER)?

On the attachment you can see what the code produces when I open the file "MASTER" with mode 1, 2 and 3.

Thanks for your help!

Karl
 

Attachments

Karl,

I recommend dispensing with the stream IO altogether and using something like fopen and fread. The stream operators won't work correctly unless you have the types defined exactly right.

I use a large buffer to read the entire MASTER all at once. I can then use the pointer types defined in the header file to access everything easily.

Here's a snippet of code I extracted from shipping code that works for me:

Code: 
FILE*                       masterFile;
MasterFilePtr               masterPtr;
MasterRecordPtr             masterRecordPtr;
int                         fileIndex;
char                        masterFilePath[ MAX_PATH_LENGTH ];

// Open the "MASTER" file.
masterFile = fopen( "MASTER", OPEN_READ_BINARY );

// Read the file.
fread( s_Buffer, sizeof(char), BUFFER_SIZE, masterFile );

// Get a pointer in the correct format.
masterPtr = (MasterFilePtr) s_Buffer;

// Add a stock info entry for each item in the MASTER file.
for ( fileIndex = 0; fileIndex < masterPtr->header.totalFiles; fileIndex ++ )
    {
    // Get a pointer to the record.
    masterRecordPtr = &masterPtr->records[ fileIndex ];

    // Trim the space characters from the symbol and description.
    masterRecordPtr->symbolEnd = '\0';
    TrimTrailingSpace( masterRecordPtr->symbol );
    masterRecordPtr->descriptionEnd = '\0';
    TrimTrailingSpace( masterRecordPtr->description );
    }

The format will be defined by the name itself for data provided by others, either MASTER, EMASTER, or XMASTER. Some vendors add DAT on the end and some don't.

Also be aware that many data vendors (especially for international markets) have bogus volume and open interest values so you need to be prepared for very large numbers that don't actually represent the volume in some cases. For further confusion, some vendors supply their stock volume data in 100s but some in actual shares.

I hope this helps.

- Curtis
 
On the Metastock //open & read tools of the trade;
highly recommend thier free 50 + page Winning Stock Strategies book also .

Its like someone told Murray Ruggerio recently, on turtletradingsoftware.com;
''you cant judge a book by its cover'' even though most of us do.
:cool:
 
Hello all.

I've been working on a open source Metastock downloader for reading and writing to metastock files using VB .NET.

Not sure if you guys might be interested.

You can check out my code and applicaton at:

http://meta-all.sourceforge.net/

The DLL is quite usable for developing your own applications as well.

The project is slowly progressing.....but anyone is welcome to help.

The API documentation shows the file mapping that I have determined with help from this thread.

Cheers!
 
You must log in or register to reply here.
Back
Top