Application of my fault-tolerant real-time technology

[prt_systems]

Quote from ktmexc20:

One nice thing to mention about pytables/hdf, is that the file itself is not opened into memory but only an active pointer.

Same goes for the hierarchial groups/tables/cols/rows// single arrays or meta data within the file. Components can then be efficiently sliced/diced/iterated in "C", and finally returned as an active object.

I really do like these utilities ........

Complete fault tolerance computing is something that I cant cost justifiy for my operations - I can however do quite well employing clustering technology and other related techniques for increasing redundany and reducing the impact of any incidents. I've never used erlang and probably would not consider it unless there was a statistical calculation that showed all of the costs involved with its use would be recovered ....

 

[Joel Reymont]

I'm considering using HDF5 instead of a proprietary file format to store price series.

 

[ktmexc20]

Quote from prt_systems:

I really do like these utilities ........

Complete fault tolerance computing is something that I cant cost justifiy for my operations - I can however do quite well employing clustering technology and other related techniques for increasing redundany and reducing the impact of any incidents. I've never used erlang and probably would not consider it unless there was a statistical calculation that showed all of the costs involved with its use would be recovered ....

How much latency cost would you think there is? My curiosity wonders what dc package you use for RT?

Erlang-White Paper

Edited in: from the white paper

Robustness - Erlang has various error detection primitives which can be used to structure fault-tolerant systems. For example, processes can monitor the status and activities of other processes, even if these processes are executing on other nodes. Processes in a distributed system can be configured to fail-over to other nodes in case of failures and automatically migrate back to recovered nodes.

Not mandatory fault tolerance...

 

[ktmexc20]

Quote from Joel Reymont:

I'm considering using HDF5 instead of a proprietary file format to store price series.

;-)

 

[ktmexc20]

Quote from ktmexc20:

One nice thing to mention about pytables/hdf, is that the file itself is not opened into memory but only an active pointer.

Same goes for the hierarchial groups/tables/cols/rows// single arrays or meta data within the file. Components can then be efficiently sliced/diced/iterated in "C", and finally returned as an active object.

Another little interesting tid bit is that there is also what they call a FileNode, In which files in the hard drive can be mapped in tables with the same structure mentioned above, if so needed.

And btw, data values can be actual referenced or bounded objects (funcs/classes,etc.)

 

[prt_systems]

Quote from ktmexc20:

.... Processes in a distributed system can be configured to fail-over to other nodes in case of failures and automatically migrate back to recovered nodes.[/size][/i]

Not mandatory fault tolerance...

..to some people this seems like black magic but producing a simple fault tolerant protocol that will run over several nodes relatively efficiently is no big deal - to run over multiple tens or hundreds (or more) is a more difficult problem.

... for most systems you can produce a great deal of fault tolerance on just a few nodes with not too much custom code. These days the cost in hardware and software is pretty low to achieve 99.99 uptime on the base systems and fault tolerance and failover on the applications side ....

All of this can be accomplished in more or less commodity computing environments.

 

[Joel Reymont]

Quote from prt_systems:

... for most systems you can produce a great deal of fault tolerance on just a few nodes with not too much custom code.

The issue is the amount of programming that you need to do. I tried implementing fail-over as you suggested and did not find it easy at. Erlang _does_ make it easy, though, and it scales.

 

[prt_systems]

Quote from Joel Reymont:

The issue is the amount of programming that you need to do. I tried implementing fail-over as you suggested and did not find it easy at. Erlang _does_ make it easy, though, and it scales.

.. but its not commodity........

Pretty risky if you need to get people to deal with this, or who are willing to deal with it on an ongoing basis without paying a premium .... I still prefer to stay with commodity types of skill sets and systems where I can get people cheap.