Recent content by Indrionas

  1. I

    Data mining challenge

    Evolutionary algorithms are not suitable for this kind of problem. They rely on a fitness function. In order for an evolutionary algorithm (such as GA) to work productively, the fitness function must be monotonic. This is not the case with searching for exact patterns. A simple example...
  2. I

    Data mining challenge

    Yes, that's what I am talking about. In this example problem I know in advance that there are 5 attributes max. The problem is what to do when it is not known in advance. One approach could be to use this "max number of attributes" as a training parameter. Then run training gradually...
  3. I

    Data mining challenge

    Yes, the search space is huge. But it doesn't mean that there are that many patterns in the data. In fact, the search space is greatly limited simply because only a very small fraction of them all are "interesting". And by "interesting" pattern I mean those that have at least some support, are...
  4. I

    Data mining challenge

    Yes, that is the core of the problem. The NN comes up with similar results (i.e. generalization estimate) no matter if I train it with all 300 variables or with the (48) relevant only: sometimes a little better, sometimes a little worse, but no significant difference. And I tried as many...
  5. I

    Data mining challenge

    Sorry for misunderstanding. You are probably viewing the file with a simple Windows text editor. The new lines in the file use Unix format. I recoded it to be viewable for Windows users (attachment). There are 10 patterns, one pattern per line, five variables per pattern. 48 variables are...
  6. I

    Data mining challenge

    Jack, what you wrote makes absolutely no sense. The result file contains 10 patterns. One pattern per line. Pattern is made up of variable numbers and sign before a number denotes if that variable's value must be true or false (1 or -1 in this case). For example, the first pattern: -175...
  7. I

    Data mining challenge

    The stupidity of the trolls in this thread is beyond belief. It turns out that I did not generate the data myself and I'm looking for a job (or have a job) in Wall Street or whatever the guy with long rants tried to say. Unbelievable :D Well, I was wrong thinking I could find an intelligent...
  8. I

    Data mining challenge

    Hi, can you elaborate on this? Sounds like an interesting idea but I have no idea how this could be applied in practice to solve the problem. Also note that the relationships between the patterns and targets are probabilistic, not deterministic. I tried to find at least one paper on the...
  9. I

    Data mining challenge

    True. True. This is part of the data generation process. The reason is very simple: to simulate data that occurs in reality and to test data mining techniques that are powerful enough to deal with this issue. This is just plain stupidity. Nice conspiracy theory you've got there :D...
  10. I

    Data mining challenge

    Sorry you worked up so much over my response. :D Didn't mean to attack you. I just don't want you to waste time over a problem you don't understand. I will reiterate: the problem is simple. There are 300 input variables and 1 target. All variables are binary. There are patterns hidden in the...
  11. I

    Data mining challenge

    The data is in semicolon separated format. You can easily rename the file to .csv and open it with Excel if that's what you really want. But the data is for machine learning, there's no point in looking at it through Excel. What particular part of the explanation did you find hard to...
  12. I

    Data mining challenge

    The data.
  13. I

    Data mining challenge

    I believe there are a few people here who do this or similar kind of work in their process of model building. Here is the deal: The data is synthetic. Means I have generated it and I know the rules (real model) that generated it. The values are separated by semicolons. There are 3000...
  14. I

    Fighting the HFT algos... daytraders and floor traders vs. HFT traders

    http://www.youtube.com/watch?v=kU0WMf465Tc http://www.cnbc.com/id/15840232?play=1&video=1504494395
See more
Back
Top