Database organization

volpunter, still the angry clueless fraud I see !!!

first, let me ask you, how is he going to fill his database if there is not some kind of programming end to parse and "clean" the data ? are you implying a simple inline load will suffice ? LOL !!! and how is he going to "manage" those data if there is no structured language to properly and easily manipulate those datasets ? Java and C++ will make things more difficult for data manipulation on a database. But you wouldn't know since you have no programming experience and you are still as clueless as ever LOL

you got owned already with your silly assumptions about Python in that other thread, and now you come back here to be beaten up some more by me :)

You have no programming and database experience, you silly clown, so stop insulting people when they make recommendations on something they do know.

and yes VB/Access for brainless Microsofties like yourself is a good way for a beginner to sharpen his skills in database design

Go back to your basement, all you can do is imagine a world of finance between your 2 sessions of XBOX360 :)

I was also wondering how long it would take you to come to this thread and throw a scene like you always do !!! LOL
 
Last edited:
cjbuckley4 said:
Thanks for your input @volpunter

That may be a bit unclear, but the issue im running into is that I don't know how to programmatically determine a 'roll algorithm' for options like I have for futures, so I figured if I just store all the options within a certain number of tranches of strike prices yesterday's close and in a certain interval of expirations then I could effectively programmatically capture all the options data necessary for backtesting. Any thoughts on how to best solve this problem? Any advice on databasing options data in general? I found futures pretty easy to solve once I looked on the exchanges' websites, but databasing options is still a question for me.
More...

a properly structured table and a well structured database will actually help you answer all those questions. I have some idea of what you are trying to do and I can see the mistakes and the assumptions you are making.

even though MonetDB is a good system, it's not going to help improve your database structure. Garbage In, Garbage Out as they say. The column store makes sense when you have 1000 of columns in a single table. Is this going to be the case here ? seems not. Another option is to go for "key-value pair" database storage system, it's a very simple approach to database engine, basically you are building tables with only a key-value column and all data are accessed through that key-value pair. MonetDB does that in some ways, but only for referencing the thousands of columns you might have. The key-value pair database has basically 2 columns in its simplest form, it's normalization to the max, and it can be very fast. Another option is to go with an "Object Database", basically you are storing data cubes, that is complex data structure into the row and columns of a table.

hence the question remains, you need to build your conceptual data model first before you can proceed, this is the first thing you learn in database design.
 
Last edited:
Yeah, I agree that optimal design is still unclear to me. How do you feel about the idea of throwing it all in one table and partitioning? I think it's best I write queries to do the heavy lifting for me.

I will have 10 columns. I've been led to believe that column orient databases are more optimal for tick data storage, maybe that's wrong, but MySQL definitely didn't impress me on a similar task.
 
dump here a sample of the data feed you are trying to store with the headers and tell us in very simple words what you want to do exactly, using the sample data feed as an example.

Like I said, your case doesn't seem very complex. Try building a cube database and it's a whole new game of complexity.
 
you will probably need to perform some OLAP queries on those data to get the results you want,

hence it's very important to build properly the tables,

OLAP solutions will usually answer all kind of crazy data questions without the need to build complex programming script for it.
 
No I havent, Is there a link you could direct me to? I've so far only been looking at CME options, but that would still be a good read. What do you think about this possibility of throwing everything on one table vs a table for each? I'm starting to like that idea myself.

@Butterfly, thanks again for your help. I'll post the data sample after school tonight.

@volpunter, you didn't find anything interesting on hat research paper discussion starter I sent you?
 
cjbuckley4 said:
No I havent, Is there a link you could direct me to? I've so far only been looking at CME options, but that would still be a good read. What do you think about this possibility of throwing everything on one table vs a table for each? I'm starting to like that idea myself.

@Butterfly, thanks again for your help. I'll post the data sample after school tonight.

@volpunter, you didn't find anything interesting on hat research paper discussion starter I sent you?
More...
I remember reading the rule when I was preparing for my series 4 test... I'll see if I can dig it up.

All in one table? As in adding it as long as u get a quote for it ? Could work...
 
dude, you continue to talk crap, from start to end. Nobody is angry here. I am simply confronting you with facts of how you are wrong on most every part you talked about:

* I never claimed what you question me with. I said most popular data bases target various APIs in many languages so limiting yourself to a specific language or even thinking about languages at the beginning is totally meaningless.

You said: "Java and C++ will make things more difficult for data manipulation on a database." -> you do this all the time: you throw some totally unsubstantiated and unproven arguments into the air and pretend as if they are fact. They are not.

* This thread has nothing to do with Python so keep python away from this thread (aside the fact that you made yourself look like a total idiot by claiming Python is used in large banks and hedge funds to run OMS (order management systems).

* Stop accusing me personally, accuse or criticize my points and arguments as I do.

* "You have no programming and database experience". -> again? How do you know? I used Kx's KDB+, and I wrote my own binary data store that can query data even faster than kdb+ can (albeit with a lot less bells and whistles). I certainly have way sufficient programming and design experience given I created a full blown trading system architecture. Stay with facts here

* By the way hos is VB related to Access? I see zero connection or overlap. Why would you mention them together? You probably meant VBA and Access? Even then VBA is completely unnecessary to write some relational database schemata. But the point is that we are dealing here with columnar data nor relational data. Hence you are off track anyways.

Basement and Xbox360? Neither I have nor own. Can we stick to the topic?

Can you please comment on the points I made:

----------------------------------

So let me understand this correctly: You pose as the big swinging dick here and cut this guy down left and right (despite him having been 100% honest and forthright in that this is something new to him). Then you go ahead and recommend an SQL based database to store columnar data structures? You still are an idiot!!!

And the programming language is one of the lowest priority concerns when thinking about storing data. Most popular database systems target multiple languages, so I have no idea what the issue regarding language choice is and why Java and C++ should be out of the question regarding API.

And then at the end you tell all of us that this amount of data can easily be handled in VB or Access?



Butterfly said:
volpunter, still the angry clueless fraud I see !!!

first, let me ask you, how is he going to fill his database if there is not some kind of programming end to parse and "clean" the data ? are you implying a simple inline load will suffice ? LOL !!! and how is he going to "manage" those data if there is no structured language to properly and easily manipulate those datasets ? Java and C++ will make things more difficult for data manipulation on a database. But you wouldn't know since you have no programming experience and you are still as clueless as ever LOL

you got owned already with your silly assumptions about Python in that other thread, and now you come back here to be beaten up some more by me :)

You have no programming and database experience, you silly clown, so stop insulting people when they make recommendations on something they do know.

and yes VB/Access for brainless Microsofties like yourself is a good way for a beginner to sharpen his skills in database design

Go back to your basement, all you can do is imagine a world of finance between your 2 sessions of XBOX360 :)

I was also wondering how long it would take you to come to this thread and throw a scene like you always do !!! LOL
More...
 
You must log in or register to reply here.
Back
Top