GPU memory arrangement is not simple and will vary widely among manufacturers, GPU models etc.
Yes GPUs have common/global memory but again it depends. It may be so slow that it's not worth accessing in parallel. You'll have to research it and do some of your own prototyping.
Choosing CUDA...