I came up with this hack in order to transfer Python code to Interactive Brokers tech support. Most support reps usually just tell you to reboot your computer. But sometimes, if you've found a legitimate problem in their code, they are willing to work closely with you; esp. if it's communicated clearly. It may take forever, and they may say it won't get fixed, but at least they're aware of the problem ;-)
As Python coders know, white-space indentation denotes code blocks. Coming from a C background there's a good reason this annoys me. With C, if your indentation gets mangled during a copy and paste (as is often the case) you can just re-indent the code with a "pretty print" program. After all, braces clearly denote code blocks in C. With Python though, if the same thing happens... you've introduced bugs into the code. There's no easy way to pretty print Python that I know of. This can be a PITA even with small code files.
Furthermore... IB restricts the types of files you can attach to a tech support ticket. Presumably the reason for this is security, which I can understand. Nonetheless, it means that neither a .txt nor .py file can be uploaded. They do allow image and PDF files however.
I don't exactly agree with this policy 100% since there are security vulnerabilities in image and PDF viewers too. But, we still have to live with the restrictions. So when I wanted to send them a Python file (which is basically plain text) and be sure white-space bugs wouldn't be introduced via copy/paste... I discovered "unoconv"; the universal office converter:
With this we can easily and quickly create a PDF document from a simple text file.
We still can't just create a PDF of the Python source directly. Because, copy and pasting from the PDF may screw with the leading white-space :-( We want to be assured that copy/pasting won't ruin meaningful content i.e. the leading white-space. And, without requiring OCR software, embedding text in an image isn't a viable solution either.
Ultimately the recipient must be able to open the allowed file type (PDF), know what to do with it at a glance, and be able to copy/paste the contents as usable code. Just a little more creativity should get us there.
To achieve this, we gzip and base64 encode the Python. This becomes our "payload". Then, we put this payload in a bash script where we can insure leading white-space is non-existent or benign. The PDF can be created from that instead.
For example, let's say we have the following demo.py file:
Obviously the leading white-space matters here.
To create a PDF file that we can attach to our support ticket, we first generate a payload based off demo.py:
The payload.sh file now contains a bunch of text, it may look like gibberish, but it's just our demo.py in a form that travels more easily through assorted tunnels.
Now, we can edit the payload.sh to add some simple human readable instructions. Put this at the very top of payload.sh:
and place another line containing PAYLOAD at the very bottom to mark the end of the heredoc.
Your payload.sh file should ultimately look like this:
Finally, convert the payload.sh into a PDF with unoconv:
View the message.pdf you created to make sure the contents resembles the payload.sh file.
When your recipient opens message.pdf they'll be able to read it and copy/paste the contents into a shell script. Then they can execute that script to print the demo Python code (or redirect it to a file). With a little luck everything will unravel nicely. Obviously you can put other types of files in the payload too.
You may be thinking there isn't a need to wrap the payload with our shell code. After all, a PDF can be created from the payload directly. But most recipients won't know what to do with base64 encoded text alone; it's confusing gibberish without additional context.
Another alternative is to generate a PDF file from a shell archive. However, our tiny custom bash script has the benefit of providing some human readable content directly in-line (via heredoc or you could use a comment instead, but heredocs are a bit nicer IMHO). In this way the user can open the PDF and have some introduction to what's going on. This is a trade-off worth considering.
Finally... I know it's a somewhat complex process in order to merely transfer a file. Unfortunately, sometimes, working around questionable security restrictions requires this kind of rigmarole. Please don't shoot the messenger!
GL/HF with this fresh approach to a rather old endeavor... sending "a message in a bottle".
As Python coders know, white-space indentation denotes code blocks. Coming from a C background there's a good reason this annoys me. With C, if your indentation gets mangled during a copy and paste (as is often the case) you can just re-indent the code with a "pretty print" program. After all, braces clearly denote code blocks in C. With Python though, if the same thing happens... you've introduced bugs into the code. There's no easy way to pretty print Python that I know of. This can be a PITA even with small code files.
Furthermore... IB restricts the types of files you can attach to a tech support ticket. Presumably the reason for this is security, which I can understand. Nonetheless, it means that neither a .txt nor .py file can be uploaded. They do allow image and PDF files however.
I don't exactly agree with this policy 100% since there are security vulnerabilities in image and PDF viewers too. But, we still have to live with the restrictions. So when I wanted to send them a Python file (which is basically plain text) and be sure white-space bugs wouldn't be introduced via copy/paste... I discovered "unoconv"; the universal office converter:
"unoconv" is a command line utility that can convert any file format that LibreOffice can import, to any file format that LibreOffice is capable of exporting.
With this we can easily and quickly create a PDF document from a simple text file.
We still can't just create a PDF of the Python source directly. Because, copy and pasting from the PDF may screw with the leading white-space :-( We want to be assured that copy/pasting won't ruin meaningful content i.e. the leading white-space. And, without requiring OCR software, embedding text in an image isn't a viable solution either.
Ultimately the recipient must be able to open the allowed file type (PDF), know what to do with it at a glance, and be able to copy/paste the contents as usable code. Just a little more creativity should get us there.
To achieve this, we gzip and base64 encode the Python. This becomes our "payload". Then, we put this payload in a bash script where we can insure leading white-space is non-existent or benign. The PDF can be created from that instead.
For example, let's say we have the following demo.py file:
Code:
#!/usr/bin/env python
import sys
print("You are using Python {}.{}.".\
format(sys.version_info.major, sys.version_info.minor))
if not sys.version_info.major == 3 and sys.version_info.minor >= 6:
print("Python 3.6 or higher is required.")
sys.exit(1)Obviously the leading white-space matters here.
To create a PDF file that we can attach to our support ticket, we first generate a payload based off demo.py:
Code:
cat demo.py | gzip | base64 > payload.shThe payload.sh file now contains a bunch of text, it may look like gibberish, but it's just our demo.py in a form that travels more easily through assorted tunnels.
Now, we can edit the payload.sh to add some simple human readable instructions. Put this at the very top of payload.sh:
Code:
#!/usr/bin/env bash
cat > /dev/null <<NOTES
Hi! Copy and paste everything you see in this PDF into a .sh file.
Then, you can run it and our Python demo code will be printed out.
NOTES
(base64 -d | zcat) < /bin/cat << PAYLOADand place another line containing PAYLOAD at the very bottom to mark the end of the heredoc.
Your payload.sh file should ultimately look like this:
Code:
#!/usr/bin/env bash
cat > /dev/null <<NOTES
Hi! Copy and paste everything you see in this PDF into a .sh file.
Then, you can run it and our Python demo code will be printed out.
NOTES
(base64 -d | zcat) < /bin/cat << PAYLOAD
H4sIANvGj2MAA3WOwQqCUBBF9+8rbrZRiCchtAjsG9oGQRg+cwJnbN5TkujfU3FXDbOae89w1qu0
85peiVPHPdoh1MLGUNOKBvjBG9MqcYijk3Qo1KHzxDcc5yJebztuZM8G81SiTRHikbO9U0/CF+JK
bFPcRTf4vhOLJomhCizhRz5xyHNkKLj8w+OQY7efDRbXxS6zO4xxTbfaKchD3aMjdaWNkrk+/XNP
CvE2MR/PFp6BCgEAAA==
PAYLOAD
Code:
unoconv -o message.pdf payload.shView the message.pdf you created to make sure the contents resembles the payload.sh file.
When your recipient opens message.pdf they'll be able to read it and copy/paste the contents into a shell script. Then they can execute that script to print the demo Python code (or redirect it to a file). With a little luck everything will unravel nicely. Obviously you can put other types of files in the payload too.
You may be thinking there isn't a need to wrap the payload with our shell code. After all, a PDF can be created from the payload directly. But most recipients won't know what to do with base64 encoded text alone; it's confusing gibberish without additional context.
Another alternative is to generate a PDF file from a shell archive. However, our tiny custom bash script has the benefit of providing some human readable content directly in-line (via heredoc or you could use a comment instead, but heredocs are a bit nicer IMHO). In this way the user can open the PDF and have some introduction to what's going on. This is a trade-off worth considering.
Finally... I know it's a somewhat complex process in order to merely transfer a file. Unfortunately, sometimes, working around questionable security restrictions requires this kind of rigmarole. Please don't shoot the messenger!
GL/HF with this fresh approach to a rather old endeavor... sending "a message in a bottle".
Last edited:
It's also an awesome demonstration of why there is so much duplication of tools in the UNIX world (