Contemporary security thinking has it that you must employ consultants or specialists to do this work.
Unfortunately this has not panned out in practice. The dangers are many, the costs are huge and the one baby that does seem to consistently get thrown out with the bath water is security. Remember, they are the experts in security, they can design better than you, they can snow better than you, and they can certainly charge better than you. But you'll be left with the responsibility, either way.
Get comfortable with having to learn yet another discipline. Here are some corollaries to help you on your way.
Lay down your requirements before you start.
I find it takes longer to lay down the security requirements of the application than to actually design the protocol to meet the needs. And, it takes longer to design than it does to code up the first version. The hard work is in knowing what to build, not in building it.
During later phases, you will find challenges that bounce you back to your original requirements.
Let that happen. Bounce things back and forward for as long as it takes to prove your requirements are really what is needed for the application. Don't be surprised if you get half-way through and discover a real doozy.
The alternative is to pick up someone else's requirements out of a textbook or off a cypherpunks list and assume they make sense. If that happens you'll be into a perpetual battle that will eventually end up in a dog's breakfast when some real canine attacker works out the joke. If you're in that game, use a psuedonym when designing, and change jobs before it gets serious.
It's OK to design your own protocol. If there is nothing out there that seems to fit, after really thrashing #6.1 for all she is worth, design one that will do the job -- your job.
Consider the big weakness of SSL - its a connection protocol!
The mantra of "you should use SSL" is just plain stupid. For a start, most applications are naturally message-oriented and not connection-oriented. Even ones that are naturally stream-based -- like voip -- are often done as message protocols because they are not stream-based enough to make it worthwhile. Another example is HTTP which is famously a request/response protocol, so it is by definition a messaging protocol layered over a connection protocol. The only real connection-oriented need I can think of is for the secure terminal work done by SSH; that is because the underlying requirement (#6.1) is for a terminal session for Unix ttys that is so solidly connection based that there is no way to avoid it. Yet, even this slavish adoption of connection-oriented terminal work is not so clearcut: If you are old enough to remember, it does not apply to terminals that did not know about character echo but instead sent lines or even entire pages in batch mode.
What this principle means is that you have to get into the soul of your application and decide what it needs (requirements! #6.1 ! again!) before you decide on the tool. Building security systems using a bottom-up toolbox approach is fraught with danger, because the limitations imposed by the popular tools cramp the flexibility needed for security. Later on, you will re-discover those tools as shackles around your limbs; they will slow you down and require lots of fumbling and shuffling and logical gymnastics to camouflage your innocent journey into security.
Security experts will tell you that if you don't build in security at the beginning, then you'll have trouble building it in later. This is true, but it hides a far worse evil: if you spend too much effort on security up front, you'll either lose time and lose the market, or you'll create a security drag on the application side before it has shown its true business model.
The security-first argument speaks to security convenience, the business-first argument speaks to survival. Make sure you understand your businesss need for survival, and pray that they understand the inconveniences you put up with to ensure that goal.
Carefully partition your protocol to be replaced. Not some portion of it (see Hypothesis #1 -- the One True Cipher Suite) but the whole sodding thing. If you like, get blatant about it and slot in ROT13 with a checksum made from your cat's ink dippings and a key exchange from putty and pigeon post.
Likely your application will not need real security for the first 1000 users or so (pick you own number here, or read up on GP). Once you've proven the business model, then it's time to secure, and start the tortuous and expensive process of rolling out a new protocol.
Here however note carefully: you are taking on a risk on behalf of the business, and the cryptoguild will massacre you! Make sure that you tell your boss and his boss and her boss too, but do it quietly.
... and we need to replace Dodgy Crypto Protocol #1 when the heat starts
Don't just tell them, write it down, email it to the auditors, tattoo it on their daughters' foreheads in a secret code that only they know.
Meanwhile, design Well Needed Protocol #2 over your private weekends. Tell your spouse you need to save the world, or better yet, make him do it. Just make sure you are ready with a replacement in advanced form.
In all probability you will need to be adept -- well versed if not expert -- in all aspects, right up to user activity. This will stretch you away from the deep cryptographic and protocol tricks that you are used to as you simply won't have time for all that. But an ability to bridge from user needs all the way across to business needs, and then down to crypto bits is probably much more valuable than knowing yet another crypto algorithm in depth.
The FC7 thesis is that you need to know Rights, Accounting, Governance, Value and Finance as well as Software Engineering and Cryptography . You need to know these things in some depth not to use them but to avoid using them ! And, more particularly, avoid their hidden costs, by asking the difficult questions about the costs that nobody else dare recognise.
Knowing the Hypotheses is a given, that's the job of a protocol engineer. That which separates out engineering from art is knowing when to breach a hypothesis.
H1 H2 H3 H4 H5 H6 H7