My colleagues and I created a document together. I want to use the document in a book and plan to make a profit. What rights do the co-authors have to my profit and is there a way to protect my profit?
Under US copyright law (17 USC 201, "The authors of a joint work are coowners of copyright in the work". Barring an explicit agreement to the contrary, each author of a joint work shares an undivided interest in the entire joint work.
Rights to profit derive from a contractual agreement with a publisher. Therefore, you have to look into the specifics of the agreement with the publisher – such agreements can be quite variable. The question that you would have to ask is whether the particular agreement that you sign is actually "true", for example, do you represent that you are the sole owner of copyright (you are not)? You can negotiate away your property rights, but you cannot negotiate away your co-authors' rights. There will almost certainly be an indemnification clause in the agreement saying that if your actions lead to the publisher getting sued then you will legally defend them (pay for their lawyer, pay whatever judgment is rendered against them...). You can negotiate a license with a publisher that send all the royalties to you as long as you are up-front about the fact that you are a co-author and therefore they do not have an exclusive license (often a deal-breaker with publishers).
In short: if you created your document together, all of you are authors and publishing the document needs permission from all authors. Therefore, unless you have a previous agreement, your colleagues can ask you whatever they want for their permission, and it's up to you to decide if giving them what they ask for is better than writing a completely different document yourself alone.
Details may vary according to jurisdiction and specifics of your situation, but the general idea is going to be the same everywhere.
Just a caveat since you mentioned your colleagues: If you created that document as a part of your work, please beware that its copyright may belong to your employer and you will need your employer's permission to publish it (possibly in addition to the permission from your colleagues).
And just a likely out-of-scope cautionary note: There are posts in https://academia.stackexchange.com/ explaining that publishing a book is unlikely to yield substantial profits for the author. Hopefully, YMMV.