It is probably a word that can be used, but not definitively.
Many words of general applicability coined by authors enter into the general lexicon.
The process is similar, but not precisely identical, to the dilution of a trademarked term to become a generic descriptive term for anything in the category of items so trademarked. Early examples of originally trademarked words that entered the general lexicon are "elevator" and "escalator".
For example, the word "grok" coined by Robert Heinlein, is now part of the general lexicon, as is the word "quark" coined by James Joyce (although its current meaning has shifted). The word "robot" from K. Čapek's play R.U.R. ‘Rossum's Universal Robots’ (1920), would be another example of a word coined in fiction that has entered general use.
The scene a faire doctrine also authorized public domain use of tropes and terms that have achieved wide use in a genre. As a rule of thumb, if three different authors have used a term to mean the same thing, or an author who is no longer in copyright has done so, it can be used freely.
For example, regardless of who originally used it, the term "mecha" would not be protected by the doctrine due to its used by multiple authors and commentators in a particular science fiction sub-genre.
"Mentat" is a close case, because it is overwhelmingly used in a specific series of books written by Frank Herbert and his son with licensing from his estate, and because the series has produced so many published properties, not only in books, but in TV and movies, it has been close to maximally protected from an IP perspective by the Herbert Estate, his publishers, and his audio-visual licensees.
There is at least a colorable argument that use of the term in a fictional setting would be a derivative work, intended to evoke and be derived from the extended series of creative properties.
The best counterargument that the term has entered the general lexicon, or in any case is not a protected derivative work, would comes from use of the term in a non-fiction sense which is not negligible, even though it isn't terribly common.
Wikipedia notes two (probably unlicensed) commercial uses, one as a trademark and the other as an IT company name, in addition to two minor fictional uses, one with a similar meaning and one with a somewhat shifted meaning.
Further, to the extent that the non-fiction use was great enough to show that it entered the general lexicon, at that point, fictional use would no longer be a derivative work because the work could be derived from many sources in which the Herbert estate does not have copyrights.
A Google N-gram search could be used to evaluate that argument empirically. The use of the word in the early 19th century (before Herbert was born) is particularly promising, although the senses in which it was used pre-Herbert would have to be evaluated, and the context in which it was used post-Dune would also have to be evaluated (if all the uses are in his licensed books, it hurts the cause).
A quick scan suggests that in the early 19th century that it was often used as a Latin or French word used untranslated for foreign flavor in English language works, so there is an argument that it is really a loan word not specific to Herbert. Much of the modern use is in fair use commentary expressly referencing Herbert's works and in obvious misspellings of the word "mental".
Particularly encouraging in the N-gram search are the uses of the word in independent science fiction novels including "The Search for Snout" (2014) by Bruce Coville, "Dorsai!" (2013) by Gordon R. Dickson, "Eye of the Storm" (2009) by John Ringo, and "The Ghost Brigades" (2007) by John Scalzi. Also, particularly notable is the use of the word in an independent non-Dune novel in "Clockwork Lives" (2015) by Kevin J. Anderson and Neil Peart, since Kevin J. Anderson was a co-author of some of the Dune novels and would have contractual obligations to the Herbert Estate as well as copyright obligations; signifying a concession of non-ownership of the term by the estate. This use in five independent fictional cases in the same genre in works published long enough ago for the statute of limitations for infringement to have lapsed (with late filed suits used to muddy the waters). This strongly advances the scene a faire doctrine validation of the word.
The term is also used in a scientific journal article in 2008 in the Indian Journal of Experimental Biology, but in a related but not identical sense of the word.
Another litmus test regarding whether a word is part of the general lexicon is whether it appears in more complete dictionaries, such as the OED, or Webster's Encyclopedic Unabridged Dictionary, or in a notable published thesaurus, None of the dictionaries and no thesaurus that I have in hard copy have that word, but this could be because mine are dated. An updated dictionary and thesaurus search would be probative evidence of this question. A thesaurus search for "genius" or "prodigy" that came up with "mentat" listed would be a good thing to look for.
A third issue for general lexicon v. derivative work analysis would be whether the word can be derived by obvious use of affixes and suffixes and retroflex letters from words that are in dictionaries. For example, a word plus -ology (for the study of), or -itis (for medical conditions) would not generally be beyond the scope of legally permissible public domain use. This again is a close call in the case of "mentat".
Single words are also harder (but not impossible) to protect as derivative works, if not trademarked as well, than extended passages either verbatim or paraphrased or translated, which would weaken the derivative work claim, although it wouldn't be entirely without merit just because it was a single word.