Any restriction of GTR model can be specified in BEAST by editing the XML.

For example to use the TPM3uf model, you can do the following: in BEAUti, select the GTR model and estimate the frequencies, set up the rest of your analysis and save the file. For the TPM model, the following rates are equal: AC=TG, AG=CT and AT=CG. To enforce this, open the XML file in a text editor, search for the GTR substitution element which looks something like this:

and remove rateAG — which equals rateCT, which is one. Then edit rateAC to be rateGT.s:x and rateAT to be rateCG.s:x, so now it looks something like this:

Since the rateAC.s:x, rateAG.s:x and rateAT.s:x parameters are not used in the model anymore you will also need to remove any element referring to these parameters from the file as well (in the state, the priors, the operators and the log).